Pavel Vlasov

Pavel Vlasov
Mar 04, 2018

How to build a GitHub bot with PhantomJS, React, and Serverless framework

How to build a GitHub bot with PhantomJS, React, and Serverless framework

This tutorial is about building a simple Serverless bot that returns a chart with top GitHub repository contributors for a selected period. It is relevant to those who have some experience with React, JavaScript, TypeScript, Node.js, Amazon Web Services (AWS), and the Serverless framework.

You can check out the code on Github.

Services and tools we’ll be using

Before jumping into coding, let’s do a quick overview of AWS services and tools that we’ll be using.

To retrieve top repository contributors, we will use GitHub stats API, the amazing Nivo to display data, Storybook to check how our chart looks and feels, PhantomJS to turn HTML into an image, and Serverless framework to interact with AWS.

Let’s get started

I’ll be using TypeScript. If you prefer ES6, you will need to configure Babel.

First, you have to create tsconfig.json in the root of your repository. Options to pay attention to include:

"module": "commonjs",
"target": "es5",
"lib": ["es6", "esnext.asynciterable"],
"moduleResolution": "node",
"jsx": "react"

Then, we’ll create a simple API to query stats from GitHub. You can follow the file structure from the GitHub repo or use your own. For example:

import * as request from "request-promise-native";

const BASE_URL = "https://api.github.com";

export interface Person {
    login: string;
    id: number;
    avatar_url: string;
    gravatar_id: string;
    url: string;
    html_url: string;
    followers_url: string;
    following_url: string;
    gists_url: string;
    starred_url: string;
    subscriptions_url: string;
    organizations_url: string;
    repos_url: string;
    events_url: string;
    received_events_url: string;
    type: string;
    site_admin: false;
}

export interface ContributorStats {
    author: Person;
    total: number;
    weeks: Array<{
        w: string;
        a: number;
        d: number;
        c: number;
    }>;
}

export interface CommitActivity {
    days: number[];
    total: number;
    week: number;
}

class GithubStatsApi {
    private _request: typeof request;

    constructor(token: string) {
        this._request = request.defaults({
            baseUrl: BASE_URL,
            headers: {
                Authorization: `token ${token}`,
                "User-Agent": "Stats Bot for Slack"
            },
            json: true
        });
    }

    public get request(): typeof request {
        return this._request;
    }

    public getContributorStats(repo: string): Promise<ContributorStats[]> {
        return this.request({
            url: `/repos/${repo}/stats/contributors`
        });
    }

    public getActivity(repo: string): Promise<CommitActivity[]> {
        return this.request({
            url: `/repos/${repo}/stats/commit_activity`
        });
    }
}

export default GithubStatsApi;

To access the GitHub API, you’ll have to create a personal access token.

This module simply sends the request with the provided token and retrieves the data.

Displaying the charts

To display the data, we’ll use Nivo and Storybook. A simple component may look like this:

import { Bar as NBar } from "@nivo/bar";
import * as React from "react";
import { render } from "react-dom";

interface Item {
    author: string;
    added: number;
    deleted: number;
    changed: number;
}

type Data = Item[];

const colors = {
    added: "#97e3d5",
    changed: "#f1e15b",
    deleted: "#f47560"
};

interface Options {
    animate?: boolean;
}

interface Node {
    id: "added" | "changed" | "deleted";
}

export const Bars = (props: { data: Data; options?: Options }) => {
    const data = props.data;
    const options = props.options || {};

    return (
        <NBar
            width={600}
            height={300}
            data={data}
            keys={["deleted", "changed", "added"]}
            indexBy="author"
            ... another props
        />
    );
};

First, setup Storybooks by running the following command in the root folder:

npm i -g @storybook/cli
getstorybook

Copy the .storybook folder into the root repository and replace all existing files. It contains the Webpack and Storybook configuration. Create a stories folder and put in a sample story for your component:

import { Bars } from '@/shared/api/chart/bar'
import { storiesOf } from '@storybook/react'
import * as React from 'react'
import data from './bars.data'

storiesOf('bars chart', module).add('default', () => {
    return <Bars data={data} />
})

Run npm run storybook and open localhost:6006 in the browser. You should see the following result:

![subscription](/imagesarssu](/images/github-bot/chart_1.png)

Try to play with the options and test data. Storybook will change the look immediately.

Turning HTML into PNG

Usually, chat systems like Facebook Messenger and Slack do not allow users to insert HTML cards in the dialog, so the next step will be to build a helper that renders HTML into a PNG image.

Using a simple script with jsdom library, we can mimic browser behavior and serialize HTML, like this:

bar.ts

import { createDomForChart } from '../utils'

export default (data: Data, options: Options = {}): string => {
    const { target, dom } = createDomForChart()
    render(<Bars data={data} options={options} />, target)

    return dom.serialize()
}

utils.ts

import * as jsdom from 'jsdom'
import { DOMElement } from 'react'

export function createDomForChart(): {
    dom: jsdom.JSDOM,
    target: DOMElement<any, any>,
} {
    const dom = new jsdom.JSDOM(`<body></body>`)
    const div: DOMElement<any, any> = dom.window.document.createElement('div')
    dom.window.document.body.appendChild(div)

    return {
        target: div,
        dom,
    }
}

createDomForChart returns a new instance of jsdom, and the chart function simply calls dom.serialize() when component rendering is done.

With PhantomJS, we can turn markup into an image using this simple script:

screenshot.js

'use strict'

var system = require('system')

var html = system.args[1]
var width = system.args[2]
var height = system.args[3]

var page = require('webpage').create()

page.viewportSize = { width: Number(width), height: Number(height) }
page.clipRect = { width: Number(width), height: Number(height) }

page.onLoadFinished = function(status) {
    try {
        if (status !== 'success') {
            return phantom.exit(1)
        }

        system.stdout.write('data:image/png;base64,' + page.renderBase64('PNG'))
        phantom.exit()
    } catch (err) {
        console.error(err)
        phantom.exit(1)
    }
}

page.setContent('<!DOCTYPE html>' + decodeURI(html), 'http://example.com')

setTimeout(function() {
    phantom.exit(1)
}, 500)

take-screenshot.ts

import {exec} from "child_process"
import * as path from "path"
const isOsx = Boolean(process.env.OSX as string);

const phantomJsPath = path.resolve(__dirname, "./phantomjs")
const PREFIX = "data:image/png;base64,"

export default (
    htmlString: string,
    width: number,
    height: number,
): Promise<Buffer> => {
    return new Promise((resolve, reject) => {
        try {
            const cmd = [
                `${path.join(phantomJsPath, isOsx ? "phantomjs_osx" : "phantomjs_linux-x86_64")}`,
                "--debug=yes --ignore-ssl-errors=true",
                `${path.join(phantomJsPath, "./screenshot.js")}`,
                `"${encodeURI(htmlString)}" ${width} ${height}`
            ].join(" ")

            exec(cmd, (err, stdout, stderr) => {
                if (err) {
                    return reject(err);
                }

                if (stdout.startsWith(PREFIX)) {
                    resolve(new Buffer(stdout.substring(PREFIX.length), "base64"))
                } else {
                    reject(new Error("Unknown error"))
                }
            })
        } catch (err) {
            reject(err)
        }
    })
}

We’re passing screenshot.js into the phantomjs executable path — along with an HTML string, width and height — and getting back buffer with the rendered image.

You may notice that I’ve been using two PhantomJS binaries (for OS X and Linux). We’ll need the Linux version further in an AWS environment. You can download them from PhantomJS.org or use files from the repository.

Tying everything up

Now, let’s create lambda to handle requests. I recommend putting PNG rendering logic into a separate service. Because PhantomJS binary is approximately 50 mb in size, it slows down deployment if you change anything in the API. Also, you can reuse this lambda for other purposes.

We’ll start by creating webpack.config.ts (to bundle source code) and serverless.base.js (to define the base serverless configuration) in the root folder.

If you want to know more about use cases of serverless JavaScript configurations, you can read about it in my previous article.

You’ll have to change deployment and image bucket names, like this:

deploymentBucket: {
    name: 'com.github-stats....deploys'
},
environment: {
    BUCKET: 'com.github-stats....images',
    GITHUB_TOKEN: '${env:GITHUB_TOKEN}',
    SLACK_TOKEN: '${env:SLACK_TOKEN},
    STAGE: '${self:provider.stage}'
},

This is because the name of the bucket has to be globally unique.

Turning HTML to PNG service

First of all, we’ll create a handler that will return a URL of the generated image. The handler should validate and process the request body:

interface Body {
    html: string;
    width: number;
    height: number;
    name: string;
}

const validateBody = (body: any): Body => {
    if (typeof body.html !== "string") {
        throw new Error("Unsupported html type");
    }
    if (typeof body.width !== "number" && typeof body.height !== "number") {
        throw new Error("Unsupported dimensions");
    }
    if (typeof body.name !== "string" && !body.name) {
        throw new Error("Name should be of type string and not empty");
    }

    return body as Body;
};

…and if everything is ok, it should generate the image and put it into an S3 bucket.

export const handler = async (
    event: Event,
    context: Context,
    callback: Callback
) => {
    try {
        const body = validateBody(event.body)

        const image = await takeScreenshot(body.html, body.width, body.height)
        const res = await new Promise((resolve, reject) => {
            const s3 = new S3({
                apiVersion: '2006-03-01',
                region: 'us-east-1',
            })
            const params = {
                Bucket: bucketName,
                Key: body.name,
                Body: image,
                ContentEncoding: 'base64',
                ContentType: 'image/png',
            }
            s3.upload(
                params,
                (err: Error, uploadResult: { Location: string }) => {
                    if (err) {
                        return reject(err)
                    }
                    resolve({ imageUri: uploadResult.Location })
                }
            )
        })

        callback(null, res)
    } catch (err) {
        callback(err)
    }
}

Let’s create webpack.config.ts to bundle source files. We’ll use the copy-webpack-plugin and webpack-permissions-plugin to include PhantomJS binaries into a bundle — and give permissions for execution. This will require us to run the deploy command with sudo since Webpack doesn’t have permissions to modify file system rights by default.

import * as CopyWebpackPlugin from 'copy-webpack-plugin'
import * as path from 'path'
import * as PermissionsOutputPlugin from 'webpack-permissions-plugin'
import getBaseConfig from '../../webpack.dev'

const baseConfig = getBaseConfig(__dirname)

module.exports = Object.assign(
    {
        plugins: [
            new CopyWebpackPlugin([
                {
                    from: 'phantomjs/phantomjs_linux-x86_64',
                    to: 'phantomjs',
                },
                {
                    from: 'phantomjs/screenshot.js',
                    to: 'phantomjs',
                },
            ]),
            new PermissionsOutputPlugin({
                buildFiles: [
                    path.join(
                        baseConfig.output.path,
                        'service',
                        'phantomjs',
                        'phantomjs_linux-x86_64'
                    ),
                ],
            }),
        ],
    },
    baseConfig
)

The last step will be using the serverless.js file to tie our handler with an API Gateway event.

const baseConfig = require('../../serverless.base')
const { defaultsDeep } = require('lodash')

module.exports = defaultsDeep(
    {
        service: 'html-to-png',
        functions: {
            renderToPng: {
                name: '${self:provider.stage}-renderToPng',
                handler: 'index.handler',
                description: 'render input html to png and put into S3 bucket',
                memorySize: 512,
                timeout: 30,
            },
        },
        custom: {
            webpack: './webpack.config.ts',
        },
    },
    baseConfig
)

Now, we need to perform the same steps for stats handler, but we don’t have to make any changes to webpack.config.ts.

const baseConfig = require('../../serverless.base')
const { defaultsDeep } = require('lodash')

module.exports = defaultsDeep(
    {
        service: 'git-stats',
        provider: {
            iamRoleStatements: [
                ...baseConfig.provider.iamRoleStatements,
                {
                    Effect: 'Allow',
                    Action: ['lambda:InvokeFunction'],
                    Resource: ['*'],
                },
            ],
        },
        functions: {
            getContributorStatsImage: {
                name: '${self:provider.stage}-getContributorStatsImage',
                handler: 'index.handler',
                events: [
                    {
                        http: {
                            path: '/stats/contributors/{owner}/{repo}',
                            method: 'get',
                            parameters: {
                                paths: {
                                    owner: true,
                                    repo: true,
                                },
                                querystrings: {
                                    start: false,
                                    end: false,
                                },
                            },
                        },
                    },
                ],
                memorySize: 256,
                timeout: 30,
            },
        },
        custom: {
            webpack: './webpack.config.ts',
        },
    },
    baseConfig
)

The only difference is an additional permission to invoke lambda:

iamRoleStatements: [
    ...baseConfig.provider.iamRoleStatements,
    {
        Effect: 'Allow',
        Action: ['lambda:InvokeFunction'],
        Resource: ['*'],
    },
]

Setting up the Slack bot

The last step will be to create a service that will handle message events for the bot. To keep it simple, we’ll handle only mention events. Let’s set up the basic event handler.

We have to handle a verification event from Slack and respond with 200 status and challenge parameters:

callback(null, {
   body: JSON.stringify({
     challenge: (slackEvent as VerificationEvent).challenge
   }),
   statusCode: 200
});

To properly handle a Slack event, the endpoint has to reply within 3000 milliseconds (3 seconds), so we’ll have to immediately respond and asynchronously send a follow-up message using postMessage API.

callback(null, { body: "OK", statusCode: 200 });

const message = mentionEvent.text;
const parts = repoRegExp.exec(message);
if (parts) {
    const repo = parts[1];

    const params = {
        FunctionName: `${stage}-getContributorStatsImage`,
        InvocationType: "RequestResponse",
        LogType: "Tail",
        Payload: JSON.stringify({
            repo
        })
    };

    const response = await lambda.invoke(params).promise();
    const payload = JSON.parse(response.Payload);

    const message = {
        channel: mentionEvent.channel,
        text: "Contributor stats",
        attachments: [
            {
                title: `Contributor stats for ${repo} repository`,
                image_url: payload.imageUri
            }
        ]
    };

    await request({
        url: "https://slack.com/api/chat.postMessage",
        method: "post",
        json: true,
        headers: {
            Authorization: `Bearer ${slackToken}`,
            "User-Agent": "Stats Bot for Slack"
        },
        body: message
    });
} else {
    throw new Error("Can not recognize repository name");
}

In the code above, we parsed the message text to extract a repository name and called out an image stats lambda to retrieve an image URL and send a message back to Slack. You can find the full code of the handler here.

Code for serverless.js and Webpack configurations would be similar to the stats service, so if you have problems with setting it up, take a look at the full source code.

Creating a Slack app

Now let’s create a new Slack app. Go to the Slack API, create a new account (if you have not already done so), create a new app, and add the bot scope in the scopes section.

Go to the “OAuth & Permissions” section in the sidebar.

features

Add the bot user scope.

features

Then, you’ll be able to install the app to your organization and get access to tokens.

tokens

Deploying the services

You’ll have to install a serverless framework version greater than 1.26 because earlier versions do not support JavaScript configuration files. And I recommend installing slx to simplify the deployment of multiple services.

npm install -g serverless
npm install -g serviceless

Copy the GitHub and Slack bot tokens, and set them to GITHUB_TOKEN and SLACK_TOKEN environment variables accordingly. Run the following command in the terminal:

sudo GITHUB_TOKEN=<your token> SLACK_TOKEN=<your slack token> slx deploy all

As mentioned above, we need sudo to set execute permissions to PhantomJS binaries.

Be patient! Deployment may take a while. At the end you should see a similar output:

Deployment completed successfuly
[app/html-to-png] [completed]:
Service Information
service: html-to-png
stage: dev
region: us-east-1
stack: html-to-png-dev
api keys:
   None
endpoints:
   None
functions:
   renderToPng: html-to-png-dev-renderToPng
Serverless: Removing old service versions...
[app/slack] [completed]:
Service Information
service: git-stats-slack
stage: dev
region: us-east-1
stack: git-stats-slack-dev
api keys:
   None
endpoints:
   POST - https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/stats/slack/event-handler
functions:
   eventHandler: git-stats-slack-dev-eventHandler
Serverless: Removing old service versions...
[app/stats] [completed]:
Service Information
service: git-stats
stage: dev
region: us-east-1
stack: git-stats-dev
api keys:
   None
endpoints:
   GET - https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/stats/contributors/{owner}/{repo}
functions:
   getContributorStatsImage: git-stats-dev-getContributorStatsImage
Serverless: Removing old service versions...

The last step will be to subscribe our endpoint to bot mention events.

Select the “Event Subscription” section in the Slack API navigation.

subscriptions

Then paste the event handler URL that you can find in the deploy command output.

Enable events

Time to play around a bit! Here are some examples of rendered images:

serverless/serverles serverless

facebook/react react

That’s it!

I hope you found this article helpful. I’d love to see in the comments other types of stats you would like to see in the service.

The article was originally posted on Medium