Poor man's CI/CD with GitHub Webhooks
Tags: github, cicd
Published: 2022-10-05 18:17:20

---
title: Poor man's CI/CD with GitHub Webhooks
published: true
date: 2022-10-05 05:02:17 UTC
tags: github, cicd
canonical_url: https://abhijithota.me/posts/poor-mans-cicd-with-github-actions/
---

## Background

In my 5th semester of college (mid 2021), I was in-charge of the Web Operations team at [E-Cell IITM](https://ecell.iitm.ac.in/home). Our responsibility was to make web applications for events, workshops and competitions. The tech stack was primarily Node.js with Express, React and MongoDB for the database. If you're a *full-stack developer*, you might know this as the MERN stack. 

The environment was fast-paced. We were building a lot of things and breaking more of them in the process. It was like a startup. The highest number of users for an application was only 6.5k. And that's the reason we could experiment a lot of stuff. Following best practices was something we tried a lot to inculcate by adding tools like linters, formatters, pre-commit hooks, etc. but again, **we were all freshers; making and breaking things** and somewhere down the line we would just push something with just a commit message of `fix stuff`.

We have an in-premise server in the institute cluster for hosting the MERN applications. It is just an Ubuntu box with [`pm2`](https://pm2.io/) installed for running Node.js apps. A lot of our time was spent in deploying very, *very* minor changes. Fix-a-typo-in-a-static-page-level minor. The process was generally this: 

0. Connect to institute's VPN if you're not in Insti's network
1. SSH into the server
2. `git pull`
3. `npm run build` if it's a React application
4. `pm2 restart frontend-server`

Easy and simple. Manual and boring. Perfect characteristics of something that needs to be automated. Automation is the mother of invention when it comes to software development and so I went looking for answers.

## Self-hosted GitHub Actions

The first and the most obvious solution was to use [GitHub Actions](https://github.com/features/actions). I've never used them before and was pretty excited to learn and use it. My excitement was soon doubled when I got to know that you can [self-host them](https://docs.github.com/en/actions/hosting-your-own-runners). I was able to learn it and setup it for our backend Node.js monolith in a day. But something felt off.

1. It required installing a lot of stuff to a rather humble machine. It also demanded a lot of memory relative to what our applications were using. To give you an idea, our main backend service uses **105 MB** at maximum whereas the runner used around **1 GB** in average.
2. The action was such that it checked out the whole repository every time. The repository was, in a way, large.[^1]
3. All our applications stayed in the home directory (`$HOME`). I could not figure out how to checkout the repository in the same place where it was and got around it by adding [post-job scripts](https://docs.github.com/en/actions/hosting-your-own-runners/running-scripts-before-or-after-a-job).
5. We weren't looking at a lot of scale so this level of sophistication felt overkill.

All we wanted to do was to run a few scripts on a `git push`. Surely, there was a minimal way of doing this? 

## The hack which seemed like the solution

Webhooks are an elegant way to communicate from server to client. They are one of the simpler ways to implement *event-driven architecture*. Think of webhooks as a third-party server sending an HTTP POST request to your own server. This doesn't require you to [keep asking that server for data](https://en.wikipedia.org/wiki/Polling_(computer_science)) or keep an open connection. GitHub offers [Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks) for a lot of repository and organisation level events. 

Here's how they work:

1. You provide a URL and a secret to GitHub. GitHub stores the hash of that secret.
2. You choose the events you want a webhook for.
3. Whenever that event happens, GitHub sends a `JSON` or `XML` payload via a `HTTP POST` request to the URL you provided in step 1. It also sends the signature which is derived from the secret you provided. 
4. In your server, you check the signature with your original signature. If it's correct, you handle the payload however you want to.

The payload being talked about contains a [lot of information related to the event](https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads). 

And there was it. A simple `HTTP POST` containing relevant data. The idea was to commit and push with a commit message starting with `deploy`. The commit message, committer, files changed, etc. would be included in the [payload for the push event](https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push). And here's how it panned out in our case:

1. Provide a URL like `https://ecell.iitm.ac.in/api/gh-webhook`
2. Choose to send the webhook only on `push` events. 
3. When we receive a webhook, check if the commit message starts with `deploy`, the committer is one of the admins and some code is changed.
4. Run a simple script:
   ```sh
   git pull
   npm run build
   pm2 restart
   ```

## Step by step instructions

Let's try to replicate this with an example repository and `localhost` as the server. Find the repository here: [`webhook-example`](https://github.com/abhijit-hota/webhook-example).

### Adding the webhook to the repository

1. Choose a repository you want to work on. You can fork the example repository. Make sure to rename `.template.env` to `.env` and change the secret string if you want to.
2. Go to `Settings` > `Webhooks` > `Add webhook`. Or just visit `https://github.com/<username>/<repo-name>/settings/hooks/new`.
3. We will start the server and expose it via a tunnel. I'm using [`cloudflared`](https://github.com/cloudflare/cloudflared) but you can use anything else if you want to. 
4. As you can see, we have chosen `application/json` as the content type cause we want the payload in JSON. For the secret, you can put any random string but make sure you remember/note what you put there. For the example repository you also have to update your `.env` file. Details in the next section.
   ![](2022-10-05-21-59-23.png)
   > **Note about SSL**: We've only disabled it because it's just a demo. Enable this without fail in your real applications.

### Listening to the webhook in your server   

1. The [`index.js`](https://github.com/abhijit-hota/webhook-example/blob/master/index.js) is the entry point for our server. It's a simple static site server in Node.js which serves a React build. We mount the webhook *listener* in the `/webhook` route. Notice how we used this route in the URL we provided to GitHub in the previous step.
   ```js
   // Mount the webhook listener
   app.use('/webhook', webhookRouter);
   ```
2. In [`webhook.js`](https://github.com/abhijit-hota/webhook-example/blob/master/webhook.js), we first parse the `.env` file which contains our secret.
   ```js
   dotenv.config();
   const SECRET = process.env.GH_WEBHOOK_SECRET;
   ```
3. Next we add 2 middlewares. The first middleware stores the raw request buffer. We do this because parsing damages the integrity of the signature. 
   ```js
   bodyParser.json({
   	verify: (req, res, buf) => {
   	req.rawBody = buf;
     	},
   })
   ```
4. The second middleware verifies the signature sent with our secret that only we and GitHub knows. This way we know the request is actually a valid one from GitHub. This is why SSL is so important because without that you're exposing this request to [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack).
   ```js
   const body = Buffer.from(req.rawBody, 'utf8');
   const ghSign = req.get('x-hub-signature-256');
   const ourSign = 'sha256=' + crypto.createHmac('sha256', SECRET).update(body).digest('hex');
   if (ghSign !== ourSign) {
      throw new Error();
   }
   ```
5. And finally, we listen to it in the `POST` handler. 
   ```js
   webhookRouter.post('/', async (req, res) => {
		console.debug(req.body)
   }
   ```
   At this point, if you push some changes, you can see the payload in your terminal where the server is running: 
   ```json
   {
       "ref": "refs/heads/master",
       "repository": {
           "name": "webhook-example",
           "full_name": "abhijit-hota/webhook-example"
       },
       "pusher": {
           "name": "abhijit-hota",
           "email": "abhihota025@gmail.com"
       },
       "head_commit": {
           "id": "6c386ca6d6f3df80ea7b0abd59c3b9a8c3983726",
           "message": "chore: add comments and increase legibility",
           "timestamp": "2022-10-05T22:19:21+05:30",
           "author": {
               "name": "abhijit-hota",
               "email": "abhihota025@gmail.com",
               "username": "abhijit-hota"
           },
           "added": [],
           "removed": [],
           "modified": [
               ".template.env",
               "index.js",
               "webhook.js"
           ]
       }
   }
   ```
   I've removed *a lot* of stuff from the payload and have just shown the relevant ones. Now all that left is to run commands based on the data we get. More on this in the next section.

### Building and executing commands

> We're only going to see how to solve it for Linux but it shouldn't be that hard to build queries for Windows systems in a similar way, if you're using Windows at all for deployments.

1. We check for the commit message and the pusher. You can of course, choose your own logic here:
   ```js
   const { head_commit: commit, pusher } = req.body;

   if (!commit.message.startsWith('deploy') || pusher.name !== 'abhijit-hota') {
       return res.send('Deployment skipped.');
   }
   ```
2. We create an array of commands to run. And we add to it as we go over the changes. The code is pretty self-explanatory and I've added comments too.
   ```js

   const commands = ["cd "];
   const changes = [...commit.modified, ...commit.added, ...commit.removed];
   
   const requiresReinstall = changes.any((change) => change.startsWith('package'));
   if (requiresReinstall) {
       commands.push('npm install');
   }
   
   const hasFrontendChanges = changes.any((change) => change.startsWith('frontend'));
   if (hasFrontendChanges) {
       commands.push('cd frontend'); 
       const requiresReinstallInFrontend = changes.any((change) => change.startsWith('frontend/package'));
       if (requiresReinstallInFrontend) {
           commands.push('npm install');
       }
       commands.push('npm run build');
   }
   commands.push('npm start');
   ```
3. Join the whole command string with ` && ` and run it.
   ```js
   const cmd = commands.join(' && ');
   console.debug("Command to run:", cmd);
   exec(cmd); // exec from child_process
   ```
4. Let's put it to test! We commit with the message `deploy: test` and we see the message:
   ```sh
   Command to run: cd /home/kreta/work/misc/webhook-example && git pull --all && git checkout main && npm start
   ```
   `npm start` in the `package.json` starts the server in `localhost:8080`. And sure enough:
   ![](2022-10-05-23-08-39.png)

## It's a hack at the end of the day

1. The performance cost is way less than GH Action runners. We run a separate Node.js server for this which only takes up like `50 MB` of RAM. I'm sure we can remove all the fluff from there and make it leaner but it works for now. 
2. For multi-repo systems, you can create organisation-level webhooks and change the command according to the `repository` field in the payload.
3. "But Abhijit, there's no CI at all!". Just add an `npm test` somewhere and log the reports, email, etc. if you care about it.

Addressing the obvious thoughts in your head, yes, it can break and there's no way of knowing if the command is faulty. But you can setup logging, alerts etc. It's your system. It's code. It cannot get more [IaC](https://en.wikipedia.org/wiki/Infrastructure_as_code) than this. I set this up on the actual server almost 2 years ago and it hasn't been changed ever since. As you can see, this is a setup once and forget kinda system.

But if your infra is changing a lot of times and requires really good monitoring, please don't use this. This is a hack and I'm not very fond of hacks. Except for this one. This one, I'm proud of.

## First principles are good

The simplest *solutions* are the best ones. If we go backwards from a solution to the problem, we can find  better ways of thinking. Some hacks and workarounds can sometimes go against the practice of using the right tools and can feel like reinventing the wheel. But if the trade-off is worth it, maybe it is the best solution for it after all. Remember to [scale with common sense](https://zerodha.tech/blog/scaling-with-common-sense/).

[^1]: The repository was larger than needed because it had a whole Bootstrap theme template.