Docker Compose Auto Update on GitHub Webhook
Published: 2020-07-04 07:28:48

In the last week's demo, I showed how to setup a <a href="https://www.socketxp.com/webhookrelay/automatically-deploy-github-local-server-git-push/"><b>Continuous Deployment(CD) pipeline for a native NodeJS application</b></a>.

In this article, I'll show you how to build a GitOps style CD pipeline that keeps a Docker Compose deployment in sync with a docker-compose YAML file hosted on a git repository.
<h2>What's new in this demo?</h2>
We have added some cool new features to SocketXP agent, such as:
<ul>
 	<li>ability to filter incoming webhooks on user specified filter rules</li>
 	<li>ability to execute a command or a script file when an incoming webhook matches with the user specified rules</li>
</ul>
Secondly, and more importantly, we'll be running our NodeJS web app inside a Docker container and bring it up using a Docker Compose YAML file.
<h2>Prerequisites</h2>
Here are the prerequisites for this demo.
<ul>
 	<li>GitHub</li>
 	<li>SocketXP Account</li>
 	<li>SocketXP Agent Installation</li>
 	<li>Docker Engine, Docker Compose and DockerHub Account</li>
</ul>
<img class="aligncenter wp-image-1148 size-full" src="https://www.socketxp.com/wp-content/uploads/2020/07/Docker-compose-auto-update-github-webhook.png" alt="Docker-compose-auto-update-github-webhook" width="1024" height="768" />

Repository with scripts that I used for this article can be found here: <a href="https://github.com/socketxp-com/docker-compose-autoupdate-on-github-webhook">https://github.com/socketxp-com/docker-compose-autoupdate-on-github-webhook</a>.
<h2>Demo App</h2>
We'll use the same nodejs http server that we used in the last week's demo.
<pre class="source-code">$cat http-server.js 
var http = require('http');
var port = 8080
var handleRequest = function(request, response) {
  console.log('Received HTTP request for URL: ' + request.url);
  response.writeHead(200);
  response.end('Hello World!, v1.0');
};
var server = http.createServer(handleRequest);
server.listen(port); 
</pre>
<h2>Dockerfile</h2>
Here is the Dockerfile we'll use to package the above nodejs app into a Docker container.
<pre class="source-code">FROM alpine:latest
RUN apk update &amp;&amp; apk add nodejs
RUN mkdir -p /usr/src/app
COPY ./http-server.js /usr/src/app
WORKDIR /usr/src/app
EXPOSE 8080 
CMD ["node","http-server.js"]
</pre>
Let's build the Docker image using the following Docker command.
<pre class="source-code">$docker build -t gvelrajan/http-server:v1.0 .
Sending build context to Docker daemon  56.83kB
Step 1/7 : FROM alpine:latest
 ---&gt; f70734b6a266
Step 2/7 : RUN apk update &amp;&amp; apk add nodejs
 ---&gt; Using cache
...
Successfully built b6b06a819b63
Successfully tagged gvelrajan/http-server:v1.0
$
</pre>
Next, let's push this Docker image to DockerHub image registry, so that it can be downloaded by the production team to deploy it.
<pre class="source-code">$docker push gvelrajan/http-server:v1.0
The push refers to repository [docker.io/gvelrajan/http-server]
3b8d0f6b058e: Layer already exists 
947da554eff8: Layer already exists 
cc403a71dcfb: Pushed 
3e207b409db3: Layer already exists 
v1.0: digest: sha256:07e70e778e1002519064503ece0d334291f4fbb7a7196b894f605ff5e3076e18 size: 1154
</pre>
<h2>Docker Compose</h2>
Docker Compose is a very helpful tool to bring up various services of an application as Docker containers using a simple single command. In our demo example we have just a single service - the http web service, packaged as a Docker container.

Here is the docker-compose yaml file we'll use to bring up our http-server docker container in production.
<pre class="source-code">$cat docker-compose.yml 
version: '3'
services:
  web:
    image: "gvelrajan/http-server:v1.0"
    ports:
      - "8080:8080"
$
</pre>
Now, bring up the app container by executing the following command.
<pre class="source-code">$docker-compose up -d
Creating network "docker-compose-autoupdate-on-github-webhook_default" with the default driver
Creating docker-compose-autoupdate-on-github-webhook_web_1_fd2bb2cedec9 ... done
</pre>
Check if the docker container is up and running, and listening on the correct port(8080).
<pre class="source-code">$docker ps
CONTAINER ID        IMAGE                        COMMAND                 CREATED             STATUS              PORTS                    NAMES
899ab99abedf        gvelrajan/http-server:v1.0   "node http-server.js"   2 seconds ago       Up 1 second         0.0.0.0:8080-&gt;8080/tcp   docker-compose-autoupdate-on-github-webhook_web_1_64dc034e55ac
$
</pre>
<h2>Curl the web server</h2>
Verify that our web app is running and serving content on port 8080.
<pre class="source-code">$curl http://localhost:8080
Hello World!, v1.0
</pre>
Our initial deployment is very successful. But this was a manual deployment. Every time a newer version of the app is released we have to manually download and deploy the application. This is not what we wanted. Our goal for this demo is to build a Continuous Deployment(CD) pipeline for our nodejs based web service running as a docker container.

Let's begin by setting up various stages in the CD pipeline, like we did in the last week's demo.
<h2>Auto deployment strategy</h2>
Our auto deployment strategy has changed and is slightly different from the one used in the last week's native app demo. Here is the auto deployment strategy for this demo:

When a new version of our app is available, we'll update the docker-compose.yml file in the GitHub and release a new version of our app in the GitHub. GitHub will trigger a webhook for this release event. We'll listen for this webhook event and automatically do a git pull on the docker-compose.yml file. We'll use the new docker-compose.yml to re-spin our nodejs web service docker container in production.

<b>Note:</b> GitHub will trigger webhooks for all sorts of push events. We don't want to react to all those webhooks and auto deploy our containers on every GitHub push. We want to auto deploy only when a new release is made.
<h2>SocketXP Webhook Relay Service</h2>
We'll use <b><a href="https://www.socketxp.com">SocketXP Webhook Relay Service</a></b> to relay webhooks from GitHub to our local production server (where the nodejs app is running as a docker container). SocketXP can also filter GitHub webhooks based on user specified rules in JSON format. We'll discuss more about it in the next section. Additionally, SocketXP can execute a script or a command, if an incoming webhook matches with the user specified rules.

Before creating webhook rules, it is better to study the webhook in detail and pick up a few fields in the webhook payload to match on. For this we need to make GitHub trigger a release webhook and capture the webhook and study it.
<h2>Webhook Capture and Study</h2>
First, let's setup the SocketXP agent to relay any webhooks from the internet to our local web service.
<pre class="source-code">$ socketxp relay http://localhost:8080

Connected.
Public URL -&gt; https://webhook.socketxp.com/ganeshvelrajan-1awy5t0h
</pre>
Pick up the above SocketXP Public URL and head to your GitHub repository. Setup the webhook settings for your GitHub project as shown below. Setup your secret and copy paste the SocketXP Public URL in the webhook URL text box. Also set the content-type to "application/json" in the drop-down box. Save the changes.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/akqgsppi4hi0c6frj6c9.png)

<h2>Triggering GitHub Release Webhook</h2>
Next, create a new release for your GitHub project in the release settings page and create a new tag for your release, as shown below.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3iqxll4jbg2tk66m6nex.png)

GitHub will trigger a webhook for this release event in your project. We can view details of the triggered webhook either in the GitHub Webhook Settings page or in the SocketXP Portal Webhook Logs page.
<pre class="source-code">Request URL: https://webhook.socketxp.com/ganeshvelrajan-1awy5t0h
Request method: POST
Accept: */*
content-type: application/json
User-Agent: GitHub-Hookshot/e509757
X-GitHub-Delivery: ca631384-bd0e-11ea-8ba6-ea05e17356b6
X-GitHub-Event: push
<b>X-Hub-Signature: sha1=c902f69547901dc0658ba54eaee5cde6c0a5ef00</b>

{
  <b>"ref": "refs/tags/v1.0",</b>
  "before": "0000000000000000000000000000000000000000",
  "after": "b5595d43b2c681257e0bae638f9de814df968901",
  "repository": {
    "id": 276817731,
    "node_id": "MDEwOlJlcG9zaXRvcnkyNzY4MTc3MzE=",
    "name": "docker-compose-autoupdate-on-github-webhook",
    "full_name": "socketxp-com/docker-compose-autoupdate-on-github-webhook",
    "private": false,
    ...
    ...
}
</pre>
In the above webhook capture, we see the signature field in the header that encodes the secret we set in the GitHub webhook settings page. Verifying this signature using the secrete we provided, authenticates the validity of the sender.

Secondly, in the above webhook capture, the payload contains the release tags ("refs/tags/v1.0") in the "ref" field.

Let's try to filter our incoming webhook based on these two fields in the webhook.
<h2>SocketXP Webhook Filter-Rules JSON File</h2>
Here is how the SocketXP filter-rules JSON file should look like to match the two fields we picked up in the webhook above.
<pre class="source-code">$cat filter-rules.json 
{
  "and":
  [
    {
      "match":
      {
        "type": "payload-hash-sha1",
        "secret": "asdfasdf23421dsaf",    
        "parameter":
        {
          "source": "header",
          "name": "X-Hub-Signature"
        }
      }
    },
    {
      "match":
      {
        "type": "value",
        "value": "refs/tags",
        "parameter":
        {
          "source": "payload",
          "name": "ref"
        }
      }
    }
  ]
}</pre>
Make sure you update the "secret" field in the filter-rules file above with your GitHub webhook secret.
<h2>SocketXP Exec Command Option</h2>
Here is the script we'd like to execute to automatically update our Docker containers running in our production or test server when a new release is made in the GitHub project.
<pre class="source-code">$cat update-docker.sh 
#!/bin/bash

git pull
docker-compose up -d
</pre>
Let's kill our SocketXP agent and re-run it with the two new arguments: "filter-rules.json" file and the "update-docker.sh" script file, as shown below.
<pre class="source-code">socketxp relay http://localhost:8080 --exec update-docker.sh --filter filter-rules.json

Connected.
Public URL -&gt; https://webhook.socketxp.com/ganeshvelrajan-1awy5t0h
</pre>
Note that the SocketXP Webhook Public URL has not changed despite killing and re-executing the command. This is because we are requesting a public URL for a webhook relay service again and that too for the same local service endpoints (http://localhost:8080). This nice feature of SocektXP saves us from updating the GitHub Webhook URL settings again.
<h2>Begin Demo</h2>
Now that the CD pipeline is all set, let's upgrade our NodeJS web app and release a version 2.0 of the artifact. We also need to update the docker-compose.yml file to reflect the updated app version.
<h3>App Upgrade</h3>
Here is a new version of our app (for the sake of this demo, we have simply updated the version string printed to 2.0).
<pre class="source-code">$cat http-server.js 
var http = require('http');
var port = 8080
var handleRequest = function(request, response) {
  console.log('Received HTTP request for URL: ' + request.url);
  response.writeHead(200);
  response.end('Hello World!, v2.0');
};
var server = http.createServer(handleRequest);
server.listen(port);
</pre>
<h3>Build Push New App Container</h3>
Build a new docker container containing this new version of our app.
<pre class="source-code">$docker build -t gvelrajan/http-server:v2.0 .
Sending build context to Docker daemon  69.63kB
Step 1/7 : FROM alpine:latest
 ---&gt; f70734b6a266
Step 2/7 : RUN apk update &amp;&amp; apk add nodejs
...
...
Successfully tagged gvelrajan/http-server:v2.0
</pre>
Push the docker image to DockerHub.
<pre class="source-code">$docker push gvelrajan/http-server:v2.0
The push refers to repository [docker.io/gvelrajan/http-server]
4a303822279a: Pushed 
947da554eff8: Layer already exists 
cc403a71dcfb: Layer already exists 
3e207b409db3: Layer already exists 
v2.0: digest: sha256:2a9ee82f76758758c77659260091380b20e3b3d095efdc6480d66e47c5842476 size: 1154
</pre>
Update the docker-compose file with the new docker image version.
<pre class="source-code">$cat docker-compose.yml 
version: '3'
services:
  web:
    image: "gvelrajan/http-server:v2.0"
    ports:
      - "8080:8080"
</pre>
<h3>Git Commit and Release Webhook</h3>
Perform git commit and push the changes to the GitHub repository. Create new release version  tag "version 2.0" for your project in GitHub.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/s6x31dvmwt32c03f96v6.png)

This event will make GitHub trigger a webhook to SocketXP agent.
<h3>Verify Webhook Received</h3>
Check the console where SocketXP agent is running. It should have received the webhook from GitHub and applied the filters on it.
<pre class="source-code">$ socketxp relay http://localhost:8080 --exec update-docker.sh --filter filter-rules.json

Connected.
Public URL -&gt; https://webhook.socketxp.com/ganeshvelrajan-1awy5t0h

Webhook received:
POST / HTTP/1.1
Host: webhook.socketxp.com
Accept: */*
content-type: application/json
User-Agent: GitHub-Hookshot/e509757
X-GitHub-Delivery: fa44ca8e-bd31-11ea-9def-3e0cd2844448
X-GitHub-Event: push
X-Hub-Signature: sha1=d32c79c8de3282e07400711c4e7d3d6542ef79eb

{
  <b>"ref": "refs/tags/2.0",</b>
  "before": "0000000000000000000000000000000000000000",
  "after": "b5595d43b2c681257e0bae638f9de814df968901",
  "repository": {
    "id": 276817731,
    "node_id": "MDEwOlJlcG9zaXRvcnkyNzY4MTc3MzE=",
    "name": "docker-compose-autoupdate-on-github-webhook",
    "full_name": "socketxp-com/docker-compose-autoupdate-on-github-webhook",
    "private": false,
	...
	...
}

<b>#Rule: Webhook signature matched.
#Rule: ref in payload matched.
All rules matched.

Executing command: [update-docker.sh]</b>
...
...
</pre>
Check if the docker container has been upgraded and running version 2.0.
<pre class="source-code">$docker ps
CONTAINER ID        IMAGE                        COMMAND                 CREATED              STATUS              PORTS                    NAMES
e7ef0e98cd45        gvelrajan/http-server:v2.0   "node http-server.js"   About a minute ago   Up About a minute   0.0.0.0:8080-&gt;8080/tcp   docker-compose-autoupdate-on-github-webhook_web_1_64dc034e55ac
</pre>
<h3>Verify the upgraded app</h3>
Curl the web service and verify if the app has been upgraded.
<pre class="source-code">$curl http://localhost:8080
<b>Hello World!, v2.0</b>
</pre>
Our web app prints "v2.0", denoting that it got updated, automatically!
<h2>Conclusion</h2>
Setting up Docker Compose to auto deploy on GitHub push or release webhook notification is a great way to set up Continuous Deployment(CD) pipeline. The challenge is in receiving, processing and filtering GitHub webhooks in the production servers or in the internal test servers. 
<br><a href="https://www.socketxp.com">SocketXP</a> is a great tool to receive, store, forward, filter and take actions based on webhooks. SocketXP Webhook Relay Service takes care of bulk of the work involved in setting up an automated CD pipeline. So that, developers can focus on their core work, while leaving the rest to SocketXP. 

Moreover, SocketXP has a free tier for developers. And our paid plans start at just $1.99/month.

<b>This article was originally published at:
<a href="https://www.socketxp.com/webhookrelay/docker-compose-auto-update-github-webhook/">https://www.socketxp.com/webhookrelay/docker-compose-auto-update-github-webhook</a></b>