Google Kubernetes Engine, CircleCI and Traefik for a full-fledged GitOps platform in the cloud - Part 2
Tags: kubernetes, circleci, traefik, gitops
Published: 2021-01-23 20:01:33

This is the second part of "Google Kubernetes Engine, CircleCI and Traefik for a full-fledged GitOps platform in the cloud" ! Part 1 is [accessible here](https://dev.to/zenika/google-kubernetes-engine-circleci-and-traefik-for-a-full-fledged-gitops-platform-in-the-cloud-part-1-2bai) and mandatory for the following. 

## 3. A first "whoami" toy project

In this section, we will illustrate how to use Traefik to expose multiple instances/branches/versions of a same app on subdomains. Suppose we chose to dedicate the `whoami` subdomain (e.g., `whoami.mywebsite.com`) to this project. Our goal is to expose the master version of the app on a `master` subdomain (e.g., `master.whoami.mywebsite.com`) and the changes of a `feature1` branch on the `feature1` subdomain (e.g., `feature1.whoami.mywebsite.com`).

For this example, we will use the containous/whoami Docker image as a dummy web application, which only serves some information about the server and the received request for any request on port 80.

### Create a namespace for the project

```
kubectl create namespace whoami
```

### Deploy the master version

Fill the following placeholders in the [whoami-master descriptor](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-master.yaml):
- `DOMAIN`: the domain you own (e.g., `mywebsite.com`)

Then apply it:
```
kubectl apply -f whoami-master.yaml
```

This will instantiate 3 instances of the `whoami` Pod, create a Service to load balance between them, and expose this Service through an IngressRoute. Notice the use of the `traefik` TLS certificate resolver created in step 2, the request for a `*.whoami.[DOMAIN]` wildcard certificate and the use of the `master.whoami.[DOMAIN]` Host rule to route traffic to this URL to our Service. In just a few seconds, the certificate is generated and your application server is exposed securely on the `master.whoami` of your domain (try it!).

### Deploy the feature1 branch

Fill the following placeholders in the [whoami-branch1 descriptor](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-branch1.yaml):
- `DOMAIN`: the domain you own (e.g., `mywebsite.com`)

Then apply it:
```
kubectl apply -f whoami-branch1.yaml
```

This will instantiate 3 new instances of the Pod, create a new Service to load balance between them, and expose this Service through an IngressRoute. Notice again the use of the `traefik` TLS certificate resolver created in step 2, the request for a `*.whoami.[DOMAIN]` wildcard certificate (which is already managed by Traefik and will be reused) and the use of the `branch1.whoami.[DOMAIN]` Host rule to route traffic to this URL to our Service. Your new app instance, distinct from the master environment is already live (try it!).

### Use a template

Notice the very few differences between the [whoami-master](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-master.yaml) and the [whoami-branch1](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-branch1.yaml) descriptors: all occurences of `master` have simply be replaced by `branch1`. In real world use cases, the Docker image will probably be different too.

Take a look at the [whoami-template descriptor template](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-template.yaml) file. Env-like placeholders (e.g., `${INSTANCE_NAME}`) are used. We will use the `envsubst` tool to instantiate our template.

Fill the following placeholders in the [whoami-template descriptor template](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/whoami-template.yaml):
- `DOMAIN`: the domain you own (e.g., `mywebsite.com`)
  
Do not touch the env-like placeholders.

Set the required environement variables, then instantiate the template and apply it:
```
export INSTANCE_NAME=branch2
export IMAGE=containous/whoami:v1.5.0
envsubst < whoami-template.yaml > whoami-branch2.yaml
kubectl apply -f whoami-branch2.yaml
```

Quick, your `branch2` instance is already up!

The `envsubst` approach will get you quite far, but look into [Kustomize](https://github.com/kubernetes-sigs/kustomize) or [Helm](https://helm.sh/) if you need a more advanced configuration management tool.



## 4. A "myapp" project with CI/CD

In this section, we will illustrate how the manual deployment described in step 3 can be integrated in about any CI/CD pipeline. Suppose we chose to dedicate the `myapp` subdomain (e.g., `myapp.mywebsite.com`) to this project. Our goal is to expose any branch X on the X subdomain (e.g., branch `feat1` exposed on `feat1.myapp.mywebsite.com`, `feat2` exposed on `feat2.myapp.mywebsite.com`, etc.).

For this example, we will build our own Docker image in a CircleCI pipeline. For the sake of simplicity, this guide will use the same containous/whoami image, but feel free to build your own for real!

### Create a namespace for the project

```
kubectl create namespace myapp
```

### Create a "myapp" project on GitHub

In the [sample_project](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/sample_project) folder, fill the following placeholders in the [myapp-template descriptor](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/sample_project/myapp-template.yaml):
- `DOMAIN`: the domain you own (e.g., `mywebsite.com`)

Then push the content of the [sample_project](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/sample_project) folder at the root of your GitHub repository (i.e., the `.circleci` folder, the `Dockerfile` and the descriptor template should be visible at the root of your repository).

Notice in particular the [.circleci/config.yml](https://github.com/Zenika/labs-tooling/blob/master/ci-cd-platform-deployment/sample_project/.circleci/config.yml) build configuration file for CircleCI (more about the syntax [here](https://circleci.com/docs/2.0/configuration-reference/)).

### Create a CircleCI service account on GCP

This can easily be done in the [Cloud Console](https://console.cloud.google.com/iam-admin/serviceaccounts).

Create a new `CircleCI` account with role `Storage Admin` and `Kubernetes Engine Admin`. Create and download a (JSON) key for this account. This technical user will have write access to the Google Container Registry (GCR) of you project and permission to deploy to your cluster.


### Build the project on CircleCI

In CircleCI, add your project in the build list in the `Add Projects` tab (use the `Add manually` button, since you have already committed a `.circleci/config.yml` file).

In the `Settings` tab, find your project in the `Projects` sub-tab and open its settings.

In `Environment Variables`, create the following variables. They are used in the CircleCI configuration and in the descriptor template.
- `GCP_PROJECT`: The id of your GCP project (visible in the [Cloud Console](https://console.cloud.google.com/home/))
- `GCP_AUTH`: The content of the JSON key file created in the previous step.
- `GCP_REGISTRY`: The GCR registry of your choice (`eu.gcr.io/[GCP_PROJECT]` for a storage in Europe)
- `GCP_ZONE`: The GCP Zone in which your cluster is hosted (visible in the [Cloud Console](https://console.cloud.google.com/kubernetes/list), e.g., `europe-west3-a`)
- `GCP_KUBE_CLUSTER`: The name of your Kubernetes cluster

Relaunch the build: the master branch is deployed on the `master` subdomain (e.g., `master.myapp.mywebsite.com`).

Create branches on your GitHub repository and push them... and let the magic happen! From source code to production in less than 20 seconds!

![Screenshot CircleCI](https://dev-to-uploads.s3.amazonaws.com/i/41xzdcvk587c538f9e9i.png)


## 5. Bonus track: delete stale instances in Kubernetes

Now you know how to deploy an instance of your app per GitHub branch in Kubernetes... But how to delete stale instances in Kubernetes (instances of the app created for branches that are since deleted in GitHub)?

### Add a second job in your CircleCI configuration

As you can see, it involves quite a bit of bash wizardry...

```
jobs:
  build:
    [...]
  delete-stale-ci-stacks:
    docker:
      - image: google/cloud-sdk
    working_directory: /home/circleci/myapp
    steps:
      - checkout
      - run:
        name: Authenticate with GCP
        command: |
          echo ${GCP_AUTH} > /home/circleci/gcp-key.json
          gcloud auth activate-service-account --key-file /home/circleci/gcp-key.json
          gcloud --quiet config set project ${GCP_PROJECT}
          gcloud config set compute/zone ${GCP_ZONE}
          gcloud --quiet container clusters get-credentials ${GCP_KUBE_CLUSTER}
      - run:
        name: Delete stale instances
        command: |
          KUBE_FEATURE_BRANCHES=`kubectl get po -n myapp -l 'app.kubernetes.io/name=myapp' -o jsonpath="{.items[*].metadata.labels['app\.kubernetes\.io/instance']}"`
          GIT_BRANCHES=`git branch --list --remote | sed 's/  origin\///g'`
          for kube_branch in $KUBE_FEATURE_BRANCHES; do
            echo "Checking branch ${kube_branch}"
            git_branch_still_exists=false
            for git_branch in $GIT_BRANCHES; do
              if [ "${git_branch}" == "${kube_branch}" ]; then
                git_branch_still_exists=true
                echo "Git branch ${git_branch} still exists, instance ${kube_branch} will not be deleted."
              fi
            done
            if [ "${git_branch_still_exists}" = false ]; then
              echo "Git branch corresponding to instance ${kube_branch} has been deleted, deleting instance..."
              kubectl delete all,ingressroute -n myapp -l app.kubernetes.io/instance=$kube_branch
            fi
          done
```

### Add the job in a new workflow triggered by a CRON

```
workflows:
  version: 2
  build-test-and-deploy:
    [...]
  delete-stale-deployments:
    triggers:
      - schedule:
        cron: "0 0,6,12,18 * * *" # Every six hours
        filters:
          branches:
            only:
              - master
    jobs:
          - delete-stale-ci-stacks
```

## Conclusion

For a toy project, the cost of this infrastructure is about $100 a month. For real-world projects with large architectures we average $500 per project per month, and the *pay-as-you-go* pricing model of the services we use guarantees this cost to be quite linear. This might look like a lot, but in fact it is ridiculously low when you take into account the billable time not spent on maintaining this type of platform ourselves (or being slowed down by it!).

As of this writing, we have been using these tools and techniques for all our projects at Zenika Labs for more than two years. Feel free to use it, share it, suggest improvements or ask any question!