Containerising and Deploying with k8s
Tags: kotlin, kubernetes, ci, docker
Published: 2020-07-25 15:55:18

---
title: Containerising and Deploying with k8s
published: true
description: 
tags: ["Kotlin", "Kubernetes", "CI", "Docker"]
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/hk1eaootda7evq3wpaf3.jpg
---

This started less an article or blog, more a journal of my many failed attempts at containerising and deploying a small app. I was keen to practice with the technologies I use every day, but without the guardrails of a fantastic platform team with all the tools and support they provide.

The following describes the (eventual!) successful steps to...

- Package a `Kotlin` web application with environment secrets in a `Docker` image

- Upload the image to a container registry, and run both manually and as part of a `CI build pipeline`

- Create a `Kubernetes` (k8s) cluster and deploy the app, both manually and as part of the CI build pipeline

- Expose the application to the internet


The code for the Tempo-Valence application used can be [seen here](https://gitlab.com/matt34Tea/tempo-valence)


------


## 1. Package app in a docker image

- Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format.

- Using `gcr.io` container registry (after failing with other registry providers - see out takes below!)

    - Enable `Cloud Registry API` in Google Cloud Platform (GCP) console
    - Set up Service Account in GCP `Service Accounts - IAM & Admin` so that this can be used to access the Google container registry (gcr). This service account will need to have a `Storage Admin` role

    - Download `key` from created GCP service account and add it to a gitlab-ci variable - called `GOOGLE_CREDENTIALS` in my example, which is accessed in the gitlab-ci.yml file below
    - (The key can also be saved to a local, non-committed keyfile somewhere if you wish to run the jib docker image locally to check)

    - Update build step of `gitlab-ci.yml` to...

```yaml
build:
    <<: *gradle-image
    stage: build
    variables:
        CLIENT_KEY: $CLIENT_KEY
        GOOGLE_CREDENTIALS: $GOOGLE_CREDENTIALS
    before_script:
        - echo "$GOOGLE_CREDENTIALS" > keyfile.json
    script:
        - export CLIENT_KEY="$CLIENT_KEY"
        - gradle jib -Djib.console=plain
        - gradle clean build
    only:
        - master
```

   - Add `jib` config in `build.gradle.kts` of the project as follows...
    
```gradle
jib {
    container {
        mainClass = application.mainClassName
        environment = mapOf(Pair("CLIENT_KEY", System.getenv("CLIENT_KEY")))
        ports = listOf("8000")
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "gcr.io/<gcp-project-name>/tempo-valence:latest"
        auth {
            username = "_json_key"
            password = file("keyfile.json").readText()
        }
    }
}
```

   (The `CLIENT_KEY` environment variable above refers to a secret env var required to access an external api from the application. The value of this env var is stored as a Variable in Gitlab CI so the pipeline can access to run the app.)


------


## 2. Upload docker image to a container registry

- See previous step

- To run locally...

```bash
docker run --rm -p 9000:9000 gcr.io/<gcp-project-name>/tempo-valence:latest

# in new terminal tab
curl http://localhost:9090
# Welcome to TempoValence!
```


------


## 3. Create a k8s container cluster

Following manual commands create cluster, deploy application and expose on an external IP

```bash
gcloud auth list

gcloud config set account m.........@gmail.com

gcloud config set project <gcp-project-name>

gcloud config set compute/zone europe-west2-a

gcloud container clusters create tempo-valence-cluster

gcloud compute instances list

kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl get deployment

kubectl get pods

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# the ports here matter - think it has something to do with how we've set up the application in the image

kubectl get service

# and use the EXTERNAL_IP of tempo-valence-service to access, e.g...
http://35.234.150.122:9000/
```


------


## 4. Deploy

- See step 3 for manual deployment

- In order to deploy from the gitlab-ci pipeline, the `gitlab-ci.yml` looked like this for the deploy stage...

```yaml
deploy:
stage: deploy
variables:
    CI_SERVICE_ACCOUNT: $CI_SERVICE_ACCOUNT
image:
    name: kiwigrid/gcloud-kubectl-helm
before_script:
    - echo "$CI_SERVICE_ACCOUNT" > key.json
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project <gcp-project-name>
    - gcloud config set container/cluster tempo-valence-cluster
    - gcloud config set compute/zone europe-west2-a
    - gcloud container clusters get-credentials tempo-valence-cluster --zone europe-west2-a
script:
    - kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
    - kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
    # Wait for the deployment to be applied
    - kubectl rollout status deployment tempo-valence --watch
```

### Few things to note here!

- Set up another service account in GCP, to be used by gitlab ci to authenticate against both the container registry and k8s cluster (may be possible just to amend the roles of previous service account, but I added a new one to keep it separate)

  - Give the service account 2 project roles -> `Kubernetes Engine Developer` and `Storage Object Admin`

  - Download the key associated to the account and add in to a new CI variable in gitlab

- The `image` is the image used to run the deployment - bit of trial and error here, as this image needed to have both `kubectl` and `gcloud` cli tools loaded (and I couldn't be bothered to create one and store it somewhere that didn't need authentication!) The image included above is open and does the job.

- For this to work, the `tempo-valence-cluster` of course needs to be up and running 


------
------


## Steps to run

As this will not be continually running here are the steps needed before this will be available on the internet


------


### To setup

```bash
gcloud container clusters create tempo-valence-cluster

# Run ci pipeline, or...
kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
```


------


### To teardown

```bash
kubectl delete service tempo-valence-service

gcloud container clusters delete tempo-valence-cluster
```


------
------


## Links and article credits

- A lot of ideas from this [Google k8s tutorial](https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app)

- And this [GCP tutorial](https://cloud.google.com/run/docs/quickstarts/build-and-deploy)

- One of our own from Matt Dowds on [Kubernetes deployment form Gitlab CI](https://medium.com/john-lewis-software-engineering/deploying-to-google-kubernetes-engine-from-gitlab-ci-feaf51dae0c1)


------
------


## Outtakes - my many failed attempts at step 1!

Package app in a docker image
- Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format. To build a Docker image, you need to have an application and a Dockerfile (in that application)
- Add `Dockerfile` to root of project. 1st attempt looked like this...

```Dockerfile
# use base image
FROM openjdk:11

# set working directory
WORKDIR /

# copy the runlocal.sh file (contains env vars and gradle run command)
COPY runlocal.sh .

# Inform Docker that the container is listening on the specified port at runtime
EXPOSE 8080

#Run the specified command within the container
RUN ./runlocal.sh

# Copy the rest of your app source code from your host to your image filesystem
COPY . .
```

- Run the following command to build and test docker image locally...
```bash
docker image build -t tempo-valence:1.0 .
```

  Sadly this exited in error...

```bash
Step 5/6 : RUN ./runlocal.sh
---> Running in 7d1ddc474a6f
./runlocal.sh: line 5: ./gradlew: No such file or directory
The command '/bin/sh -c ./runlocal.sh' returned a non-zero code: 127
``` 

- So next I tried working with a gradle base image...

```Dockerfile
FROM gradle:5.4.1-jdk11 AS build
```

Gradle started, but sadly a config error was returned...

```bash
FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project ''.
```

- Then looked in to `--env-file` and using other ways of getting the relevant client key into the image without exposing it. Finally added the env_var to a `.env` file and accessed in `Dockerfile` as follows...

```Dockerfile
FROM openjdk:11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY .env

COPY . .
```

This successfully builds and tags the image, but my concern is now that this `.env` file is only available locally, as it is .gitignore'd
    

- Running the image as a container using

```bash
docker container run --publish 8000:8080 --detach --name tv tempo-valence:1.0

# --publish 8000:8080 asks Docker to forward traffic incoming on the host’s port 8000, to the container’s port 8080. Containers have their own private set of ports, so if you want to reach one from the network, you have to forward traffic to it in this way. Otherwise, firewall rules will prevent all network traffic from reaching your container, as default

# --detach asks Docker to run this container in the background

# --name specifies a name with which you can refer to your container in subsequent commands, in this case tv
```

didn't run the app, as we have no current run command in the `Dockerfile`


So, added a couple of bits back in the `Dockerfile`...

```Dockerfile
FROM gradle:5.4.1-jdk11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY $CLIENT_KEY

CMD [ "gradle", "run" ]

COPY . .
```

This successfully runs the container, but I cannot access the app via the browser on `http://localhost:8000` as expected


Some links being used so far...
- https://docs.docker.com/get-started/part2/
- https://www.reddit.com/r/docker/comments/7dhszj/dockercompose_how_do_i_expose_a_port_to_my/

Some useful commands...
- `docker exec tv env` (lists env vars set in the container)


------

**New attempts... (13.03.20)**

- Tried using [`jib`](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin) to containerise the app

- Added the following into `build.gradle` file...

```gradle
plugins {
    id 'com.google.cloud.tools.jib' version '2.1.0'
}

jib {
    container {
        mainClass = application.mainClassName
        environment = [CLIENT_KEY:System.getenv("CLIENT_KEY")]
        ports = ['8000']
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "matttea/matttea-images/tempo-valence:latest"
    }
}
```

- Run the following commands to build...

```bash
export CLIENT_KEY=<client_key_value>

gradle jib
```

- Returned error related to authentication...

```bash
Containerizing application to matttea/matttea-images/tempo-valence...
Base image 'openjdk:11' does not use a specific image digest - build may not be reproducible
The credential helper (docker-credential-desktop) has nothing for server URL: registry-1.docker.io

Got output:

credentials not found in native keychain

...

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build image failed, perhaps you should make sure your credentials for 'registry-1.docker.io/matttea/matttea-images/tempo-valence' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help
```

- Using [this link](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#authentication-methods)

- See also [this journal entry on docker credentials helper](https://github.com/mattTea/journal/blob/master/2020/03-March/2020-03-14.md)

------

**New attempts... (25.05.2020)**

- tried changing job image to detail to `image  = "https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest"` based on `docker logout` command url response

- didn't work...

```bash
> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> Invalid image reference https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest, perhaps you should check that the reference is formatted correctly according to https://docs.docker.com/engine/reference/commandline/tag/#extended-description
For example, slash-separated name components cannot have uppercase letters
```

_(Originally posted on https://logical-progression.matttea.com)_