Automating GKE creation on CircleCI builds
Tags: devops, kubernetes, intersystems, circleci
Published: 2020-07-31 07:34:28

Creating GKE cluster manually (or through [gcloud](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster)) is easy, but the modern [Infrastructure-as-Code (IaC) approach](https://martinfowler.com/bliki/InfrastructureAsCode.html) advises that the description of the Kubernetes cluster should be stored in the repository as code as well. How to write this code is determined by the tool that’s used for IaC.

In the case of Google Cloud, there are [several options](https://cloud.google.com/solutions/infrastructure-as-code/#cards), among them [Deployment Manager](https://cloud.google.com/deployment-manager/docs/) and [Terraform](https://www.terraform.io/). Opinions are divided as to which is better: if you want to learn more, read this Reddit thread [Opinions on Terraform vs. Deployment Manager?](https://www.reddit.com/r/googlecloud/comments/9nzchm/opinions_on_terraform_vs_deployment_manager/) and the Medium article [Comparing GCP Deployment Manager and Terraform](https://medium.com/@kari.marttila/comparing-gcp-deployment-manager-and-terraform-3bc6e1b3aa2d). 

For this article we’ll choose Terraform, since it’s less tied to a specific vendor and you can use your IaC with different cloud providers.

We’ll assume you already have a [Google account](https://console.cloud.google.com/), and that you’ve created a project named, for instance, "Development". In this article, its ID is shown as \<PROJECT_ID\>. In the examples below, change it to [the ID of your own project](https://support.google.com/googleapi/answer/7014113?hl=en). 

Keep in mind that Google isn’t free, although it has a [free tier](https://cloud.google.com/free/). Be sure to [control your expenses](https://cloud.google.com/billing/docs/).

You should fork the [original repository](https://github.com/intersystems-community/objectscript-rest-docker-template). We’ll call this fork “my-objectscript-rest-docker-template” and refer to its root directory as "\<root_repo_dir\>" throughout this article.

All code samples are stored in [this repo](https://github.com/intersystems-community/gke-terraform-circleci-objectscript-rest-docker-template) to simplify copying and pasting.

The following diagram depicts the whole deployment process in one picture:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ymei7t8qldkuwg9d84xd.png)

So, let's [install](https://learn.hashicorp.com/terraform/getting-started/install.html) the latest version of Terraform at the time of initial writing:

```
$ terraform version
Terraform v0.12.17
```

The version is important here, because many examples on the Internet use earlier versions, and 0.12 brought [many changes](https://www.hashicorp.com/blog/announcing-terraform-0-12/). **Update:** 0.13 brought yet [more changes](https://www.hashicorp.com/blog/announcing-the-terraform-0-13-beta/).

We want Terraform to perform certain actions (use certain APIs) in our GCP account. To enable this, [create a Service Account](https://cloud.google.com/iam/docs/service-accounts) with the name 'terraform', and enable the Kubernetes Engine API. Don’t worry about how we’re going to achieve this — just read further and your questions will be addressed.

Let's try an example with the [gcloud utility](https://cloud.google.com/sdk/gcloud/), although we could also use the [Web Console](https://console.cloud.google.com/iam-admin/serviceaccounts).

We're going to use a couple different commands in the following examples. See the following documentation topics for more details on these commands and features:
* [gcloud iam service-accounts create](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-gcloud)
* [Granting roles to a service account for specific resources](https://cloud.google.com/iam/docs/granting-roles-to-service-accounts#granting_access_to_a_service_account_for_a_resource)
* [gcloud iam service-accounts keys create](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-gcloud)
* [Enabling an API in your Google Cloud project](https://cloud.google.com/endpoints/docs/openapi/enable-api)

Now let's walk through the example.

```
$ gcloud init
```
We won’t discuss all of the setup details here. You can read a little more in [this article](https://community.intersystems.com/post/deploying-intersystems-iris-solution-gcp-kubernetes-cluster-gke-using-circleci). For this example, run the following commands:

```
$ cd <root_repo_dir>
$ mkdir terraform; cd terraform
$ gcloud iam service-accounts create terraform --description "Terraform" --display-name "terraform"
```

Now let's add a few roles to the terraform service account besides “Kubernetes Engine Admin” (container.admin). These roles will be useful to us in the future.

```
$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/container.admin

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/iam.serviceAccountUser

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/compute.viewer

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/storage.admin

$ gcloud iam service-accounts keys create account.json \
  --iam-account terraform@<PROJECT_ID>.iam.gserviceaccount.com

```
Note that the last entry creates your *account.json* file. Be sure to keep this file secret.

```
$ gcloud projects list
$ gcloud config set project <PROJECT_ID>
$ gcloud services list --available | grep 'Kubernetes Engine'
$ gcloud services enable container.googleapis.com
$ gcloud services list --enabled | grep 'Kubernetes Engine'
container.googleapis.com Kubernetes Engine API
```

Next, let’s describe the GKE cluster in Terraform’s [HCL](https://www.terraform.io/docs/glossary.html#hcl) language. Note that we use several placeholders here; replace them with your values:

| Placeholder | Meaning | Example |
| --- | --- | --- |
| \<PROJECT_ID\> | GCP project ID | possible-symbol-254507 |
| \<BUCKET_NAME\> | Storage for Terraform state - should be [unique](https://cloud.google.com/storage/docs/naming) | circleci-gke-terraform-demo |
| \<REGION\> | [Region](https://cloud.google.com/compute/docs/regions-zones/) where resources will be created | europe-west1 |
| \<LOCATION\> | [Zone](https://cloud.google.com/compute/docs/regions-zones/) where resources will be created | europe-west1-b |
| \<CLUSTER_NAME\> | GKE cluster name | dev-cluster |
| \<NODES_POOL_NAME\> | GKE worker nodes pool name | dev-cluster-node-pool |

Here’s the HCL configuration for the cluster in practice:

```
$ cat main.tf
terraform {
  required_version = "~> 0.12"
  backend "gcs" {
    bucket = "<BUCKET_NAME>"
    prefix = "terraform/state"
    credentials = "account.json"
  }
}

provider "google" {
  credentials = file("account.json")
  project = "<PROJECT_ID>"
  region = "<REGION>"
}

resource "google_container_cluster" "gke-cluster" {
  name = "<CLUSTER_NAME>"
  location = "<LOCATION>"
  remove_default_node_pool = true
  # In regional cluster (location is region, not zone) 
  # this is a number of nodes per zone 
  initial_node_count = 1
}

resource "google_container_node_pool" "preemptible_node_pool" {
  name = "<NODES_POOL_NAME>"
  location = "<LOCATION>"
  cluster = google_container_cluster.gke-cluster.name
  # In regional cluster (location is region, not zone) 
  # this is a number of nodes per zone
  node_count = 1

  node_config {
    preemptible = true
    machine_type = "n1-standard-1"
    oauth_scopes = [
      "storage-ro",
      "logging-write",
      "monitoring"
    ]
  }
}

```

To make sure the HCL code is in the proper format, Terraform provides a handy formatting command you can use:

```
$ terraform fmt
```

The code snippet shown above indicates that the created resources will be [provided by Google](https://www.terraform.io/docs/providers/google/guides/provider_reference.html), and the resources themselves are [google_container_cluster](https://www.terraform.io/docs/providers/google/r/container_cluster.html) and google_container_node_pool, which we designate [preemptible](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms) for costs savings. We also choose to create [our own pool](https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools) instead of using the default.

Let’s focus briefly on the following setting:
```
terraform {
  required_version = "~> 0.12"
  backend "gcs" {
    Bucket = "<BUCKET_NAME>"
    Prefix = "terraform/state"
    credentials = "account.json"
  }
}
```
Terraform writes everything it's done into the status file and then uses this file for other work. For convenient sharing, it’s better to store this file somewhere in a remote place. A typical place is a [Google Bucket](https://cloud.google.com/storage/docs/key-terms#buckets).

Let's create this bucket. Use the name of your bucket instead of the placeholder \<BUCKET_NAME\>. Before bucket creation let’s check if \<BUCKET_NAME\> is available as it has to be unique across all GCP:

```
$ gsutil acl get gs://<BUCKET_NAME>
```

Good answer:

```
BucketNotFoundException: 404 gs://<BUCKET_NAME> bucket does not exist
```

"Busy" answer means you have to choose another name:

```
AccessDeniedException: 403 <YOUR_ACCOUNT> does not have storage.buckets.get access to <BUCKET_NAME>
```

Let’s also enable versioning, as [Terraform recommends](https://www.terraform.io/docs/backends/types/gcs.html).

```
$ gsutil mb -l EU gs://<BUCKET_NAME>

$ gsutil versioning get gs://<BUCKET_NAME>
gs://<BUCKET_NAME>: Suspended

$ gsutil versioning set on gs://<BUCKET_NAME>

$ gsutil versioning get gs://<BUCKET_NAME>
gs://<BUCKET_NAME>: Enabled
```
Terraform is modular and needs to add a Google provider plugin to create something in GCP. We use the following command to do this:

```
$ terraform init
```
Let's look at what Terraform is going to do to create a GKE cluster:

```
$ terraform plan -out dev-cluster.plan
```

The command output includes details of the plan. If you have no objections, let's implement this plan:

```
$ terraform apply dev-cluster.plan
```

By the way, to delete the resources created by Terraform, run this command from the \<root_repo_dir\>/terraform/ directory:

```
$ terraform destroy -auto-approve
```

Let’s leave the cluster as is for a while and move on. But first note that we don’t want to push everything into the repository, so we’ll add several files to the exceptions:

```
$ cat <root_repo_dir>/.gitignore
.DS_Store
terraform/.terraform/
terraform/*.plan
terraform/*.json
```

#### Using Helm

As described in [this article](https://community.intersystems.com/post/deploying-intersystems-iris-solution-gcp-kubernetes-cluster-gke-using-circleci), we could store Kubernetes manifests as yaml files in the \<root_repo_dir\>/k8s/ directory, which we then sent to the cluster using the "kubectl apply" command. 

This time we'll try a different approach: using the Kubernetes package manager [Helm](https://helm.sh/docs/), which currently has [version 3](https://helm.sh/blog/helm-3-released/). Please, use version 3 or later because version 2 had Kubernetes-side security issues (see [Running Helm in production: Security best practices](https://engineering.bitnami.com/articles/running-helm-in-production.html) for details). First, we’ll pack the Kubernetes manifests from our k8s/ directory into a Helm package, which is known as a [chart](https://helm.sh/docs/topics/charts/#the-chart-file-structure). A Helm chart installed in Kubernetes is called a release. In a minimal configuration, a chart will consist of several files:

```
$ mkdir <root_repo_dir>/helm; cd <root_repo_dir>/helm
$ tree <root_repo_dir>/helm/
helm/
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   └── service.yaml
└── values.yaml
```

Their purpose is well-described on [the official site](https://helm.sh/docs/topics/charts/). The best practices for creating your own charts are described in the [The Chart Best Practices Guide](https://helm.sh/docs/chart_best_practices/) in the Helm documentation. 

Here’s what the contents of our files look like:

```
$ cat Chart.yaml
apiVersion: v2
name: iris-rest
version: 0.1.0
appVersion: 1.0.3
description: Helm for ObjectScript-REST-Docker-template application
sources:
- https://github.com/intersystems-community/objectscript-rest-docker-template
- https://github.com/intersystems-community/gke-terraform-circleci-objectscript-rest-docker-template
```

```
$ cat templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "iris-rest.name" . }}
  labels:
    app: {{ template "iris-rest.name" . }}
    chart: {{ template "iris-rest.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  strategy:
    {{- .Values.strategy | nindent 4 }}
  selector:
    matchLabels:
      app: {{ template "iris-rest.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "iris-rest.name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
      - image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        name: {{ template "iris-rest.name" . }}
        ports:
        - containerPort: {{ .Values.webPort.value }}
          name: {{ .Values.webPort.name }}
```
```
$ cat templates/service.yaml
{{- if .Values.service.enabled }}
apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name }}
  labels:
    app: {{ template "iris-rest.name" . }}
    chart: {{ template "iris-rest.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  selector:
    app: {{ template "iris-rest.name" . }}
    release: {{ .Release.Name }}
  ports:
  {{- range $key, $value := .Values.service.ports }}
    - name: {{ $key }}
{{ toYaml $value | indent 6 }}
  {{- end }}
  type: {{ .Values.service.type }}
  {{- if ne .Values.service.loadBalancerIP "" }}
  loadBalancerIP: {{ .Values.service.loadBalancerIP }}
  {{- end }}
{{- end }}
```

```
$ cat templates/_helpers.tpl
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}

{{- define "iris-rest.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "iris-rest.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

```

```
$ cat values.yaml
namespaceOverride: iris-rest

replicaCount: 1

strategy: |
  type: Recreate

image:
  repository: eu.gcr.io/iris-rest
  tag: v1

webPort:
  name: web
  value: 52773

service:
  enabled: true
  name: iris-rest
  type: LoadBalancer
  loadBalancerIP: ""
  ports:
    web:
      port: 52773
      targetPort: 52773
      protocol: TCP

```

To create the Helm charts, install the [Helm client](https://github.com/helm/helm/releases/tag/v3.0.1) and the [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) command-line utility.

```
$ helm version
version.BuildInfo{Version:"v3.0.1", GitCommit:"7c22ef9ce89e0ebeb7125ba2ebf7d421f3e82ffa", GitTreeState:"clean", GoVersion:"go1.13.4"}
```
Create a namespace called "iris". It would be nice if this was created during the deployment, but [initially it was not the case](https://github.com/helm/helm/issues/6794). **Update**: looks like, *--create-namespace* flag is currently supported.

First, add credentials for the cluster created by Terraform to kube-config:

```
$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>
$ kubectl create ns iris
```

Confirm (without kicking off a real deploy) that Helm is going to create the following in Kubernetes:

```
$ cd <root_repo_dir>/helm
$ helm upgrade iris-rest \
  --install \
  . \
  --namespace iris \
  --debug \
  --dry-run
```

The output - the Kubernetes manifests - has been omitted for space here. If everything looks good, let’s deploy:

```
$ helm upgrade iris-rest --install . --namespace iris

$ helm list -n iris --all
iris-rest  iris  1  2019-12-14 15:24:19.292227564  +0200  EET  deployed    iris-rest-0.1.0  1.0.3
```

We see that Helm has deployed our application, but since we haven’t created the Docker image *eu.gcr.io/iris-rest:v1* yet, Kubernetes can’t pull it (ImagePullBackOff):

```
$ kubectl -n iris get po
NAME                       READY  STATUS            RESTARTS AGE
iris-rest-59b748c577-6cnrt 0/1    ImagePullBackOff  0         10m
```

Let’s finish with it for now:

```
$ helm delete iris-rest -n iris
```

#### The CircleCI Side

Now that we’ve tried out Terraform and the Helm client, let’s put them to use during the deployment process on the CircleCI side.

```
$ cat <root_repo_dir>/.circleci/config.yml
version: 2.1

orbs:
  gcp-gcr: circleci/gcp-gcr@0.6.1

jobs:
  terraform:
    docker:
    # Terraform image version should be the same as when
    # you run terraform before from the local machine
      - image: hashicorp/terraform:0.12.17
    steps:
      - checkout
      - run:
          name: Create Service Account key file from environment variable
          working_directory: terraform
          command: echo ${TF_SERVICE_ACCOUNT_KEY} > account.json
      - run:
          name: Show Terraform version
          command: terraform version
      - run:
          name: Download required Terraform plugins
          working_directory: terraform
          command: terraform init
      - run:
          name: Validate Terraform configuration
          working_directory: terraform
          command: terraform validate
      - run:
          name: Create Terraform plan
          working_directory: terraform
          command: terraform plan -out /tmp/tf.plan
      - run:
          name: Run Terraform plan
          working_directory: terraform
          command: terraform apply /tmp/tf.plan
  k8s_deploy:
    docker:
      - image: kiwigrid/gcloud-kubectl-helm:3.0.1-272.0.0-218
    steps:
      - checkout
      - run:
          name: Authorize gcloud on GKE
          working_directory: helm
          command: |
            echo ${GCLOUD_SERVICE_KEY} > gcloud-service-key.json
            gcloud auth activate-service-account --key-file=gcloud-service-key.json
            gcloud container clusters get-credentials ${GKE_CLUSTER_NAME} --zone ${GOOGLE_COMPUTE_ZONE} --project ${GOOGLE_PROJECT_ID}
      - run:
          name: Wait a little until k8s worker nodes up
          command: sleep 30 # It’s a place for improvement
      - run:
          name: Create IRIS namespace if it doesn't exist
          command: kubectl get ns iris || kubectl create ns iris
      - run:
          name: Run Helm release deployment
          working_directory: helm
          command: |
            helm upgrade iris-rest \
              --install \
              . \
              --namespace iris \
              --wait \
              --timeout 300s \
              --atomic \
              --set image.repository=eu.gcr.io/${GOOGLE_PROJECT_ID}/iris-rest \
              --set image.tag=${CIRCLE_SHA1}
      - run:
          name: Check Helm release status
          command: helm list --all-namespaces --all
      - run:
          name: Check Kubernetes resources status
          command: |
            kubectl -n iris get pods
            echo
            kubectl -n iris get services
workflows:
  main:
    jobs:
      - terraform
      - gcp-gcr/build-and-push-image:
          dockerfile: Dockerfile
          gcloud-service-key: GCLOUD_SERVICE_KEY
          google-compute-zone: GOOGLE_COMPUTE_ZONE
          google-project-id: GOOGLE_PROJECT_ID
          registry-url: eu.gcr.io
          image: iris-rest
          path: .
          tag: ${CIRCLE_SHA1}
      - k8s_deploy:
          requires:
            - terraform
            - gcp-gcr/build-and-push-image

```
You’ll need to add several [environment variables](https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-project) to your project on CircleCI side:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/w4mb2wnytd1soafu7714.png)

The GCLOUD_SERVICE_KEY is the CircleCI service account key, and TF_SERVICE_ACCOUNT_KEY is the Terraform service account key. Recall that the service account key is the whole content of *account.json* file.

Next, let’s push our changes to a repository:
```
$ cd <root_repo_dir>
$ git add .circleci/ helm/ terraform/ .gitignore
$ git commit -m "Add Terraform and Helm"
$ git push
```

The CircleCI UI dashboard should show that everything is ok:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/nja60f4tjy45qi8gtmr5.png)

Terraform is an idempotent tool and if the GKE cluster is present, the "terraform" job won’t do anything. If the cluster doesn’t exist, it will be created before Kubernetes deployment.
Finally, let’s check IRIS availability:

```
$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>

$ kubectl -n iris get svc
NAME      TYPE          CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE   
iris-rest LoadBalancer  10.23.249.42  34.76.130.11  52773:31603/TCP   53s

$ curl -XPOST -H "Content-Type: application/json" -u _system:SYS 34.76.130.11:52773/person/ -d '{"Name":"John Dou"}'

$ curl -XGET -u _system:SYS 34.76.130.11:52773/person/all
[{"Name":"John Dou"},]

```

#### Conclusion

Terraform and Helm are standard DevOps tools and should be fine integrated with IRIS deployment.

They do require some learning, but after some practice, they can really save you time and effort.