Applying Workload Identity With A Demo
Tags: googlecloud, cloud, diy, terraform
Published: 2023-03-13 04:20:19

## Overview

As part of the [first part](https://dev.to/kameshsampath/what-is-workload-identity--120l) of the series we understood what is **Workload Identity**. In this DIY blog we will apply [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) to our GKE workloads by deploying a demo application called **lingua-greeter** to GKE, the lingua-greeter will call [Translate API](https://cloud.google.com/translate) to translate the greeting text passed to it.

## Glossary

| Abbreviation | Expansion
| :----:    |:----------:
| API | Application Programming Interface
| ACL | Access Control List|
| GCS | Google Cloud Storage
| GKE | Google Kubernetes Engine
| GSA | Google Service Account
| IAM | Identity and Access Management
| KSA | Kubernetes Service Account
| RBAC| Role Based Access Control
| SA  | Service Account
| VPC | Virtual Private Cloud


## Pre-requisites

- [Google Cloud Account](https://cloud.google.com)
- With a [Service Account](https://cloud.google.com/iam/docs/service-account-overview) with roles:
  - `Kubernetes Engine Admin` - to create GKE cluster
  - `Service Account` roles used to create/update/delete Service Account
    - *iam.serviceAccounts.actAs*
    - *iam.serviceAccounts.get*
    - *iam.serviceAccounts.create*
    - *iam.serviceAccounts.delete*
    - *iam.serviceAccounts.update*
    - *iam.serviceAccounts.get*
    - *iam.serviceAccounts.getIamPolicy*
    - *iam.serviceAccounts.setIamPolicy*
   Or simply you can add `Service Account Admin` and `Service Account User` roles
  - `Compute Network Admin`   - to create the VPC networks
- Enable [Translation API](https://console.cloud.google.com/apis/library/translate.googleapis.com) on your Google Cloud Account
- [Google Cloud SDK](https://cloud.google.com/sdk)
- [terraform](https://terraform.build)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Taskfile](https://taskfile.dev)

### Optional

- [kustomize](https://kustomize.io)(Optional)
- [direnv](https://direnv.net)(Optional)

## Download Sources

```shell
git clone https://github.com/kameshsampath/workload-identiy-gke-demo && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"
```
## Environment Setup

### Variables

When working with Google Cloud the following environment variables helps in setting the right Google Cloud context like Service Account Key file, project etc., You can use [direnv](https://direnv.net) or set the following variables on your shell,

```shell
export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube"
```

(e.g.)

```shell
export CLOUDSDK_ACTIVE_CONFIG_NAME=personal
export GOOGLE_APPLICATION_CREDENTIALS=~/.ssh/my-sa-key.json
export GOOGLE_CLOUD_PROJECT=my-awesome-project
export KUBECONFIG="$DEMO_HOME/.kube"
```

> **TIP** If you are using direnv you can then create file `.envrc.local` and add the environment variables. They can then be loaded using `direnv allow .`

You can find more information about gcloud cli configurations at <https://cloud.google.com/sdk/docs/configurations>.

We will be using [terraform](https://terraform.build) to create the Google Cloud resources e.g. GKE Cluster with **Workload Identity** enabled, Google Service Accounts(GSA), [IAM](https://cloud.google.com/iam/docs/overview) policies and bindings.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called `.local.tfvars`. Set the following environment variable to make terraform use the variable values form the file `.local.tfvars`

```shell
export TFVARS_FILE=.local.tfvars
```
Check the [Inputs](https://github.com/kameshsampath/workload-identity-gke-demo#inputs) section for all possible terraform variables that are configurable.

### Example
  
An example `.local.tfvars` that will use a Google Cloud project **my-awesome-project**, create a two node GKE cluster named **wi-demo** in region **asia-south1** with Kubernetes version **1.24.** from **stable** release channel. The machine type of each cluster node will be **e2-standard-4**. The demo will be deployed in Kubernetes namespace **demo-apps**, will use **lingua-greeter** as the Kubernetes Service Account.

```hcl
app_ksa            = "lingua-greeter"
app_namespace      = "demo-apps"
cluster_name       = "wi-demo"
configure_app_workload_identity = false
gke_num_nodes      = 2
kubernetes_version = "1.24."
machine_type       = "e2-standard-4"
project_id         = "my-awesome-project"
region             = "asia-south1"
release_channel    = "stable"
```

> **NOTE**: For rest of the section we assume that your tfvars file is called `.local.tfvars`

## Application Overview

As part of the demo, let us deploy a Kubernetes application called `lingua-greeter`. The application exposes a REST API `/:lang` , that allows you to translate a text `Hello World!` into the language `:lang` using Google Translate client.

> **NOTE**: The `:lang` is the [BCP 47](https://en.wikipedia.org/wiki/IETF_language_tag) language code.
>

![Apply Workload Identity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cwxe3acyxlgloo8b17rq.png)

## Create Environment

We will use terraform to create a GKE cluster with `WorkloadIdentity` enabled for its nodes,

Initialize terraform and download its modules,

```shell
task init
```

### Create GKE cluster

The terraform apply will creates a Kubernetes(GKE) Cluster,

```shell
task create_cluster
```

The terraform apply will create the following Google Cloud resources,

- A Kubernetes cluster on GKE
- A Google Cloud VPC that will be used with GKE
- GKE is configured to use [Workload Identity Pool](https://github.com/kameshsampath/workload-identity-gke-demo/blob/main/gke.tf#L41). As refresher check [How Workload Identity Works](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#how_works).

## Deploy Application

To see **Workload Identity** in action we will deploy the application(workload) on to GKE in two parts,

- Application is **not** enabled for Workload Identity
- Application **enabled** for Workload Identity

### Without Workload Identity Enabled

Create the namespace `demo-apps` to deploy the `lingua-greeter` application,

```shell
kubectl create ns demo-apps
```

Run the following command to deploy the application,

```shell
kubectl apply -n demo-apps -k $DEMO_HOME/app/config
```

Wait for application to be ready,

```shell
kubectl rollout status -n demo-apps deployment/lingua-greeter --timeout=60s
```

Get the application service LoadBalancer IP,

```shell
kubectl get svc -n demo-apps lingua-greeter
```

> **NOTE**: If the `EXTERNAL-IP` is `<pending>` then wait for the IP to be assigned. It will take few minutes for the `EXTERNAL-IP` to be assigned.
> You can use the following command to wait until `External-IP` is assigned,
>
>```shell
> while [ -z $(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}") ]; do sleep .3; done;
>```

### Call Service

```shell
export SERVICE_IP=$(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}")
```

Call the service to return the translation of `Hello World!` in [Tamil(ta)](https://en.wikipedia.org/wiki/Tamil_language),

```shell
curl "http://$SERVICE_IP/ta"
```

The service should fail with a message,

```text
{"message":"Internal Server Error"}
```

When you check the logs of the `lingua-greeter` pod,

```shell
kubectl logs -n demo-apps -lapp=lingua-greeter
```

You should see a message like,

```text
 ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.10.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8080
time="2023-03-10T07:36:35Z" level=error msg="googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\nMore details:\nReason: authError, Message: Invalid Credentials\n"
```

As it describes you don't have authentication credentials to call the API. All Google Cloud API requires `GOOGLE_APPLICATION_CREDENTIALS` to allow client to authenticate itself before calling the API. If you check the [deployment manifest](./../app/config/deployment.yaml) we dont have one configured.

### Configure Application To Use Workload Identity

Run the following command to configure the application to use **Workload Identity**,

```shell
task use_workload_identity
```

The terraform script [rbac.tf](https://github.com/kameshsampath/workload-identity-gke-demo/blob/main/rbac.tf) does the following,

1. Create a GSA called `translator` 
2. Add role `roles/iam.workloadIdentityUser` to `translator` with a single member `serviceAccount:$GOOGLE_CLOUD_PROJECT.svc.id.goog[demo-apps/lingua-greeter]`. This basically allows KSA to impersonate itself as GSA `translator` thereby allowing it to call Google Cloud services that are allowed for GSA `translator`, in this case to use Google Translate API.
3. Add IAM policy binding to `translator` for role `roles/cloudtranslate.user` which allows it to call the Google Translate API.
4. Finally an updated `lingua-greeter` KSA manifest `$DEMO_HOME/k8s/sa.yaml`, that is annotated with the GSA `client_email` that it is allowed to use, in this case it will be something like `translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com`.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: lingua-greeter
  namespace: demo-apps
  annotations:
    iam.gke.io/gcp-service-account: "translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com"
```

Run the following command to update the Kubernetes SA `lingua-greeter` to use the Google IAM Service Account(GSA) `translator` using **Workload Identity mechanics** and call the Google Translate API,

```shell
kubectl apply -n demo-apps -f "$DEMO_HOME/k8s/sa.yaml"
```

[Call the service](#call-service) again, the service should succeed with a response,

```json
{"text":"Hello World!","translation":"வணக்கம் உலகம்!","translationLanguage":"ta"}
```

> NOTE: Sometimes it may take few seconds for the pods to refresh the metadata, in such cases try to call the service after few seconds.

## Cleanup 

To clean up all the Google Cloud resources that were created as part of this demo, 

```shell
task destroy
```

## Summary

- Deploy GKE cluster with Workload Identity enabled
- Deploy `lingua-greeter` application to GKE
- Create Google Service Account `translator` with permissions to call Google Translate API
- Annotating Kubernetes Service Account `lingua-greeter` with Google Service Account `translator` allowing it to impersonate Google Service Account.