Securing access to Scaleway Elements API Keys from Gitlab CI
Tags: scaleway, security, devops, gitlab
Published: 2021-10-15 11:26:30

How many api keys are stored per day as variables in the Gitlab CI configuration? 

When a `Scaleway Elements API Key` is saved in Gitlab, we face all the security issues of storing credentials outside of the cloud infrastructure: Access, authorization, key rotation, age, destruction, location, etc.

There are 2 common reasons for developers to store GCP credentials in Gitlab CI:

* They use `shared runners`.
* They use `specific/group runners` deployed in a `Scaleway Kubernetes Kapsule` cluster but do not use (or do not know about) [Gitlab additional configurations][gitlab-1].

The alternative that Gitlab CI proposes for users is to mount a secret volume to the runner pods that are created for each build.

# Prerequisites

Install the following tools:

- [Scaleway cli][scw-1]
- [kubectl](https://kubernetes.io/docs/tasks/tools/) 
- [Helm](https://helm.sh/docs/intro/install/)

And create 2 projects in your Scaleway organization:

* devops
* development

# Working with Kapsule

The first step is to create the Kapsule devops cluster and configuring our environment [1]. 

> Generate the API Key for the `devops` project. You can create one by following the documentation [How to generate an API key](scw-2).

```shell
scw init
scw k8s cluster create name=kapsule-devops
```

Let's create a pool for the runner jobs:

```shell
scw k8s pool create cluster-id=$(scw k8s cluster list | grep kapsule-devops | awk '{ print $1 }') name=dev node-type=GP1_XS size=2
```

Add a taint

```shell
kubectl taint nodes gitlab-runner-jobs-dev-reserved=true:NoSchedule --selector=k8s.scaleway.com/pool-name=dev
```

* Configure `kubectl` to communicate with the cluster:

```shell
scw k8s kubeconfig install kapsule-devops
```

* Create the namespace for the dev runner:

```shell
kubectl create namespace dev
```

* Create the Kubernetes service account to use for specific runner:

```shell
kubectl create serviceaccount --namespace dev app-deployer
```

* Generate an [API Key](scw-2) for the specific runner.

> Note: For easier visibility and auditing, I recommend to centrally store API keys in a dedicated project and in an external tools like Vault.

* To allow the specific runner to impersonate the API Key we need to store the credentials in a Kubernetes secret.

```shell
kubectl create secret generic dev-api-key --from-file ~/.config/scw/config.yaml -n dev
```

![Binding Kubernetes Secrets with Scaleway API Keys](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ohnv5z1batvn9ju6z2wn.png)

# Assign the API Key to the Gitlab runner

The next step is to mount the secret as a data volume [4].

* Start by installing Helm:

```shell
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```

* Add Gitlab Helm package:

```shell
helm repo add gitlab https://charts.gitlab.io
```

* Configure the runner: 

Create the file `values.yaml`:

```yaml
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
metrics:
  enabled: true
runners:
  image: ubuntu:18.04
  config: |
   [[runners]]
     [runners.kubernetes]
        [[runners.kubernetes.volumes.secret]]
          name = "dev-api-key"
          mount_path = "/root/.config/scw"
          read_only = true
  locked: true
  pollTimeout: 360
  protected: true
  serviceAccountName: app-deployer
  privileged: false
  secret: dev-runner-tokens
  namespace: dev
  builds:
    cpuRequests: 100m
    memoryRequests: 128Mi
  services:
    cpuRequests: 100m
    memoryRequests: 128Mi
  helpers:
    cpuRequests: 100m
    memoryRequests: 128Mi
  tags: "k8s-dev-runner"
  nodeSelector: 
     k8s.scaleway.com/pool-name: dev
  nodeTolerations:
    - key: "gitlab-runner-jobs-dev-reserved"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
```

> You can find the description of each attribute in the Gitlab runner charts repository [2]

* Get the Gitlab registration token from `Project -> Settings -> CI/CD -> Runners` in the `Setup a specific Runner manually` section and create the following secret:

```shell
kubectl create secret generic dev-runner-tokens --from-literal=runner-token='' --from-literal=runner-registration-token='<TOKEN>' -n dev
```

* Install the runner:

```shell
helm install -n dev app-dev-runner -f values.yaml gitlab/gitlab-runner
```

![Specific Runner with Kapsule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/42qhdf7jw44vb4o2io6u.png)

# Using the specific runner in Gitlab CI

Create the pipeline `.gitlab-ci.yml`:

```yaml
stages:
  - dev

infra:
  stage: dev
  image:
    name: scaleway/cli:v2.3.1
  script: 
    - /scw k8s cluster create name=kapsule-dev
    - /scw k8s pool create cluster-id=$(/scw k8s cluster list | grep kapsule-dev | awk '{ print $1 }') name=apps node-type=DEV1_M size=2
  tags:
    - k8s-dev-runner
  only:
    - main
```

The job will create a Kapsule cluster in the `development` project. We can follow the same steps for a `production` environment.

![Deploy Kapsule Dev from Kapsule DevOps using Gitlab CI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/24xnad2z7a75wtharkhx.png)

> If you want to change the API Key, you just need to delete and recreate the `dev-api-key` secret.

# Access the Kapsule cluster

You can follow the same steps to connect to the Kapsule Cluster from your Gitlab job. 

* Create a secret:

```shell
kubectl create secret generic dev-kapsule-config --from-file ~/.kube/config -n dev
```

* and mount it as a data volume:

```yaml
        [[runners.kubernetes.volumes.secret]]
          name = "dev-kapsule-config"
          mount_path = "/root/.kube"
          read_only = true
```

> In this example, we used a kubeconfig with admin access. You should use [RBAC][k8s-1] to restrict access to only specific Kubernetes resources that the runner needs.

# Implementing Gitlab Flow

[Environment branches with Gitlab Flow][gitlab-2] is a branching strategy and workflow. Suppose you have additional environments like a pre-production environment and a production environment. Deploy the main branch to your development environment. To deploy to pre-production, create a merge request [3] from the main branch to the pre-prod branch. Go live by merging the pre-prod branch into the production branch.

![Gitlab flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n1ychv25gbzt6zhwfsvq.png)

Add the following script `./utils/autoMergeRequest.sh`:

```shell
#!/bin/bash

[[ $CI_PROJECT_URL =~ ^https?://[^/]+ ]] && CI_PROJECT_URL="${BASH_REMATCH[0]}/api/v4/projects/"

BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"Deployment to ${TARGET_BRANCH}\",
    \"assignee_id\":\"${GITLAB_USER_MAILINGLIST_ID}\"
}";

LISTMR=`curl --silent "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests?state=opened" --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}"`;
COUNTBRANCHES=`echo ${LISTMR} | grep -o "\"source_branch\":\"${CI_COMMIT_REF_NAME}\"" | wc -l`;

if [ ${COUNTBRANCHES} -eq "0" ]; then
    curl -X POST "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests" \
        --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}" \
        --header "Content-Type: application/json" \
        --data "${BODY}";

    echo "Opened a new merge request: Deployment to ${TARGET_BRANCH} and assigned to you";
    exit;
fi

echo "No new merge request opened";
```

It's a common pattern to receive the merge request from a mailing list. Create a Gitlab user with this mailing list and add the variable CI/CD `GITLAB_USER_MAILINGLIST_ID` with the user ID. You can get the ID using the URL `https://gitlab.com/api/v4/users?username=<USERNAME>`.

Add the Variable CI/CD `PRIVATE_TOKEN`, you can create one from `Settings > Project Access Tokens`.

Example of a `gitlab-ci.yml`:

```yaml
stages:
  - build
  - dev
  - preprod
  - prod

build:
  stage: build
  script: 
    - echo 'build'
    - TARGET_BRANCH=main ./utils/autoMergeRequest.sh
  only:
    - /^feature\/*/
    - /^hotfix\/*/

dev:
  stage: dev
  script: 
    - echo 'deploy dev'
    - TARGET_BRANCH=preprod ./utils/autoMergeRequest.sh
  tags:
    - k8s-dev-runner
  only:
    - main

preprod:
  stage: preprod
  script: 
    - echo 'deploy preprod'
    - TARGET_BRANCH=prod ./utils/autoMergeRequest.sh
  tags:
    - k8s-preprod-runner
  only:
    - preprod

prod:
  stage: prod
  script: 
    - echo 'deploy prod'
  tags:
    - k8s-prod-runner
  only:
    - prod
```

> The `preprod` and `prod` branches need to be marked as protected. Go to `Settings > Repository > Protected branches`.

# Conclusion

In this post, we created a devops cluster, we mounted the config.yml file to the specific runner, and we ended up deploying our Scaleway Elements and Kubernetes resources in an environment project.

This mechanism guarantees end-to-end security for your API Keys resources in Scaleway Elements.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I've convinced you to remove your API keys from Gitlab CI variables.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

# Documentation

[1] https://www.scaleway.com/en/docs/compute/kubernetes/api-cli/creating-managing-kubernetes-lifecycle-cliv2/
[2] https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml
[3] https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
[4] https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

[gitlab-1]: https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
[gitlab-2]: https://docs.gitlab.com/ee/topics/gitlab_flow.html#environment-branches-with-gitlab-flow
[scw-1]: https://github.com/scaleway/scaleway-cli
[scw-2]: https://www.scaleway.com/en/docs/console/my-project/how-to/generate-api-key/
[k8s-1]: https://kubernetes.io/docs/reference/access-authn-authz/