Securing access to AWS IAM Roles from Gitlab CI
Tags: aws, security, devops, gitlab
Published: 2021-03-22 11:36:52

How many access and secret keys are stored per day as variables in the Gitlab CI configuration? 

When an `AWS access key` is saved in Gitlab, we face all the security issues of storing credentials outside of the cloud infrastructure: Access, authorization, key rotation, age, destruction, location, etc.

There are 2 common reasons for developers to store AWS credentials in Gitlab CI:

* They use `shared runners`.
* They use `specific runners` deployed in a `Amazon Elastic Kubernetes Service` cluster but do not use (or do not know about) [IAM Roles for Service Accounts][irsa] feature.

You can continue to use AWS Access keys in Gitlab CI and secure the keys with external tools like Vault and Forseti, but this will add additional tools to manage.

The alternative that AWS proposes for customers is to use IAM Roles for Service Accounts (IRSA) feature.

IRSA feature allows you to bind the `Kubernetes Service Account` associated with the specific runner to an `AWS IAM Role`.

# Working with IRSA

The first step is to create and configure our EKS devops cluster.

* We start by creating our EKS cluster [1] using [eksctl][eksctl]:

```shell
export AWS_PROFILE=<AWS_PROFILE>
export AWS_REGION=eu-west-1
export EKS_CLUSTER_NAME=devops
export EKS_VERSION=1.19

eksctl create cluster \
 --name $EKS_CLUSTER_NAME \
 --version $EKS_VERSION \
 --region $AWS_REGION \
 --managed \
 --node-labels "nodepool=dev"
```

* Create an IAM OIDC identity provider for the cluster 

```
eksctl utils associate-iam-oidc-provider --cluster=$EKS_CLUSTER_NAME --approve

ISSUER_URL=$(aws eks describe-cluster \
                       --name $EKS_CLUSTER_NAME \
                       --query cluster.identity.oidc.issuer \
                       --output text)
```

* Configure `kubectl` to communicate with the cluster:

```shell
aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
```

* Create the namespace to use for the Kubernetes service account.

```shell
kubectl create namespace dev
```
* Create the Kubernetes service account to use for specific runner:

```shell
kubectl create serviceaccount --namespace dev app-deployer
```

* Allow the Kubernetes service account to impersonate the IAM Role by using an IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the IAM Role.

```shell
ISSUER_HOSTPATH=$(echo $ISSUER_URL | cut -f 3- -d'/')

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

PROVIDER_ARN="arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$ISSUER_HOSTPATH"

cat > trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:dev:app-deployer",
          "${ISSUER_HOSTPATH}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

ROLE_NAME=eks-cluster-role

aws iam create-role \
          --role-name $ROLE_NAME  \
          --assume-role-policy-document file://trust-policy.json

EKS_ROLE_ARN=$(aws iam get-role \
                        --role-name $ROLE_NAME \
                        --query Role.Arn --output text)
```

* Add the `eks.amazonaws.com/role-arn=$EKS_ROLE_ARN` annotation to the Kubernetes service account, using the IAM role ARN.

```shell
kubectl annotate serviceAccount app-deployer -n dev eks.amazonaws.com/role-arn=$EKS_ROLE_ARN
```

> You could also use `eksctl create iamserviceaccount [..]` [2]

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sl9fm53y2dj8tytflsi3.png)

# Assign KSA to Gitlab runner

The next step is to assign the KSA to our Gitlab runner.

* Start by installing Helm:

```shell
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```

* Add Gitlab Helm package:

```shell
helm repo add gitlab https://charts.gitlab.io
```

* Configure the runner: 

Create the file `values.yaml`:

```yaml
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "<REGISTRATION_TOKEN>"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
metrics:
  enabled: true
runners:
  image: ubuntu:18.04
  locked: true
  pollTimeout: 360
  protected: true
  serviceAccountName: app-deployer
  privileged: false
  namespace: dev
  builds:
    cpuRequests: 100m
    memoryRequests: 128Mi
  services:
    cpuRequests: 100m
    memoryRequests: 128Mi
  helpers:
    cpuRequests: 100m
    memoryRequests: 128Mi
  tags: "k8s-dev-runner"
  nodeSelector: 
    nodepool: dev
```

> You can find the description of each attribute in the Gitlab runner charts repository [3]

* Get the Gitlab registration token in `Project -> Settings -> CI/CD -> Runners` in the `Setup a specific Runner manually` section.

* Install the runner:

```shell
helm install -n dev app-dev-runner -f values.yaml gitlab/gitlab-runner
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ehc72gupvmk40kwxqkd.png)

# Using the specific runner in Gitlab CI

Before running our first pipeline in Gitlab CI, let's add the Kubernetes cluster administrator permission to the IAM role we created earlier.

```shell
cat > eks-admin-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "eksadministrator",
            "Effect": "Allow",
            "Action": [
                 "eks:*",
                 "cloudformation:*",
                 "ec2:*",
                 "ssm:*",
                 "iam:*"
          
            ],
            "Resource": "*"
        }
    ]
}
EOF

aws iam  put-role-policy \
--role-name $ROLE_NAME \
--policy-name eks-admin-policy \
--policy-document file://eks-admin-policy.json
```

> Note: This policy is used as an example. AWS recommends you to use fine grained permissions.

Now we can run our pipeline `.gitlab-ci.yaml`:

```yaml
stages:
  - dev

before_script:
    - yum install -y tar gzip
    - curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
    - mv /tmp/eksctl /usr/local/bin
infra:
  stage: dev
  image: 
    name: amazon/aws-cli
  script: 
    - eksctl create cluster --name=business --region=eu-west-1 --managed --instance-types t3.medium
  tags:
    - k8s-dev-runner
```

The job will create an EKS cluster in the AWS Account using `eksctl`. We can follow the same steps for a `prod` environment.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jyqnnt4iny2mkgfyu2vc.png)

# Conclusion

This mechanism guarantees end-to-end security for your IAM role resources. You can easily create a cron job that delete the IAM roles in the evening and re-create them in the morning of a working day.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I've convinced you to remove your Access & secret keys from Gitlab CI variables and use specific runners in a EKS with IRSA configured.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

# Documentation

[1] https://aws.amazon.com/fr/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
[2] https://eksctl.io/usage/iamserviceaccounts
[3] https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml

[irsa]: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
[eksctl]: https://eksctl.io/