Kubernetes `cronjob` to refresh .dockerconfig secret
Tags: k8s
Published: 2024-03-21 14:02:20

---
title: Kubernetes `cronjob` to refresh .dockerconfig secret
published: true
description: Kubernetes `cronjob` to refresh .dockerconfig secret
tags: k8s
---
A lot is taken from https://linuxhandbook.com/auto-update-aws-ecr-token-kubernetes/, with few modifications here and there.

# Overview
While [AWS ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html) offers a secure way to store container images, its authentication system relies on tokens that expire every 12 hours. This frequent renewal can be cumbersome if we want to use images stored in AWS ECR for our deployment outside the AWS network where the AWS IAM Role is not accessible.

This post proposes a solution to automate the AWS ECR authentication token generation. The solution will be divided into areas:
- Preparations: collect all the necessary information, and store it as environment variables.
- Create resource: Create all the resources needed (including the Cronjob), by using the information collected earlier.
- Test and verify: if the CronJob working as expected, and the resulting token can be used to access the AWS ECR. 

Once CronJob installed, triggered and completed successfully, it will (re)create a (new) secret `regcred-aws` (type `kubernetes.io/dockerconfigjson`) within the targeted namespace. Once the secret available, it can be referred from `kind: deployment` as `spec.template.spec.imagePullSecrets`, or from `kind: pod` as `spec.imagePullSecrets`

# Preparations
0. The ECRPULL user referred in the following step, are configured with the following permissions:
    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowPull",
                "Effect": "Allow",
                "Action": [
                    "ecr:BatchGetImage",
                    "ecr:GetDownloadUrlForLayer",
                    "ecr:GetAuthorizationToken"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
    ```
1. Acquire the AWS credential by capturing the output of command `aws iam create-access-key --user-name ECRPULL`. The output may look similar to this:
    ```json
    {
       "AccessKey": {
           "UserName": "ECRPull",
           "AccessKeyId": "AAAAAAAAAAAAAAAAAAAA",
           "Status": "Active",
           "SecretAccessKey": "BBBBBBBBBBBBBBBBBBBBBBBBBBBB",
           "CreateDate": "2024-01-01T00:00:00+00:00"
       }
    }
    ```
2. Create enviroment variables as follow based on the output above.
    ```bash
    export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
    export AWS_SECRET_ACCESS_KEY=BBBBBBBBBBBBBBBBBBBBBBBBBBBB
    export AWS_ACCOUNT=`aws sts get-caller-identity --query='Account' --output text`
    export AWS_REGION="ap-southeast-3"
    export APP_NAMESPACE="TargetNameSpace" # Namespace to keep the generated secret
    ```

# Create Resources
1. Create secret `ecr-registry-helper-secrets` to store the AWS credentials.
    ```bash
    kubectl create secret generic ecr-registry-helper-secrets \
       --namespace=${APP_NAMESPACE} \
       --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
       --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
       --from-literal=AWS_ACCOUNT=${AWS_ACCOUNT}
    ```
2. Create configmap & service-account:
    ```bash
    cat <<EOL | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ecr-registry-helper-cm
      namespace: ${APP_NAMESPACE}
    data:
      AWS_REGION: ${AWS_REGION}
      DOCKER_SECRET_NAME: regcred-aws
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: ecr-registry-helper-sa-${APP_NAMESPACE}
      namespace: ${APP_NAMESPACE}
    EOL
    ```
3. Create the role and its binding in the cluster
    ```bash
    cat <<EOL | kubectl apply -f -
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: ${APP_NAMESPACE}
      name: role-manage_regcred-aws_secrets
    rules:
    - apiGroups: [""]
      resources: ["secrets"]
      resourceNames: ["regcred-aws"] # Replace with your desired ECR token secret name
      verbs: ["delete"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: ${APP_NAMESPACE}-role-binding-ecr-registry-helper
      namespace: ${APP_NAMESPACE}
    subjects:
    - kind: ServiceAccount
      name: ecr-registry-helper-sa-${APP_NAMESPACE} # Replace with your service account name if different
      namespace: ${APP_NAMESPACE}
      apiGroup: ""
    roleRef:
      kind: Role
      name: role-manage_regcred-aws_secrets # Replace with your role name if different 
      apiGroup: ""
    EOL
    ```
4. Deploy the cronjob, that will execute itself every 8th hours since the script installed (`schedule: "0 */8 * * *"`).
    ```bash
    cat <<EOL | kubectl apply -f -
    apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: regcred-aws
      namespace: ${APP_NAMESPACE}
    spec:
      schedule: "0 */8 * * *"
      jobTemplate:
        spec:
          template:
            spec:
              serviceAccountName: ecr-registry-helper-sa-${APP_NAMESPACE}
              volumes:
              - name: data
                emptyDir:
                  medium: Memory
              initContainers:
              - name: getecr-registry-helper-get-token
                image: amazon/aws-cli
                imagePullPolicy: IfNotPresent
                volumeMounts:
                - mountPath: /data
                  name: data
                envFrom:
                - secretRef:
                    name: ecr-registry-helper-secrets
                - configMapRef:
                    name: ecr-registry-helper-cm
                command:
                  - /bin/bash
                  - -c
                  - |-
                    aws ecr get-login-password --region \${AWS_REGION} > /data/token
              containers:
              - name: ecr-registry-helper-get-docker-registry-credentials
                image: bitnami/kubectl
                imagePullPolicy: IfNotPresent
                volumeMounts:
                - mountPath: /data
                  name: data
                envFrom:
                  - secretRef:
                      name: ecr-registry-helper-secrets
                  - configMapRef:
                      name: ecr-registry-helper-cm
                command:
                  - /bin/bash
                  - -c
                  - |-
                    ECR_TOKEN=\$(cat /data/token)
                    NAMESPACE_NAME=${APP_NAMESPACE}
                    kubectl delete secret --ignore-not-found \${DOCKER_SECRET_NAME} -n \${NAMESPACE_NAME}
                    kubectl create secret docker-registry \${DOCKER_SECRET_NAME} --docker-server=https://\${AWS_ACCOUNT}.dkr.ecr.\${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=\${ECR_TOKEN} --namespace=\${NAMESPACE_NAME}
                    echo -n "Secret was successfully updated at "; date
              restartPolicy: Never
    EOL
    ```
# Test and verify
Run the first job interactively for testing. The last command output should show string resemblence the `~/.docker/config` file, which confirm our cronjob is correctly configured.
```bash
kubectl create job --from=cronjob/regcred-aws regcred-aws-manual --namespace=${APP_NAMESPACE}
kubectl describe jobs.batch regcred-aws-manual --namespace=${APP_NAMESPACE}
kubectl get secret regcred-aws --namespace=${APP_NAMESPACE} -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq
```