Configuring ArgoCD on Amazon EKS
Tags: aws, kubernetes, argocd, eks
Published: 2021-04-17 21:36:06

[Argo CD][doc-2] is a declarative, GitOps continuous delivery tool for Kubernetes. In this blog post we will see how to install, configure and manage ArgoCD on [Amazon Elastic Container Service for Kubernetes][doc-3] (Amazon EKS).

# Prerequisites

* [Installing][install-1] and [configuring][install-6] AWS CLI
* [eksctl][install-2]
* [Kubectl][install-3]
* [ArgoCD CLI][install-4]
* Create a public hosted zone in Route 53. [See tutorial][tutorial-1]
* Request a public certificate with AWS Certificate Manager. [See tutorial][tutorial-2]

# EKS configuration

We start by creating the EKS cluster. 

```shell
export AWS_PROFILE=<AWS_PROFILE>
export AWS_REGION=eu-west-1
export EKS_CLUSTER_NAME=devops
export EKS_VERSION=1.19

eksctl create cluster \
 --name $EKS_CLUSTER_NAME \
 --version $EKS_VERSION \
 --region $AWS_REGION \
 --managed
```

# ArgoCD installation

Before installing ArgoCD in Kubernetes, we need to authenticate on Amazon EKS:

```shell
aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
```

Install ArgoCD using the official yaml

```shell
$ kubectl create namespace argocd
$ kubectl config set-context --current --namespace=argocd 
$ kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

# ArgoCD configuration

Associate the public certificate that you created earlier to the ArgoCD server.

```shell
cat > argocd-server.patch.yaml << EOF
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "<ACM_ARGOCD_ARN>"
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
  - "<LOCAL_IP_RANGES>"
EOF

ACM_ARGOCD_ARN=<ACM_ARGOCD_ARN>
sed -i "s,<ACM_ARGOCD_ARN>,${ACM_ARGOCD_ARN},g; s/<LOCAL_IP_RANGES>/$(curl -s http://checkip.amazonaws.com/)\/32/g; " argocd-server.patch.yaml
$ kubectl patch svc argocd-server -p "$(cat argocd-server.patch.yaml)"
```

Create a Record Set in your hosted zone that you created earlier. The CNAME record points to the ingress hostname of the ArgoCD server.

```shell
PUBLIC_DNS_NAME=<PUBLIC_DNS_NAME>
R53_HOSTED_ZONE_ID=<R53_HOSTED_ZONE_ID>
cat > argocd-recordset.json << EOF
{
            "Changes": [{
            "Action": "CREATE",
                        "ResourceRecordSet": {
                                    "Name": "argocd.${PUBLIC_DNS_NAME}.",
                                    "Type": "CNAME",
                                    "TTL": 300,
                                 "ResourceRecords": [{ "Value": "$(kubectl get services argocd-server --output jsonpath='{.status.loadBalancer.ingress[0].hostname}')"}]
}}]
}
EOF

aws route53 change-resource-record-sets --hosted-zone-id $R53_HOSTED_ZONE_ID --change-batch file://argocd-recordset.json
```

As the AWS Elastic Load Balancer is doing the SSL we need to start Argo CD in HTTP (insecure) mode by disabling the TLS. Edit the `argocd-server` deployment to add the `--insecure` flag to the `argocd-server` command:

```shell
cat > argocd-deployment-server.patch.yaml << EOF
spec:
  template:
    spec:
      containers:
        - command:
          - argocd-server
          - --staticassets
          - /shared/app
          - --insecure
          name: argocd-server
EOF

$ kubectl patch deployment argocd-server -p "$(cat argocd-deployment-server.patch.yaml)" 
```

Amazon EKS creates a Classic Load Balancer. 

> An AWS Application Load Balancer can be used as Load Balancer for both UI and gRPC traffic. See [ArgoCD Ingress Configuration][doc-1].

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1unkgxdu4k07q3w92ifj.png)

# ArgoCD Web UI configuration

The initial admin password is autogenerated to be the pod name of the Argo CD API server. Let's change it.

Edit the `argocd-secret` secret and update the `admin.password` field with a new bcrypt hash. You can use a site like https://www.browserling.com/tools/bcrypt to generate a new hash.

```shell
ARGOCD_ADDR="argocd.${PUBLIC_DNS_NAME}"

$ kubectl patch secret argocd-secret \
  -p '{"stringData": {
    "admin.password": "<BCRYPT_HASH>",
    "admin.passwordMtime": "'$(date +%FT%T%Z)'"
  }}'
```

Log in now using the username admin and the new password.

```shell
argocd login $ARGOCD_ADDR
```

The admin user is a `superuser` and it has unrestricted access to the system. ArgoCD recommends to not use the admin user in daily work.

Let's add two new users `demo` and `ci`:

* User `demo` will have read only access to the Web UI, 
* User `ci` will have write privileges and will be used to generate access tokens to execute `argocd` commands in CI / CD pipelines.

If you have a Git repository, you can specify it in the `repositories` attribute.

```shell
cat > argocd-configmap.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  repositories: |
    - url: <GIT_REPOSITORY_URL>
      passwordSecret:
        name: demo
        key: password
      usernameSecret:
        name: demo
        key: username
  admin.enabled: "true"
  accounts.demo.enabled: "true"
  accounts.demo: login
  accounts.ci.enabled: "true"
  accounts.ci: apiKey
EOF

GIT_USERNAME=<GIT_USERNAME>
GIT_TOKEN=<GIT_TOKEN>

$ kubectl create secret generic demo \
--from-literal=username=$GIT_USERNAME \
--from-literal=password=$GIT_TOKEN
```

Let's create the users:

```shell
sed -i "s,<GIT_REPOSITORY_URL>,$GIT_REPOSITORY_URL,g" argocd-configmap.yaml

$ kubectl apply -f argocd-configmap.yaml
```

Add a password to the `demo` user:

```shell
argocd account update-password --account demo --current-password "${ADMIN_PASSWORD}" --new-password "<DEMO_PASSWORD>"
```

We can now assign the roles to the users:

* `demo` user will have read only access,
* `ci` user will manage projects, repositories, clusters and applications.

```shell
cat > argocd-rbac-configmap.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    p, role:ci, applications, sync, *, allow
    p, role:ci, applications, update, *, allow
    p, role:ci, applications, override, *, allow
    p, role:ci, applications, create, *, allow
    p, role:ci, applications, get, *, allow
    p, role:ci, applications, list, *, allow
    p, role:ci, clusters, create, *, allow
    p, role:ci, clusters, get, *, allow
    p, role:ci, clusters, list, *, allow
    p, role:ci, projects, create, *, allow
    p, role:ci, projects, get, *, allow
    p, role:ci, projects, list, *, allow
    p, role:ci, repositories, create, *, allow
    p, role:ci, repositories, get, *, allow
    p, role:ci, repositories, list, *, allow

    g, ci, role:ci
EOF

$ kubectl apply -f argocd-rbac-configmap.yaml
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zh2bndmjklyalq1ffss4.png)

# CI / CD integration

To create a token using the `ci` user you can run the command:

```shell
AROGOCD_TOKEN=$(argocd account generate-token --account ci)
```

The token can be stored in AWS Secret Manager and used in a CI / CD pipeline::

```shell
aws secretsmanager create-secret --name argocd-token \
    --description "ArgoCD Token" \
    --secret-string "${AROGOCD_TOKEN}"
```

The following Gitlab example demonstrates the use of this token to create a cluster, a project, and synchronize an application in ArgoCD.

> If you want to understand how an IAM role can be attached to a Gitlab runner, please refer to my previous post on [Securing access to AWS IAM Roles from Gitlab CI][doc-6]

```yaml
stages:
  - init
  - deploy

variables: 
   KUBECTL_VERSION: 1.20.5
   ARGOCD_VERSION: 1.7.4
   ARGOCD_ADDR: argocd.example.com

# Get ArgoCD credentials from Secret Manager
before_script:
    - export AROGOCD_TOKEN="$(aws secretsmanager get-secret-value --secret-id argocd-token --version-stage AWSCURRENT --query SecretString --output text)"
    # install kubectl
    - curl -L "https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o /usr/bin/kubectl 
    # install argocd
    - curl -sSL -o /usr/local/bin/argocd "https://github.com/argoproj/argo-cd/releases/download/v${ARGOCD_VERSION}/argocd-linux-amd64"

init demo project 🔬:
  stage: init
  when: manual
  image:
    name: amazon/aws-cli
  script:
    - argocd cluster add $BUSINESS_K8S_CONTEXT --name business-cluster-dev --kubeconfig $KUBE_CONFIG --auth-token=${AROGOCD_TOKEN} --server ${ARGOCD_ADDR} || echo 'cluster already added'
  tags:
    - k8s-dev-runner
  only:
    - master 

deploy demo project 🚀:
  stage: init
  when: manual
  image:
    name: amazon/aws-cli
  script:
    - sed -i "s,<KUBERNETES_CLUSTER_URL>,$BUSINESS_K8S_CLUSTER_URL,g;s,<GIT_REPOSITORY_URL>,$CI_PROJECT_URL.git,g" application.yaml
    # Connect to aws eks devops cluster
    - aws eks update-kubeconfig --region $AWS_REGION --name $EKS_CLUSTER_NAME
    # Create ArgoCD project
    - argocd proj create demo-dev -d $BUSINESS_K8S_CLUSTER_URL,app-dev -s $CI_PROJECT_URL.git --auth-token=${AROGOCD_TOKEN} --server ${ARGOCD_ADDR} || echo 'project already created' 
    # Create ArgoCD application
    - kubectl apply -n argocd -f application.yaml
  tags:
    - k8s-dev-runner
  only:
    - master

deploy demo app 🌐:
  stage: deploy
  image:
    name: amazon/aws-cli
  script:
    - cd envs/dev
    - argocd app sync demo-dev --auth-token=${AROGOCD_TOKEN} --server ${ARGOCD_ADDR}
  tags:
    - k8s-dev-runner
  only:
    - tags
```

And here the configuration of the ArgoCD application:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-dev
  namespace: argocd
spec:
  project: demo-dev
  source:
    repoURL: <GIT_REPOSITORY_URL>
    targetRevision: HEAD
    path: envs/dev
  destination:
    server: <KUBERNETES_CLUSTER_URL>
    namespace: app-dev
```

Before running such a pipeline, the Gitlab runner must have access to the `argoproj.io` API.

Create the RBAC:

```shell
cat - <<EOF | kubectl apply -f - --namespace "argocd"
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: $GITLAB_RUNNER_IAM_ROLE_NAME
  namespace: argocd
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["clusters", "projects", "applications", "repositories", "certificates", "accounts", "gpgkeys"]
    verbs: ["get", "create", "update", "delete", "sync", "override", "action"]

EOF    

cat - <<EOF | kubectl apply -f - --namespace "argocd"
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: $GITLAB_RUNNER_IAM_ROLE_NAME
  namespace: argocd
subjects:
- kind: User
  name: $GITLAB_RUNNER_IAM_ROLE_NAME
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: $GITLAB_RUNNER_IAM_ROLE_NAME
  apiGroup: rbac.authorization.k8s.io

EOF
```

> `GITLAB_RUNNER_IAM_ROLE_NAME` is the name of the IAM role linked to the kubernetes service account attached to the runner.

Update the `aws-auth` configmap:

```shell
$ kubectl get configmap -n kube-system aws-auth -o yaml > aws-auth.yaml
```

Complete the config map with the following role:

```yaml
  mapRoles: | 
    - rolearn: $GITLAB_RUNNER_IAM_ROLE_ARN
      username: $GITLAB_RUNNER_IAM_ROLE_NAME
```

Apply the change

```shell
$ kubectl apply -f aws-auth.yaml
```

That's it!

# Conclusion

In this blog post we configured the ArgoCD server, ArgoCD Web UI and we ended up integrating ArgoCD into a CI / CD tool.

Hope you enjoyed reading this blog post.

If you have any questions or feedback, please feel free to leave a comment.

Thanks for reading!

[install-1]: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
[install-2]: https://github.com/weaveworks/eksctl#installation
[install-3]: https://kubernetes.io/docs/tasks/tools/
[install-4]: https://argoproj.github.io/argo-cd/getting_started/#1-install-argo-cd
[install-6]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
[tutorial-1]: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingHostedZone.html
[tutorial-2]: https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html
[doc-1]: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#aws-application-load-balancers-albs-and-classic-elb-http-mode
[doc-2]: https://argoproj.github.io/argo-cd/
[doc-3]: https://aws.amazon.com/eks/
[doc-6]: https://dev.to/stack-labs/securing-access-to-aws-iam-roles-from-gitlab-ci-3an0