Implementing GitOps with Argo CD, GitHub, and Azure Kubernetes Service
Tags: azure, devops, kubernetes, gitops
Published: 2023-11-13 10:51:08

Many organizations have already shifted to DevOps as part of their digital transformation strategy. This promotes an environment where collaboration and shared responsibility foster a more cross-functional setting, necessitating processes that enable developers to perform more IT operations-related tasks. In this article, I will explain what GitOps is and demonstrate its application using ArgoCD, GitHub, and Azure Kubernetes Service (AKS).

# What’s GitOps?
GitOps is a developer-centric framework for operational practices that is declarative and based on the version control system Git, a term coined by Weaveworks in 2017. It facilitates the automation of infrastructure provisioning and software deployment. Developers store infrastructure as code (IaC) files, configuration files, and application codes in Git repositories, which represent the desired state of their infrastructure. When a commit is pushed to a monitored branch in the source control, it triggers an update that aligns the cloud infrastructure or application to match the new state.

# How GitOps works?
The GitOps workflow involves the following steps:

1. The developer begins a new task by creating a branch from the main production branch.
2. The developer writes configuration files that define the infrastructure and resources needed.
3. Once the development is completed, a pull request (PR) is created for team review and approval.
4. Upon approval, the code is merged into the production branch, which the GitOps tool monitors. The tool detects discrepancies between the current deployed state and the repository state, raising an alert for manual synchronization or automatically updating the infrastructure.

There are many tools available for adopting GitOps, such as Azure DevOps Pipelines, GitHub Actions, and CircleCI. Some tools, like Azure DevOps, CircleCI, and Jenkins, have a broader scope, while others are more Kubernetes-centric. For this demonstration, I will be using ArgoCD.

# What’s Argo CD?
ArgoCD is an open-source GitOps tool that is installed within a Kubernetes cluster. It offers features like RBAC, multi-cluster deployment, and automatic synchronization with Git repositories. ArgoCD streamlines continuous delivery workflows by keeping applications in sync with their desired state as defined in Git. Its web interface and CLI are designed for accessibility, promoting collaboration between developers and operations teams and enhancing reliability for Kubernetes applications.

# Initial setup
Before deploying ArgoCD, we must provision an AKS cluster. Below is a Terraform configuration for setting up two AKS clusters — one for ArgoCD and another for deploying applications. Save this configuration in a file named `main.tf`.
```
provider "azurerm" {
  features = {}
}
resource "azurerm_resource_group" "aks_rg" {
  name     = "rg-training-eus-01"
  location = "East US"
}
resource "azurerm_virtual_network" "aks_vnet" {
  name                = "vnet-training-argocd-eus-01"
  address_space       = ["10.0.0.0/8"]
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
}
resource "azurerm_kubernetes_cluster" "aks_argocd" {
  name                = "aks-training-dev"
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
  dns_prefix          = "argocdcluster"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  identity {
    type = "SystemAssigned"
  }
}
resource "azurerm_kubernetes_cluster" "aks_apps" {
  name                = "aks-training-dev-02"
  resource_group_name = azurerm_resource_group.aks_rg.name
  location            = azurerm_resource_group.aks_rg.location
  dns_prefix          = "appscluster"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  identity {
    type = "SystemAssigned"
  }
}
resource "azurerm_container_registry" "acr" {
  name                     = "acrtrainingeus01"
  resource_group_name      = azurerm_resource_group.aks_rg.name
  location                 = azurerm_resource_group.aks_rg.location
  sku                      = "Standard"
  admin_enabled            = false
}
resource "azurerm_role_assignment" "acr_pull" {
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.aks_argocd.kubelet_identity[0].object_id
}
resource "azurerm_role_assignment" "acr_pull_second" {
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.aks_apps.kubelet_identity[0].object_id
}
```

To initialize the Terraform configuration, use the following command:
```
$ terraform init
```
Deploy the configuration with:
```
$ terraform apply — auto-approve
```
With the AKS cluster in place, you can proceed to install ArgoCD.

# Install Argo CD
To install ArgoCD, first retrieve your cluster credentials using the Azure CLI:
```
$ $ az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
$ az aks get-credentials --resource-group rg-training-dev --name aks-training-dev
```
The installation of Argo CD is straightforward and can be accomplished by deploying the official manifest file from GitHub at the following URL: `https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`. It's best practice to deploy it in a separate namespace. You can create a new namespace and apply the manifest file with these commands:
```
$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
The above commands will create three Custom Resource Definitions (CRDs) for Applications, ApplicationSets, and AppProjects, as well as multiple roles, ClusterRoles with their respective bindings, and two Secrets. These Secrets are treated as opaque blobs, which Kubernetes does not decode or process specifically. The commands also create multiple ConfigMaps and Deployments.

# Login to Argo CD
By default, Argo CD’s API server is not exposed to the internet. You can set up an Ingress Controller or expose the API server by changing the service type to `LoadBalancer` with the following command:
```
kubectl patch svc argocd-server -n argocd — type=’json’ -p=”[{‘op’:’replace’, ‘path’:’/spec/type’, ‘value’:’LoadBalancer’}]”
```
(Optional) If you want to connect to Argo CD directly from your localhost, you can use the port-forward method to create a streaming connection from your computer to the Argo CD service running on AKS:
```
kubectl port-forward svc/argocd-server -n argocd 8080:443
```
However, you will need to keep this connection open in your terminal or command line instance.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2sthk2tkdvy54zpszeky.png)

Alternatively, obtain the external IP address of the load balancer to access the Argo CD UI directly with:
```
$ kubectl get svc argocd-server -n argocd
```
Then browse to the LoadBalancer’s external IP address.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rjcn2sb6ol6gg7653v7f.png)

The default login credentials for Argo CD use the admin username and an auto-generated password, which can be retrieved as follows:

For PowerShell:
```
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | ForEach-Object { [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_)) };
```
For Bash:
```
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
```
# Adding an additional cluster to Argo CD
To add an additional AKS cluster to Argo CD, you must first install the Argo CD CLI on your machine. Retrieve the latest version and download the corresponding executable with these commands:
```
$version = (Invoke-RestMethod https://api.github.com/repos/argoproj/argo-cd/releases/latest).tag_name
Invoke-WebRequest -Uri "https://github.com/argoproj/argo-cd/releases/download/$version/argocd-windows-amd64.exe" -OutFile "argocd.exe"
```
Then, add the path of the downloaded executable to your system’s environment variables to facilitate access:
```
$ [Environment]::SetEnvironmentVariable(“Path”, “$env:Path;C:\Users\Ivan\ArgoCD-CLI”, “User”)
```
To add a cluster, update your kubectl context with the new cluster's details using the commands you previously used, this time pointing to the applications' AKS:
```
$ az account set — subscription 36e90881–8db4–4f13-aa7a-8e9d83febdf4
$ az aks get-credentials — resource-group rg-training-dev — name aks-training-dev-01
```
You can verify the new cluster’s context with the following command:
```
$ kubectl config get-contexts
CURRENT   NAME                                                         CLUSTER                                                      AUTHINFO                                                     NAMESPACE
*         aks-training-dev                                             aks-training-dev                                             clusterUser_rg-training-dev_aks-training-dev
          aks-training-dev-02                                          aks-training-dev-02                                          clusterUser_rg-training-dev_aks-training-dev-
```
Once the context is set, log in to the Argo CD CLI using the same credentials you use for the web application:
```
$ argocd login 20.253.8.230
WARNING: server certificate had error: tls: failed to verify certificate: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
Username: admin
Password:
'admin:login' logged in successfully
Context '20.253.8.230' updated
```
Acknowledge any certificate warnings and proceed to log in. After successful login, you can add the cluster to Argo CD with:
```
$ argocd cluster add aks-training-dev-02
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `aks-training-dev-02` with full cluster level privileges. Do you want to continue [y/N]? y
time="2023-11-10T10:04:55+01:00" level=info msg="ServiceAccount \"argocd-manager\" created in namespace \"kube-system\""time="2023-11-10T10:04:55+01:00" level=info msg="ClusterRole \"argocd-manager-role\" created"
time="2023-11-10T10:04:56+01:00" level=info msg="ClusterRoleBinding \"argocd-manager-role-binding\" created"
time="2023-11-10T10:05:01+01:00" level=info msg="Created bearer token secret for ServiceAccount \"argocd-manager\""
Cluster 'https://aks-training-dev-02-dns-uxajaze6.hcp.eastus.azmk8s.io:443' added
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/if7ls6mz8abbgh6ofbtb.png)

This will create a service account argocd-manager on the cluster referenced by the context aks-training-dev-02 with full cluster-level privileges. Confirm when prompted to continue. After the service account argocd-manager is created, along with the associated ClusterRole and ClusterRoleBinding, verify that the new cluster is managed by Argo CD via both the CLI and the web application with:
```
$ argocd cluster list
SERVER                                                             NAME                 VERSION  STATUS      MESSAGE  PROJECT
https://aks-training-dev-02-dns-uxajaze6.hcp.eastus.azmk8s.io:443  aks-training-dev-02  1.26     Successful
https://kubernetes.default.svc                                     in-cluster           1.26     Successful
```
# Add a new user
When you first install Argo CD, it automatically creates a built-in admin user with full access. A best practice is to create local users and disable the built-in admin account. To create a new user, define a new ConfigMap resource as follows:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  accounts.gtrekter: apiKey, login
  accounts.gtrekter.enabled: "true"
```
The username follows the ‘accounts’ string and accepts two values:
- `apiKey` to allow generating API keys
- `login` to allow login using the UI.
As soon as the user is created, you can proceed with disabling the admin account. Edit the ConfigMap and set the `admin.enabled` key to `false` as shown below:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  accounts.gtrekter: apiKey, login
  accounts.gtrekter.enabled: "true"
  admin.enabled: "false"
```
The password of the newly created user it’s going to be the same as the admin. However, you can easily change it by using the following command:
```
$ argocd account update-password --account gtrekter --current-password xxxxxxxxxxxxxxxx --new-password xxxxxxxxxxxxxxxx
```
Even if we allow our new user to log in to the UI and generate a new token, there is little they can do without RBAC permissions. RBAC roles are managed via the `argocd-rbac-cm`, so the first step is to pull it using:
```
$ kubectl get cm argocd-rbac-cm -n argocd -o yaml > argocd-rbac.yml
```
The content of the file will look as follows:
```
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-rbac-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-rbac-cm","namespace":"argocd"}}
  creationTimestamp: "2023-11-09T20:06:17Z"
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
  resourceVersion: "2778"
  uid: b8369700-a83e-47d9-8ae6-5f2eb2b0110d
```
The permissions are defined with the following formats:
```
p, <role/user/group>, <resource>, <action>, <object>
```
You can specify a variety of resources, such as `clusters`, `projects`, `applications`, `applicationsets`, `repositories`, `certificates`, `accounts`, `gpgkeys`, `logs`, `exec`, and `extensions`. The actions available for these resources include `get`, `create`, `update`, `delete`, `sync`, `override`, and `action/<group/kind/action-name>`.

In this demo, we will grant the user ‘gtrekter’ the rights to create new applications and view clusters. We will add the following policy to the ConfigMap that we have obtained:
```
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-rbac-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-rbac-cm","namespace":"argocd"}}
  creationTimestamp: "2023-11-09T20:06:17Z"
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
  resourceVersion: "2778"
  uid: b8369700-a83e-47d9-8ae6-5f2eb2b0110d
data:
  policy.csv: |
    p, role:developer, applications, *, */*, allow
    p, role:developer, clusters, get, *, allow
    p, role:developer, repositories, get, *, allow
    g, gtrekter, role:developer
```
To apply the updated RBAC configuration, run the following command:
```
$ kubectl apply -f argcd-rbac.yml
```
# Deploying your first ArgoCD Application
You can deploy a new application using either the Argo CD CLI or the Argo CD UI. In this demo, we will focus on using the Argo CD UI. After logging into the UI, click on the **New App** button.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kojroaxmxnej7jjop6e1.png)

In the panel that appears, configure several settings in the General section: name the application, select the project for grouping and access control, choose the sync policy (automatic or manual), and decide whether to enable cascading deletion with the “Set deletion finalizer” option. Sync options also allow you to manage schema validation, resource pruning order, differential ignores, namespace creation, and server-side application, as well as applying changes only when they are out of sync.

- **Skip schema validation:** Bypasses schema validation steps during the sync process in Argo CD, useful for custom resources not recognized by Argo CD.
- **Prune last:** Deletes resources no longer present in the desired state after other changes are applied, maintaining service availability during updates.
- **Respect Ignore Differences:** Prevents differences between desired and live states that are intentional from triggering a sync.
- **Auto-create namespace: Automatically creates the specified Kubernetes namespace for the application if it does not already exist.
- **Apply out of sync only:** Applies changes only to resources that are out of sync with the repository’s desired state, allowing for selective syncing.
- **Server-side apply:** Utilizes Kubernetes’s server-side apply feature during the sync process.

Regarding the prune propagation policy, there are three options:

- **Foreground:** Deletes “dependent” objects first, ensuring all related objects are cleaned up before the main object is removed from the cluster.
- **Background:** Deletes the main object immediately, with Kubernetes cleaning up dependent objects in the background.
- **Orphan:** Leaves dependent objects in the cluster when the main object is deleted, effectively “orphaning” them.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vz0s6wybutzhi096mrns.png)

The next section should contain the repository URL, the revision (branch or tag) to be monitored by Argo CD, and the path to the folder with relevant files. For this demo, the public training repository with manifests stored in the ‘argocd’ subdirectory will be used.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/78a5ltrys5xk0mchjimg.png)

In the Destination section, specify where to deploy the application and its related namespace.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6kis6g9e27e09b4hpwuh.png)

The Directory section offers configuration options for managing the deployment:

- **Top-Level arguments:** Parameterize the application definition to override configurations within Argo CD directly.
- **Directory recurse:** Specify whether to recursively sync all subdirectories within the path.
- **External variables:** Define external variables used by templates or configurations.
- **Include/Exclude:** Specify which files or directories to include or exclude during the sync process.

After filling out the configuration, click the Create button.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gbytx59exjczma1c8n6q.png)

Once you completed filling up the configuration, click the create button. The modal will close, and your new application will be listed with the status `OutOfSync` and the health status `Missing`.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hrv0ihihkxazxrgtdom1.png)

This indicates that the Argo CD application has been created but not yet deployed. In fact, if we list all the deployments in the cluster, we will see just the default and the Argo CD related resources
```
$ kubectl get applications -n argocd
NAME        SYNC STATUS   HEALTH STATUS
guestbook   OutOfSync     Missing
$ kubectl get deployments --all-namespaces
NAMESPACE         NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
argocd            argocd-applicationset-controller   1/1     1            1           2d22h
argocd            argocd-dex-server                  1/1     1            1           2d22h
argocd            argocd-notifications-controller    1/1     1            1           2d22h
argocd            argocd-redis                       1/1     1            1           2d22h
argocd            argocd-repo-server                 1/1     1            1           2d22h
argocd            argocd-server                      1/1     1            1           2d22h
calico-system     calico-kube-controllers            1/1     1            1           2d22h
calico-system     calico-typha                       1/1     1            1           2d22h
kube-system       coredns                            2/2     2            2           2d22h
kube-system       coredns-autoscaler                 1/1     1            1           2d22h
kube-system       konnectivity-agent                 2/2     2            2           2d22h
kube-system       metrics-server                     2/2     2            2           2d22h
tigera-operator   tigera-operator                    1/1     1            1           2d22h
```
To deploy the application, click the **Sync** button on the application card.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rezizcy8wy2uj128x4pb.png)

Then, in the panel that appears, select the resources to synchronize, define the synchronization and prune propagation options, and click the **Synchronize** button.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ro6mlx8s2tsao5xntpgv.png)

Upon completion, the application’s status will change to `Synced`, and its health status will update to `Healthy`.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a44zt8y9df5616er6ekh.png)

Clicking on the card will redirect you to the application detail view, where you can see the various resources and their relationships.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/al9ed05nrw8yxjx66x4g.png)

# Update the application
Let’s assume we are anticipating an increase in traffic and need to update the number of replicas for our application. We will create a new branch and update the number of replicas from 2 to 5 as shown below:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: guestbook-ui
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: guestbook-ui
  template:
    metadata:
      labels:
        app: guestbook-ui
    spec:
      containers:
      - image: gcr.io/heptio-images/ks-guestbook-demo:0.2
        name: guestbook-ui
        ports:
        - containerPort: 80
```
The next step is to raise a new Pull Request so other team members can review and either approve or reject the changes before they are merged into the production branch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sohitqbl799p4fuhle8h.png)

Assuming the Pull Request is approved and our code is merged, the desired state (with 5 replicas) will no longer match the live state. Since we set the sync policy of our application to Manual, this discrepancy will highlight that the application is out of sync.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7cjeutsjkmt6wol4oxrl.png)

To update the live state, we simply click the **Sync** button on the application card, and Argo CD will handle the rest.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z9v299ah78cyfsjoqtti.png)

# References
- **Argo CD Installation:** https://argo-cd.readthedocs.io/en/stable/cli_installation/#download-with-powershell-invoke-webrequest
- **ArgoCD Projects:** https://argo-cd.readthedocs.io/en/stable/user-guide/projects/
- **ArgoCD User management:** https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/
- **ArgoCD RBAC:** https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/#rbac-permission-structure