ArgoCD: declarative Projects, Applications, and ArgoCD deploy from Jenkins
Tags: devops, kubernetes, tutorial, todayilearned
Published: 2021-07-12 10:57:07

---
title: ArgoCD: declarative Projects, Applications, and ArgoCD deploy from Jenkins
published: true
date: 2021-05-19 04:26:45 UTC
tags: devops,kubernetes,tutorial,todayilearned
canonical_url: https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/
---

![](https://cdn-images-1.medium.com/max/1024/1*LydFAwy_HJjw8lGCsi1Iqg.png)

An application, cluster, or repository can be created In ArgoCD from its WebUI, CLI, or by writing a Kubernetes manifest that then can be passed to `kubectl` to create resources.

For example, Applications are Kubernetes CustomResources and described in Kubernetes CRD [`applications.argoproj.io`](https://github.com/argoproj/argo-cd/blob/master/manifests/crds/application-crd.yaml):

```
$ kubectl get crd applications.argoproj.io
NAME CREATED AT
applications.argoproj.io 2020–11–27T15:55:29Z
```

And are accessible in ArgoCD’s namespaces as common Kubernetes resources:

```
$ kubectl -n dev-1–18-devops-argocd-ns get applications
NAME SYNC STATUS HEALTH STATUS
backend-app OutOfSync Missing
dev-1–18-web-payment-service-ns Synced Healthy
web-fe-github-actions Synced Healthy
```

Such an approach would give an ability to create all necessary applications when creating a new ArgoCD instance. For example, when we are upgrading the Kubernetes version on our AWS Elastic Kubernetes Service, we are creating a new cluster with Jenkins, Ansible, and Helm (check [AWS Elastic Kubernetes Service: a cluster creation automation, part 1 — CloudFormation](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-a-cluster-creation-automation-part-1-cloudformation/) and [AWS Elastic Kubernetes Service: a cluster creation automation, part 2 — Ansible, eksctl](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-a-cluster-creation-automation-part-2-ansible-eksctl/) for more details). where we have all necessary controllers and operators installed, including ArgoCD.

By using the Declarative approach for ArgoCD, we also can create all applications and other settings configured during a new EKS cluster provisioning.

So, our task, for now, is to create:

- Projects with roles and permissions for namespaces Backend and Web teams
- Applications — for our Backend and Web projects
- Repositories — set up authentication for Github repositories

At first, let’s see how it’s working in general by creating manifest and resources manually, and then will apply this to the Ansible role and will create a Jenkins job.

- [Projects](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Projects)
- [Applications](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Applications)
- [Repositories](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Repositories)
- [Repository Secret](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Repository_Secret)
- [Jenkins, Ansible, and ArgoCD](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Jenkins_Ansible_and_ArgoCD)
- [Ansible Kubernetes Secret](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Ansible_Kubernetes_Secret)
- [Ansible ArgoCD Projects and Applications](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Ansible_ArgoCD_Projects_and_Applications)
- [Jenkins](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/#Jenkins)

### Projects

Documentation — [Projects](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#projects).

We need to have two projects for two of our teams — Backend, and Wb, and each needs to have limits for used Namespaces and must have a role configured for access to applications in those projects.

Create a new file with the Backend’s Project:

```
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: backend
  namespace: dev-1-18-devops-argocd-ns
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  description: "Backend project"
  sourceRepos:
  - '*'
  destinations:
  - namespace: 'dev-1-18-backend-*'
    server: [https://kubernetes.default.svc](https://kubernetes.default.svc)
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceWhitelist:
  - group: "*"
    kind: "*"
  roles:
  - name: "backend-app-admin"
    description: "Backend team's deployment role"
    policies:
    - p, proj:backend:backend-app-admin, applications, *, backend/*, allow
    groups:
    - "Backend"
  orphanedResources:
    warn: true
```

Here:

- `sourceRepos`: allow deploying to the project from any repositories
- `destinations`: a cluster and namespaces in it that are allowed to deploy to. In the example above we are using the _dev-1-18-backend-*_ mask, so the Backend team will be allowed to deploy to any namespace started from the mask, and for the Web project we will use a similar mask _dev-1-18-web-*_
- `clusterResourceWhitelist`: on a cluster level allow creating Namespaces only
- `namespaceResourceWhitelist`: on Namespaces level allow creating any resources
- `roles`: create a _backend-app-admin_ role with full access to applications in this project. See more at [ArgoCD: users, access, and RBAC](https://rtfm.co.ua/en/argocd-users-access-and-rbac/) and [ArgoCD: Okta integration, and user groups](https://rtfm.co.ua/en/argocd-okta-integration-and-user-groups/)
- `orphanedResources`: enable notifications about stale resources, check the [Orphaned Resources Monitoring](https://argoproj.github.io/argo-cd/user-guide/orphaned-resources/)

Deploy it:

```
$ kubectl apply -f project.yaml
appproject.argoproj.io/backend created
```

And check:

```
$ argocd proj get backend
Name: backend
Description: Backend project
Destinations: <none>
Repositories: *
Whitelisted Cluster Resources: /Namespace
Blacklisted Namespaced Resources: <none>
Signature keys: <none>
Orphaned Resources: enabled (warn=true)
```

### Applications

Documentation — [Applications](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#applications).

Next, let’s add a manifest for a test application in the Backend project:

```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backend-app
  namespace: dev-1-18-devops-argocd-ns
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: "backend"
  source:
    repoURL: [https://github.com/projectname/devops-kubernetes.git](https://github.com/projectname/devops-kubernetes.git)
    targetRevision: DVPS-967-ArgoCD-declarative-bootstrap
    path: tests/test-svc
    helm:
      parameters:
      - name: serviceType
        value: LoadBalancer
  destination:
    server: [https://kubernetes.default.svc](https://kubernetes.default.svc)
    namespace: dev-1-18-backend-argocdapp-test-ns
  syncPolicy:
    automated:
      prune: false
      selfHeal: false
      allowEmpty: false
    syncOptions:
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
```

Here:

- `finalizers`: set to delete all Kubernetes resources when deleting the application from ArgoCD (_cascade deletion_, see [App Deletion](https://argoproj.github.io/argo-cd/user-guide/app_deletion/))
- `project`: a project to add the application to and which will set limits for the application (namespaces, resources, etc)
- `source`: set a repository to deploy from, and a branch
- in the `helm.parameters` - a set values for a Helm chart (`helm --set`)
- `destination`: a cluster and namespace to deploy to. The Namespace here must be in allowed in the Project that the application belongs to
- `syncPolicy`: synchronization settings, check [Automated Sync Policy](https://argoproj.github.io/argo-cd/user-guide/auto_sync/) and [Sync Options](https://argoproj.github.io/argo-cd/user-guide/sync-options/)

Also, ArgoCD supports the so-called App of Apps, when you are creating a manifest for an application that will create a set of other applications. We will not use it 9yet), but looks interesting, see the [Cluster Bootstrapping](https://argoproj.github.io/argo-cd/operator-manual/cluster-bootstrapping/).

Deploy the application:

```
$ kubectl apply -f application.yaml
application.argoproj.io/backend-app created
```

Check it:

```
$ kubectl -n dev-1–18-devops-argocd-ns get application backend-app
NAME SYNC STATUS HEALTH STATUS
backend-app Unknown Healthy
```

And its status:

```
$ argocd app get backend-app
Name: backend-app
Project: backend
Server: [https://kubernetes.default.svc](https://kubernetes.default.svc)
Namespace: dev-1–18-backend-argocdapp-test-ns
URL: https://dev-1–18.argocd.example.com/applications/backend-app
Repo: [https://github.com/projectname/devops-kubernetes.git](https://github.com/projectname/devops-kubernetes.git)
Target: DVPS-967-ArgoCD-declarative-bootstrap
Path: tests/test-svc
SyncWindow: Sync Allowed
Sync Policy: Automated
Sync Status: Unknown
Health Status: Healthy

CONDITION MESSAGE LAST TRANSITION

ComparisonError rpc error: code = Unknown desc = authentication required 2021–05–18 09:11:13 +0300 EEST
OrphanedResourceWarning Application has 1 orphaned resources 2021–05–18 09:11:13 +0300 EEST
```

![](https://cdn-images-1.medium.com/max/1024/0*fCZ-CfH9-nb6KjJW.png)

Here, we have the “ **_ComparisonError = authentication required_** ”, as ArgoCD can not connect to the repository because it is private.

Let’s go to the repository authentication.

### Repositories

Documentation — [Repositories](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#repositories).

Suddenly, repositories are added to the `argocd-cm` ConfigMap, instead of being a dedicated resource such as Applications or Projects.

As for me, not too good solution as if a developers want to add a repository, I’ll have to give him access to the “system” ConfigMap.

Also, authentication for a Git server uses Kubernetes secrets stored in an ArgoCD’s namespace, so the developer will need to have access there too.

Still, ArgoCD has a way to authenticate on a Git server for different repositories by using the same authentication setting, see the [Repository Credentials](https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#repository-credentials).

The idea behind is that we can set a “mask” for repositories, i.e [_https://github.com/projectname_,](https://github.com/projectname,) and attach login and password or an SSK private key. Then, a developer can set a repository like [_https://github.com/orgname/reponame_,](https://github.com/orgname/reponame,) and ArgoCD will use the _github.com/projectname_ mask and will perform authentication on the server for the _github.com/projectname/reponame_ repository.

In this way, we can create a “single point of authentication” and all our developers will use it for their repositories, as all our repositories are located in the same Github Organization _orgname_.

#### Repository Secret

Here, we will use HTTPS and [Github Access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token).

Go to the Github profile and create a token (best to create a dedicated user for ArgoCD):

![](https://cdn-images-1.medium.com/max/942/0*VUw7j4dZWPtBdc30.png)

Give repositories permissions:

![](https://cdn-images-1.medium.com/max/798/0*873qYwJZnTLTSXn8.png)

Encode the token with base64 in terminal or by using the [https://www.base64encode.org](https://www.base64encode.org):

```
$ echo -n ghp_GE***Vx911 | base64
Z2h***xMQo=
```

And user name:

```
$ echo -n username | base64
c2V***eTI=
```

Add a Kubernetes Secret in the ArgoCD Namespace:

```
apiVersion: v1
kind: Secret
metadata:
  name: github-access
  namespace: dev-1-18-devops-argocd-ns
data:
  username: c2V***eTI=
  password: Z2h***xMQ==
```

Create it:

```
$ kubectl apply -f secret.yaml
secret/github-access created
```

Add its use to the `argocd-cm` ConfigMap:

```
...
  repository.credentials: |
    - url: [https://github.com/orgname](https://github.com/orgname)
      passwordSecret:
        name: github-access
        key: password
      usernameSecret:
        name: github-access
        key: username
...
```

Check:

```
$ argocd repocreds list
URL PATTERN USERNAME SSH_CREDS TLS_CREDS
[https://github.com/projectname](https://github.com/projectname) username false false
```

![](https://cdn-images-1.medium.com/max/728/0*qbgatkTv9e0_zcTX.png)

And try to synchronize the application:

```
$ argocd app sync backend-app
…
Message: successfully synced (all tasks run)
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service dev-1–18-backend-argocdapp-test-ns test-svc Synced Progressing service/test-svc created
```

### Jenkins, Ansible, and ArgoCD

Now it’s time to think about automation.

We have our ArgoCD installed from its Helm chart with an Ansible role by using the community.kubernetes module similar to the [Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS](https://rtfm.co.ua/ansible-modul-community-kubernetes-i-ustanovka-helm-charta-s-externaldns/#Ansible_%D1%80%D0%BE%D0%BB%D1%8C) (_Rus_).

What we need to create:

- to the Ansible role add a Secret for Github
- in the ArgoCD settings need to add the repository.credentials
- create directories for Projects and Applications manifests, and in the Ansible role add [`community.kubernetes.k8s`](https://docs.ansible.com/ansible/latest/collections/community/kubernetes/k8s_module.html) that will use those manifests. in this way, applications will be created automatically during new ArgCD instance provisioning, and developers will be able to add a new application by themself

#### Ansible Kubernetes Secret

In a variables file, in our case, this will be the `group_vars/all.yaml`, add two new variables and encrypt them with the [`ansible-vault`](https://rtfm.co.ua/ansible-ispolzovanie-vault-zashifrovannogo-xranilishha/). Do not forget to encode them to base64 before encrypting:

```
...
argocd_github_access_username: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          63623436326661333236383064636431333532303436323735363063333264306535313934373464
          ...
          3132663634633764360a666162616233663034366536333765666364643363336130323137613333
          3636
argocd_github_access_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          61393931663234653839326232383435653562333435353435333962363361643634626664643062
          ...
          6239623265306462343031653834353562613264336230613466
...
```

In the role tasks, in this case `roles/argocd/tasks/main.yml`, add the Secret creation:

```
...
- name: "Create github-access Secret"
  community.kubernetes.k8s:
    definition:
      kind: Secret
      apiVersion: v1
      metadata:
        name: "github-access"
        namespace: "{{ eks_env }}-devops-argocd-2-0-ns"
      data:
        username: "{{ argocd_github_access_username }}"
        password : "{{ argocd_github_access_password }}"
...
```

In the values of the Helm chart add `repository.credentials`, see [`values.yaml`](https://github.com/argoproj/argo-helm/blob/master/charts/argo-cd/values.yaml#L574):

```
...
- name: "Deploy ArgoCD chart to the {{ eks_env }}-devops-argocd-2-0-ns namespace"
  community.kubernetes.helm:
    kubeconfig: "{{ kube_config_path }}"
    name: "argocd20"
    chart_ref: "argo/argo-cd"
    release_namespace: "{{ eks_env }}-devops-argocd-2-0-ns"
    create_namespace: true
    values:
      ...
      server:
        service:
          type: "LoadBalancer"
          loadBalancerSourceRanges:
            ...
        config:
          url: "https://{{ eks_env }}.argocd-2-0.example.com"
          repository.credentials: |
            - url: "https://github.com/projectname/"
              passwordSecret:
                name: github-access
                key: password
              usernameSecret:
                name: github-access
                key: username
...
```

#### Ansible ArgoCD Projects and Applications

Create directories to store manifest files:

```
$ mkdir -p roles/argocd/templates/{projects,applications/{backend,web}}
```

In the `roles/argocd/templates/projects/` directory create two files for two projects:

```
$ vim -p roles/argocd/templates/projects/backend-project.yaml.j2 roles/argocd/templates/projects/web-project.yaml.j2
```

Describe the Backend Project:

```
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: "backend"
  namespace: "{{ eks_env }}-devops-argocd-2-0-ns"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  description: "Backend project"
  sourceRepos:
  - '*'
  destinations:
  - namespace: "{{ eks_env }}-backend-*"
    server: "https://kubernetes.default.svc"
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceWhitelist:
  - group: "*"
    kind: "*"
  roles:  
  - name: "backend-app-admin"
    description: "Backend team's deployment role"
    policies:
    - p, proj:backend:backend-app-admin, applications, *, backend/*, allow
    groups:
    - "Backend" 
  orphanedResources:
    warn: true
```

Repeat for the Web

And create applications.

Add a `roles/argocd/templates/applications/backend/backend-test-app.yaml.j2` file:

```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: "backend-app-1"
  namespace: "{{ eks_env }}-devops-argocd-2-0-ns"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: "backend"
  source:
    repoURL: "https://github.com/projectname/devops-kubernetes.git"
    targetRevision: "DVPS-967-ArgoCD-declarative-bootstrap"
    path: "tests/test-svc"
    helm:
      parameters:
      - name: serviceType
        value: LoadBalancer
  destination:
    server: "https://kubernetes.default.svc"
    namespace: "{{ eks_env }}-backend-argocdapp-test-ns"
  syncPolicy:
    automated:
      prune: true
      selfHeal: false
    syncOptions:
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    retry:
      limit: 2
```

At the end of the tasks, add those manifests apply — first, Projects, then Applications:

```
...
- name: "Create a Backend project"
  community.kubernetes.k8s:
    kubeconfig: "{{ kube_config_path }}"
    state: present
    template: roles/argocd/templates/projects/backend-project.yaml.j2
          
- name: "Create a Web project"
  community.kubernetes.k8s: 
    kubeconfig: "{{ kube_config_path }}"
    state: present
    template: roles/argocd/templates/projects/web-project.yaml.j2
          
- name: "Create Backend applications"
  community.kubernetes.k8s:
    kubeconfig: "{{ kube_config_path }}"
    state: present
    template: "{{ item }}"
  with_fileglob:
    - roles/argocd/templates/applications/backend/*.yaml.j2

- name: "Create Web applications"
  community.kubernetes.k8s:
    kubeconfig: "{{ kube_config_path }}"
    state: present
    template: "{{ item }}"
  with_fileglob:
    - roles/argocd/templates/applications/web/*.yaml.j2
```

For Applications use the [`with_fileglob`](https://rtfm.co.ua/ansible-with_fileglob-kopirovanie-neskolkix-shablonov-iz-direktorii/) to get all files from the `roles/argocd/templates/applications/`, as it's planned to have dedicated files per each Application to make it easier for developers to manage them.

#### Jenkins

See examples in the [Jenkins: миграция RTFM 2.6 — Jenkins Pipeline для Ansible](https://rtfm.co.ua/jenkins-migraciya-rtfm-2-6-jenkins-pipeline-dlya-ansible/) and [Helm: пошаговое создание чарта и деплоймента из Jenkins](https://rtfm.co.ua/helm-poshagovoe-sozdanie-charta-i-deplojmenta-iz-jenkins/#Jenkins_job) (both in Rus, unfortunately).

I will not describe this in details, but in short, we are using a Scripted Pipeline with a `provision.ansibleApply()` function call:

```
...
            stage("Applly") {
                // ansibleApply((playbookFile='1', tags='2', passfile_id='3', limit='4')
                provision.ansibleApply( "${PLAYBOOK}", "${env.TAGS}", "${PASSFILE_ID}", "${EKS_ENV}")
            }
...
```

The function is the next:

```
...
def ansibleApply(playbookFile='1', tags='2', passfile_id='3', limit='4') {
        
    withCredentials([file(credentialsId: "${passfile_id}", variable: 'passfile')]) {

        docker.image('projectname/kubectl-aws:4.5').inside('-v /var/run/docker.sock:/var/run/docker.sock --net=host') {
            
            sh """
                aws sts get-caller-identity
                ansible-playbook ${playbookFile} --tags ${tags} --vault-password-file ${passfile} --limit ${limit}
            """ 
        }   
    }
}.
...
```

Here, with Docker, we are spinning up a container for our image with the AWS CLI and Ansible, which is running an Ansible playbook, passes a tag, and executes necessary Ansible role.

In the playbook, we have tags set for each role, so it’s easy to execute only one role:

```
...
    - role: argocd
      tags: argocd, create-cluster
...
```

As the result, we have a Jenkins job with such parameters:

![](https://cdn-images-1.medium.com/max/514/0*emAZtIpd59g8_exC.png)

Run it:

![](https://cdn-images-1.medium.com/max/1024/0*QP2svJTm9VqVSGFl.png)

Log in to the new ArgoCD instance:

```
$ argocd login dev-1–18.argocd-2–0.example.com --name admin@dev-1–18.argocd-2–0.example.com
```

Check applications:

![](https://cdn-images-1.medium.com/max/863/0*5ZPT3weZYcVRlLgh.png)

Done.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/argocd-declarative-projects-applications-and-argocd-deploy-from-jenkins/)_._

* * *