A GitOps-Powered Kubernetes Testing Machine with ArgoCD and Testkube — Kubeshop
Tags: kubernetes, argo, testing, gitops
Published: 2022-04-21 08:51:23

One of the major trends in contemporary cloud native application development is the adoption of GitOps; managing the state of your Kubernetes cluster(s) in Git — with all the bells and whistles provided by modern Git platforms like GitHub and GitLab in regard to workflows, auditing, security, tooling, etc. Tools like ArgoCD or Flux are used to do the heavy lifting of keeping your Kubernetes cluster in sync with your Git repository; as soon as difference is detected between Git and your cluster it is deployed to ensure that your repository is the source-of-truth for your runtime environment.

Don’t you agree that it’s time to move testing and related activities into this paradigm also? Exactly! We at Kubeshop are working hard to provide you with the first GitOps-friendly Cloud-native test orchestration/execution framework — Testkube — to ensure that your QA efforts align with this new and shiny approach to application/cluster configuration management. Combined with the GitOps approach described above, Testkube will include your test artifacts and configuration in the state of your cluster and make git the source of truth for these test artifacts. And it’s Open-Source too.

Key wins achieved with this approach:
  1. Since your tests are included in the state of your cluster you are always able to validate that your application components/services work as required.
  2. Since tests are executed from inside your cluster there is no need to expose services under test externally purely for the purpose of being able to test them.
  3. Tests in your cluster are always in sync with the external tooling used for authoring
  4. Test execution is not strictly tied to CI but can also be triggered manually for ad-hoc validations or via internal triggers (Kubernetes events)
  5. You can leverage all your existing test automation assets from Postman, or Cypress, or through executor plugins.

From a conceptual perspective this can be illustrated as follows:


![Diagram of how the entire process looks like](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pjn03hsb6hrg7dkp3mhj.jpeg)

## Walkthrough

Enough talk - let’s see this in action - here comes a step-by-step walkthrough to get this in place for the automated deployment and execution of Postman collections in a local Minikube cluster to test.

Let’s start with setting things up for our GitOps-powered testing machine!

### Installations

**1. Install Minikube**

You can follow the minikube installation for your operating system [here](https://minikube.sigs.k8s.io/docs/start/).

**2. Install ArgoCD**

Follow the [ArgoCD installation guide](https://argo-cd.readthedocs.io/en/stable/getting_started).

**Note**: For step 3 “ Access The Argo CD API Server”, choose the “Port Forwarding” method, as that is the easiest way to connect to it with a Minikube cluster. 

**3. Install Testkube**

Follow the installation guide for Testkube [here](https://kubeshop.github.io/testkube/installing). Make sure to install the CLI client and the components in your cluster.

### Setting up application and tests

Install a “Hello Kubernetes!” application in your cluster
We will create a YAML file for a simple “Hello Kubernetes” application that we will then create our integration tests against. 

Create the file `hello-kubernetes.yaml`

{% embed https://gist.github.com/aabedraba/8752143657bbcf13795215d5d7c88675 %}

And deploy the _Hello Kubernetes_ deployment with:

```
kubectl apply -f hello-kubernetes.yaml‍
```

You can test that your application has been correctly installed by running:

```
minikube service hello-kubernetes-service
```

**5. Set up a Git Repository containing some Postman collections**

We are going to use tests created by Postman and exported in a [Postman collections](https://www.postman.com/collection/) file.

We can upload this to the same Git Repository as our application, but in practice the repository could be the same repository hosting the application or it could also be in a separate repository where you manage all your test artifacts.

So let’s create our `postman-collections.json` and push it to the repository.

{% embed https://gist.github.com/aabedraba/ed643af55e5b50e223d461fe1bb4a062 %}

You can see an example of how the repository should look like [here](https://github.com/aabedraba/testkube-argocd-tests).

### Configure ArgoCD to work with Testkube

**6. Configure ArgoCD to use the Testkube plugin**

To get ArgoCD to use Testkube, we need to add Testkube as a [plugin](https://argo-cd.readthedocs.io/en/stable/user-guide/config-management-plugins/). For that we have built a customized `argocd-repo-server` container image that will include Testkube as a binary.

Patch the ArgoCD repo-server pod image. 

Create a deployment patch manifest patch.yaml, this will be of type [JSON-patch](http://jsonpatch.com/):

{% embed https://gist.github.com/aabedraba/4e635ba4e48f49670122778056114df7 %}

And apply it with the following command:

```sh
kubectl patch deployments.apps -n argocd argocd-repo-server — type json — patch-file patch.yaml
```

**7. Define Testkube as a plugin in ArgoCD’s Configuration Management Plugin**

Create the file `argocd-plugins.yaml`

{% embed https://gist.github.com/aabedraba/d352a66c153bc31e850693b432ecaa97 %}

‍As you can see here, we’re using the command `testkube crd` which creates the Custom Resources (manifests) that ArgoCD will then add to our cluster. Apply the patch with the command:

```sh
kubectl patch -n argocd configmaps argocd-cm — patch-file argocd-plugin.yaml
```

**8. Configure an ArgoCD application to manage test collections in your cluster**

Create the file `testkube-application.yaml`

{% embed https://gist.github.com/aabedraba/8017ee780fe44f15a92afa8ad0130945 %}

Notice that we have defined `path: postman-collections` which is the test folder with our Postman collections from the steps earlier. With Testkube you can use multiple test executors like curl for example, so it is convenient to have a folder for each. We have also defined the `.destination.namespace` to be `testkube`, which is where the tests should be deployed in our cluster.‍

Now let’s create the application with:

```sh
kubectl apply -f testkube-application.yaml
```

**9. Run the initial ArgoCD sync and check your cluster**

On ArgoCD’s dashboard, we will now see the newly created application. Let’s click to get into it and sync our tests.


![Clicking on Testkube application in ArgoCD](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i0s4ccoyb0hhpo1znj5a.png)

And now click on _Sync_ to see your tests created.


![Clicking on Sync in ArgoCD](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1b8fp0ewm5xuq4sw8kv0.png)

And voilà, there’s our test collection created and managed by ArgoCD with every new test created and updated in the Github repository containing the tests!

![hello-kubernetes successfully deployed in ArgoCD](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d6doefm75rogzqjiw86l.png)

### Run your tests!

**10. Run ad-hoc tests from the CLI**

Now that we’re all set - let’s try some ad-hoc test execution using Testkube’s CLI

List the tests in your cluster with:

```sh
kubectl-testkube tests list
```

You should see your deployed test artifacts 

![listing testkube tests in terminal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3yldfr9z01h4ov6167v.png)

To run those tests execute the following command:

```
kubectl-testkube tests run hello-kubernetes
``` 

The test execution will start in the background, you now need to copy the command from the image below to check the result of the execution of the test

![running a testkube test in terminal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fgih8l1bkhpz2rkzkvxn.png)

```
kubectl testkube tests execution EXECUTION_ID
```

And you should see that the tests have run successfully, just like in the image below.

![Results of a Testkube test in terminal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5kfw6gj4ad7nwqr5dz9.png)

**11. See test results in the Testkube dashboard**

You can also see the results of your tests in a nice dashboard. Just open the Testkube dashboard with the following command

```
kubectl-testkube dashboard
```
‍

And you will be able to see the results of the execution in the Executions tab as seen in the image below.
 
![Testkube dashboard showing the tests run](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tr73924tztywg19s8pkw.png)

**12. Test the flow: update the test and deploy the updated test with ArgoCD**

Let’s add an additional test to our collection. Replace the content our existing test in `hello-kubernetes.json` with the following: 

{% embed https://gist.github.com/aabedraba/808c38bff8dd88b436b2aa3453f29857 %}

As you can see, we have added a request status check. Now commit this change to the Github repository. 

If you now go to ArgoCD’s dashboard you’ll see that your tests are out of sync with the deployed artifacts.

![ArgoCD dashboard showing OutOfSync message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/79upqadsxkv0jk4ee1mp.png)

Click on Sync again and apply the changes. With that, your test artifacts are back in sync!

![ArgoCD dashboard showing Synced message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vibcnzf18gug7pqxmx73.png)

Wow - that was quite a lot to get through but we ended up with something really neat - an automated test deployment and execution pipeline based on GitOps principles!

## Wrapping up

Once fully realized - using GitOps for testing of Kubernetes applications as described above provides a powerful alternative to a more traditional approach where orchestration is tied to your current CI/CD tooling and not closely aligned with the lifecycle of Kubernetes applications.

Would love to get your thoughts on the above approach - over-engineering done right? Waste of time? Let us know!

Check Testkube on GitHub — and let us know if you’re missing something we should be adding to make your k8s resource testing easier.

- [Download the latest release](https://github.com/kubeshop/testkube/releases) on GitHub
- Check out the [documentation](https://kubeshop.github.io/testkube/)
- Get in touch with us on our [Discord server](https://discord.gg/uNuhy6GDyn) 

Thank you!