QA - no excuses!
Tags: ci, qa, devops, docker
Published: 2022-01-24 17:17:19

I made a very interesting journey into the world of Continuous Integration and QA. The result is a [project](https://github.com/andreav/jenkins-docker-qa) based on Docker, Jenkins and some really interesting tools I've tried to put together.

Not only do I ask you to download and try it, but above all to recommend new tools to integrate to get the best pipeline ever!

This project is a starter kit for setting up a Jenkins docker installation for CI and QA.  
*I developed on Windows, with Docker Desktop with WSL 2 engine and VS Code, but with minor adjustments (before all slash notation for command line commands) it should run on Linux too.*  

## Tools integrated

| Tool                                 | Usage                   |
|--------------------------------------|-------------------------|
| [Docker](https://www.docker.com/) | Jenkins is run inside Docker and pipelines run inside docker agents |
| [Jenkins](https://www.jenkins.io/)   |  The automation server  |
| [Jenkins Configuration as Code](https://www.jenkins.io/projects/jcasc/) | Jenkins plugin for run an image pre-provisioned / pre-configured |
| [Jenkins plugins](https://plugins.jenkins.io/) | Many Jenkins plugins |
| [SonarQube](https://www.sonarqube.org/) | Static code analyser | 
| [PostgreSQL](https://www.postgresql.org/) | Database for storing Sonarqube results |
| [xUnit.Net](https://xunit.net/) | Testing framework - used for Unit test, UI tests, Coverage |
| [Selenium](https://www.selenium.dev/) |  Web browser automation for UI tests |
| [k6](https://k6.io/) | Load testing tool |
| [Grafana](https://grafana.com/) | Visualization tool for analyzing load test results |
| [influxdb](https://www.influxdata.com/) | Database for storing k6 results used by k6 and Grafana |

## docker compose up

Project ships with some docker compose files for bringing up main parts of the system:

* /docker-compose.yml
  Brings up the core infrastructure: 
    * jenkins
    * local docker registry (insecure version)

* /sonarqube/docker-compose.yml
  Brings up sonarqube infrastructure
    * postregs database
    * sonarqube server

* /selenium/docker-compose.yml
  Brings up selenium infrastructure
    * selenium-hub
    * chrome node
    * firefox node
    * edge node

* /k6/docker-compose.yml
  Brings up Grafana infrastructure
    * influxdb:1.8
    * Grafana with pre-provisioned dashboards
  
If you want the full system up&running with just one command you can issue (**from the root of the repo**):

    docker compose -f .\docker-compose.yaml -f .\sonarqube\docker-compose.yaml -f .\k6\docker-compose.yaml up

You should see all containers coming up (following picture is taken from VS Code Docker extension):

![Containers up&running](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m4sxqi4tn0oqsdhx79ut.png "Containers up&running")

*Note: selenium containers are not run with this command because the grid will be automatically run on demand by the Jenkins before running UI tests.*

*Note: before running Sonarqube, be sure to meet [Docker Host Requirements](https://hub.docker.com/_/sonarqube)
Usually you should issue these command:*
```
  sysctl -w vm.max_map_count=524288
  sysctl -w fs.file-max=131072`
  ulimit -n 131072
  ulimit -u 8192
```

<br />

If everything went well, you now have many systems ready:

| URL | System |
|-----|--------|
| http://localhost:8181 | Jenkins |
| http://localhost:9000 | Sonarqube server - User: admin / Pasw: admin |
| http://localhost:4444 | Selenium Grid (only available while UI tests runs) |
| http://localhost:3000 | Grafana |

<br />

:warning: Security Notes :warning:
This stack is not production ready:
* For running docker agents from Jenkins, I exposed docker daemon on tcp://localhost:2375 without TLS locally 
* I relaxed some Jenkins [CSP](https://content-security-policy.com/) to correctly visualize k6 HTML reports.  
If you do not need this feature, feel free to remove `hudson.model.DirectoryBrowserSupport.CSP` directive inside /jenkins/Dockerfile  file.
* Docker registry is just meant for hosting local built images ( used as Jenkins Pipeline Docker agents) so it runs insecure (without certificates)

## Jenkins

The first piece of the puzzle is Jenkins, the automation tool.
A custom docker image is built starting from the official one for minor adjustments:
* disable wizard setup
* setting up Jenkins Configuration as Code plugin directory
* setting up pre-provisioned jobs
* modifying Content Security Policy for supporting k6 HTML reports
* adding docker and docker-compose to the image. This is useful when using the default agent, for running docker commands like bringing up a selenium grid
* pre-provisioning the image with a bunch of plugins

Thanks to the JCasC and Seed Job plugins when we start Jenkins we will find all the job folder containing test jobs alredy in place:

![Jenkins job folders](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/alb2clq2vd5k9q84b6bb.png "Jenkins job folders")

*Note: all the Jenkins configuration (JCasC and Jobs) is stored into separate files into /jenkins/casc_configs folder. You can add files here to automatically create new jobs or modify Jenkins configuration*

*Note: remember to [approve](https://stackoverflow.com/a/45771737/1966366) scripts before runnig them otherwise they'll fail.
You can also edit and re-save job configuration for approving jobs.*

<br />

## Test job folder *"docker"*:

Here you will find some tests for basic usage.
* testing the default agent
* testing a docker agent with an image from [docker hub](https://hub.docker.com/)
* locally building a test image and pushing it lo the local Docker registry
* testing a docker agent with a custom local image pulled by the local registry (the one previously built)

All these pieces are used in following pipelines, so it seems a good idea to test the basic blocks

<br />

## Test job folder *"sonarqube"*:

In this folder you'll find jobs for executing *static code analysis*.

For this purpose I used Sonarqube. Sonarqube has a client/server architecture:
* server - started with the docker compose command previously described
* client - the scanner - analyses the project and sends data to the server

In Jenkins, I'm running [dotnet core](https://dotnet.microsoft.com/en-us/) static analysis in two falvours:

* with a netcore scanner built on the fly
* with a netcore scanner pre-built, pulled from the local registry. This is meant to speed up this phase of the pipeline, using an agent with java and the netcore scanner already installed

I chose the very interesting [pitstop netcore project](https://github.com/EdwinVW/pitstop) to be analyzed.

Before running Sonarqube jobs, you need to create a Sonarqube token for the scanner to login into the server.
* In Sonarqube - create a token for admin user  
  - http://localhost:9000, user: admin, password: admin  
  - Go to: Administration -> Security -> Users -> "Administrator" Tokens
* In jenkins - config Sonarqube credential.  
  - JCasC has already provisioned a placeholder Sonarqube token, just update its password  with the token just created.  
  -  Go to  Manage Jenkins -> Manage Credentials -> sonarqube-token-admin -> Update -> Change Password

These are the jobs in the Sonarqube folder: 

* 01_build_netcore_sonarscanner_image_and_push_to_local_registry:
Builds a netcore image with java & sonarscanner already installed and puh it to local registry
* 02_sonarscanner-in-docker-agent:
Example of integrating jenkins - netcore - sonarqube
After running this job login to Sonarqube http://localhost:9000 and you'll see:
    * Static code analysis
    * Code coverage
    * Test results
* 03_sonarscanner-in-docker-agent-netcore-scanner:
Like the job before but faster.
This job uses a docker agent pre-configured for scanning net core projects.

In Sonarqube, the result should looks like this:

![sonarqube-project-overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kxbpzk86cwjgjn6x63h7.png "sonarqube-project-overview")


![sonarqube-project-details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5h3pz9tnst1omf21ffo1.png "sonarqube-project-details")

<br />

## Test job folder *"unit tests"*:

Here you will find a job example for running netcore unit tests with xUnit.

We again use the [pitstop netcore project](https://github.com/EdwinVW/pitstop) which ships with xUnit unit tests.

This job collects tests results and publishes results using [xUnit Jenkins plugin](https://plugins.jenkins.io/xunit/).

This job also collects cove coverage and publishes results using [cobertura Jenkins plugin](https://plugins.jenkins.io/cobertura/)

Results should look like this after running a couple of builds:

![Jenkins xUnit cobertura results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wvpktzqpogh66zwh1lah.png "Jenkins xUnit cobertura results")
 
<br />

## Test job folder *"selenium"*:

The job runs xunit + selenium UI tests against a basic grid (3 browsers)

*Note:* selenium grid is automatically run on demand by the jenkins job and discarded when the pipeline ends.

You can follow tests progress pointing your browser to http://localhost:4444

![Selenium Hub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8yswdodrfj8tsnrivpu7.png "Selenium Hub")

<br />

## Test job folder *"k6"*:

k6 is a wonderful tool for performance and load testing.

In this job folder we try two integrations:
* a pipeline running k6 load test and publishing HTML report directly in Jenkins. 
  After job runs, you should see a new report published like this:
![Jenkins HTML report](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80i2t72icure36pmhit7.png "Jenkins HTML report")
 
* a pipeline running k6 load test and publishing k6 metrics to Grafana  
If you browse to the dashboard from Grafana ("Dashboards" -> "Browse" -> "k6 Load Testing Results") after you run the load test, you will see the dashboard populated like this:

![Grafana k6 dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tuosvko48xogu4a9bnql.png "Grafana k6 dashboard")

<br />
HTML Results are obtained converting k6 output to HTML. 
This is done using the [k6-reporter project](https://github.com/benc-uk/k6-reporter).
The HTML generated is not compliant with Jenkins Content Security Policy so I [modified it](https://www.jenkins.io/doc/book/security/configuring-content-security-policy/) for allowing correct visualization.
If you do not need this feature, feel free to remove `hudson.model.DirectoryBrowserSupport.CSP` directive inside /jenkins/Dockerfile file.

For visualizing k6 results into Grafana I chose [influxdb+grafana integration](https://k6.io/docs/results-visualization/influxdb-+-grafana/).
Grafana is pre-provisioned with:
* a datasource pointing to influxdb (version 1.8 because version 2.0 is not still supported by k6)
* a [k6 dashboard](https://grafana.com/grafana/dashboards/2587)

Due to [this issue](https://github.com/grafana/grafana/issues/10786) it is not still possible to provision dashboard with variables in grafana.
So I modified the k6 dashbord replacing the variable "${DS_K6}" with the provisioned datasource UID:  "influxdb-k6-uid".


## Conclusions

I always take great inspiration from open-source projects.
This time I hope to be able to reciprocate.

This project is still a "work-in-progress", 
please let me know which tools can be integrated to release more and more quality software!

Thank you for reading.