Continuous Container Vulnerability Testing With Trivy
Tags: tutorial, kubernetes, security, docker
Published: 2021-11-04 13:45:17

Without security in your CI/CD, sooner or later, a vulnerability will sneak in, and before you know it, someone will be mining at your expense.


💛 *Huge thanks to [Teppei Fukuda](https://github.com/knqyf263) for reviewing this tutorial.*

You’ve adopted [continuous integration](https://semaphoreci.com/continuous-integration) (CI), followed TDD and BDD principles — having mastered [testing](https://semaphoreci.com/blog/20-types-of-testing-developers-should-know), you deploy anytime, at a moment’s notice. You are a CI guru. You can live happily ever after.

Like all fairy tales, the story ends abruptly; before real life intrudes. Before we realize that **a vital component has been left out**, security.

In this tutorial, I’ll propose ways of integrating security at every stage of your [CI/CD workflow](https://semaphoreci.com/cicd). I can’t promise a happy life, but I guarantee you’ll sleep better at night.

## Reactive vs. proactive security

When we do vulnerability or penetration testing in a live production system, we say we’re doing *reactive security*. We’re in a race — trying to find the issues before anyone else can exploit them.

Even if the security scan is very comprehensive, when all is said and done, we’re just patching holes. Better it would be if holes wouldn’t be there in the first place. 

![Reactive Security](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4pl6tz90duod34gfd3nc.png)

On the other end of the spectrum, we have *proactive security*. Proactive security is all you do before release or deployment. We want to prevent the system from ever being exposed to a vulnerability.

![Proactive Security](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/909qyn5kwv6srac5z4o4.png)

There’s nothing wrong with being reactive, as long as it’s not the only solution you're using.

## Proactive security with Trivy

[Trivy](https://github.com/aquasecurity/trivy) is an open-source security and misconfiguration scanner. It works at every level: it can check the code in a Git repository, examine container images, advise regarding configuration files, look into Kubernetes deployments, and verify Infrastructure as Code (IaC).

Trivy is maintained by [Aqua](https://www.aquasec.com/), and feeds from their [vulnerability database](https://avd.aquasec.com/) and [many other data sources](ttps://aquasecurity.github.io/trivy/v0.20.2/vulnerability/detection/data-source/). It runs on Linux, macOS, Docker, as Helm Chart, and a VS Code Extension.

## Automated security in your CI/CD

There are at least four security checkpoints in every CI/CD. We'll learn how to use Trivy at each one of these stages:

1.  The source code **dependencies**.
2.  **Artifacts** such as Docker images. Attackers exploit vulnerabilities deep down in the application or the supporting libraries to break out from the container.
3.  **Configuration files**.
4.  **Infrastructure code** describing cloud services that power the application.

## Vulnerability testing for dependencies

Before we start, we need something to test. In this tutorial, we’ll be using a “Hello, World” Kubernetes demo. So go ahead and fork it hereÇ

-   [semaphore-demo-ruby-kubernetes](https://github.com/semaphoreci-demos/semaphore-demo-ruby-kubernetes).

For maximum effect, you can combine these steps with the ones laid down in past tutorials:

-   With [Structure Tests for Docker Containers](https://semaphoreci.com/blog/structure-testing-for-docker-containers) we run custom tests inside the container image.
-   In [Secure Kubernetes deployments](https://semaphoreci.com/blog/kubernetes-deployments) we confirm that the Kubernetes manifests are using sane deployment practices.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pfyoqz3utgeydcarr6cc.png)

Next, install [Aqua Trivy](https://aquasecurity.github.io/trivy/v0.20.2/getting-started/installation/). The first time Trivy runs, it downloads the vulnerability database and creates a cache folder for results. You can clean it up with `trivy --reset`.

To run a dependency scan use `trivy fs`. Trivy detects the `Gemfile` in our project and searches for vulnerabilities.

``` bash
$ trivy fs .

Need to update DB
Downloading DB...
Number of language-specific files: 1
Detecting bundler vulnerabilities...

Gemfile.lock (bundler)
======================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
```

You can even scan a repository without cloning it with `trivy repo`.

``` bash
$ trivy repo https://github.com/knqyf263/trivy-ci-test
```

You may simultaneously run a scan for vulnerabilities and misconfigurations by adding `--security-checks vuln,config`. The scan will include any Dockerfiles, Kubernetes, and Terraform files in the repo.

``` bash
$ trivy fs --security-checks vuln,config .
```

💡You can see all the languages and package managers supported by Trivy [here](https://aquasecurity.github.io/trivy/v0.20.2/vulnerability/detection/language/).

## Scanning dependencies with CI/CD

The dependencies on our demo are clean since the repository receives frequent [dependabot pull requests](https://docs.github.com/en/code-security/supply-chain-security/managing-vulnerabilities-in-your-projects-dependencies/about-dependabot-security-updates#about-dependabot-security-updates) from GitHub. But sependabot, being asynchronous, is reactive so it's not enough to prevent us from shipping unsafe code. We can switch to proactive mode by integrating Trivy into our [CI/CD pipeline](https://semaphoreci.com/blog/cicd-pipeline).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2mdsgizhkpdt8rgnplv.png)

Let's see how it works. After [adding your repository to Semaphore](https://docs.semaphoreci.com/guided-tour/getting-started/), click on **Add Block** and type the following commands in the job:

``` bash
wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
trivy fs --exit-code 1 .
```

The first two lines install Trivy in the [CI machine](https://docs.semaphoreci.com/ci-cd-environment/machine-types/). The third, `checkout`, clones the repository. The last one runs Trivy with `--exit-code 1` to force the pipeline to stop when some problem is detected.

The CI pipeline looks like this after adding the Trivy scan:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/139wxrgb50dwe1u6c4vj.png)

For extra security, we can verify the checksum of the Trivy package. The checksum file is included in [releases](https://github.com/aquasecurity/trivy/releases). The following line will make the pipeline stop if checksums don't match. 

``` bash
$ echo "46119ad9571f740201461b7529059afacb01ff74549de60bca657deba2f556cd  trivy_0.20.1_Linux-64bit.deb" | sh
asum -c -a 256
```

Adjust the filename and checksum as needed.

## Scanning Docker images

Before we can deploy the application, we have to package it with Docker. Since container images may have [misconfigurations that leave the system open](https://aquasecurity.github.io/trivy/v0.20.2/misconfiguration/), we can't go on without scanning them. We'll break down the task into three steps:

1. The base image we’re building from

2. The Dockerfile that packages the application.
3. The final container image.

You can learn more about Docker and Kubernetes with our free ebook: [CI/CD with Docker and Kubernetes](https://semaphoreci.com/resources/cicd-docker-kubernetes).

### Vulnerability testing for base images

We're base image for our application's container is a Debian 11.1 "Bullseye". We use `trivy image` to scan it directly from the image registry. You'll need a Docker installation for this to work.

``` bash
$ trivy image ruby:2.7
```

**TIP** 💡: You can get other output formats with `-f FORMAT` (for example `-f json`).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1hqne9aqbtfdeeoiv9t5.png)

In all likelihood, there are dozens of vulnerabilities in the image ranging from low to critical. At this point we need to start applying risk management strategies and think what level of security is "good enough" for our needs. We'll worry for now about high and critical issues. By adding `--severity HIGH,CRITICAL` to the command we can filter results.

The next step is to go through the list of issues and assess what actions, if any, are needed. Some of the vulnerable packages are not actually needed and can be removed. Let’s do that by adding these lines into `Dockerfile` and `Dockerfile.ci`:

``` docker
# remove unneeded packages with vulnerabilities
RUN apt-get purge -y curl "libcurl*" libaom0 python3.9
RUN apt-get autoremove -y
```

And build a new “base image” to see if the vulnerabilities are gone.

``` bash
$ docker build -t my-test-image .
$ trivy image --severity HIGH,CRITICAL my-test-image
```

 Other vulnerabilities may be patched or ignored. We have two ways of skipping vulnerabilities with Trivy:

-   Adding `--ignore-unfixed` to the command hides vulnerabilities that do not have a fix or patch.
-   In `.trivignore` we list the CVEs we want to skip. 

For instance, if after reading about a particular issue and deciding I can safely ignore it, I add the following line into `.trivyignore`:

``` text
# a libc vulnerability in the base image, currently unfixed
CVE-2021-33574
```

Commit the changed files once satisfied with the results.

### Vulnerability testing container images

With the base image and the dependencies scanned, the resulting container should be pretty safe. Let’s confirm this by testing the final Docker image in the pipeline

The check runs after the “Docker build” job in the continuous delivery pipeline. As you can see in the screenshot below, I added an image scan [in parallel](https://semaphoreci.com/blog/revving-up-continuous-integration-with-parallel-testing) with [Container Structure Tests](https://semaphoreci.com/blog/structure-testing-for-docker-containers).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6r6vgevzha7x140i05j6.png)

The job pulls the recently-built Docker image and scans it with `trivy image`. You may need to add `docker login` and a [secret](https://docs.semaphoreci.com/essentials/environment-variables/) if your image is private.

``` bash
wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
docker pull "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
trivy image --severity HIGH,CRITICAL "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
```

### Scan the Dockerfiles

Scanning the image artifact is not the end of the story; there’s one more thing we can do to improve its security: check if we’re using good practices in our Dockerfiles. Trivy understands Dockerfiles and can offer suggestions on improving them.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/79cljpbeoex1u1c9dfvo.png)

Dockerfile-scanning is done with `trivy config`. We have to copy the files into a temporary folder to avoid scanning all config files in the repository:

``` bash
$ mkdir -p audit-dockerfiles
$ cp Dockerfile* audit-dockerfiles
$ cd audit-dockerfiles
$ trivy config .
$ cd -
```

The analogous job in the pipeline looks like this:

``` bash
wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
dockerdir=$(mktemp -d)
cp Dockerfile* $dockerdir
(cd $dockerdir; trivy config --exit-code 1 .)
```

The new job should run **before** “Docker build” in the pipeline.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fxykxnslnf5dh2iw9v63.png)

## Scan Kubernetes

As important as securing the application is keeping the infrastructure it runs on safe. Where and how we run it must also come under a magnifying glass.

We’re running our demo project in a Kubernetes cluster. This means we have at least two levels of security checks ahead of us: the cluster **infrastructure** and the **deployment**.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sojvghj9rw7bxsyyii7f.png)

### Test the Infrastructure

Being proactive in this area means using IaC tools such as Terraform, so Trivy can enforce a set of rules that [encode good security practices](https://github.com/aquasecurity/appshield).

Our demo project doesn’t include any IaC files, but this is where the Trivy job would go if it did.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o47oh92w64arsorwpyya.png)

As with Dockerfiles, we use `trivy config` to scan the Terraform files:

``` bash
wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
cd terraform
terraform init
trivy config --exit-code 1 --severity MEDIUM,HIGH,CRITICAL .
```

### Check Kubernetes manifests

You may not manage or have access to the cluster, so infrastructure tests may not apply to you. But if you’re using Kubernetes, for sure you'll have to worry about manifests. Manifest are configuration files that completely describe how Kubernetes runs the application. `trivy config` serves as an excellent way of rounding out deployment checks.

We’ll add this test after the infrastructure scanning and next to other manifest tests done in a previous post: [Secure Your Kubernetes Deployments](https://semaphoreci.com/blog/kubernetes-deployments).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ucqy5boovyf2oheovmp7.png)

The job runs `trivy config` on the deployment files in a temporary folder:

``` bash
wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
k8sdir=$(mktemp -d)
cp deployment*.yml $k8sdir
(cd $k8sdir; trivy config --exit-code 1 --severity HIGH,CRITICAL .)
```

## Extending Trivy

Let me close up this post by mentioning that Trivy can be extended with plugins and custom policies. For example, Aqua provides the [kubectl plugin](https://github.com/aquasecurity/trivy-plugin-kubectl) to better integrate Trivy with Kubectl. The plugin lets us scan images running in a Kubernetes pod or deployment:

``` bash
$ trivy plugin install github.com/aquasecurity/trivy-plugin-kubectl

# Scan a pod
$ trivy kubectl pod mypod

# Scan a deployment
$ trivy kubectl trivy deployment mydeployment
```

Custom policies are written in the [rego language](https://www.openpolicyagent.org/docs/latest/policy-language/) and may be used to enforce specific rules in your organization. To learn about how to write policies, check out the [custom policies doc page](https://aquasecurity.github.io/trivy/v0.20.2/misconfiguration/custom/).

## Conclusion

A 100% secure system is a chimera. But that doesn't mean we can't use every tool at hand to reduce the odds of having a security breach.

You have added a total of five new security checkpoints into your pipeline. You can tweak the `severity` levels until you reach the right balance between functionality and security.

Thanks for reading!

**Related reads**:

- [Podcast] [Justin Cormack on Integrating Security into Software Building](https://semaphoreci.com/blog/justin-cormack-on-building-secure-software).
- [Design an Effective Build Stage for Continuous Integration](https://semaphoreci.com/blog/build-stage).
- [Kubernetes vs Docker: Understanding Containers in 2021](https://semaphoreci.com/blog/kubernetes-vs-docker).
- [Beyond Docker with Earthly](https://semaphoreci.com/blog/beyond-docker-with-earthly).