API's From Dev to Production - Part 4 - Image Scanning
Tags: dotnet, docker, csharp, github
Published: 2021-03-05 12:20:11

## Series Introduction
Welcome to Part 4 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a [**shift-left**](https://dev.to/peteking/shift-left-engineering-4fpp) mindset. We will use **Azure**, **Docker**, **GitHub**, **GitHub Actions** for CI/C-Deployment and Infrastructure as Code using **Pulumi**.

In this post we will be looking at:

* Add container scanning to our GitHub Action workflow

* Resolve security vulnerabilities

---

## TL;DR
We add the **Azure Container** Scan GitHub Action to our actions' workflow file, before pushing the container image to our registry. Add a group and user to our `Dockerfile` and use the `USER` command to specify the user for the process to run; ensuring our copied files can be accessed by this user. The security attack surface is reduced by running as non-root, and this is raised in CIS; the linting provided by **Dockle**. We resolved some slightly older CVE’s as examples; these can be raised by **Trivy** and can be allowed through or not based on configuration. Both Trivy and Dockle are wrapped by the Azure Container Scan GitHub Action.

---

## GitHub Repository

{% github https://github.com/peteking/Samples.WeatherForecast-Part-4 no-readme %}

---

## Introduction
In Part 3 we were able to get into GitHub Actions and build and push our Docker image to the GitHub Container registry. This is a great start, we are producing a Docker image in CI and it’s a consistent image, however, **what about vulnerabilities**?

---

## Requirements
We will be picking-up where we left off in [Part 3](https://dev.to/peteking/api-s-from-dev-to-production-part-3-7dn), which means you’ll need the end-result from [GitHub Repo - Part 3](https://github.com/peteking/Samples.WeatherForecast-Part-3) to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

---

## Vulnerabilities
A software vulnerability is a glitch, flaw, or weakness that is present in software or in an Operating System. CVE (**C**ommon **V**ulnerabilities and **E**xposures), contains a [list](https://cve.mitre.org/cve/) of records each containing an identification number, a description, and at least one public reference - for publicly known cybersecurity vulnerabilities.

CVE Records are used in numerous cybersecurity [products and services](https://cve.mitre.org/about/faqs.html#what_types_of_products_use_cve) from around the world, including the U.S. National Vulnerability Database [(NVD)](https://cve.mitre.org/about/cve_and_nvd_relationship.html).

There are many open source and commercial tools specifically designed for CI purposes; to be used before pushing a Docker image to a container registry. This can ensure any CVE’s of certain thresholds can be caught a build time.

> Commercial tools can also scan containers at runtime and apply custom mediating actions.

---

## CIS (Center for Internet Security)
CIS is a community of cybersecurity experts, who have developed CIS Benchmarks: more than 100 configuration guidelines across 25+ vendor product families to safeguard systems against today’s evolving cyber threats.

> CIS also produced a list of hardened images - https://www.cisecurity.org/cis-hardened-image-list/

---

## Why is this important?
It’s important because the software we design, build, and ship needs to be secure, we want to ensure safety of our end users' data and be responsible. We **must** take security seriously and both CVE and CIS provide a trusted source of information.

The open source community has done an amazing job of utilising the information and provided fantastic tooling we can all take advantage of.

---

## What can we do?
In terms of CVE’s, [AquaSec](https://www.aquasec.com) has created not only a great commercial product but an open source product called [Trivy](https://github.com/aquasecurity/trivy). This is a Docker image scanner that detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is easy to use, you can just install the binary and scan, all that’s required is to specify a target such as an image name of the container.

For CIS, there is an open source tool called [Dockle](https://github.com/goodwithtech/dockle) - Dockle is a Container Image Linter for Security, Helping build the Best-Practice Docker Image, which is ideal for CI.

---

## GitHub Actions to the rescue
**Microsoft** has create a GitHub Action which wraps both *Trivy* and *Dockle* called container-image-scan, you can find it on the GitHub Actions Marketplace and the open source repo, [Azure/container-scan](https://github.com/Azure/container-scan).

As of writing, the GitHub Container Registry does not scan images you push there, but other container registries do, such as the Azure Container Registry (ACR). It’s good that images will be scanned once they are in the registry, however, in terms of shifting-left, it is hugely beneficial to scan images during CI, if a vulnerability is found during build, engineers in the team are notified immediately, they can fix it themselves or consult with experts to help them. 

> This promotes a faster feedback loop and potentially stops images being pushed to a registry if that have vulnerabilities.

---

## Let’s use Azure Container Scan
In out workflow file we previously called, `build-and-publish.yml`, it’s located in the `.gihub/workflows` folder. We can add an extra step, we’ve given it a name, `Scan docker image`, specified the `azure/container-scan@v0` GitHub Action, and set the required and optional parameters. We’ve set the `severity-threshold: MEDIUM`, this is completely up to you, for this example, I’ve simply chosen MEDIUM. Please read the documentation on the Azure Container Scan repository and set it to what you believe is adequate for you specific situation.

```yaml
- name: Scan docker image
  uses: azure/container-scan@v0
  with:
    image-name: ${{ env.image-name }}
    severity-threshold: MEDIUM
    username: ${{ github.repository_owner }}
    password: ${{ secrets.GH_CR }}
```

### Full Workflow file
```yaml
name: Build and Publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

env:
  image-name: ghcr.io/peterjking/samples-weatherforecast-part-4:${{ github.sha }}

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2

      - name: Build docker image
        run: docker build . -t ${{ env.image-name }}

      - name: Scan docker image
        uses: azure/container-scan@v0
        with:
          image-name: ${{ env.image-name }}
          severity-threshold: MEDIUM
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Push docker image
        run: docker push ${{ env.image-name }}
```

> If you commit the changes, your GitHub Action will kick-off and build and scan.

> If you have not specified your repository secret GH_CR because you may be using a new repository perhaps for this, don’t forget to add the secret to your repo.

---

### Scan results
#### CVE
Depending on when you are running this and what the state of the base image will determine what you see. For me at the time of writing, I have no CVE’s, which is great! Only a week before writing this, I made a change for a similar workflow and had a HIGH CVE, it was all to do with OpenSSL in Alpine Linux. The screenshot is below for when it came up.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uiqt6k6257c9w46vfcco.png)

---

We can check what CVE this is with its number, a quick Google will reveal some interesting information.

For this particular issue, I found the following link: 
{% github https://github.com/docker-library/official-images/pull/9295 %}

> Whilst writing this blog series, this issue was open.
> Since then, the issue has been closed and merged :)

> It was essentially a simple update, OpenSSL needed to be updated in Alpine Linux to resolve this vulnerability.

Since we have zero *known* vulnerabilities at the moment, from another API, I did come across another issue, again, with Alpine but earlier in December 2020. The scan results are below as another example:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7izr1fp1e7lcw6fd6232.png)

---

In early December this came up, and this is because I was using the same base image we are using in this blog post. You should always investigate CVE and determine if they apply to you, and the reason why I’m showing you this is for that reason. Let’s go through it and see what it is.

Again, a quick Google on CVE-2020-28928, and I decide to settle on the information over at AquaSec:

https://avd.aquasec.com/nvd/cve-2020-28928/

You can see it is indeed MEDIUM, 5.5 severity, and when it was published, November 24th 2020.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s5cxwn6z13jygr9ffizf.png)

---

If you scroll down the page you’ll see a nice thoughtful list on mitigations. If you look even closer you can see the potential mitigation is essentially memory management. Luckily for us, we are using a language and framework which prides itself on being memory safe and good memory management (on our behalf of course). Given we are using C# in this case and we are not using C or C++ for instance and using string.h etc.  We can safely ignore this. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/83qn0o175i7ib7pauqd3.png)

---

How do we ignore this CVE you ask? Well, if you check the documentation on the Azure Container Scan repository, it shows you how…

Simply create a new folder in your `.github/workflows/` called `containerscan`, in there create a new YAML file called, `allowedlist.yaml`.

```yaml
general:
  vulnerabilities:
    - CVE-2020-28928
  bestPracticeViolations:
```

> **Good Practice**

> If you do this, you need to have a regular review going forward of what you are allowing through.

Given the noted good practice above, this is a *suburb example*, Alpine Linux resolved this issue. How did I know this? Well, looking at Alpine’s GitHub repo tells all in this GitHub issue: 

{% github https://github.com/alpinelinux/docker-alpine/issues/123 %}

Now, for my repo or a repo that had this vulnerability, I can safely remove it from the, `allowedlist.yaml` file.

#### CIS
In terms of CIS, we have a single 1 x WARN, and 2 x INFO.

Personally, I believe CIS-DI-0001 should really be higher, you **really should** in my view run your process as **non-root**. It’s easily solved too!

When the processes inside the container run as root, the same as the host machine, this give unprecedented privileges; we should always opt for least-privilege as a matter of practice. For example, if a user is able to gain access to the host as root 

By default, root in a container is the same root (`uid 0`) as the host machine. If a user succeeds in breaking out of an application running as root in a container, this malicious user may be able to gain access to the host with the same root user.

> **Good Practice**
> It is good practice to launch processes with a non-root user.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kqri3na8buxx9mt8vcr2.png)

> **TIP**

> Some official images already create a user for you to use, it’s always good to inspect the `Dockerfile` your using as your base to see.

In our case, a user is not created for us, so let’s fix that now, you can add the following code to your `Dockerfile`.

```dockerfile
RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet
```
Once we do this, when the process runs, it should be running as this, `dotnet` user. Therefore, we should also make sure when we `COPY` files we change the owner to the `dotnet` user using `chown` like so:

`COPY --chown=dotnet:dotnet --from=publish /out .`

If we do **not** do this, we run the risk of the `dotnet` user not being able to see the files and execute - *That would be bad for us*.

#### Full Dockerfile
```dockerfile
ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /app

# Copy and restore as distinct layers
COPY . .
WORKDIR /app/src/Samples.WeatherForecast.Api
RUN dotnet restore Samples.WeatherForecast.Api.csproj -r linux-musl-x64

FROM build AS publish
RUN dotnet publish \
    -c Release \
    -o /out \
    -r linux-musl-x64 \
    --self-contained=true \
    --no-restore \
    -p:PublishReadyToRun=true \
    -p:PublishTrimmed=true

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}

RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

WORKDIR /app
COPY --chown=dotnet:dotnetgroup --from=publish /out .

USER dotnet
EXPOSE 8080
ENV ASPNETCORE_URLS=http://+:8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]
```

---

Now you have a slightly refactored `Dockerfile`, let’s give this a go locally to make sure all is well.

Build the image locally:

`docker build -t samples-weatherforecast:v4 .`

Run the image locally:

`docker run -it --rm -p 8080:8080 samples-weatherforecast:v4`

Test the image in Postman or REST Client - https://marketplace.visualstudio.com/items?itemName=humao.rest-client

Once you’ve verified that it works, it’s now time to commit those changes. This should kick-off the GitHub Action and build, scan and push the Docker image container registry.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cqh3rw0ibkvt5njm65gk.png)

---

We can verify our API is running under our dotnet user but getting inside the container.

How you ask? Well, there are two ways, the **CLI** way or you can use **Docker Desktop**.

> I’ll *show off* the Docker Desktop feature.

*Open* **Docker Desktop** → *Move* you mouse of your running container → *Click*, **CLI**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qb8qudvib85watjbw7d8.png)

---

A command prompt should open and you should be inside your container!

Now, *execute*, `whoami` and it should come back saying, `dotnet`; this is your `USER`.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/irlazehy0co4h3fl88lt.png)

---

You can also execute the `top` command, let’s do this too:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oeb58v67l2c2fljby01g.png)

> Cool! All running as `dotnet`.

---

#### Build results
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aasv1mvml16ixe9n12aa.png)

---

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vuu0kznx24clye7eq32w.png)

---

> Packages → Container Registry
> Let’s double check that our image was pushed by checking our GitHub Packages area.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pp4vptpq9sfzi637l4sc.png)

---

***Success!***

In addition, I have also pulled this image down and tested it too just like we did in [API's From Dev to Production - Part 3](https://dev.to/peteking/api-s-from-dev-to-production-part-3-7dn)

---

## What have we learned?
We have learned how to scan our container images before we end up pushing them to our container registry. We understand how important this is and why we should do it, the open source tools that are available to you. We have also learned how to decrease our security attack surface by running our process as a user by adding a group and user in our Dockerfile.

There are many open source tools that are very capable, however, at times you may wish to opt for a paid product.

Here is a short list:
* [Snyk](https://snyk.io/)
* [Acqua](https://www.aquasec.com/)
* [Prisma Cloud](https://www.paloaltonetworks.com/prisma/cloud) (formally Twistlock)
* [Anchore](https://anchore.com/)
* ...and many more, don't forget some container registries also scan images.

---

## Up next
[Part 5](https://dev.to/peteking/api-s-from-dev-to-production-part-5-26cp) in this series will be about **CIS Issues**:

* Add a healthcheck to the Dockerfile

* Add an item to the CIS allowedlist.yaml

---

## More information
* https://www.azure.com
* https://dotnet.microsoft.com/
* https://www.github.com
* https://www.docker.com
* https://www.pulumi.com