Automate tagging of vulnerable Docker images in ECR
Tags: aws, cloud, ecr, containers
Published: 2022-06-19 17:12:44

Amazon Elastic Container Registry is a fully-managed Docker container registry. It makes it easy for developers to store and manage Docker images inside their AWS environment. 
ECR supports two types of image scanning. Enhanced image scanning requires an integration with Amazon Inspector. It will scan your repositories continuously. Basic image scanning will use the Common Vulnerabilities and Exposures (CVEs) database ([open-source Clair](https://github.com/quay/clair)) to find vulnerabilities in your images. You can trigger scans on image push or manually. 

Although it's a very useful feature to find vulnerabilities in your images, it's still up to you to do something with this information.
By using Amazon Eventbridge, you can configure some action in response to the result of a scan.

In this case we will automatically tag images which contain `CRITICAL` vulnerabilities with a vulnerable prefix. Images with this prefix will be removed from your repository in 5 days. You can not give the same tag to multiple images in one repository. That's why we will use prefixes.

![Architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a1tp4zka2fhbooo9zbhp.png)
 

The Eventbridge rule (*) to create this condition:

```yaml
EventTrigger:
  Type: "AWS::Events::Rule"
  Properties:
    Description: Trigger Lambda if a scan contains critical vulnerabilities
    EventPattern:
      source:
        - "aws.ecr" 
      detail-type:
        - "ECR Image Scan" 
      detail:
        finding-severity-counts:
          CRITICAL:
            - exists: true
            - numeric: [ ">", 0 ]
    State:  "ENABLED"
    Targets:
    - Arn: !GetAtt LambdaFunction.Arn
      Id: scan-ecr
```
_(*) I was not able to make this working with an "OR" condition to tag the image when it contains critical "OR" high vulnerabilities. If you need an OR condition you can create multiple rules (with the risk to trigger your Lambda twice when you have both high and critical vulnerabilities). Another solution could be to forward all scan events and do the filtering inside the Lambda._

As you can see we've configured a `LambdaFunction` as target of our rule. To make this work we still need to provide the permission for our Eventbridge rule to trigger the Lambda.

```yaml
PermissionForEventsToInvokeLambda: 
  Type: "AWS::Lambda::Permission"
  Properties: 
    FunctionName: !GetAtt LambdaFunction.Arn
    Action: "lambda:InvokeFunction"
    Principal: "events.amazonaws.com"
    SourceArn: !GetAtt EventTrigger.Arn
```

The Lambda will add the `vulnerable` tag prefix by using the ecr client its `put_image` function. By using this function we can add a tag to the image without the need to pull or push the Docker image. 

```Python
def add_image_tag(repo, manifest, digest):
    """Add 'vulnerable' tag prefix to image."""
    try:
        response = client.put_image(
            repositoryName=repo,
            imageManifest=manifest,
            imageTag="vulnerable-" + digest[7:],
            imageDigest=digest,
        )
    except ClientError as error:
        logging.error("Unexpected error")
        raise error

    LOGGER.info("Image tag 'vulnerable' added!")
    return response
```

As you can see, we only assign the following policies to our Lambda role.

- "ecr:BatchGetImage"
- "ecr:PutImage"

We don't need to assign any additional policies like `ecr:InitiateLayerUpload` or `ecr:UploadLayerPart` which are required when you really want to push a Docker image.

To test our setup we can pull, tag and push an image to our registry.

```shell
$ docker pull node:11-stretch-slim
$ docker tag node:11-stretch-slim 123456789012.dkr.ecr.eu-west-1.amazonaws.com/node:11-stretch-slim
$ docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/node:11-stretch-slim
```

Just after the push, the scan is still running.
![Before ECR scan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnta2cnnmcp37r7gx6mx.png)
 
After the scan the image is tagged with a vulnerable prefix![After ECR scan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pa5l9wkzbamcgjs2d1nl.png)

We can push some additional images to test the implementation
![Pushed more images](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fg39km0d01s2fjhvm331.png)

The solution is working! Still there are some "missing" features. As already mentioned, the `OR` Condition in the Eventbridge rule is [not working](https://github.com/aws/containers-roadmap/issues/565). So we're currently only filtering for `CRITICAL` images. Next it would be nice if we could use Docker image tag prefixes in IAM policies, so we can deny that vulnerable images are being pulled. In my solution I'm removing images with this tag prefix after 5 days.

```json
{
  "rulePriority": 1,
  "description": "Remove vulnerable images after 5 days",
  "selection": {
    "tagStatus": "tagged",
    "tagPrefixList": ["vulnerable"],
    "countType": "sinceImagePushed",
    "countUnit": "days",
    "countNumber": 5
  },
  "action": { "type": "expire" }
}
```

That being said, I hope you enjoyed reading this post and that you will start thinking about vulnerabilities in Docker images! The full solution is available on [my GitHub](https://github.com/lvthillo/ecr-vulnerable-image-tagger).