This is how you can test your cfn-guard rules
Tags: cfnguard, testing, aws
Published: 2022-12-16 08:32:23

In my previous blog, [How do you prove that your infrastructure is compliant](https://binx.io/blog/2021/11/23/how-do-you-prove-that-your-infrastructure-is-compliant/).  
I explained how you can prove your infrastructure is compliant using [CloudFormation Guard](https://github.com/aws-cloudformation/cloudformation-guard).
But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository [CloudFormation Guard](https://github.com/aws-cloudformation/cloudformation-guard).  
You will notice that the project itself offers a testing framework.
Alright! Let’s build a ruleset and write some tests for it!

Let’s build a ruleset for S3
----------------------------

We will start with a sample for S3, create a file called: `rules/s3.guard`.

```
let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]

rule BucketEncryption when %buckets !empty {
  %buckets.Properties {
    BucketEncryption.ServerSideEncryptionConfiguration[*] {
      ServerSideEncryptionByDefault.SSEAlgorithm IN ["AES-256", "aws:kms"] <<Ensure all S3 buckets use encryption-at-rest>>
    }
  }
}
```

Now we will write the test, called: `rules/s3_tests.yaml`.

```yaml
---
- name: "Skip if not present"
  input:
     Resources: {}
  expectations:
    rules:
      BucketEncryption: SKIP
- name: "No encryption used"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
  expectations:
    rules:
      BucketEncryption: FAIL

- name: "Bucket with KMS Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: aws:kms
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with AES-256 Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: AES-256
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with UNKNOWN Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: UNKNOWN
  expectations:
    rules:
      BucketEncryption: FAIL
```

Now it’s time to run the tests:

```bash
cfn-guard test --rules-file rules/s3.guard --test-data rules/s3_tests.yaml
```

This will give the following output:

```txt
Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL
```
You might be thinking, that looks nice! But the output is quite verbose, and you need to execute a command per rule/test set.  
It gets even worse when a test does fail. For this example I altered the `rules/s3.guard` file to use `aws:kmz` instead of `aws:kms`.  

Then it looks like this:

```txt
Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  FAILED Rules:
    BucketEncryption: Expected = PASS, Evaluated = FAIL

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL
```
If you look at `Test Case #3` it says: `BucketEncryption: Expected = PASS, Evaluated = FAIL`. That makes it hard to spot what test failed.

cfn-guard-test
--------------

For this reason I created a python package called [cfn-guard-test](https://github.com/Nr18/cfn-guard-test). This package  
makes it easier to run many rule/test sets. This is especially useful in the CI/CD pipelines.
 
> Please read the repository for the installation instructions.

Let’s run the tests again!

```bash
cfn-guard-test --verbose
```

The output now looks something like:

```txt
===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 5
Failed 0
```

That is a nice and clean overview. It lists the number of passed and failed tests. For the next example I altered the  

`rules/s3.guard` file to use `aws:kmz` instead of `aws:kms`. This will again trigger a failure:

```txt
===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 4
Failed 1

Rule BucketEncryption failed on #3 "Bucket with KMS Encryption" in rules/s3.guard
```

Well you have to agree. That immediately sums up how many passed and failed. And more it shows that the `BucketEncryption`  
rule fails. And it failed on the test case "Bucket with KMS Encryption".
You don’t want to spend time figuring out what is wrong. This is why we use tooling, to make our life easier.
This tool also has the ability to generate JUnit report.

```bash
cfn-guard-test --verbose \
  --junit-path "reports/cfn-guard.xml"
```

Conclusion
----------

[cfn-guard-test](https://github.com/Nr18/cfn-guard-test) can help you run many tests. And get an aggregated report of  
the results. This tool will help you detect bugs in your rules easier and quicker.