Automating your CloudQuery Policies with CircleCI
Tags: devops, circleci, tutorial, security
Published: 2022-03-01 10:48:20

This blog is going to cover running CloudQuery in CircleCI as part of your continuous integration or continuous delivery pipeline.

We all know how awesome a tool CloudQuery is, with a bit of configuration you very quickly get access to the `cloudquery fetch` command to get information on your cloud assets, and then you can start running `cloudquery policy run <policy>` against that to get amazing guidance on your cloud infrastructure. But what about automating the process, or making it accessible to the rest of your team?

You could use our [helm chart](https://docs.cloudquery.io/docs/deployment/helm-chart) to deploy a dedicated and persistent version of CloudQuery, or if you want to experiment with you could add this to your CI/CD processes. I personally really like using CircleCI and have been an active user since about 2017, so for this example, we’ll be creating a CircleCI workflow (and [template](https://github.com/ukmadlz/cloudquery-circleci-template)) to achieve this.

## What is CI/CD?

CI, or continuous integration, in its simplest form is the means of continually adding and integrating code into a shared body of code. In Git terms, this is the creation of changes in a branch, followed by the testing, and then merging into the main branch.

CD, or continuous delivery, is the methodology by which you can deploy your codebase at any time. This means making sure that the codebase is tested, and has an automated release process, rather than manually grabbing the code and loading it to a location.

## How to add CloudQuery to CircleCI?

### Prerequisites

Make sure you’ve followed one of our [Getting Started Guides](https://docs.cloudquery.io/docs/getting-started/getting-started-with-aws) to make sure you have a valid and functional `config.hcl` for CloudQuery to use.

### Getting started with CircleCI

If you’ve never used CircleCI before, rather than walk through all the steps here I’d recommend reading through the [Your First Green Build](https://circleci.com/docs/2.0/getting-started/) guide in the CircleCI documentation. This guide will take you through the creation of your `.circleci/config.yml` and what the core parts are.

Once you’ve got that started, here is how to add CloudQuery to your CircleCI flow.

The first step is to add to the list of `jobs`, each of these jobs is a distinct task and process to complete. Here is the job to fetch data to a Postgres instance:

```yaml
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:
  build:
    # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/2.0/configuration-reference/#docker-machine-macos-windows-executor
    docker:
      - image: cimg/base:stable
        environment:
          CQ_DSN: postgresql://postgres:pass@localhost/postgres?sslmode=disable
      - image: postgres:11
        environment:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: pass
          POSTGRES_DB: postgres
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - checkout
      - run:
          name: "Install CloudQuery"
          command: curl -L https://github.com/cloudquery/cloudquery/releases/latest/download/cloudquery_linux_x86_64 -o cloudquery && chmod a+x cloudquery
      #  Wait for Postgres to be ready before proceeding
      - run:
          name: "Waiting for Postgres to be ready"
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run: 
          name: "Fetch AWS Resources"
          command: ./cloudquery fetch
```

When you have a new job, you need to set the container that will execute the task and any supporting containers. In this case, we have an image of `cimg/base:stable` which is a basic Linux image to execute tasks as our primary container with a supporting `postgres:11` image.

We also have to provide the `CQ_DSN` variable, following this [ENV substitution guide](https://docs.cloudquery.io/docs/configuration/overview#environment-variable-substitution), to point to the Postgres instance within the CircleCI job.

Next, we have the `steps`, these are the commands executed on the primary container to add, configure, and perform tasks. The first task is usually `checkout`, this command will clone your codebase from your git repository to run against.

The following step is `Install CloudQuery`, this follows the [Download and Install](https://docs.cloudquery.io/docs/getting-started/getting-started-with-aws#download-and-install) process of adding CloudQuery to a Linux instance and configuring it to be executable.

```yaml
      - run:
          name: "Install CloudQuery"
          command: curl -L https://github.com/cloudquery/cloudquery/releases/latest/download/cloudquery_linux_x86_64 -o cloudquery && chmod a+x cloudquery
```

Due to the way CircleCI parallelises its container, it does take a moment for the Postgres image to be ready so we do have a wait step.

```yaml
      - run:
          name: "Waiting for Postgres to be ready"
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
```

This step is using `dockerize` to make sure the Postgres image is available to use before executing any commands against it.

And finally, we run CloudQuery itself.

```yaml
      - run: 
          name: "Fetch Resources"
          command: ./cloudquery fetch
```

Using `cloudquery fetch` the container either uses the `.cq` information if committed to the repository or downloads a fresh version of the providers configured and grabs the data from your provider to use.

### Configuring CircleCI

Now for this to work, you’ll need to add the access keys etc in CircleCI. It’s usually best to do this on a per-project basis. Within CircleCI go to projects, then select the project in question:

![Select a Project within CircleCI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/43d4odhxn5vi6wiuh5b4.png)

From here navigate to `Project Settings`:

![Navigate to Project Settings](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kolu07j2zrinog9x2vmc.png)

From here select `Environment Variables`, and from here you can add your ENV variables as a key-value pair:

![Add relevant ENV variables to the CircleCI project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3kylqjs1xpt5cis9v6uw.png)

To make this work, you now need to add a `workflows` reference. This is how you chain jobs together to accomplish tasks:

```yaml
# Invoke jobs via workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  fetch-workflow:
    jobs:
      - build
```

This workflow `fetch-workflow` will now execute on every branch pushed to your git repository. But once the `fetch` is complete the data is lost as the Postgres image is destroyed after use, which isn’t much use for sharing the information gathered from the provider.

## Adding a policy check

Once you have added a policy from the [CloudQuery Hub](https://hub.cloudquery.io/policies) or written your [Custom Policy](https://docs.cloudquery.io/docs/tutorials/policies/policies-overview), you’ll want to run this nightly or after deployment and to this we can extend the example above quickly:

```yaml
jobs:
  build:
    […]
  policy:
    # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/2.0/configuration-reference/#docker-machine-macos-windows-executor
    docker:
      - image: cimg/base:stable
        environment:
          CQ_DSN: postgresql://postgres:pass@localhost/postgres?sslmode=disable
      - image: postgres:11
        environment:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: pass
          POSTGRES_DB: postgres
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - checkout
      - run:
          name: "Install CloudQuery"
          command: curl -L https://github.com/cloudquery/cloudquery/releases/latest/download/cloudquery_linux_x86_64 -o cloudquery && chmod a+x cloudquery
      - run:
          name: "Waiting for Postgres to be ready"
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run: 
          name: "Fetch Resources"
          command: ./cloudquery fetch
      - run:
          name: "Check AWS Policy"
          command: ./cloudquery policy run aws --output-dir ./
      - store_artifacts:
          path: ./aws.json
          destination: aws-policy
```

Our new `policy` job is the same as the previous `build` job except it has two additional steps. The first of which is `Check AWS Policy` step, this is running the entire [AWS Policy](https://hub.cloudquery.io/policies/cloudquery/aws/latest) set against the gathered data.

```yaml
      - run:
          name: "Check AWS Policy"
          command: ./cloudquery policy run aws --output-dir ./
```

And because of the use of the `--output-dir` flag, it’s creating an `aws.json` for the results of the policy checks.

The final step is storing the `aws.json` as an artifact so that after the execution of the check, and the subsequent destruction of the image you can still review the results of the policy check:

```yaml
      - store_artifacts:
          path: ./aws.json
          destination: aws-policy
```

Early we mentioned running this as a nightly task, CircleCI workflows have a `trigger` for this:

```yaml
# Invoke jobs via workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  fetch-workflow:
    […]
  policy-check:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - main
    jobs:
      - build
      - policy
```

Once you add the `schedule` to the `triggers` you can follow the `crontab` standard to decide when it’s executed, and any other `filters` you may want to limit this execution by. This is a legacy method of automating within CircleCI, for a more up-to-date approach that requires a little more configuration I’d use [Scheduled Pipelines](https://circleci.com/blog/using-scheduled-pipelines/).

## Summary

This blog should have covered all the aspects of using CloudQuery with CircleCI to automate your policy checks, and with that make it shareable with your teammates. If you’d like to discuss other ways to use CloudQuery in CI you can join our [Discord](https://cloudquery.io/discord) and talk to us about your needs. And if we’ve missed anything we’d happily follow up this blog with more details, or even other CI/CD providers.