Smoke testing in CI/CD pipelines
Tags: cicd, testing, circleci
Published: 2021-10-21 00:26:12

---
title: Smoke testing in CI/CD pipelines
published: true
date: 2021-10-18 17:00:00 UTC
tags: cicd,testing,circleci
cover_image: https://production-cci-com.imgix.net/blog/media/Tutorial-Advanced-C.jpg?ixlib=rb-3.2.1&auto=format&fit=max&q=60&ch=DPR%2CWidth%2CViewport-Width%2CSave-Data&w=750
canonical_url: https://circleci.com/blog/smoke-tests-in-cicd-pipelines/
---

Here’s a common situation that plagues many development teams. You run an application through your CI/CD pipeline and all of the tests pass, which is great. But when you deploy it to a live target environment the application just does not function as expected. You can’t always predict what will happen when your application is pushed live. The solution? Smoke tests are designed to reveal these types of failures early by running test cases that cover the critical components and functionality of the application. They also ensure that the application will function as expected in a deployed scenario. When implemented, smoke tests are often executed on every application build to verify that basic but critical functionality passes before jumping into more extensive and time-consuming testing. Smoke tests help create the fast feedback loops that are vital to the software development life cycle.

In this post I’ll demonstrate how to add smoke testing to the deployment stage of a CI/CD pipeline. The smoke testing will test simple aspects of the application post deployment.

## Technologies used for smoke testing

This post will reference the following technologies:

- [GitHub](https://github.com)
- [CircleCI](https://circleci.com/)
- [Docker](https://www.docker.com/)
- [Kubernetes](https://kubernetes.io/)
- [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/)
- [Bash](https://www.gnu.org/software/bash/manual/html_node/What-is-Bash_003f.html)
- [smoke.sh - open source smoke testing framework by asm89](https://github.com/asm89/smoke.sh)
- [Pulumi](https://www.pulumi.com/)

## Prerequisites

This post relies on configurations and code that are featured in my previous post [Automate releases from your pipelines using Infrastructure as Code](https://dev.to/ronpowelljr/automate-releases-from-your-pipelines-using-infrastructure-as-code-23b1-temp-slug-1721005). The full source code can be found in [this repo](https://github.com/datapunkz/orb-pulumi-gcp).

## Getting the most from smoke tests

Smoke tests are great for exposing unexpected build errors, connection errors, and for validating a server’s expected response after a new release is deployed to a target environment. For example, a quick, simple smoke test could validate that an application is accessible and is responding with expected response codes like `OK 200`, `300`, `301`, `404`, etc. The examples in this post will test that the deployed app responds with an `OK 200` server code and will also validate that the default page content renders the expected text.

## Running CI/CD pipelines without smoke tests

Let’s take a look at an example pipeline config that is designed to run unit tests, build, and push a Docker image to Docker Hub. The pipeline also uses [infrastructure as code](https://dev.to/circleci/an-intro-to-infrastructure-as-code-104d-temp-slug-9937351) ([Pulumi](https://www.pulumi.com/)) to provision a new Google Kubernetes Engine (GKE) cluster and to deploy this release to the cluster. This pipeline config example does not implement smoke tests. Please be aware that if you run this specific pipeline example, a new GKE cluster will be created and will live on until you manually run the `pulumi destroy` command to terminate all the infrastructure it created.

**Caution:** _Not terminating the infrastructure will result in unexpected costs._

```
version: 2.1
orbs:
  pulumi: pulumi/pulumi@2.0.0
jobs:
  build_test:
    docker:
      - image: cimg/python:3.8.1
        environment:
          PIPENV_VENV_IN_PROJECT: 'true'
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            pipenv install --skip-lock
      - run:
          name: Run Tests
          command: |
            pipenv run pytest
  build_push_image:
    docker:
      - image: cimg/python:3.8.1
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and push Docker image
          command: |       
            pipenv install --skip-lock
            pipenv run pip install --upgrade 'setuptools<45.0.0'
            pipenv run pyinstaller -F hello_world.py
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-pulumi-gcp' >> $BASH_ENV
            source $BASH_ENV
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME
  deploy_to_gcp:
    docker:
      - image: cimg/python:3.8.1
        environment:
          CLOUDSDK_PYTHON: '/usr/bin/python2.7'
          GOOGLE_SDK_PATH: '~/google-cloud-sdk/'
    steps:
      - checkout
      - pulumi/login:
          version: "2.0.0"
          access-token: ${PULUMI_ACCESS_TOKEN}
      - run:
          name: Install dependencies
          command: |
            cd ~/
            pip install --user -r project/requirements.txt
            curl -o gcp-cli.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz
            tar -xzvf gcp-cli.tar.gz
            echo ${GOOGLE_CLOUD_KEYS} | base64 --decode --ignore-garbage > ${HOME}/project/pulumi/gcp/gke/cicd_demo_gcp_creds.json
            ./google-cloud-sdk/install.sh --quiet
            echo 'export PATH=$PATH:~/google-cloud-sdk/bin' >> $BASH_ENV
            source $BASH_ENV
            gcloud auth activate-service-account --key-file ${HOME}/project/pulumi/gcp/gke/cicd_demo_gcp_creds.json
      - pulumi/update:
          stack: k8s
          working_directory: ${HOME}/project/pulumi/gcp/gke/
workflows:
  build_test_deploy:
    jobs:
      - build_test
      - build_push_image
      - deploy_to_gcp:
          requires:
          - build_test
          - build_push_image

```

This pipeline deploys the new app release to a new GKE cluster, but we do not know if the application is actually up and running after all of this automation completes. How do we find out whether the application has been deployed and is functioning properly in this new GKE cluster? Smoke tests are a great way to quickly and easily validate the application’s status after deployment.

## How do I write a smoke test?

The first step is to develop test cases that define the steps required to validate an application’s functionality. Identify the functionality that you want to validate, and then create scenarios to test it. In this tutorial, I’m intentionally describing a very minimal scope for testing. For our sample project, my biggest concern is validating that the application is accessible after deployment and that the default page that is served renders the expected static text.

I prefer to outline and list the items I want to test because it suits my style of development. The outline shows the factors I considered when developing the smoke tests for this app. Here is an example of how I developed test cases for this smoke test:

- What language/test framework? 
  - Bash
  - smoke.sh
- When should this test be executed? 
  - After the GKE cluster has been created
- What will be tested? 
  - Test: Is the application accessible after it is deployed? 
    - Expected Result: Server responds with code `200`
  - Test: Does the default page render the text “Welcome to CI/CD” 
    - Expected Result: `TRUE`
  - Test: Does the default page render the text “Version Number: “ 
    - Expected Results: `TRUE`
- Post test actions (must occur regardless of pass or fail) 
  - Write test results to standard output
  - Destroy the GKE cluster and related infrastructure 
    - Run `pulumi destroy`

My test case outline (also called a test script) is complete for this tutorial and clearly shows what I’m interested in testing. For this post, I will write smoke tests using a bash-based, open source smoke test framework called [`smoke.sh`](https://github.com/asm89/smoke.sh) by [asm89](https://github.com/asm89). For your own projects, you can write smoke tests in what ever language or framework you prefer. I picked `smoke.sh` because it’s an easy framework to implement and it’s open source. Now let’s explore how to express this test script using the `smoke.sh` framework.

## Create smoke test using smoke.sh

The `smoke.sh` framework’s [documentation](https://github.com/asm89/smoke.sh#smokesh) describes how to use it. The next block of sample code shows how I used the `smoke_test` file found in the `test/` directory of the [example code’s repo](https://github.com/datapunkz/orb-pulumi-gcp).

```
#!/bin/bash

. tests/smoke.sh

TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="k8s"
PULUMI_CWD="pulumi/gcp/gke/"
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"

while true
do
  STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
  if [$STATUS -eq 200]; then
    smoke_url_ok $SMOKE_URL
    smoke_assert_body "Welcome to CI/CD"
    smoke_assert_body "Version Number:"
    smoke_report
    echo "\n\n"
    echo 'Smoke Tests Successfully Completed.'
    echo 'Terminating the Kubernetes Cluster in 300 second...'
    sleep 300
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    break
  elif [[$TIME_OUT_COUNT -gt $TIME_OUT]]; then
    echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    exit 1
  else
    echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
    TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
  fi
  sleep 10
done

```

Next, I’ll explain what’s going on in this smoke\_test file.

### Line by line description of the smoke\_test file

Let’s start at the top of the file.

```
#!/bin/bash

. tests/smoke.sh

```

This snippet specifies the Bash binary to use and also specifies the file path to the core `smoke.sh` framework to import/include in the `smoke_test` script.

```
TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="k8s"
PULUMI_CWD="pulumi/gcp/gke/"
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"

```

This snippet defines environment variables that will be used throughout the `smoke_test` script. Here is a list of each environment variable and its purpose:

- `PULUMI_STACK="k8s"` is used by Pulumi to specify the Pulumi app stack.
- `PULUMI_CWD="pulumi/gcp/gke/"` is the path to the Pulumi infrastructure code.
- `SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)` is the Pulumi command used to retrieve the public IP address of the application on the GKE cluster. This variable is referenced throughout the script.
- `SMOKE_URL="http://$SMOKE_IP"` specifies the url endpoint of the application on the GKE cluster.

```
while true
do
  STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
  if [$STATUS -eq 200]; then
    smoke_url_ok $SMOKE_URL
    smoke_assert_body "Welcome to CI/CD"
    smoke_assert_body "Version Number:"
    smoke_report
    echo "\n\n"
    echo 'Smoke Tests Successfully Completed.'
    echo 'Terminating the Kubernetes Cluster in 300 second...'
    sleep 300
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    break
  elif [[$TIME_OUT_COUNT -gt $TIME_OUT]]; then
    echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    exit 1
  else
    echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
    TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
  fi
  sleep 10
done

```

This snippet is where all the magic happens. It’s a `while` loop that executes until a condition is true or the script exits. In this case, the loop uses a `curl` command to test if the application returns an `OK 200` response code. Because this pipeline is creating a brand new GKE cluster from scratch, there are transactions in the Google Cloud Platform that need to be complete before we begin smoke testing.

- The GKE cluster and application service must be up and running.
- The `$STATUS` variable is populated with the results of the curl requests then tested for the value of `200`. Otherwise, the loop increments the `$TIME_OUT_COUNT` variable by 10 seconds, then waits for 10 seconds to repeat the `curl` request until the application is responding.
- Once the cluster and app are up, running, and responding, the `STATUS` variable will produce a `200` response code and the remainder of the tests will proceed.

The `smoke_assert_body "Welcome to CI/CD"` and `smoke_assert_body "Version Number: "` statements are where I test that the welcome and version number texts are being rendered on the webpage being called. If the result is false, the test will fail, which will cause the pipeline to fail. If the result is true, then the application will return a `200` response code and our text tests will result in `TRUE`. Our smoke test will pass and execute the `pulumi destroy` command that terminates all of the infrastructure created for this test case. Since there is no further need for this cluster, it will terminate all the infrastructure created in this test.

This loop also has an `elif` (else if) statement that checks to see if the application has exceeded the `$TIME_OUT` value. The `elif` statement is an example of [exception handling](https://en.wikipedia.org/wiki/Exception_handling) which enables us to control what happens when unexpected results occur. If the `$TIME_OUT_COUNT` value exceeds the `TIME_OUT` value, then the `pulumi destroy` command is executed and terminates the newly created infrastructure. The `exit 1` command then fails your pipeline build process. Regardless of test results, the GKE cluster will be terminated because there really isn’t a need for this infrastructure to exist outside of testing.

## Adding smoke tests to pipelines

I’ve explained the smoke test example and my process for developing the test case. Now it’s time to integrate it into the CI/CD pipeline configuration above. We’ll add a new `run` step below the `pulumi/update` step of the `deploy_to_gcp` job:

```
      ...
      - run:
          name: Run Smoke Test against GKE
          command: |
            echo 'Initializing Smoke Tests on the GKE Cluster'
            ./tests/smoke_test
            echo "GKE Cluster Tested & Destroyed"
      ...

```

This snippet demonstrates how to integrate and execute the `smoke_test` script into an existing CI/CD pipeline. Adding this new run block ensures that every pipeline build will test the application on a live GKE cluster and provide a validation that the application passed all test cases. You can be confident that the specific release will perform nominally when deployed to the tested target environment which in this case, is a Google Kubernetes cluster.

## Wrapping up

In summary, I’ve discussed and demonstrated the advantages of using smoke tests and Infrastructure as Code in CI/CD pipelines to test builds in their target deployment environments. Testing an application in its target environment provides valuable insight into how it will behave when it’s deployed. Integrating smoke testing into CI/CD pipelines adds another layer of confidence in application builds.

If you have any questions, comments, or feedback please feel free to ping me on Twitter [@punkdata](https://twitter.com/punkdata).

Thanks for reading!