On-demand running of tests via the CI pipeline
Tags: buildkite, ci, devops, playwright
Published: 2023-07-24 13:51:24

In all the CI/CD tools I've used (Jenkins, TeamCity, Bitbucket Pipelines, Gitlab), I would say Buildkite is my favourite so far 👍. Being a fan of the sayings _"right tool for the right problem"_ and _"it doesn't matter what tool you use"_, it still helps that your DevOps tool offer flexibility with a cleaner and minimalist interface unlike the very cluttered Gitlab 🫣.

In one of its flexible feature, it allows the user with a conditional modal before the pipeline proceeds to the next steps. They call it [block step](https://buildkite.com/docs/pipelines/block-step) 🧱 in Buildkite.

Not all our automated tests are embedded into our build pipelines. We usually enable happy path smoke tests to make our micro-service pipelines run faster. Hence, we allow users to run ad-hoc/on-demand tests based on their preferences.

We created a standalone test pipeline called **"tests"**
![ad-hoc tests](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vtqsr66do0qpfy5tgvd5.png)

We set the CI variables / steps with default values then call the `pipeline.yml` in our code repository:
```yaml
  STAGING_BASE_URL: https://staging.mydomain.net   
  STAGING_PASSWORD: def456
  TEST_BASE_URL: https://test.mydomain.net 
  TEST_PASSWORD_TEST: abc123
  SPECS: tests/**/*.spec.ts # will run the whole test suite
  BUILDKITE_PARALLEL_RUN: 1 # default to 1 but can be overriden
  ENVIRONMENT: test

steps:
  - label: ':pipeline: Initiate pipeline'
    command: buildkite-agent pipeline upload .buildkite/pipeline.yml
```

In our code repository, we have two files, the `.buildkite/pipeline.yml` which triggers the block modal:
```yaml
steps:
  - block: ':playwright: Run tests on demand'
    # branches: '!master'
    fields:
      - select: ':earth_americas: Select environment'
        key: 'environment'
        default: 'test'
        options:
          - label: 'test'
            value: 'test'
          - label: 'staging'
            value: 'staging'
      - text: ':test_tube: Test spec path / file (e.g. tests/api/**/*.ts)'
        default: 'tests/e2e/**/*.ts'
        key: 'specs'
      - select: ':man-running: Parallel runs'
        key: 'parallelism'
        default: '3'
        options:
          - label: '1'
            value: '1'
          - label: '2'
            value: '2'
          - label: '3'
            value: '3'
  - label: ':buildkite: Trigger tests'
    commands:
      - chmod 777 .buildkite/pipeline-adhoc.sh
      - .buildkite/pipeline-adhoc.sh | buildkite-agent pipeline upload
```
... and the `.buildkite/pipeline-adhoc.sh`, which captures the entered values in the block modal and also allows parameterising the `parallelism` option in Buildkite as the integer value (currently defaulted to **1**) can only be updated via a script:
```bash
#!/bin/bash

set -e

ENVIRONMENT=$(buildkite-agent meta-data get "environment")
SPECS=$(buildkite-agent meta-data get "specs")
PARALLEL_RUN=$(buildkite-agent meta-data get "parallelism")

if [ "${ENVIRONMENT}" = "test" ]; then
  PASSWORD=${TEST_PASSWORD}
  BASE_URL=${TEST_BASE_URL}
else
  PASSWORD=${STAGING_PASSWORD}
  BASE_URL=${STAGING_BASE_URL}
fi

cat <<YAML
env:
  ENVIRONMENT: "${ENVIRONMENT}"
  SPECS: "${SPECS}"
  PASSWORD: "${PASSWORD}"
  BASE_URL: "${BASE_URL}"

steps:
  - label: ':playwright: Run tests'
    if: build.env('SPECS') != null # some components might not have spec files specified so need to skip running tests
    commands:
    - 'npx playwright test'
    parallelism: ${PARALLEL_RUN} # this is defaulted to > 1 so need to set PARALLEL_RUN as a CI variable for repo's with only 1 test spec file so CI doesn't fail with 'no spec files' error
    retry:
      manual: 
        allowed: false
        reason: 'Integration with other platforms means retries should not be performed for tests.'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'test-results/junit-*.xml'
          format: 'junit'
    artifact_paths:
      - 'results/**/*'
YAML
```

Once all these are set up, the pipeline can be triggered anytime and will stop on the step with **block**
![pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80e2j8wecczknz92ycm9.png)

Clicking the "Run tests on demand" button would take you to the modal where you can choose your own adventure 🤠 then clicking "Continue" button on the modal would trigger the tests based on your selections ✅
![block](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/clq0473d5icj5vfofnmm.png)