rptf 0.1.12
End-to-end pipeline test framework for GitLab
Requires Python: >=3.6

![ICON](https://assets.gitlab-static.net/uploads/-/system/project/avatar/22927059/icon-snek.png)

# Remote Pipeline Test Framework

End-to-end test framework for GitLab pipeline template authors.

Trigger pipelines, make assertions.

## Why

As a GitLab pipeline author, your pipeline and jobs will be used across several projects under varying configurations. Your pipeline template understands the context (branches, tags, vars, source files etc.) and responds accordingly.

The resultant pipeline's status might vary, the number of stages and jobs might change, statuses of the jobs might vary, artifacts may or may not be produced and environments may or may not be provisioned etc.

Should you not setup proper tests for each of these scenarios? Should these tests not be executed on every change of the pipeline template?

Thus, `rptf`.

## Table of Contents

1. [Quickstart](#quickstart)
1. [Using RPTF with GitLab](#using-rptf-with-gitlab)
1. [Setup GitLab Access Token](#setup-gitlab-access-token)
1. [Usage With Self-Managed GitLab](#usage-with-self-managed-gitlab)
1. [Authoring Tests](#authoring-tests)
1. [Detailed Example](#detailed-example)
1. [Test Output](#test-output)
1. [CLI Options](#cli-options)
1. [JUnit Reports](#junit-reports)
1. [License](#license)
1. [Contributing](#contributing)
1. [Who Uses RPTF](#who-uses-rptf)

## Quickstart

Being authored in Python, `rptf` is distributed via PyPi: https://pypi.org/project/rptf/

Install `rptf` locally with `pip`.

```bash
pip install rptf
```

This makes `rptf` available for local execution:

```bash
rptf --help
```

Test cases are defined in a Yaml file, which by default is called `rptf.yml`.

```yaml
# rptf.yml

should_succeed_with_25_jobs:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: success
        job_count: 25

should_fail_with_1_job:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: failed
        job_count: 1

should_fail_with_3_jobs_deploy_skipped:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: failed
        job_count: 3
        job_status:
            build: success
            test: failed
            deploy: skipped
```

Each test case contains two sections: `trigger` and `assertions`. The example above shows two test project's pipelines being triggered and asserts being make expect the pipelines have `25` jobs and `1` job respectively.

Pipeline status is also asserted, and the second test case expects a failed pipeline.

## Using RPTF with GitLab

We can execute `rptf` within a GitLab pipeline, thus, you have a primary test project which contains the `rptf.yml` file and includes a `.gitlab-ci.yml` definition that triggers pipelines in other GitLab projects and makes assertions.

```yaml
# .gitlab-ci.yml

stages:
    - test

rptf:
    stage: test
    image: python:3
    before_script:
        - pip install rptf
    script:
        - rptf
```

The `rptf` job above triggers pipelines defined in `rptf.yml` as long as the `GITLAB_ACCESS_TOKEN` provided has sufficient permissions. It's advisable to set the token at the project or group level via `GitLab -> Settings -> CICD`.

## Setup GitLab Access Token

The project containing the `rptf` job needs to provide `GITLAB_ACCESS_TOKEN`. This token must be authorized to execute pipelines across all projects defined in `rptf.yml`.

## Usage With Self-Managed GitLab

Environment variable `GITLAB_API_URL` is read to determine the GitLab instance where the pipelines are created.

You can set this to get `rptf` to work with your self-managed GitLab instance.

```shell
# set GITLAB_API_URL
export GITLAB_API_URL=https://path-to-your-instance/api/v4

# execute rptf 
rptf
```

## Authoring Tests

The Yaml specification declares the following attributes that can be used to author test cases:

- `trigger` defines the target project where pipeline should be triggered
    - `project_id` required
    - `ref` defaults to `master`
    - `branch` deprecated in favor of `ref`
    - `variables` defaults to an emppty list `[]'
- `assertions` defines assertions to make after pipeline finishes execution
    - `pipeline_status` required, expected pipeline status
    - `job_count` expected number of jobs
    - `job_status` dict of expected job status
    - TODO `artifact_count` expected number of artifacts
    - TODO `environment_status` dict of expected environment status

## Detailed Example

```yaml
two_jobs_two_artifacts:
    trigger:
        project_id: 11112222
        ref: feature-branch-abc
        variables:
            hello: world
            foo: bar
    assertions:
        pipeline_status: success
        job_count: 2
        job_status:
            artifact-one: success
            artifact-two: success
        artifact_count: 2               # TODO
        environment_status:             # TODO
            success: available          # TODO
            review_env_one: stopped     # TODO
```

## CLI Options

```bash
# help
rptf --help             # help

# run tests
rptf                    # by default, executes rptf.yml
rptf --target FILE      # path to test definition yaml

# toggle JUnit test reporting
rptf                    # generates JUnit report
rptf --junit            # generates JUnit report
rptf --no-junit         # JUnit reporting skipped
```

## Test Output

[![asciicast](https://asciinema.org/a/407815.svg)](https://asciinema.org/a/407815)

![](https://i.imgur.com/5xm7lKu.png)

![](https://i.imgur.com/JpZ8RhZ.png)

## JUnit Reports

By default, all test failures are reported in a JUnit compatible XML file called `rptf-report.junit.xml`.

This can be integrated in the `rptf` job to enable deep integration with GitLab (or any other tool that supports JUnit reporting format).

```yaml
# .gitlab-ci.yml

stages:
    - test

rptf:
    stage: test
    image: python:3
    before_script:
        - pip install rptf
    script:
        - rptf
    artifacts:
        reports:
            junit: rptf-report.junit.xml
```

Example of RPTF has failing tests integrated back into a GitLab Merge Request:
- https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/merge_requests/1

JUnit reporting can be turned off by passing the `rptf --no-junit` flag.

## License

[MIT](./license)

## Contributing

Contributions welcome [here](https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/) in the form of feature requests, feature implementations, documentation improvements, examples etc.

```shell
# execute locally
pipenv install
pipenv run python -m rptf.rptf

```

## Who Uses RPTF

In-exhaustive list of RPTF users:

- GitLab Five Minute Production App
- LeapMotion
- APNIC
- Brainbits
- Everbridge
- Bolt

[Add](https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/edit/master/readme.md) your organization.