Announcing the Tracetest integration with Keptn, the control plane for cloud-native application life-cycle orchestration
Tags: devops, gitops, observability, testing
Published: 2023-03-09 16:00:26

Today, we're happily announcing that Tracetest now works with Keptn, a control plane and micro platform for cloud-native application delivery and operations.

> *Check out this [hands-on Demo example](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration) of how Tracetest works with Keptn!*

## What is Keptn?

[Keptn](https://keptn.sh/) is an [open-source CNCF project](https://github.com/keptn/keptn). It’s a control plane and micro platform for cloud-native application delivery and operations following best practices of putting observability data into the driving seat of making automated decisions.

Keptn augments any standard Kubernetes cluster to support delivery based on automated quality gates and self-healing operations workflows. Enabled by a fully event-driven architecture, Keptn can be easily extended to integrate with new toolchains. The declarative approach versus traditional scripts enables enterprise-wide deployments and easy maintenance.

If you are interested in Keptn, also take a look at the latest evolution called [KLT (Keptn Lifecycle Toolkit)](https://lifecycle.keptn.sh/). KLT uses a K8s Operator approach which brings automated deployment observability and orchestration to your GitOps deployments.

## What is Tracetest?

[Tracetest](https://tracetest.io/) is an [open-source CNCF project](https://github.com/kubeshop/tracetest). It allows you to quickly build integration and end-to-end tests, powered by your OpenTelemetry traces.

Tracetest leverages your OpenTelemetry traces to enable trace-based testing with assertions against your trace data at every point of the request transaction. To use Tracetest, simply point it to your existing trace data source or send traces directly to Tracetest.

We make it possible to

- Define tests and assertions for every microservice that a request passes through.
- This solution works with your existing distributed tracing, allowing you to build tests based on your already-instrumented system.
- Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
- Return both the response data and a distributed trace.
- Define assertions against both the response and trace data. This will ensure that both your response and underlying processes work correctly, quickly, and without errors.
- Save tests.
- Run the tests manually or via CI build jobs with the Tracetest CLI.

## Tracetest Now Works with Keptn!

[Tracetest now works with Keptn](https://artifacthub.io/packages/keptn/keptn-integrations/tracetest), by using the [Keptn Job Executor Service](https://github.com/keptn-contrib/job-executor-service) plugin. You can now upload a Tracetest test definition and a CLI configuration to a service and run a test with this simple job definition below.

```yaml
apiVersion: v2
actions:
- name: "Run tracetest on your service"
  events:
    - name: "sh.keptn.event.test.triggered"
  tasks:
    - name: "Run tracetest"
      files:
        - data/test-definition.yaml
        - data/tracetest-cli-config.yaml
      image: "kubeshop/tracetest:latest"
      cmd:
        - tracetest
      args:
        - --config
        - /keptn/data/tracetest-cli-config.yaml
        - test
        - run
        - --definition
        - /keptn/data/test-definition.yaml
        - --wait-for-result
```

**Why is it important?**

When running integration tests, it can be difficult to pinpoint where, within a network of microservices, an HTTP transaction fails. With tracing enabled, Tracetest can run tests with assertions against existing data throughout every service involved in the transaction. These tests can be used as part of your CI/CD process to ensure system functionality and catch regressions.

![https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124387/Blogposts/keptn-integration/image_45_amw06p.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124387/Blogposts/keptn-integration/image_45_amw06p.png)

By enabling trace-based testing from within your Keptn application life-cycle orchestration, you can ensure every part of your multi-stage delivery is covered to match SLOs.

![https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124437/Blogposts/keptn-integration/screely-1678122433950_1_mbavxs.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124437/Blogposts/keptn-integration/screely-1678122433950_1_mbavxs.png)

Read more about the Keptn integration with Tracetest, [here](https://tracetest.io/integrations/keptn).

## Try Tracetest with Keptn

Install Keptn in your Kubernetes cluster by following the official docs, [here](https://keptn.sh/docs/1.0.x/install/helm-install/). This shows both a simple Helm install and a full install of Keptn. Next, you need to [install](https://github.com/keptn-contrib/job-executor-service/blob/main/docs/INSTALL.md) the `Job Executor Service 0.3.x`.

Finally, on your local machine, you need:

1. `Keptn CLI` [installed](https://keptn.sh/docs/1.0.x/install/cli-install/)
2. and already [authenticated](https://keptn.sh/docs/1.0.x/install/authenticate-cli-bridge/) with Keptn API.

With Keptn set up, you can configure Keptn and Tracetest.

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI:

```yaml
brew install kubeshop/tracetest/tracetest
```

> **Note**: Check out the [download page](https://tracetest.io/download) for more info.

From here, follow the [official documentation](https://docs.tracetest.io/getting-started/installation) to install the Tracetest server in your Kubernetes cluster. Once the server is installed, open up Tracetest Web UI in the browser and follow the instructions for connecting your trace data store with Tracetest.

## Configure Tracetest with Keptn

Keptn works with two [concepts](https://keptn.sh/docs/concepts/glossary/). A **Project** and a **Service**. A **Project** is an element that holds multiple services forming an application in stages. And, a **Service** is the smallest deployable unit and is deployed in all project stages according to the defined order.

Usually, these resources are managed by Keptn during **Sequences** — a set of tasks for realizing a delivery or operations process.

Start by creating a **Shipyard** to contain the **Project**. Then, create a skeletal [`shipyard.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/shipyard.yaml) file with the following content:

```yaml
apiVersion: "spec.keptn.sh/0.2.2"
kind: "Shipyard"
metadata:
  name: "<shipyard_name>"
spec:
  stages:
    - name: "production"
      sequences:
        - name: "validate-app"
          tasks:
            - name: "test"
```

Create a new project using the `shipyard.yaml` file:

```bash
keptn create project <project_name> -y -s shipyard.yaml
```

> **Note:** Keptn may ask you to have a Git repository for this project to enable GitOps. If so, you need to create an empty Git repository and a Git token and pass it through the flags `--git-remote-url`, `--git-user`, and `--git-token`. More details about this setup can be seen on [Keptn docs/Git-based upstream](https://keptn.sh/docs/1.0.x/manage/git_upstream).

![https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124541/Blogposts/keptn-integration/screely-1678122385629_1_frgqac.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124541/Blogposts/keptn-integration/screely-1678122385629_1_frgqac.png)

Now you can create a service:

```bash
keptn create service <service_name> --project <project_name> -y
```

Now, we will set up a job associated with the service, listening to a task event.

Create the [`tracetest-cli-config.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/tracetest-cli-config.yaml) configuration file for the Tracetest CLI in your current directory, identifying the Tracetest instance that should run the tests:

```yaml
scheme: http
endpoint: tracetest.tracetest.svc.cluster.local:11633
```

Create the [`test-definition.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/test-definition.yaml) Tracetest test definition in your current directory:

```yaml
type: Test
spec:
  id: apdCx-h4g
  name: Pokeshop - List Pokemon
  description: Get a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
      method: GET
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[name = “Tracetest trigger”]
    assertions:
    - attr:tracetest.span.duration < 500ms
```

> **Note:** The Tracetest test definition will vary based on your own test definitions.

Add these files as resources for your service in the `production` stage:

```bash
keptn add-resource --project <project_name> --service <service_name> --stage production --resource test-definition.yaml --resourceUri data/test-definition.yaml

keptn add-resource --project <project_name> --service <service_name> --stage production --resource tracetest-cli-config.yaml --resourceUri data/tracetest-cli-config.yaml
```

These files will be located in the folder `data` and will be injected into our Keptn job that we will set up in the next step.

Create a [`job-config.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/job-config.yaml):

```yaml
apiVersion: v2
actions:
- name: "Run tracetest on your service"
  events:
    - name: "sh.keptn.event.test.triggered"
  tasks:
    - name: "Run tracetest"
      files:
        - data/test-definition.yaml
        - data/tracetest-cli-config.yaml
      image: "kubeshop/tracetest:latest"
      cmd:
        - tracetest
      args:
        - --config
        - /keptn/data/tracetest-cli-config.yaml
        - test
        - run
        - --definition
        - /keptn/data/test-definition.yaml
        - --wait-for-result

```

This job will run Tracetest every time a `test` event happens, listening to the event `sh.keptn.event.test.triggered`. This event is emitted by the `test` task on the `validate-app` sequence.

Add this job as a resource in Keptn:

```bash
keptn add-resource --project <project_name> --service <service_name> --stage production --resource job-config.yaml --resourceUri job/config.yaml
```

![https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124714/Blogposts/keptn-integration/screely-1678122514961_1_mvxsv7.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124714/Blogposts/keptn-integration/screely-1678122514961_1_mvxsv7.png)

To guarantee that the job will be called by Keptn when you execute the `production` sequence, you need to configure the `Job Executor Service` integration in your project to listen to `sh.keptn.event.test.triggered` events.

You do that through the Keptn Bridge, the Web UI, by going to the project, choosing the `Settings` option, and `Integrations`.

Choose the `job-executor-service` integration, and add a subscription to the event `sh.keptn.event.test.triggered` and your project.

Finally, to see the integration running, we only need to execute the following command:

```bash
keptn trigger sequence validate-app --project <project_name> --service <service_name> --stage production
```

Now you should be able to see the sequence running in Keptn Bridge.

![https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124437/Blogposts/keptn-integration/screely-1678122433950_1_mbavxs.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1678124437/Blogposts/keptn-integration/screely-1678122433950_1_mbavxs.png)

## What's next?

Would you like to learn more about Tracetest and what it brings to the table? Check the [docs](https://docs.tracetest.io/tools-and-integrations/keptn) and try it out today by [downloading](https://tracetest.io/download) it today! Want to learn more about Keptn? [Read more here](https://keptn.sh/), or check out [more integrations here](https://keptn.sh/docs/integrations/).

Also, please feel free to join our [Discord community](https://discord.com/channels/884464549347074049/963470167327772703), give [Tracetest a star on GitHub](https://github.com/kubeshop/tracetest), or schedule a [time to chat 1:1](http://calendly.com/ken-kubeshop/otel-user-interview-w-tracetest).