Monitor GitLab with OpenTelemetry and New Relic
Tags: monitoring, gitlab, otel, devops
Published: 2023-03-06 18:33:43

Shifting left isn't just about testing your code and automating the build and deployment process. It's about monitoring and testing your CI and CD pipelines, too.

In this post, you’ll learn how to monitor your GitLab jobs and pipelines using OpenTelemetry and the [New Relic GitLab quickstart](https://newrelic.com/instant-observability/gitlab?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab). After you're finished, you'll be able to:

- Track key metrics on your GitLab pipelines in a pre-built dashboard, such as how long your jobs are taking and how often they are failing.
- Visualize jobs and pipeline executions as distributed traces with logs in context.
- Pinpoint where issues are coming from in your pipelines.
- Create alerts on your GitLab pipelines.

The next image shows a New Relic dashboard with some of the GitLab metrics you’ll be able to see.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nz16v3hrm01aqzof1ip9.png)

With this integration, you can shift left and get observability for the build and deploy phases of the DevOps lifecycle. 

## How the integration works
The [New Relic GitLab integration](https://github.com/newrelic-experimental/gitlab) is split into two parts for ease of use. Both parts run as jobs in your existing or new GitLab pipelines. You’ll learn how to integrate each part in the next section.

  **New Relic exporter**: Add this job to the end of each of your pipelines to export your pipeline data as logs and traces to New Relic. You’ll be able to use the distributed trace viewer to inspect each pipeline run as well as explorer logs in the log explorer.
  **New Relic metrics exporter**: This job should be scheduled. It exports GitLab data for all your projects, pipelines, jobs, runners, environments, deployment, and releases. You can configure which groups and projects are included. You can also run this as a standalone container outside GitLab.

Let’s look at some examples that show what you can visualize in New Relic after you’ve enabled the integration.

The following distributed trace captured by the New Relic exporter shows a job that contains an error with associated logs in context.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kmhr2r1xe5ad1u407on2.jpg)

The next two images show metrics captured by the New Relic metrics exporter. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k160uftkvqo02zsr36di.jpg)

The next image shows GitLab resources as log events with the configured attributes.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d3qrd4o27nrzheiohg61.jpg)


## Configuring the exporters
Both exporter integrations are set up similarly, but with one key difference:

- The New Relic exporter needs to be set up for each pipeline you want to report data for.
- The New Relic metrics exporter only needs to be set up once.

## Prerequisites:

Before setting up the integration, you will need both a GitLab token and a New Relic ingest API key.

1. Follow these steps in the [API documentation](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) to generate a New Relic ingest API key.
2. Generate a GitLab API Token with “read_api” access.

- In the top-right corner, select your avatar.
- Select **Edit profile**.
- On the left sidebar, select **Access Tokens**.
- Enter a name and optional expiration date for the token.
- Select the “read_api” scope.
- Select **Create personal access token**.

## Setting up the New Relic exporter

The New Relic exporter should be configured to run as a child job at the end of your project pipeline. It will ship the data from the pipeline to New Relic as a distributed trace with the logs being reported as [logs in context](https://docs.newrelic.com/docs/logs/logs-context/logs-in-context/?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab).



## Step one: Create CI/CD variables in GitLab
First, you need to set up your CI/CD variables in GitLab.  In addition to the required GitLab token and New Relic API key, there are also several optional variables.

To add or update variables in GitLab for your project:

1. Go to your project’s **Settings > CI/CD** and expand the **Variables** section.
2. Select **Add variable** and fill in the details for the following required variables. Make sure that both variables are masked.
_GLAB_TOKEN_ provides access to the GitLab API. Add the GitLab API token you created in the previous section.
_NEW_RELIC_API_KEY_ is the ingest API key you created in the previous section.
3. You can also update the following optional variables. Each configuration option should be set up in GitLab as a pipeline environment variable. By default, these optional variables are already set to automatically export job logs to New Relic.
  1._OTEL_EXPORTER_OTEL_ENDPOINT_ is the New Relic OTEL endpoint and includes a port. By default, it’s set to https://otlp.nr-data.net:4318 in the US and https://otlp.eu01.nr-data.net:4318 in the EU.
2._GLAB_EXPORT_LOGS_ is used to export job logs to New Relic. This variable is a boolean that’s set to true by default.
3. _GLAB_ENDPOINT_ is the GitLab API endpoint. It’s set to https://gitlab.com by default.

## Step two: Add the pipeline configuration for the exporter 

- Create a file called _new-relic-exporter.yml_ in the root directory of the GitLab project, repository, or branch you want to monitor.

Add the following pipeline code. The New Relic image is provided for testing purposes. It’s recommended that you use your own image of the exporter in production. You can build your own image based on the [repository](https://github.com/newrelic-experimental/gitlab).

```
stages:
- new-relic-exporter
new-relic-exporter:
 rules:
   - when: always
 image:
   name: docker.io/dpacheconr/gitlab-exporter:1.0.0
 stage: new-relic-exporter
 script:
   - echo "Done"
```

Ensure that the rule is set to always so that the exporter runs even if previous jobs fail.

## Step three: Add the exporter job to your existing pipeline 

In your _.gitlab-ci.yml_  file, add a new stage with the name _new-relic-exporter_ to your _stages_ block as shown in the next code snippet. This should be set as the last stage in the list.

```
stages:
…existing stages…
- new-relic-exporter
```

Configure the _new-relic-exporter_ stage block as shown in the next code snippet:

```

new-relic-exporter:
 stage: new-relic-exporter
 inherit:
   variables: true
 variables:
   CI_PARENT_PIPELINE: $CI_PIPELINE_ID
 trigger:
   include:
     - local: new-relic-exporter.yml
```

## Step four: Test and confirm that New Relic is receiving data

Commit your code to run your pipeline and confirm that the New Relic Exporter job runs as the final job in the pipeline.

[Log into New Relic](https://one.newrelic.com/nr1-core?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab), go to **Services - OpenTelemetry**, and select the entity. It will have the same name as the project, repo, or branch that you are monitoring. Select the **Distributed Traces** tab and confirm that you can see a trace for your pipeline. Sometimes it takes a few minutes for the first trace to appear. The next image shows an example of a trace from the exporter.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g8n63ezcafim89rqr3wc.jpg)

An example of a trace from the exporter in the distributed traces dashboard in New Relic.

Next, go to the **Logs** tab and confirm that you can see your pipeline logs appearing.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c7x7luhiocfn6b5vjj3v.jpg)
Log events in New Relic show GitLab resources.

If there is no data, ensure that you have correctly set up the environment variables with the appropriate keys. In GitLab, you can check the New Relic exporter job output for additional debugging information.

## Setting up the New Relic metrics exporter

The New Relic metrics exporter is standalone and does not need to be attached to an existing project pipeline. You should run it in its own project pipeline, but you can add it to an existing pipeline if you prefer.

This gathers data from the GitLab API about your projects, pipelines, jobs, runners, environments, deployments, and releases, then exports the data to New Relic as a mixture of metrics and logs.

To get the latest data from your accounts, you should run the New Relic metrics exporter on a regular basis. At least once per hour is recommended.

## Step one: Create a new project

Create a new GitLab project and give it a descriptive name like New Relic metrics exporter.


## Step two: Create CI/CD variables in GitLab

The exporter job has a number of configurable options which are provided as CI/CD environment variables. You must supply a GitLab token and New Relic API key. 

You also need to provide a [regular expression](https://docs.python.org/3/library/re.html) to define which projects and groups you’d like the exporter to monitor. Your GitLab token may have access to data you don’t want to export so take care to set this up so that only the projects and groups you are interested in are captured. 

The other configurations are optional. Each configuration option should be set up as a pipeline environment variable. See the [README](https://github.com/newrelic-experimental/gitlab) for a list of the configuration options available along with their default values. The defaults with no additional configuration will run the job every 60 minutes.

To add or update variables in project settings:

1. Go to your project’s **Settings > CI/CD** and expand the **Variables** section.
2. Select **Add variable** and add your _GLAB_TOKEN_ and _NEW_RELIC_API_KEY_. Don’t forget to mask both variables.
3. Add or update any optional variables as needed for your project. If you want to change the frequency of the job to a value other than 60 minutes, you can do so with the optional variable _GLAB_EXPORT_LAST_MINUTES_.

##Step three:   Create your pipeline definition
If you have created a new project for this exporter as recommended, you will need to create the pipeline definition file _.gitlab-ci.yml_ in the root of your project. Add the pipeline code shown in the next snippet.

```
stages:
- new-relic-metrics-exporter
 new-relic-metrics-exporter:
 rules:
   - when: always
 image:
   name: docker.io/dpacheconr/gitlab-metrics-exporter:1.0.0
   pull_policy: always
 stage: new-relic-metrics-exporter
 script:
   - echo "Done"
```

## Step four: Test the pipeline

Run the pipeline and confirm that the New Relic metrics exporter job runs correctly with no errors, by checking the New Relic metrics exporter job output.

To execute a pipeline schedule manually, follow these steps in GitLab:

1. On the top bar, select **Main menu > Projects** and find your project.
2. On the left sidebar, select **CI/CD > Schedules**.
3. Select **Play** to run the desired pipeline.

In the New Relic [query builder](https://one.newrelic.com/data-exploration/query-builder?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab), run the following [NRQL query](https://docs.newrelic.com/docs/query-your-data/nrql-new-relic-query-language/get-started/introduction-nrql-new-relics-query-language/?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) to ensure that New Relic is receiving data from the exporter:

> SELECT * FROM Log where gitlab.source ='gitlab-metrics-exporter'

Go to [Metrics & Events](https://one.newrelic.com/data-exploration/data-explorer/explorer?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) in New Relic and your GitLab metrics in the list of available metrics. These metrics will be prefixed with _gitlab_. 

Metrics include pipeline, job duration, and queue duration. The next image shows the information on the duration of GitLab jobs.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6izx8hs9z7zkq16yxeb6.jpg)
New Relic dashboard shows duration of GitLab jobs.

Go to [Logs](https://one.newrelic.com/logger?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) in New Relic to see your log events. Apply the filter _gitlab.source:"gitlab-metrics-exporter"_ to only see data sourced from _new-relic-metrics-exporter_. You will be able to see log data on all of your GitLab resources, including your projects, deployments, and runners.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dhzcayiesesu6sywrjez.jpg)
All log data showing in New Relic, including GitLab projects, deployments, and runners.

## Step five: Create a pipeline schedule

Once you have confirmed the job is correctly sending data, you need to set up a schedule in GitLab to run the job regularly. The schedule pattern should match the value from _GLAB_EXPORT_LAST_MINUTES_. As discussed earlier in this post, the default is 60 minutes. Setting values that are different from the value in _GLAB_EXPORT_LAST_MINUTES_ can lead to duplicate or missing data in New Relic.

To add a pipeline schedule, do the following in GitLab:

    On the top bar, select **Main menu > Projects** and find your project.

    On the left sidebar, select **CI/CD > Schedules**.

    Select **New schedule** and add an interval pattern To run hourly, set the pattern to  _0 * * * *_.

## Step six: Import the quickstart dashboard

We’ve built a starter dashboard to visualize your GitLab data and help you monitor your jobs and pipelines. You can install this dashboard using the [GitLab](https://newrelic.com/instant-observability/gitlab?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) [quickstart](https://newrelic.com/instant-observability/gitlab?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) and then further customize it as needed. To get this dashboard, simply follow these steps:

1. Go to the [Gitlab quickstart](https://newrelic.com/instant-observability/gitlab?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) in New Relic Instant Observability and select **Install now**.
2. Choose an account and select **Begin installation**.
3. If you've already completed the earlier steps in this tutorial, select **Done** to move on to the next step.
4. The quickstart deploys the resources to your account. Select **See your data** to get to the dashboard.

The following image shows the dashboard, which provides a summary of key metrics such as the number of runners available, the number of successful jobs, the top errors, and the job queue status.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3msrj9nidro1o0xj0vfq.png)

Install the GitLab quickstart to get a dashboard template with prebuilt views to monitor your GitLab pipelines.

## Conclusion

After you're done, you'll have complete visibility into your GitLab estate, from high-level metrics to logs and distributed traces. The next step is to consider [adding alerts](https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/get-started/your-first-nrql-condition?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab) when certain thresholds are met, such as when jobs are taking too long, or when the number pipeline failures gets too high. 

By sending your GitLab data to New Relic, you get observability into the building and deployment phases of the DevOps lifecycle, providing you with more insights into your DevOps practices and helping you shift left.

---

_Read the full blog post at New Relic: [Monitor GitLab with OpenTelemetry and New Relic](https://newrelic.com/blog/how-to-relic/monitor-gitlab-with-opentelemetry?utm_source=devto&utm_medium=community&utm_campaign=global-fy-23q4-dev_to_post_monitorgitlab)._

_Not an existing New Relic user? [Sign up for a free account](https://newrelic.com/signup?utm_source=devto&utm_medium=community&utm_campaign=global-fy23-q4-dev_to_post_monitorgitlab) to get started!_ 👨‍💻