Add deployment annotations to grafana dashboards
Tags: monitoring
Published: 2023-05-19 14:57:59

When noticing spikes in exceptions on our dashboards, one of the first things we do is check if there has been a recent deployment. Checking to see if a deployment caused the spikes.

This can be straightforward, but if you have many teams deploying many apps all throughout the day, it may be a little trickier tracking down a related deployment. 

I am going to talk you through how to add a grafana annotation to visualize when a deployment has taken place. This will be a simple annotation with some tags. 

Here is what we are gonna create

![grafana annotation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ucnjhqm1665g0bql2zl8.png)

First of all, I am going to be running this using my grafana playground which you can find here

{% embed https://dev.to/wesley_skeen/setup-grafana-locally-with-some-data-sources-4mig %}

Once you have that set up, there are a few steps we need to do to enable annotations to display on a dashboard.

### Create a service account
Browse [here](http://localhost:3000/org/serviceaccounts) and click **Add service account**. If you have the grafana playground running, you should be able to access the link.

![add service account](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vj8qu7yogk8eu5g7tcvj.png)

Give it a name and the **Editor** role

![add name and editor role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vv8mgt721kktlri3i5w6.png)

### Create a service account token
Click **Add service account token**

![Add service account token button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jtio368690i9d6eob1x2.png)

Then click **Generate token**

![generate token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mr8kndkbuxecrly9si8a.png)

Once you have this token, copy it and keep it save.

### Create a dashboard
For this example, I am just gonna create a really simple dashboard here with a random metric. Just to have something to show.

![dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnp6f7hhwpvu5aohie4k.png)

### Create annotations
Next we will create some annotations to display. We are going to send them as a http post request. The URL we create annotations on is `http://localhost:3000/api/annotations`.

We next have to add the `Authorization` header. The value for this will be `Bearer {your service account token we created}`.

The last thing to do is send a request. The payload will look like

```
{
  "created": EPOCH with milliseconds, // eg 1684502867000
  "updated": EPOCH with milliseconds,
  "time": EPOCH with milliseconds,
  "timeEnd": EPOCH with milliseconds,
  "text": "Deployment", // text to display on the annotation pop up
  "tags": [ // tags to display on the annotation pop up
    "kind:deployment",
    "project:your-project-or-app-name",
    "env:prod",
    "region:eu-west-1"
  ]
}
```

The response should be 

```
{
    "id": {annotation_id},
    "message": "Annotation added"
}
```

Now that we have created an annotation, it is time to set up our dashboard to view them.

### Set up dashboard annotations
Go to the dashboard you created previously and go to it's settings.

Next click on **Annotations** and then **Add annotation query**.

![add annotation query](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hi5pv58mg6hd2611ebag.png)

Chose **Grafana** as the data source.

![Grafana datasource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4gh24ktq8sgzdgk0gipx.png)

Next under **Filter by**, choose **Tags** and then turn on **Match any**.

![tags filter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mve439o4n10yszo8gq4t.png)

You can then filter by tags you sent when sending the http request. I am just gonna filter on one tag `kind:deployment`.

![tags](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1gpulpttfemaxw0a6bhy.png)

Click **Apply** and **Save dashboard**. 

Browse back to your dashboard and you now should see an annotation being displayed

![grafana annotation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ucnjhqm1665g0bql2zl8.png)

 If you don't see any annotations check the following things
- You are using a recent timestamp for your EPOCH time when creating your annotations
- You are filtering by the correct tags
- Your dashboard has annotations turned on

![annotations switch](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/abm4337tm059wkguaazf.png)

### Adding to Azure DevOps pipeline
I have created a [gist](https://gist.github.com/WesleySkeen/c197ead0e0cc3887d1f56945e83abaea) that can be added to your Azure DevOps pipeline.