Creating Custom Incident Response Workflows with n8n
Tags: devops, pagerduty, mattermost, tutorial
Published: 2020-05-14 12:29:37

---
title: Creating Custom Incident Response Workflows with n8n
published: true
date: 2020-05-03 14:29:52 UTC
tags: devops,pagerduty,mattermost,tutorial
canonical_url: https://n8n.io/blog/creating-custom-incident-response-workflows-with-n8n/
---

I’ve been involved in the DevOps world for a while and yet, I finished reading [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592) only recently. This piqued my interest in how teams execute their incident response playbooks. It’s enlightening to see the different approaches teams take, to hone what works best for them.

![](https://cdn-images-1.medium.com/max/1024/1*9TfFECFPp-Pm5leShq9pIg.png)<figcaption>Incident response management in Mattermost powered by n8n and PagerDuty</figcaption>

I wanted to test how automating a minimalist incident response playbook would look like and I decided to test it out with three of my favorite tools [n8n](https://n8n.io/), [PagerDuty](https://www.pagerduty.com/) and [Mattermost](https://mattermost.com/). Here’s a quick introduction to the three tools, in case you aren’t aware of them:

1. n8n is a [fair-code](https://faircode.io/) licensed tool that helps you automate tasks, sync data between various sources, and react to events all via a visual workflow editor.
2. PagerDuty is a SaaS incident response platform for IT departments in companies.
3. Mattermost is a flexible and open-source messaging alternative to Slack.

To avoid panic during an incident, a lot of companies have an incident response playbook. I created a minimalist six-step playbook for this tutorial. Whenever, a service goes down or something unexpected happens, the on-call team would follow this high-level protocol:

1. Triage issue in Jira
2. Create auxiliary channel
3. Invite the on-call team to the channel
4. Acknowledge the issue
5. Fix the issue
6. Resolve the ticket

We will automate this playbook with three workflows (these workflows were built using n8n@0.111.0) in n8n and this is how the result shall look like once we are done.

![](https://cdn-images-1.medium.com/max/640/1*bv7T0z4uVHKxrCUfpLOomQ.gif)<figcaption>Incident response workflow in play</figcaption>

## Workflow 1 — Make sure everyone knows what happened

Our first workflow will cover the first three steps of the playbook. Whenever a service goes down and creates an incident report on PagerDuty, we want the workflow to automate the following tasks for us:

1. A webhook gets triggered and informs a general incidents channel on Mattermost that something is wrong.
2. Create an auxiliary channel for the specific incident, invite the on-call team to it and share its link for those interested in the incident.
3. Triage an issue on Jira.
4. Share the links of the auxiliary channel, PagerDuty incident and the Jira issue in the Incidents channel, and the auxiliary channel.
5. Share action buttons in the auxiliary channel to acknowledge and resolve the incident.

Let’s get started with the nodes of the first workflow. I have also submitted [Workflow 1](https://n8n.io/workflows/353) on n8n.io, in case you’d like to skim through this workflow. Please note that you’ll still need to configure a couple of things like your credentials, channels on Mattermost as well as the settings of the nodes. You can find information on how to setup n8n in the [documentation](https://docs.n8n.io/#/setup). Alternatively, you can sign-up for n8n.cloud to get access to our hosted service.

### 1. Webhook node: Get data from PagerDuty

First of all, we need to pull in the new incident reports from PagerDuty. To do that start n8n with the tunnel parameter:

`n8n start --tunnel`

> **Note:** Make sure that you don’t forget to add the `--tunnel` parameter. If you’re using n8n.cloud, click on the Open button on your dashboard

Add a new node by clicking on the '***+***' button on the top right of the Editor UI. Select the _Webhook_ node under the ***Triggers*** section.

In the Node Editor view, set the ***HTTP Method*** to `POST`. Now, you’ll need to save the workflow. I named it ‘Incident Response Workflow’. Once the workflow is saved, click on ***Webhook URLs***, select ***Test***, and then click on the URL to copy it to the clipboard.

> **Note:** Don’t forget to save the workflow first before copying the Webhook URLs.

Here’s a video of me following the steps mentioned above.

{% youtube ZCNXpUvUXfQ %}

Now that we have our *Webhook* node ready on n8n, we’ll need to configure the settings on PagerDuty, so that it sends the new incident reports to the webhook.

Unless your team already uses PagerDuty, you can create a free trial account on PagerDuty. If you are creating a new account, you’ll also have to create a service that PagerDuty will be monitoring.

PagerDuty has integrations with a lot of services, to monitor them, in case something goes wrong. Once you have created your service, let’s configure the webhooks for the service.

To do that, click on ***Services***. Click on the ***More*** button on the right side and select ***View Integrations*** from the menu (do this for the service that you want to configure the webhook for). Now, under the section called ***Extensions*** and ***Add-Ons***, click on ***Add or manage extensions***. Click on the ***+ New Extension*** button and select 'Generic V2 Webhook' as the ***Extension Type***. I entered `n8n` as the name and entered the URL that I copied from the *Webhook* node. Click on the ***Save*** button and we are done!

Here’s a video of me following the steps mentioned above.

{% youtube yTHOCy9Tcx0 %}

Now, click on the _Execute Workflow_ button to register the webhook. Once you’ve done that, you can create a new incident at PagerDuty. Your Webhook node will receive all the details. Keep in mind that the Test webhooks are only valid for 120 seconds. It should look something like in the following image.

![](https://n8n.io/blog/content/images/size/w1600/2021/03/webhook-node-output.png)<figcaption>Response received by the Webhook node upon creation of an incident on PagerDuty</figcaption>

At times, when you are sending too many requests from PagerDuty, it will disable the webhook. You’ll have to re-enable it by going to the list of extensions and clicking on the ***Re-enable*** button.

![](https://n8n.io/blog/content/images/size/w1600/2021/03/enable.png)<figcaption>Re-enabling the webhook on PagerDuty</figcaption>

### 2. Mattermost node: Create an auxiliary channel

Now, we need to create a *Mattermost* node that will create an auxiliary channel so that the on-call team can coordinate on a fix for the incident.

To do that, click on the '***+***' button and click on the _Mattermost_ node. In the Node Editor, enter your Mattermost credentials. Here’s some detailed [information](https://docs.n8n.io/credentials/mattermost) on how to create an access token for the credentials. I have used an access token from a bot account, but you can also use the access token from your account.

> _Note:_ Throughout the tutorial, please make sure that the nodes are connected properly before you start the configuration in the Node Editor. If you don’t do this, the variables mentioned in the tutorials might not be visible to you.

Once you are all sorted out with the credentials, select ‘Channel’ as the ***Resource*** in the Node Editor. Now select your team as the ***Team ID*** (in case you are unable to acquire that, please check with your system admin). We now need to enter a ***Display Name*** for the channel. Since this would be a dynamic piece of information, click on the gears icon next to the field and select ***Add Expression***. Select the following in the Variable Selector:

`Current Node > Input Data > JSON > body > messages > [Item: 0] > log_entries > [Item: 0] > incident > summary`

Quite some indentation, I know! This will make sure that the display name of the channel would be the same as the incident summary on PagerDuty to keep things coherent. Now you need to enter a ***Name***. This needs to be a unique value, so we’ll select the `id` from the Incident report. Click on ***Add Expression*** and select the following in the Variable Selector:

`Current Node > Input Data > JSON > body > messages > [Item: 0] > id`

Perfect, now click on ***Execute Node*** and this will create an auxiliary channel on Mattermost. Here’s a video of me following the steps mentioned above.

{% youtube hlSWUcY2UHA %}

### 3. Mattermost node: Add on-call team to auxiliary channel

Once the auxiliary channel has been created, we need to make sure that all the on-call team members have been added to the channel. However, right now we‘ll add a single user to the channel.

To do that create another *Mattermost* node. Select the credentials that you entered earlier. Select ‘Channel’ as the ***Resource*** and click on ‘Add User’ for ***Operation***. Now we have to specify the ***Channel ID*** where the user should be added. Since this is another dynamic piece of information, click on _Add Expression_ and in the Variable Selector, select the following:

`Current Node > Input Data > JSON > id`

Now we will specify a user by selecting ourselves from the dropdown list for ***User ID***. Click on the ***Execute Node*** button and you will notice that you will be added to the channel. This node ensures that the specified user is always added to the auxiliary channel created by the workflow.

Here’s a video of me following the steps mentioned above.

{% youtube BN6PXD6FwpA %}

As an exercise, try using the PagerDuty API to pull a list of the email IDs of the people who are on-call and add them to the auxiliary channel in Mattermost. Feel free to pick this up once you are finished with the tutorial.

### 4. Jira Software node: Triage the issue in Jira

Since the playbook specifies that the issue should also be triaged in Jira, we’ll need to add a node that creates a ticket in Jira. To do that, create a Jira node by clicking on the + button on the top right.

In the Node Editor, enter the _Credentials_ for Jira. Here’s detailed [information](https://docs.n8n.io/credentials/jira/) on how you can create a new API Token for the credentials.

Once you are sorted out with the _Credentials_, select the _Project_ where the tickets would be created. I selected a test project that I created specifically for this tutorial. In the ***Issue Type***, I selected ‘Bug’ but feel free to select something else. **Summary*** is a dynamic piece of information, select _Add Expressions_ and pick the `summary` variable just like you did for the ***Display Name*** section while configuring the *Mattermost* node to create a channel.

Click on ***Execute Node*** and this will create a Jira ticket for you. Here’s a video of me following the steps mentioned above.

{% youtube SdlmxP-Q5vc %}

### 5. Mattermost node: Post details in the Incidents channel

The next thing that needs to be done is to post the details of the incident in the Incidents channel. We will need to share the following information in the channel:

1. Summary of the incident
2. Link to the Auxiliary channel
3. Link to the PagerDuty incident
4. Link to the Jira ticket

Sharing these pieces of information will ensure that if someone outside of the on-call team is interested to check out what is going on, they can get this information from the Incidents channel.

To do this, create a new *Mattermost* node. In the Node Editor, select your _Credentials_. Now we need to enter the ***Channel ID***. Since this is not a dynamic piece of information (the Incidents channel would always be there and hence, the ID will remain the same), we need to grab its ***Channel ID***.

If you don’t already have a channel like this for the tutorial, you can manually create a new channel on Mattermost. To get its ID, click on the down arrow next to the channel name and click on the ***View Info*** option. This will reveal the ID of the channel. You can then copy and paste that in the ***Channel ID*** field in the node. In the message section, I entered the following expression to include the information that we mentioned in the list above.

```
🚨 New incident: {{$node["Webhook"].json["body"]["messages"][0]["incident"]["summary"]}}

Auxiliary Channel -> https://mattermost.internal.n8n.io/test/channels/{{$node["Mattermost"].json["name"]}}

PagerDuty Incident -> {{$node["Webhook"].json["body"]["messages"][0]["incident"]["html_url"]}}

Jira Issue -> https://n8n.atlassian.net/browse/{{$json["key"]}}
```

Finally, click on the ***Execute Node*** button to send this information to your Incidents channel. Here’s a video of me following the steps mentioned above.

{% youtube KWYOn2RKg7U %}

### 6. Mattermost node: Post details and action buttons in the auxiliary channel

As a last step of this workflow, we need to provide the information that we talked about in the previous node to the auxiliary channel as well. Moreover, we will need to provide the following two buttons in the channel:

1. **Acknowledge:** Clicking this button will change the status of the incident on PagerDuty from ‘Triggered’ to ‘Acknowledged’.
2. **Resolve:** Clicking this button will change the status of the incident on PagerDuty from ‘Acknowledged’ to ‘Resolved’ and mark the ticket in Jira to ‘Done’.

To do this, create a new *Mattermost* node and connect it to the *Jira* node. This will ensure that this and the previous Mattermost node can run in parallel. In the Node Editor, select your ***Credentials***. Next, you’ll need to enter the ***Channel ID*** of the auxiliary channel. You can follow the steps mentioned in _Workflow 1, Step 3_ to do that. In the ***Message*** section, I entered the following expression (this is quite similar to the _Message_ from the previous node):

```
⚠️ {{$node["Webhook"].json["body"]["messages"][0]["log_entries"][0]["incident"]["summary"]}}

PagerDuty incident: {{$node["Webhook"].json["body"]["messages"][0]["log_entries"][0]["incident"]["html_url"]}}

Jira issue: https://n8n.atlassian.net/browse/{{$json["key"]}}
```

Now, we need to create the buttons which will trigger the actions that we talked about. To do that, under ***Attachments***, click on the ***Add Attachment*** button, click on ***Add Attachment Item***, and select ***Actions***. Then click on the ***Add Actions*** button and name it `Acknowledge`.

Now click on the ***Add Integration*** button. This will allow us to give the URL of the webhook this button will trigger on being clicked. We’ll leave this empty for now.

We’ll also need to send details (to the next workflow) about the PagerDuty incident to mark as resolved when the button is clicked. To do that, click on the ***Add Context to Integration*** button under the ***Context*** section. We’ll enter `pagerduty_incident` as the ***Property Name***. Since the ***Property Value*** is a dynamic piece of information, click on ***Add Expression***. In the Variable Selector, select the following:

`Nodes > Webhook > Output Data > JSON > body > messages > [Item: 0] > incident > id`

Now, add another button called `Resolve` and following the same steps mentioned above. For this button, we’ll need to add the context of the pager duty incident and the Jira ticket key. I’ll leave this as an exercise for you. For the sake of uniformity, you can name the ***Property Name*** `jira_key`.

> In case you were wondering, it is important to send the context with the buttons as there might be multiple auxiliary channels at any given time and multiple people clicking on different Acknowledge and Resolve buttons. We need the correct context so that we don’t close up the wrong PagerDuty incidents and Jira tickets by mistake.

Click on the ***Execute Node*** button to send all this information to the auxiliary channel. Here’s a GIF of me following the steps mentioned above.

{% youtube 5WznYDhhwF0 %}

## Workflow 2 — Make sure that the incident is acknowledged

Our second workflow will cover the fourth step of the playbook. Once all the people responsible get notified that an incident has occurred, we need to make sure that there is a quick and easy way to acknowledge the incident so that it is clear that someone in the on-call team has got it.

Let’s get started with the nodes of the second workflow. I have also submitted [Workflow 2](https://n8n.io/workflows/354) on n8n.io, in case you’d like to skim through this workflow. Please note that you’ll still need to configure a couple of things like your credentials as well as the settings of the nodes.

### 1. Webhook node: Get data from the Acknowledge button

We now need to set up a Webhook node that listens to the event when somebody clicks on the ***Acknowledge*** button in the auxiliary channel.

Create a Webhook node the same way you did in _Workflow 1, Step 1_. Now copy the link of the _Test_ webhook from this Webhook node, go to the node from _Workflow 1, Step 6_ and paste it in the ***URL*** field in the ***Integration*** section of the Acknowledge button under ***Actions***.

Once you are done with that, click on the ***Execute Node*** button to register the webhook and test it by clicking on the Acknowledge button in the auxiliary channel. Here’s a video of me following the steps mentioned above.

{% youtube 9-069p0Q9Co %}

### 2. PagerDuty node: Acknowledge the incident on PagerDuty

Now we need to get the ID of the incident from the webhook node to know which incident to mark as acknowledged. We get this information from the context that we added to the _Integration_ of the button.

Add a *PagerDuty* node by clicking on the '***+***' button on the right side. In the Node Editor view, first of all, you’ll have to enter the _Credentials_ for PagerDuty. Here’s detailed [information](https://docs.n8n.io/credentials/pagerDuty/) on how you can create a new API Token for the credentials. Once you are done with that, select ‘Update’ as the ***Operation***. Since the ***Incident ID*** is a dynamic piece of information, click on _Add Expression_ and select the following in the Variable Selector:

`Current Node > Input Data > JSON > body > context > pagerduty_incident`

In the ***Email*** field, I have just entered my email. In the ***Update Fields*** section, click on the ***Add Field*** button and select _Status_. From the dropdown list in the ***Status*** field, select ‘Acknowledged’. Now, click on the ***Execute Workflow*** button. Go to the auxiliary channel and click on the ***Acknowledge*** button. This will change the status of your incident report from ‘Triggered’ to ‘Acknowledged’. Here’s a video of me following the steps mentioned above.

{% youtube muTOqpLTVbk %}

### 3. Mattermost node: Confirm the acknowledgment

Now we just need to confirm the change of status of the PagerDuty incident by sending a message to the auxiliary channel. I’ll leave this as an exercise for you. In case you run into any troubles, here’s a GIF of me creating this node.

{% youtube 5pem8UUwR64 %}

## Workflow 3 — Make sure that everything is marked resolved after the fix

Our third workflow will cover the sixth step of the playbook. Once the issue has been fixed, we need to make sure that the incident on PagerDuty has been marked as ‘Resolved’ and the ticket on Jira has been marked as ‘Done’. We also need to ensure that everyone in the Incidents and the auxiliary channel is aware of the resolution as well.

Let’s get started with the nodes of the third workflow. The nodes of this workflow have been left as an exercise for you. I have added GIFs for the nodes and have also submitted [Workflow 3](https://n8n.io/workflows/355) on n8n.io, in case you run into any troubles. Please note that you’ll still need to configure a couple of things like your credentials as well as the settings of the nodes.

### 1. Webhook node: Get details from the Resolve button

Just like in the last workflow, we need a *Webhook* node that listens to the event when somebody clicks on the ***Resolve*** button in the auxiliary channel. Here’s a video of me creating this node.

{% youtube Rm6SiK8PpqQ %}

### 2. PagerDuty node: Resolve the incident on PagerDuty

Now we need to change the status of the PagerDuty incident from ‘Acknowledged’ to ‘Resolved’. This is very similar to the _Workflow 2, Step 2_. Here’s a video of me creating this node.

{% youtube qxPaCWBDzns %}

### 3. Jira Software node: Resolve the incident on Jira

Now we need to update the status of the Jira ticket to ‘Done’. Here’s a video of me creating this node.

{% youtube PJDrH--SWRo %}

### 4. Mattermost nodes: Announce the resolution in the auxiliary and Incidents channel

Lastly, we need to create two *Mattermost* nodes:

1. To acknowledge in the auxiliary channel that the incident report on PagerDuty and the ticket on Jira have been resolved.
2. To announce in the Incidents channel that the incident has been resolved.

Here’s a video of me creating this node.

{% youtube W8Ly-jVwLj0 %}

Congratulations, you successfully built an automated incident response workflow using n8n, PagerDuty and Mattermost 🎉

Let’s run the whole system end to end. First of all, you’ll have to click on the ***Execute Workflow*** button on all three workflows to register the Webhook nodes. Go ahead and get started by creating a new incident on PagerDuty.

{% youtube dwpdW5kJdSU %}

Now, to make sure that the workflow runs permanently without you having to press the _Execute Workflow_ on all three workflows before each incident creation, we’ll need to use the _Production_ webhook.

To do that, you’ll just need to get the Production webhook URL from the different Webhook nodes, update the URLs on PagerDuty and the Mattermost node from _Workflow 1, Step 6_, save the workflows and finally activate the workflows. This will make your workflows ready to use.

> **_Note:_** _When working with a_ Production _webhook, please ensure that you have saved and activated the workflow. Don’t forget that the data flowing through the webhook won’t be visible in the Editor UI with the_ Production _webhook._

## Conclusion

Today we created an automatic incident workflow using a variety of n8n nodes. The first-class support for webhooks and APIs allows n8n to integrate a very wide array of services and products, to create powerful workflows in a simplified way. This was an example of automating a minimalist incident response playbook. Which other services are you using for managing incidents in your organization? In case you have created other workflows with n8n that use different nodes, I’d love to check them out, please consider [sharing](https://n8n.io/workflows) those workflows with the community.

In case you’ve run into an issue while following the tutorial, feel free to reach out to me on [Twitter](https://twitter.com/tanay1337) or ask for help on our [forum](https://community.n8n.io/) 💙

> This [post](https://n8n.io/blog/creating-custom-incident-response-workflows-with-n8n/) originally appeared on n8n.io [blog](https://n8n.io/blog/).

* * *