Learn How to Setup Incident Management for Your CI/CD Pipeline
Tags: devops, incidentmanagement, tutorial
Published: 2023-03-06 07:26:24

Every organization today has a team of engineers working hard to keep their systems up and running, and they must create processes for handling incidents that inevitably arise. Even if you have the best developers and site reliability engineers (SREs), managing and addressing these incidents will take time and effort. However, as with everything, incident response is much easier to manage if you have the right tools and strategy. A mix of best-of-the-breed continuous integration and continuous delivery (CI/CD) platforms integrated with the best incident management tool is what every organization needs.

## What is Incident Management? 
Incident management is a process of identifying, investigating, and resolving incidents affecting a service's availability. The incident management process begins with an outage or event that triggers an alert from monitoring tools or from manual detection by operations staff. The incident manager then assesses the impact on customers and the business before deciding how to respond based on the severity of the problem, who is impacted, and how long it will take to resolve it. Incident management is a key component of DevOps. It encompasses the steps that are taken to respond to incidents and reduce their impact on business operations.

## What is PagerDuty?
PagerDuty is a cloud-based IT incident management tool that helps notify and coordinate responses to incidents across an organization. Here are some of the features and advantages of PagerDuty:
- Helps determine the urgency of an incident and assigns it to the engineers accordingly, so they can take appropriate action
- Provides live notifications when there is an update on an incident
- Allows users to assign tasks and track their progress
- Enables notifications for specific incidents

## Tutorial: How to Send PagerDuty Alerts Using Harness

In this tutorial, we’ll demonstrate how to send PagerDuty alerts in case incidents are detected during a pipeline execution. 

When a developer commits a code to a source control management (SCM) system, like GitHub, the CI/CD tool will build and test the code for any misconfigurations, bugs, or vulnerabilities. The code then gets deployed to a target environment, such as Kubernetes. Here, we will configure the alert notification so that when an unsuccessful deployment happens, the pipeline will notify PagerDuty to alert the developers and the team responsible for taking further actions. 

![Pagerduty Harness](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xsq1eg941n09nhosmjji.png)

Harness is the heart of this whole setup, from cloning the code to continuous integration to deploying it to the target environment and alerting the responsible people. 

## Prerequisites:
- [Free Harness cloud](https://app.harness.io/auth/#/signup/?module=cd&utm_medium=social&utm_campaign=community-devrel-post&utm_content=pager-duty-dev-article&utm_term=cd) account
- [PagerDuty free](https://www.pagerduty.com/home/) account
- A [sample application](https://github.com/pavanbelagatti/Simple-Node-App) to connect with the Harness platform (you can fork my repo and use the same)
- A Kubernetes cluster from any cloud provider to deploy our application (you can use [Minikube](https://minikube.sigs.k8s.io/docs/start/) or [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) to create a single node cluster)

**Note**: Built-in notifications are a capability of the Harness platform, and as such, they can be applied to any stage types.

## Harness Setup
If you are new to Harness, you need to have a [free Harness account](https://www.harness.io/products/continuous-delivery?utm_source=internal&utm_medium=social&utm_campaign=community-devrel-post&utm_content=pager-duty-dev-article&utm_term=cd). 

Next, sign in to your Harness account and select the CD module. Set up a simple pipeline to deploy your application. We will use a Kubernetes cluster to deploy our sample application. 
![Harness module](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9wgvhufp8cw6p2xh0751.png)

Add the required connectors for your project, such as your GitHub repository (we use a [sample application](https://github.com/pavanbelagatti/Simple-Node-App); you can fork and use it) and a Delegate. The [Harness Delegate](https://developer.harness.io/docs/platform/Delegates/get-started-with-delegates/delegates-overview) is a service you run in your local network or VPC to connect all of your artifact, infrastructure, collaboration, verification, and other providers with Harness.

Once you set up your project and the pipeline to deploy your application, it is time to set up notifications. On the right-hand side of the dashboard, you will see a notify icon. Click the icon and set up your incident notification strategy. 
![deploy service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1zioung5n0ylkqi5f5po.png)

As we discussed above, I’ll show you how you can integrate the PagerDuty notifications.
![deploy notifications](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l4cfhscjwyqyabdayemj.png)

Add notifications.
![name of the rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gx2wnia5mjvrq7wz9a08.png)

The setup will ask you what event you would like to receive notifications about. I am simply setting up notifications for any deployment, i.e., for any successful deployment of the application/service, notifications will be generated.
![pipeline events](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q2os6x2x9ch4ktoklcob.png)

Next, we will pick PagerDuty as the notification method.
![notification method](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c370hot5lu86ax6agaml.png)

Once you pick PagerDuty as your notifications tool, you are required to provide the PagerDuty Key in the next step.
![events pipe](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fztdxfjgip3yhwpl38e0.png)

To get the PagerDuty key, you need to access your PagerDuty account.

### Setting Up PagerDuty
Create an account on PagerDuty and provide a service name to continue.
![welcome to pagerduty](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jotp6lukmgabjt74fh7i.png)

In the next step, you’ll set up integrations. Go to the “All” tab and search for Harness, and select Harness as shown below.
![add Harness integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dpqtucptwzbm9e1i14zg.png)

The next step will ask you how you would like to get notified. I put my email address. See the example email below sent by PagerDuty.
![Triggered PagerDuty](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1447gnyixblf8qarlvmq.png)

Next, go to Service and click Service Directory to see your services configured. You should see the added service by you. 
![service discovery](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1lqotobrdjx3z31sgzkj.png)

Click on Manage and then on View Integration to see your key.
![integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lqv5v9ikozqglk5q563j.png)

The integration key is what we need. Copy it and save it.
![proper Harness integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/odes22rkpi8599it4m6a.png)

### Adding PagerDuty to Harness
Back in the Harness app, add the key to your PagerDuty notification tab.
![pagerduty key setup](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/letcnw2k5xcji0j3nkss.png)

![pagerduty keys](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b2kbddpe2szchgrmtlpx.png)

Save the changes and apply them. Then, go to your Harness dashboard, and run the pipeline. 
![run pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4t174d5of4uqfvo5i17l.png)

On the successful deployment of our application, we should now see the event notification on PagerDuty and receive an email response.
![successful deploy notify](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9wk2ir36pwp4w2ug0qdp.png)

Similarly, you can customize your workflow and events, assign people to look into the incidents, and more. The PagerDuty integration can help your support and engineering team keep track of what is happening to resolve issues as quickly as possible. 

#### Incident Management is Critical for CI/CD
Once you deploy your service, you need to know what is happening and report different events to your team. Having an incident management tool is an important addition to any CI/CD platform to ensure everything is going as expected. PagerDuty is an excellent example of one of the many notification tools that can be integrated with Harness, taking your incident management to the next level.