Auto Close Dynatrace alert
Asked by Donny Gison:

We are using Dynatrace API to send Alerts to OpsGenie. Alerting works, but seems we are unable to auto close when DT alert is resolved. Any recommendations how to enable, auto closing based on DT resolution alert?

Does it conflict with Schedule's acknowledgement?


Answers:
author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)

This is Mubeen. I will be happy to help you.

As I understand you want to automatically close the alert when it's resolved in Dynatrace.  

You can achieve this by using the Dynatrace "Advanced" integration settings. Please note advanced settings for integrations are available only on the Standard and Enterprise plans.   

From the Advanced Integration settings, you can set a "Close Alert" action which will close an existing Opsgenie alert based on its alias. The functionality to close an alert is reliant upon the "Alias" field of the alert. Here is the documentation to help understand the "[Advanced Integration Settings](https://support.atlassian.com/opsgenie/docs/use-advanced-integration-settings/)"   

<br />

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian

comments: author: Donny Gison
body: [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601)

Seems to be set-up but not working? Thanks.

![Screen Shot 2022-03-13 at 1.13.21 AM.png](https://community.atlassian.com/t5/image/serverpage/image-id/187884iE27DC13A2BE28361/image-dimensions/507x383?v=v2 "Screen Shot 2022-03-13 at 1.13.21 AM.png")


author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)   
Hope your day is great,  
As I can see from the screenshot shared you have just set the initial close alert rule.   

So as mentioned in my previous response, you can set a "Close Alert" action which will close an existing Opsgenie open alert based on its "alias" value.

<br />

\[The alias is the unique identifier for "open" alerts. There can only be one "open" alert with the same alias in Opsgenie.\]

<br />

So for the Close Alert rule to work, the below conditions are required to match, and the setup will be as follows.  

1. You will initially configure a "Create alert" action. A "Create Alert" action will create a new Opsgenie alert if the condition set of its filter matches the data. You need to make sure to set a custom "Alias" value for this "Create alert" action.

![Create Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188002i4920D449CCAB175C/image-size/large?v=v2&px=999 "Create Alert.png")  
2. A "Close Alert" action will close an existing Opsgenie alert based on its "Alias" which means, for the "Close alert" action to work you need to have an Open alert with the same Alias field value configured in both "Create alert" and Close alert" action.  
![Close Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188003iBB61A6268DA0BF60/image-size/large?v=v2&px=999 "Close Alert.png")  

3. The actions are evaluated in order, each time the alert payload reaches Opsgenie it will be first evaluating the first action configured under your account.   

<br />

Ex: You have created "Create alert -1", "Create alert -2", "Close alert -1" and "Close alert -2" actions under your integration settings.   

Alert triggered from Dynatrace,   
- It first checks the "Create alert -1" action if it matches alert will be created   
- if not then It will go to the next action "Create alert -2" if this alert action is matching then based on its rules alert will be created. If there is already an alert open with the same information It will get de-duplicated.  

- If not it will then route to "Close alert -1" at this point if this action matches then Opsgenie will check if there is any open alert with the "Alias" matching, and It will close that alert accordingly and It will not check any further with the last action "Close alert -2"  

For any action set up for integration, an Alias value must be defined and matched exactly in order for the action to be executed.  

Hope this information is helpful, please let us know if you have any further questions. I will be happy to assist.  

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian


author: Donny Gison
body: Hi [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601) ,

That's helpful. Made me understand Actions more. May I please seek further feedback.

In your example the Alias to match is hard coded. What if it changes like DT problem number.

DT Alert Creation Message -

\[PROD\] SmartQ UW: OPEN Problem P-220343162

DT Alert Closure Message -

\[PROD\] SmartQ UW: RESOLVED Problem P-220343162

May Alias values is likely not right :)

![Screenshot 2022-03-15 174147.png](https://community.atlassian.com/t5/image/serverpage/image-id/188160i3B9388B13641D2C6/image-size/large?v=v2&px=999 "Screenshot 2022-03-15 174147.png")


author: Donny Gison
body: Apologies :) I think I misunderstood the value for Alias. I have updated and trying the below .

Just waiting for the next alert to test it out. Opted to not use Create Alert, for now, to not mess with the count of alerts.

![Screenshot 2022-03-15 174931.png](https://community.atlassian.com/t5/image/serverpage/image-id/188163iC531CEC1147C8BB2/image-size/large?v=v2&px=999 "Screenshot 2022-03-15 174931.png")


author: Donny Gison
body: Unfortunately this did not work as well.

Won't the DT Alert for Resolved be logged somewhere as well? We seem to not be getting one. The closure stopped for us sometime Feb. Now I am wondering if its the Integration API. Not sure

Here is a sample log when the Auto Closure from a Resolved DT alert was still working when received, hope this helps.

![Screenshot 2022-03-15 191949.png](https://community.atlassian.com/t5/image/serverpage/image-id/188177i0D58F7DEBE5C9B16/image-size/large?v=v2&px=999 "Screenshot 2022-03-15 191949.png")


author: Donny Gison
body: The creation seems to work. We just had 2 DT alerts today that Opsgenie was able to tag duplicate.

![Screenshot 2022-03-15 192654.png](https://community.atlassian.com/t5/image/serverpage/image-id/188179i49EBCA61A3269618/image-size/large?v=v2&px=999 "Screenshot 2022-03-15 192654.png")

We got 2 DT alert emails with different email subjects but the same body of text, Opsgenie was able to have them in one alert only, just increased the count to 2.

Creation works including duplicates but closure doesn't. I still don't fully get Alias haha

Here's the Create set-up. Maybe I need to define exactly the same fields for Close alert?

Close just have Alias, User and Notes.

Help will be appreciated.

![Screenshot 2022-03-15 193009.png](https://community.atlassian.com/t5/image/serverpage/image-id/188181i95D71F161F403768/image-size/large?v=v2&px=999 "Screenshot 2022-03-15 193009.png")


author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)

I can help explain the concept of "Alias", as mentioned earlier The **Alias** field is a **user-defined unique identifier for "open"** alerts.   

From the last deduplication message screenshot you sent, you can see it has the Message: ++**"\[Dynatrace\]: \[PROD\]: SmartQ: U/W ................... low disk space"**++

And you can see the count increased to 2. You got this as "Alias" because you have set the "Create alert" action **alias** field as "\[Dynatrace\]: {{message}}".  

Since the alert was received with the **same alias value** and it matched with the "Create alert" action so it got deduplicated.   

**Please note:** As mentioned earlier in my previous response,  
The payload received from Dynatrace will be checked in the order your alert actions are created.  
![Screenshot 2022-03-16 at 12.03.46 PM.png](https://community.atlassian.com/t5/image/serverpage/image-id/188351iBB55957436468A5A/image-size/large?v=v2&px=999 "Screenshot 2022-03-16 at 12.03.46 PM.png")   

As per the order, the first section is "Create alert" when the alert payload comes and it matched with the first action "Create alert" it will create the alert. So you need to send a different payload that matches only with "close alert" to be able to close the alert which is already created with the same "Alias" value.   

<br />

Incoming payload from Dynatrace \> -- matches with -- \> "Create alert" = Alert will be created.

Incoming payload from Dynatrace \>-- matches with -- \> "Create alert" and an alert is already open = It will be deduplicated  

Incoming payload from Dynatrace \>-- matches with -- \> "Close alert" and an alert with the same alias value is already in open state = Alert will be closed.   

For your testing try to set the "alias" values in both create alert and close alert action as I specified in the previous response screenshots.   

Hope this answers your queries.   

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian

<br />

<br />


author: Donny Gison
body: Hi [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601)

Thanks for continuing to respond. I am slow in understanding Aliases I think. Let me try the below.

* Create Alert

![Create Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188601iFDB266ED693993A6/image-size/large?v=v2&px=999 "Create Alert.png")

* In the sample deduplication, the Message are not actually identical.

1st DT Alert has "Asia ESB"

![DT alert for Create 1.png](https://community.atlassian.com/t5/image/serverpage/image-id/188602i89EB241EB525B71F/image-size/large?v=v2&px=999 "DT alert for Create 1.png")

2nd DT alert that became deduplicated has "SmartQ UW"

![DT alert for Create 2.png](https://community.atlassian.com/t5/image/serverpage/image-id/188603i9974CF0DB8952760/image-size/large?v=v2&px=999 "DT alert for Create 2.png")

Opsgenie was successful to create and deduplicate even with some words different on the description of the 2 DT alerts.

Now, for Close alert, DT sent these

For the first alert - different is just the word "RESOLVED" instead of "OPEN"

![DT alert for Resolved 1.png](https://community.atlassian.com/t5/image/serverpage/image-id/188604i733DFAB01DBA8F2E/image-size/large?v=v2&px=999 "DT alert for Resolved 1.png")

For the second alert - different is just the word "RESOLVED" instead of "OPEN"

![DT alert for Resolved 2.png](https://community.atlassian.com/t5/image/serverpage/image-id/188605iC6D5FBF88E4F8A09/image-size/large?v=v2&px=999 "DT alert for Resolved 2.png")

This confused me on how Alias works. Deduplicate was able to work with different words on the Description of the 2 DT alerts received. But Close was not able to with just "Resolved" as the difference. I might be understanding Alias incorrectly.

We tried adding a filter of looking for "Resolved" in the Close alert because the default of just Alias someshow stopped working in February.

![Close Alert .png](https://community.atlassian.com/t5/image/serverpage/image-id/188606iB36D7E8E781476BF/image-size/large?v=v2&px=999 "Close Alert .png")

Maybe given the DT alerts we receive you can be very specific on what may work as Close alert set-up? To make it work again. And other Action set-up I need to do based on what we get above.

Also, is there a way to confirm that the "Resolved" alert from DT did reach Opsgenie to go through the Actions? This is one thing I can't find so far.

Hope you continue to help me solve this. Thank you.


author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)   
No worries I can help clarify your queries with Alias.

<br />

For the alerts which got deduplicated with the Alias field and from the screenshot I see you have just set "{{alias}}". To help understand how you got two different alerts message content was de-duplicated, I have provided the information in my previous responses which is a common setup for most of the Opsgenie configuration.  

But your recent query requires checking configuration and logs within from your Opsgenie instance because it will be a unique setup/configuration for individual instances. You can reach out to me at <https://support.atlassian.com/contact/#/> so I can further check account-specific details to help further.   

Also, I would like to share a specific example configuration that will allow you to create and close an alert, so you can test this further to understand/implement it accordingly.  

For our example, let's just consider you are getting the following data for an instance state change and you need to set Create and Close actions accordingly.

Ex:   
Server.abc is currently not responding. (Create alert)  
The issue with Server.abc is resolved now. (Close alert)  

Initial setup   
![Screenshot 2022-03-17 at 3.52.14 PM.png](https://community.atlassian.com/t5/image/serverpage/image-id/188618i20068706E86FE125/image-size/large?v=v2&px=999 "Screenshot 2022-03-17 at 3.52.14 PM.png")  

As per this setup when I receive the payload as "Server.abc is currently not responding" it will create an alert for me as I have set a message containing "not responding" as a condition to get the alert to create.  

<br />

Now the "close alert" configuration setup in relation to the alert created by the above "Create alert" condition.

![Screenshot 2022-03-17 at 3.53.15 PM.png](https://community.atlassian.com/t5/image/serverpage/image-id/188619i03AFDBBA2CF41D20/image-size/large?v=v2&px=999 "Screenshot 2022-03-17 at 3.53.15 PM.png")  

As you can see from the screenshot I have set the condition as a message that contains "resolved now" should match and with the same alias value if there is any open alert then close it.   

Hope this clarified the Alias role and alert conditions. You can try to test with the exact value to understand how it works.   

Just want to add another payload example in relation to the examples shared above to clear your understanding.  

Let's say you have sent the **resolved** payload a little differently as follows **"The server.abc which was not responding that is resolved** now"  

This technically resolves the alert as it has "**resolved now"** in it, **but it will not** instead it will **de-duplicate** the alert. Because before the alert condition match the "Close alert" action it matched with the "Create alert" action "**not responding"** hence it de-duplicated**.
<br />** I have specified in my previous responses Payload checks the alert action in order, the first action it matches will be executed and it will stop from there.   

<br />

<br />

Hope this example clarified your query with "Alias".   

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian

<br />

<br />


author: Donny Gison
body: Hi [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601) , I raised

<https://getsupport.atlassian.com/servicedesk/customer/portal/45/OGSP-86337>

to investigate the logs and confirm that DT Resolved alerts do reach our Opsgenie instance.

Tried defining an Alias of \[PROD\] for both Create and Close, unfortunately it did not work.

![Prod Create Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188673iA072EC7C2FEF6606/image-size/large?v=v2&px=999 "Prod Create Alert.png")

![Prod Close Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188674i689D54DC6E06AE31/image-size/large?v=v2&px=999 "Prod Close Alert.png")

The DT alerts

For Create

![PROD Open.png](https://community.atlassian.com/t5/image/serverpage/image-id/188675i9C6C1B1927B72988/image-size/large?v=v2&px=999 "PROD Open.png")

For Close, but seem to not consumed by Opsgenie

![PROD Resolved.png](https://community.atlassian.com/t5/image/serverpage/image-id/188676i29748B8CFAC1C638/image-size/large?v=v2&px=999 "PROD Resolved.png")


author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)   

Yes, I got the ticket you have created, I will further help with the ticket you have opened :)   

Also to clarify on the screenshot you shared above for the "create alert" action rule you have mentioned the filter condition as "**Match all alerts** "   

![Prod Create Alert.png](https://community.atlassian.com/t5/image/serverpage/image-id/188796iBA2BDD541F8135C3/image-size/large?v=v2&px=999 "Prod Create Alert.png")  

<br />

As mentioned in my previous responses all the Alert actions for CREATE or CLOSE will be checked in order, if the alert payload matches with the first one then the next alert action will not be checked.   

Since you have set the alert filter as "**Match all alerts** " it will match with all alerts incoming payload, including the "Resolved" payload, hence no "Close Alert" actions were executed.  

Instead of "**Match all alerts**" you can set a specific value as one of the alert conditions to test it again.

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian


author: Donny Gison
body: Noted [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601)

We can explore the Match alert options for Create next. For now, let's analyze first the Resolved alerts from DT via the raised ticket. If Opsgenie is receiving them, and if possible to have a view of what we are receiving on the API. Thanks.


author: Mubeen Mohammed
body: Sure :)  
I have already sent the required information on the ticket to further investigate this.  

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian


author: Donny Gison
body: Thanks for all the help [@Mubeen Mohammed](/t5/user/viewprofilepage/user-id/4779601) , our DT team has switched to webhook API and closure is now working. We'll just have to work on our Policies to ensure it's being routed to the right schedule and people.


author: Jason Selby
body: Was this an issue with Dynatrace or Atlassian ? I am also experiencing it


author: Mubeen Mohammed
body: Hello [@Jason Selby](/t5/user/viewprofilepage/user-id/3356262)   
This is Mubeen. I will be happy to help you.  

If you are currently facing an issue with closing the Opsgenie alert when the original alert is resolved in Dynatrace, it is possible that your Opsgenie configuration settings for the Dynatrace integration are not configured correctly or it is also possible that the alert closure payload from Dynatrace is not reaching Opsgenie to work as per your Integration settings.   

- You can cross verify the validation options discussed in this ticket to check accordingly.

- From the Opsgenie debug logs section validate if the Dynatrace alert closure payload is reaching Opsgenie. (If not you are required to work with the Dynatrace admin who is currently working on your Dynatrace configuration)

- If you are able to receive the closure alert payload from Dynatrace to Opsgenie(By checking in Opsgenie Debug logs) then you can validate the "Close Alert" action in the Dynatrace integration settings of your Opsgenie account.

Hope this answers your queries.   

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian


author: Jason Selby
body: Mohammed,

The issue i am having has been introduced recently, i have checked everything above and it is setup in the correct way, the close just doesnt work. Would it be possible to have a support call?


author: Mubeen Mohammed
body: Hello [@Jason Selby](/t5/user/viewprofilepage/user-id/3356262)   
This requires checking configuration and logs within your Opsgenie account because it will be a unique setup/configuration for individual instances. You can reach out to us at <https://support.atlassian.com/contact/#/> with the complete integration/alert details so we can help check/investigate the issue accordingly.

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian


author: Mubeen Mohammed
body: Hello [@Donny Gison](/t5/user/viewprofilepage/user-id/1165413)   
Thank you for the confirmation and I am glad it is working as expected for you now.  

Regards,

Mohammed Mubeen  
Support Engineer, Atlassian