How to set Opsgenie alert priority from Prometheus alert priority
Asked by Ashma Shriyan:

Hello

I have the priority level in Prometheus set as 'severity' with values low,medium and critical. I tried using Opsgenie's 'Custom Value to Opsgenie priority' option with value {{severity.substring(0,2)}} but that does not seem to change the priority level of the Opsgenie alert. What am I missing here? Can someone please help me.

Thanks!


Answers:
author: Shiv Pathak
body: Hi Folks,

Posting my configuration which works well with severity mapping to priority in opsgenie as follows:

critical =\> P1

warning =\> P2

info =\> P3

none =\> P4

Configuration for Alertmanager I used as follows:

```
global:
  resolve_timeout: 1m
  opsgenie_api_url: https://api.eu.opsgenie.com/
  opsgenie_api_key: <my-top-secret-api-key>
receivers:
  - opsgenie_configs:
    - teams: <my-team-name>
      priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'
    name: opsgenie
route:
  group_by: ['...']
  receiver: opsgenie
  repeat_interval: 5m
```

I tested this with Opsgenie free tier with Prometheus Alertmanager.

Please mind indentations while configuring.

**Final Note:**please input api-key, team name accordingly, also group_by etc fields are not optimised, I tested this with bare 4 alerts having specific severity what I mentioned above.

comments: author: Prabhu Thangaraj
body: ```
global:
  resolve_timeout: 1m
  opsgenie_api_url: https://api.eu.opsgenie.com/
  opsgenie_api_key: "XXXXXXXXXX"
receivers:
  - opsgenie_configs:
    - responders:
        - type: team
          name: "<team name>"
    priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'
  name: opsgenie
route:
  group_by: ['alertname','service']
  receiver: opsgenie
  repeat_interval: 5m
```


author: Prabhu Thangaraj
body: The above configuration worked for me i was working with a Atlassian Support, and we tried the above config it worked for us


author: Jason Walton
body: The above will work, but only if if you only have one alert! The problem is this line:  

```
priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else if eq .Labels.severity "warning"}}P2{{else if eq .Labels.severity "info"}}P3{{else}}P4{{end}}{{end}}'
```

<br />

The "range" here will iterate over all alerts, so if you have more than one alert this will generate a priority like "P2P2", which opsgenie will reject:  

```
level=error ts=2022-03-09T14:59:23.350Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="opsgenie/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 422: {\"message\":\"Request body is not processable. Please check the errors.\",\"errors\":{\"priority\":\"should be one of [ P1, P2, P3, P4, P5 ]\"},\"took\":0.001,\"requestId\":\"c4390ce3-f99c-4149-820d-1886421ef213\"}"
```

<br />

I'm using this configuration to group together alerts of the same priority and send them to opsgenie as a batch:

```
receivers:
  - name: "null"
  - name: "opsgenie - critical"
    opsgenie_configs:
    - priority: P1
  - name: "opsgenie - warning"
    opsgenie_configs:
    - priority: P2
  - name: "opsgenie"
    opsgenie_configs:
    - priority: P3
route:
  receiver: "null"
  group_by:
    - job
  routes:
    # Alerts in this receiver are ignored.
    - receiver: "null"
      match:
        alertname: Watchdog
    - receiver: "opsgenie - critical"
      match:
        severity: "critical"
    - receiver: "opsgenie - warning"
      match:
        severity: "warning"
    - receiver: "opsgenie"
```


author: Daniel Leite
body: My prometheus+alertmanager work fine with priorities!

Example:

in prometheus, i have this alert:

```

 - alert: Instance_Free_Inodes
   expr: (disk_inodes_used * 100 / disk_inodes_total) > 75
   for: 5m
   labels:
     severity: page
     alert_target: '{{ $labels.host }}'
   annotations:
     description: '{{ $labels.alias }} of job {{ $labels.job }} ({{ $labels.path }}) has ({{  $value | printf "%.2f" }})% used inodes for more than 5 minutes.'
     summary: {{ $labels.alias }} using more than 75% inodes on {{ $labels.path }}
     impact: Services may fail
     action: Check the server disks and remove some files, if possible
     dashboard: https://grafana.example.com/d/pjhLJOzmk/infrastructure-hosts-stats-3hours
     priority: P3
```

Then in alertmanager:

```

- name: opsgenie_techops_team
? opsgenie_configs:
??? - send_resolved: true
????? http_config: {}
? ? ? api_key: <secret>>
? ??? api_url: https://api.opsgenie.com/
????? message: '{{ template "opsgenie.default.message" . }}'
? ? ? description: '{{ template "opsgenie.default.description" . }}'
? ? ? source: '{{ template "opsgenie.default.source" . }}'
? ? ? responders:
? ? ? - name: techops_team
?? ????? type: team
????? priority: '{{ if .CommonAnnotations.priority }}{{ .CommonAnnotations.priority
}}{{ else }}P3{{ end }}'
```

This way i can setup the priority in the prometheus alert and it is pushed to opsgenie correctly


author: Nick H
body: With [Prometheus' Alert Manager](https://docs.opsgenie.com/docs/prometheus-integration#section-configuration-on-prometheus) config file you should be able to map the severity field with something customer like this:  

priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else}}P3{{end}}{{end}}'  

You can always change "P1" "P3" to their preference.

![image.png](https://mail.google.com/mail/u/0?ui=2&ik=7b12d6bccc&attid=0.1&permmsgid=msg-a:r-142476994724882587&th=16e1918bf427edda&view=fimg&sz=s0-l75-ft&attbid=ANGjdJ_j4pGjrDf-WZl_0kondQ7HCdXfJEl-dY9vdxh_OxoHBWOk5feDgzOz0raCo04sAykXGtHTNwCgf71S8zFU7d8nTTIUrhhDkfIeW48Aj7Q3Dkl7916xrV5_14E&disp=emb&realattid=ii_k2c9rk3b0)

comments: author: Ashma Shriyan
body: [@Nick H](/t5/user/viewprofilepage/user-id/2540930) Thanks! I tried that out and it maps priority to P1 for critical alerts but I still wonder why does the one on Opsgenie portal did not work out. ie, Setting 'Custom Value to Opsgenie priority' option with value {{severity.substring(0,2)}}


author: Ashma Shriyan
body: Any update on this yet?


author: Nick H
body: I'm realizing Prometheus unfortunately is not an integration that can have a custom value to Opsgenie property configured such as a value {{severity.substring(0,2)}}:

![priority image.jpg](https://community.atlassian.com/t5/image/serverpage/image-id/73219iD40A90CE1F384754/image-size/large?v=v2&px=999 "priority image.jpg")

So the best option is mapping this on the Alert Manager side like shown above. We also see customers using Alert Policies to parsed for the severity field, and modify/map the alert's priority this way, but these policies are only included in our Standard and Enterprise plans: <https://docs.opsgenie.com/docs/global-policies#section-alert-policy>


author: Ashma Shriyan
body: apologies, but i'm not satisfied with your answer. If you say Prometheus is not an integration that can have a custom value to Opsgenie property, why is the option 'Custom Value to Opsgenie priority' even included in Prometheus integration page? If that cannot have values like '{{severity.substring(0,2)}}' what other values can it have? Can you please give me an example so that I understand the feature better?


author: Nick H
body: It's not an option (unless you are seeing something else with your integration):

![priorty 2.jpg](https://community.atlassian.com/t5/image/serverpage/image-id/73237i9C93B983A86C3B54/image-size/large?v=v2&px=999 "priorty 2.jpg")

Prometheus is using an API key to post to Opsgenie and because of this, looking for the values that are priority is (P1-5).


author: Ashma Shriyan
body: This is what I see and I would like to know how to use it and what values does it take.![Opsgenie.png](https://community.atlassian.com/t5/image/serverpage/image-id/73326i92E26BC1604B8597/image-size/large?v=v2&px=999 "Opsgenie.png")


author: Nick H
body: You can use this option (Custom Value to Opsgenie Priority) with other webhook-based integrations. Unfortunately it does not work with our API integrations such as Prometheus.

Prometheus is an integration that sends notifications through our API (<https://prometheus.io/docs/alerting/configuration/#opsgenie_config>) and due to this - Opsgenie will be looking for exactly this value being sent from the alert API.

So for your use-case, you'd need to map this in Alert Manager (since you are on a Free plan with limited functionality to other ways of mapping this in Opsgenie).


author: Nick H
body: Hi Ashman,

I think you are on the right track, but missing one character. Can you try this string instead:

P{{severity.substring(0,2)}}

The priority is touchy in this sense that it needs a hardcoded 'P' to map to our priorities.

Let us know if this does the trick! If not, I may need to take a look at what the logs are parsing or what the severity looks like.

comments: author: Ashma Shriyan
body: [@Nick H](/t5/user/viewprofilepage/user-id/2540930) Thanks for the response. I tried what you suggested but that did not seem to work. Please take a look and let me know about the workaround. Thanks!


Tags: atl-orange