Zulip Datadog Integration
Tags: datadog, zulip
Published: 2021-08-02 02:59:22

# Zulip Datadog Integration

This is to record my findings during the research for Zulip Datadog integration.  
TL;DR is at the end  

## Slack Compatible Incoming Webhook
Might be used for custom format message on Slack too though I have not tested that  
Also the formatting on Zulip is still limited and not to mention the translation of Slack message formatting to Zulip one  
After testing this I go for native one instead  


To integrate Datadog with Zulip, the webhook integration on Datadog should be used  
And the payload can be sent to one of your Zulip organization's endpoint which is either:
- Slack Compatible Incoming Webhook
- Zulip Native API - Send Message Endpoint

**References**
- https://docs.datadoghq.com/monitors/notifications
- https://docs.datadoghq.com/integrations/webhooks
- https://zulip.com/integrations/doc/slack_incoming

**URL**
https://your_org.zulipchat.com/api/v1/external/slack_incoming?api_key=your_bot_api_key&stream=your_stream_name

### Monitor Notification
The following one only tested for Monitor Notification  
Not sure what else a webhook can be used for  

**Example Payload**
Example JSON payload with similar format to DataDog Slack Integration one  
Can be verified via https://app.slack.com/block-kit-builder  
```json
{
	"blocks": [
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "<https://duck.com|Triggered: CPU load is very high on your.host.name -  - [non China]>"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "Check CPU usage just in case  @slack-datadog"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "system.cpu.user over host:your.host.name was > 70.0 on average during the last 5m."
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "Metric value: 84.513"
			},
			"fields": [
				{
					"type": "mrkdwn",
					"text": "*Tags*\nhost:your.host.name"
				},
				{
					"type": "mrkdwn",
					"text": "*Notified*\n@slack-datadog"
				}
			]
		},
		{
			"type": "image",
			"image_url": "http://placekitten.com/700/500",
			"alt_text": "Multiple cute kittens"
		}
	]
}

```


**Actual Payload**
```json
{
	"blocks": [
		{
			"type": "header",
			"text": {
				"type": "mrkdwn",
				"text": "TEST MESSAGE ONLY"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "<$LINK|$EVENT_TITLE>"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "$EVENT_MSG"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "$ALERT_STATUS"
			}
		},
		{
			"type": "section",
			"text": {
				"type": "mrkdwn",
				"text": "Metric value: {{value}}"
			},
			"fields": [
				{
					"type": "mrkdwn",
					"text": "*Tags*\nhost:$HOSTNAME"
				},
				{
					"type": "mrkdwn",
					"text": "*Name of the metric*\n$ALERT_METRIC"
				}
			]
		},
		{
			"type": "image",
			"image_url": "$SNAPSHOT",
			"alt_text": "$ALERT_METRIC: {{value}}"
		}
	]
}

```


## Zulip Native API - Send Message Endpoint
Notice: due to an issue in Markdown image syntax implementation  
it's impossible to put inline image in one message with presence of other content  
(only shows and image in link format when one message contains **only** the link for an image)  

**URL**
https://your_bot_email_address:your_bot_api_key@your_org.zulipchat.com/api/v1/messages

### Monitor Notification

**Example Payload**
```json
{
  "type": "stream",
  "to": "your_stream_name",
  "topic": "Castle",
  "content": "I come not, friends, to steal away your hearts."
}
```

#### Message Topic
`$ALERT_TITLE`

#### Message Content
```md
[$EVENT_TITLE]($LINK)

$EVENT_MSG

**Alert Status**
$ALERT_STATUS

[$ALERT_METRIC Snapshot]($SNAPSHOT)"
```

**Actual Payload**
This is webhook payload content be sent in **encoded form**  
(Check `Encode as form` when creating Datadog Webhook, Zulip API requires it)  
JSON disallows multiline strings so `\n` is used instead
```json
{
  "type": "stream",
  "to": "your_stream_name",
  "topic": "$ALERT_TITLE",
  "content": "[$EVENT_TITLE]($LINK)\n\n$EVENT_MSG\n\n**Alert Status**\n$ALERT_STATUS\n\n$ALERT_METRIC Snapshot Image:\n$SNAPSHOT"
}

```

**References**
- https://zulip.com/api/send-message
- https://zulip.com/help/format-your-message-using-markdown
- https://github.com/zulip/zulip/issues/16444

## TL;DR
It seems using Zulip Native API - Send Message Endpoint is better for Datadog integration  
Though the current status of Zulip image handling is frustrating  
The general formatting of messages also needs significant improvement to be "nice/comfortable to use"  

I will keep on finding a better messaging solution with channel/stream first design  
And maybe find another solution or fallbacks to Slack for service related notifications