Alertmanager configuration with PyYAML
Tags: prometheus, alertmanager, devops, monitoring
Published: 2019-09-09 17:24:51

---
title: Alertmanager configuration with PyYAML
published: true
description: A story about myself trying to get PyYAML and pyyaml-include to work for configuring Alertmanager
tags: prometheus, alertmanager, devops, monitoring
cover_image: https://thepracticaldev.s3.amazonaws.com/i/ftavhrr2rdqd571ne601.png
---

**What my nightmares were like trying to use multiple configuration files with Alertmanager** :skull::eyes:

I recently started working on monitoring some of the internal tools and services at [We Provide](https://www.weprovide.com). I decided to tinker with the "Prometheus, Grafana and Alertmanager" stack as a first attempt.

*Obviously, there we some hiccups... (literally and figuratively)* :hankey:

![I over engineered...](https://thepracticaldev.s3.amazonaws.com/i/lo8hgm03jfsac1yrgsy8.jpg)

Originally I started thinking about a tool I could write, something like `am-stitcher` (*ssooo original, not like `amtool`, at all...*), but... I realized that I was over-engineering something that was basically a matter of merging multiple YAML files into one.

**Google, here I come, gimme something useful or copy-pastable please! :heart:**
I didn't really find anything sufficient for this use-case in terms of literally merging.

Some of the tools I came across did what I wanted them to do, but didn't give me control over in what order to merge them. Some did, but they required me to specify every file individually instead of being able to recursively search a directory.

**PyYAML with pyyaml-include** :love_letter: :two_hearts:
I found an article about `!include` in YAML, and as naive as I am, I thought it was something that might work in YAML natively... But, of course, it didn't!

Support for `!include` is added by the `pyyaml-include` extension, which adds this "directive" to PyYAML.

So... I tried installing PyYAML and the `pyyaml-include` extension using PIP, hoping that the parsing with PyYAML would work with only the CLI, and it'd automatically use the `pyyaml-include` extension. But I couldn't find a way to use PyYAML only in the CLI...

*I should really stop being so naive... Don't you think so too?* :grimacing::sweat_smile:

What I ended up writing was a small Python script that imports PyYAML and uses `pyyaml-include` to parse `!include` "directives".


*Don't be fooled, this was a lot of copy-paste in combination with trial and error!* :fire:
```python
import sys
import os
import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir=os.getcwd())

data = ''

for line in sys.stdin:
    data += line

data = yaml.load(data, Loader=yaml.FullLoader)
yaml.dump(data, sys.stdout, default_flow_style=False)
```

I decided to use the standard input (`sys.stdin`) and output (`sys.stdout`) streams to read and write. I didn't want to bother with using arguments when I didn't even know the Python syntax :sweat_smile:

```python
python parse-yaml.py < alertmanager.yml > merged.yml
```

**Shall we add a tiny bit of automation?!** :raised_hands:

At We Provide we mostly use Bitbucket, for open-source we use GitHub (obviously...), but for most of the stuff we use Bitbucket.

It consists out of three steps, "Parse & Merge", "Validate" and "Deploy". I think those are kinda self-explanatory, don't you think? :blush:


```yaml
pipelines:
  default:
    - step:
        name: Parse & Merge
        image: python:3.5.1
        script:
          - pip install pyyaml pyyaml-include
          - python parse-yaml.py < alertmanager.yml > merged.yml
        artifacts:
          - merged.yml
    - step:
        name: Validate
        image: serializator/amtool
        script:
          - amtool check-config merged.yml
    - step:
        name: Deploy
        image: eeacms/rsync
        script:
          - rsync merged.yml $USER@$HOST:$ALERTMANAGER_YAML
          - curl -L -X POST $HOST$RELOAD_PATH
```

As you can see, one of the images is `serializator/amtool`. Originally I tried using the [`amtool-docker` image from Cisco Metacloud (Anthony Rogliano)](https://hub.docker.com/r/metacloud/amtool-docker). But the fact that it was built on top of `golang:1.9-alpine3.6` caused issues with the usage of `/bin/bash` instead of `/bin/sh` in the Bitbucket pipeline. 

*^ If you know a solution for this, or if I overlooked something in the Bitbucket pipeline that I could've configured, please let me know!* :heart:

What I ended up doing was forking the repository and build the image on top of `golang:1.13-stretch` instead of `golang:1.9-alpine3.6`.

*^ I thought about the increase in size by not using the alpine tag, but it's negligible in my opinion, looking at the use-case.*

**I'm in love with the "/-/reload" endpoint** :see_no_evil::speak_no_evil:
I thought that I'd had to use SSH and use Docker Compose to restart the Alertmanager service, but it luckily didn't end up being that ugly.

Somewhere in 2016 the `/-/reload` endpoint was introduced by [ZhenyangZhao](https://github.com/ZhenyangZhao), and I'm glad he made that pull request! I don't know how I'd live with myself knowing that one of my deployments didn't have cURL in it...

So, without much hesitation, I quickly wrote one line more to send a POST request to the `/-/reload` endpoint, configured some stuff using environment variables, and pushed it to master!

**I didn't know that this kinda happiness existed!** :blush::fire:
I finally laid down on bed, closed my eyes and began dreaming about the many files, folders and `!include`'s I was going to write with a happy smile on my face! :laughing:

I hope you got something useful out of this, and if not, I hope you at least enjoyed reading it!

I thank you, @yamadashy for [this emoji cheatsheet :heart:](https://dev.to/yamadashy/emoji-cheatsheet-of-githubslackdiscordqiita-4d2f)