Managing health checks at scale
Tags: architecture, devops, sre, showdev
Published: 2021-02-05 17:39:00

At [15Five](https://www.15five.com), like many other companies, we use cron to schedule regularly occurring jobs. If a job fails to complete at the expected time or exits with the error code then we get an alert via [healthchecks.io](https://healthchecks.io).

![failure healthcheck alert](https://dev-to-uploads.s3.amazonaws.com/i/bitubmqacvhty6ras6do.png)

For a while this worked pretty well.

But soon we faced a challenge. We wanted to expand health checks to all of our testing  environments, but that would require creating hundreds and hundreds of new checks. That's a lot of manual work, and why work when you can automate? 

I pondered different solutions like writing a script to [copy the checks from one project to another](https://github.com/healthchecks/healthchecks/issues/266), or a script to parse all the ping endpoints into a format that could easily be inserted into our codebase, or managing the health checks [via terraform](https://github.com/kristofferahl/terraform-provider-healthchecksio). But none of these solutions were perfect. I wanted something to completely automate the process.

Fortunately, after talking with the Healthchecks  maintainer, it turns out they [already had a basic script for that](https://github.com/healthchecks/healthchecks/issues/343#issuecomment-599259779). Every time a job completed, it would do a simple call to the Healthcheck API, creating the check if it didn't already exist and returning the check endpoint. Ping the endpoint, and you're done! Easy peasy, no human work involved.

```shell
#!/bin/bash

API_KEY=your-api-key-here

# Check's parameters. This example uses system's hostname for check's name.
PAYLOAD='{"name": "'`hostname`'", "timeout": 60, "grace": 60, "unique": ["name"]}'

# Create the check if it does not exist.
# Grab the ping_url from JSON response using the jq utility:
URL=`curl -s https://healthchecks.io/api/v1/checks/  -H "X-Api-Key: $API_KEY" -d "$PAYLOAD"  | jq -r .ping_url`

# Finally, send a ping:
curl -m 10 --retry 5 $URL
```

I expanded on this to work with alert channels, measure start and end times, and to record failures.

{% gist https://gist.github.com/caleb15/1a817ef5e58e8a8caf65190cff33806e %}

Install [jq](https://stedolan.github.io/jq/), replace `healthchecks_api_key` with your API key and ship the file with your cron jobs. Now you can pass the cron command to the script and it will do all your work for you. Creating a new environment? Just update the API key and you're good to go!

We use Ansible to install jq and templatize the crontab and healthcheck file. Feel free to use Puppet instead, or manual distribution, or docker, or heck, redstone blocks in Minecraft, who am I to judge?

This has been working well in our production environment for months now. As a bonus when we go into Healthchecks.io we can see a detailed history of job start and end times:

![Image of start and end pings with timing](https://dev-to-uploads.s3.amazonaws.com/i/wfjoo4bohkxuqosqfulr.png)

It's great to be able to set up a new datacenter and take comfort in a vibrant field of solid green checks, and the only thing I had to do to set it up was to update an API key!

<!--

Runtime?

A 15-5 we have about 100 different cron jobs we needed to monitor. We use healthcheck.io to alert us when the jobs failed or stopped running. Whenever we needed a new cron job a engineer would ask me for the ping URL and I would give it to them and they would place it in the code. Easy peasy.

But I wanted to expand the checks to several new environments. I could manually recreate all the checks, again and again and again, hundreds of times. Afterwards I would need to keep all the checks in sync, and creating a new check would require me repeating myself

But in the process of migrating to terraform we were recreating all our environments. 

Talk about problem. Need consistent scalable way of managing and creating health checks across different environments.

Health checks through Terraform?
https://github.com/kristofferahl/terraform-provider-healthchecksio

Problem is that healthchecks are more of a application-engineering thing than a devops / infrastructure thing. Also how would my code know what healthcheck url's to ping? I would either have to scrape url's available, manually input all url's, or save it to a SSM var. Now that I think about it not actually sure why I didn't use a SSM var. 

I thought about scraping health check info but you would still have to add the check to each env... unless you combined it w/ terraform approach I guess but it would still require manual labor. 

Decided on python script that would create & update healthcheck for me. Whipped up a quick prototype in [arepl](https://www.github.com/almenon/arepl-vscode)

```python
from arepl_dump import dump

#$end

import requests

API_URL = "https://healthchecks.io/api/v1/checks/"
API_READONLY_KEY = "..."
API_KEY = "..."

# just saving this for arepl
mock_api_checks = requests.get(API_URL, headers={"X-Api-Key": API_READONLY_KEY})

cache = {
    "foof": "https://hc-ping.com/d10a4052-128f-4784-9570-1b55499547a2"
}

# args: key
key = "fah"
endpoint = cache.get(key)
if not endpoint:
    # check if healthcheck was manually created
    checks = mock_api_checks
    # this will need to be refactored to handle duplicate names
    check_names = {check["name"]:check for check in checks.json()["checks"]}
    endpoint = check_names.get(key)['unique_key']
    if not endpoint:
        # We have to make check
        print('pretend making check for key ' + key)
        endpoint = "foo"
    else:
        print('check already exists')
    cache[key] = endpoint
print('endpoint is ' + endpoint)

```

Wanted to create python package. Problem is we operate via monolith and I had never created package before. Talk about CI/CD package. Reference article.

Switched over to bash script cause script author gave me was in bash and I wanted to simplify things. 

Benefit is I can see timing info now! Show image.

Switched *back* to a python script because of helm / kubernetes templating. 

![healthchecks](https://dev-to-uploads.s3.amazonaws.com/i/sfkofw6ulh4qsefsddy2.png)
*Example of two successful checks above and a failure alert message below*

-->

<!--
Problem: Need consistent scalable way of managing healthchecks.
Need to spin up different environments.

Solution: Script that automatically creates checks

Struggle: how to create such a script? What language? How to save state?

Adopt in Paul's changes

Switch to python?

https://github.com/bdd/runitor

Scraping healthcheck info:
https://15five-dev.atlassian.net/browse/DEV-841

Create healthchecks manager:
https://15five-dev.atlassian.net/browse/DEV-856

Healthchecks manager PR:
https://github.com/15five/fifteen5-deploy/pull/306

Enabling healthchecks in prod:
https://15five-dev.atlassian.net/browse/DEV-943

```python
# def get_endpoint():
#     return "foo"

# def start():
#     endpoint = get_endpoint()
#     try:
#         requests.get(endpoint + "/start", timeout=5)
#     except requests.exceptions.RequestException:
#         # If the network request fails for any reason, we don't want
#         # it to prevent the main job from running
#         pass
```

-->