Monitor home Internet connectivity with Cloudprober
Tags: homelab
Published: 2023-10-07 03:49:00

## Background

From time to time, my home is disconnected from the Internet, typically due to an outage of the Internet Service Provider. I do receive notification about my cameras going offline, but I have no idea how long it lasts; it can range from 10 minutes to 12 hours, and Comcast won’t tell you how long (you don’t even have access to the outage information unless you log in with the Comcast account). In addition, sometimes the Internet isn’t fully down, but the quality drops, so I’d like to set up something to monitor my Internet connection.

## Journey of solution-hunting

The principle is simple: I set up something to ping (or cURL) https://google.com. Given the historical availability of that page, this probe is going to tell me whether I can reach Internet. Now the real question is, what tool can do the queries, save the data, and visualize the latency for me.

I did some research, and asked around. Most people don’t care; the ones who care typically uses SmokePing:

{% embed https://github.com/oetiker/SmokePing %}

SmokePing has its own web UI to show the data. The UI looks good enough for most users, but I’m not an average user, but this is insufficient for an SRE-SWE. I’m more of a backend guy, so not very good at implementing the interactive features myself. Another option I found was Nagios:

{% embed https://en.wikipedia.org/wiki/Nagios %}

I tried it briefly, before I had another idea: simply make a daemon (or a scheduled task) to post to some data-hosting service, such as [Google Cloud Monitoring](https://cloud.google.com/monitoring) (also known as “StackDriver”. (Disclaimer: I’m a Googler.) I did some more research, and summarized my options in a GitHub issue:

{% embed https://github.com/FranklinYu/snippets/issues/23 %}

When I was collecting libraries for this idea, I found Cloudprober.

## Cloudprober

{% embed https://github.com/cloudprober/cloudprober %}

Cloudprober was created by Googlers (probably as a side project), and it supports uploading data to Google Cloud Monitoring (exactly what I needed). The tool was designed for black-box service monitoring of owned services, but it can actually be pointed to any service. For example, I monitor the Google homepage with this configuration:

```textproto
# proto-file: https://github.com/cloudprober/cloudprober/blob/master/config/proto/config.proto
# proto-message: cloudprober.ProberConfig

probe: {
  name: "google_homepage"
  type: HTTP
  targets: {
    host_names: "www.google.com"
  }
  interval: "30s"
  timeout: "1s"
  latency_distribution: {
    exponential_buckets: {
      scale_factor: 100
      base: 1.1
      num_buckets: 25
      # last bucket starts at: 100 * 1.1^24 = 985 ms
    }
  }
  latency_unit: "ms"
  http_probe: {
    protocol: HTTPS
  }
}

surfacer: {
  type: STACKDRIVER
  stackdriver_surfacer: {
    project: "franklinyu-home"
  }
}
```

This creates nice charts on Google Cloud console like

![heatmap of latency](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/21ug1v2ccf6uvfvl6b9j.png)

## Starting Cloudprober yourself

First, of course you need to download Cloudprober. You can follow [the official guide](https://cloudprober.org/docs/overview/getting-started/) to download the pre-built binary, or (if you are using Arch Linux) use [my AUR package](https://aur.archlinux.org/packages/cloudprober).

Then you need a Google Cloud project, and [create a service account key](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key). Follow the Google Cloud documentation to set the environment variable.

Now save the configuration file somewhere, for example in `~/Desktop/cloudprober.textproto`, and run Cloudprober like

```sh
cloudprober --config_file ~/Desktop/cloudprober.textproto
```

Wait for a while, and the metric should appear in Google Cloud Console as

```plain
custom/cloudprober/http/google_homepage/latency
```

And you can see the data in [Metric Explorer](https://console.cloud.google.com/monitoring/metrics-explorer).

## Configuration

The configuration file is specified in [the text format of Procotol Buffer](https://protobuf.dev/reference/protobuf/textformat-spec/), also known (in Google) as “Text-Proto”. Most parts are straight forward; the `latency_distribution` stanza is explained in [Google Cloud documentation](https://cloud.google.com/monitoring/api/v3/distribution-metrics). If we denote the scale-factor as “k” and base as “a”, then basically the buckets are the right-open intervals

{% katex %}
[k a^i, k a^{i+1})
{% endkatex %}

except the first and last bucket (which has to cover the minimum and maximum). My strategy of choosing the base and the scale factor boils down to the “target interval”, which is the interval of latency that I care about. If we denote the number of buckets as “n”, then my strategy is that the entire “target interval” should be covered by buckets from second to the “second to last”. In other words, the “target interval” is supposed to be a subset of

{% katex %}
[k, k a^{n-2})
{% endkatex %}