Monitoring Any System with StatsD and AppSignal's Standalone Agent
Tags: monitoring, devops, operations, performance
Published: 2020-10-29 16:02:01


Application monitoring alone is not always enough to get the full picture of your systems. Frequently, services running in satellite apps (or supporting apps) can have an acute impact on your day-to-day operations. Database servers are well-known examples of this. Backup scripts and other background jobs can also slow systems and are often overlooked.

AppSignal does not watch these satellite processes by default. To extend monitoring everywhere and have all your data in a single app, you can install AppSignal’s [standalone agent](https://docs.appsignal.com/standalone-agent/installation.html).

## AppSignal's Standalone Agent

The standalone agent is based on the [same software](https://docs.appsignal.com/appsignal/how-appsignal-operates.html#agent) with which we usually instrument Ruby, Elixir, or JavaScript applications. This software can also run in standalone mode.

The standalone agent can be used to monitor:

- **Infrastructure**: machines that are part of our system but do not run application code.
- **Background jobs**: such as intensive cron jobs or long-running data-processing scripts. If these background jobs are written in a supported language (Ruby, Elixir, or Node.js) you can use the standard integration.
- **More languages**: programs written in languages other than those supported out of the box.

For instance, with the standalone agent, we could track a machine learning model written in Python, instrument backup scripts, [monitor Kafka brokers](https://blog.appsignal.com/2020/07/07/how-appsignal-monitors-their-own-kafka-brokers.html), or collect [host metrics](https://docs.appsignal.com/metrics/host-metrics/index.html) in our web farms. You can view all this information on AppSignal to complement the metrics you already have for your main applications.

## How It Works

The agent is shipped as a deb or rpm package and doesn’t have any language dependencies. It runs in any Debian/Ubuntu or Red Hat-based systems. For detailed installation instructions, check the [agent documentation](https://docs.appsignal.com/standalone-agent/installation.html#installation).

Once installed, the agent is configured in a few minutes and keeps running forever as a daemon, silently monitoring your infrastructure. What is more, the agent includes a [StatsD](https://docs.appsignal.com/standalone-agent/statsd.html) server that relays any custom data you log into your AppSignal dashboard.

### What Is StatsD?

StatsD is a standard for collecting and aggregating arbitrary data. It focuses on logging metric and performance information. It uses a lightweight text protocol over UDP connections that have a small footprint in your machine.

A StatsD message looks like this:

```
KEY:VALUE|TYPE
```

Where `KEY` is any arbitrary string, and `VALUE` is a number. The type value defines how the number is processed.

We support three types of metrics:

- **c**: this is a *counter* that increments each time it is called. For instance, `active_users:1|c` adds 1 to the `active_users` counter.
- **g**: a *gauge* takes a numeric value and maintains it until updated. This is useful to record values that change up and down over time, such as throughput, number of active users, or number of pending tasks in a queue.
- **t**: stores *timing* values. This type is ideal for tracking durations. AppSignal calculates means, count, and percentiles for all the logged timings.

Other unsupported metric types will be silently ignored.

## Sending Data to StatsD

The standalone agent listens for UDP packets on port 8125. We can send StatsD-formatted strings from the command line using [netcat](http://netcat.sourceforge.net/):

```bash
echo -n "myscript.myevent.counter:1|c" | nc -4u -w0 localhost 8125
```

Since we’re using UDP, we don’t have to wait for a response.

[Socat](http://www.dest-unreach.org/socat/) also works:

```bash
echo -n "myscript.myevent.counter:1|c" | socat - udp:localhost:8125
```

This makes it easy to instrument any batch or cron jobs. For instance, the following lines use a gauge to log how much data a backup job has generated:

```bash
backup_size=$(du -m /backups | cut -f1)
echo -n "backup.data:$backup_size|g" | nc -4u -w0 localhost 8125
```

We’re not limited to integers. StatsD also works with floating-point numbers:

```bash
echo -n "network.latency:0.2|g" | nc -4u -w0 localhost 8125
```

## Using Tags

You can add tags to your metrics. The StatsD server supports optional tags at the end of the message:

```
KEY:VALUE|TYPE|#TAGS
```

We can apply several tags in the same message and assign values for filtering later:

```bash
echo -n "backup.data:$backup_size|g|#backups,env:production" | nc -4u -w0 localhost 8125
```

We’ll learn how to see the data in AppSignal in a bit.

## Instrumenting Languages

The StatsD server is compatible with any language that can send UDP packets. Let’s see a few examples.

Python is a popular language for data mining and machine learning. We can add instrumentation into Python applications using the build-in `socket` library:

```python
import time

# measure time taken for function
start = time.process_time()
train_my_model()
training_time = time.process_time() - start

# send value to statsd
import socket

UDP_IP = "127.0.0.1"
UDP_PORT = 8125
MESSAGE = b"model.training.time:" + bytes(str(training_time), 'utf-8') + b"|t"

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
```

We could achieve similar results in Java with the `java.net` library:

```java
byte[] buffer = "mybatch.exception.counter:1|c".getBytes();
InetAddress address = InetAddress.getByName("127.0.0.1");
DatagramPacket packet = new DatagramPacket(
    buffer, buffer.length, address, 8125
);
DatagramSocket datagramSocket = new DatagramSocket();
datagramSocket.send(packet);
```

PHP is another of the incredibly popular language. We can send UDP packets with `socket_sendto`:

```php
$msg = "mywebsite.active_users:$ACTIVE_USERS|g";
$len = strlen($msg);

$sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_sendto($sock, $msg, $len, 0, '127.0.0.1', 8125);
socket_close($sock);
```

### StatsD Clients

Thus far, we used the built-in networking capabilities in every language. But there’s more. Many languages and products include third-party StatsD clients or addons. Any StatsD-compliant client should work, at least for the supported data types. You can find a list of clients at the [StatsD project wiki](https://github.com/statsd/statsd/wiki).

## Viewing Host Data in AppSignal

The **Host Metrics** dashboard shows your machine’s resource utilization:

![Hosts metrics dashboard location](https://blog.appsignal.com/images/blog/2020-09/hosts-dashboard.png)

AppSignal adds an entry for every machine running the standalone agent.

![Available hosts](https://blog.appsignal.com/images/blog/2020-09/host-ov.png)

The dashboard shows the load and CPU averages disk, network, memory, and swap usage:

![Host dashboard](https://blog.appsignal.com/images/blog/2020-09/hosts1.png)

![](https://blog.appsignal.com/images/blog/2020-09/hosts2.png)

## Creating Dashboards

AppSignal doesn’t automatically generate dashboards for the StatsD value you sent—you’ll need to create custom dashboards for this.

First, click on **Add dashboard** under the dashboard menu:

![Location of the create dashboard button](https://blog.appsignal.com/images/blog/2020-09/dash0.png)

Give a name to the dashboard:

![Creating a dashboard](https://blog.appsignal.com/images/blog/2020-09/dash1.png)

Clicking the **Add graph** button shows the graph builder:

![Empty custom dashboard](https://blog.appsignal.com/images/blog/2020-09/dash2.png)

Give a name to the graph, and click on **Add metric**:

![Creating a graph](https://blog.appsignal.com/images/blog/2020-09/create1.png)

Pick the metric you’re interested in from the menu.

![Adding metrics to the graph](https://blog.appsignal.com/images/blog/2020-09/backup1.png)

Optionally, use tags for filtering:

![Filtering by tags](https://blog.appsignal.com/images/blog/2020-09/backup2.png)

Next, click **Back to overview**. You might want to try different graph types and value units to find out which one best suits the data you want to represent.

Timing data looks better with area graphs because the mean and percentiles are shown more clearly.

![Changing display settings](https://blog.appsignal.com/images/blog/2020-09/stats-timer.png)

Line graphs work great for counters and gauges.

![](https://blog.appsignal.com/images/blog/2020-09/create2.png)

Once you’re happy with the result, click on **Create graph**.

![New graph created](https://blog.appsignal.com/images/blog/2020-09/backup3.png)

## Wrapping Up

We’ve learned how to use AppSignal’s standalone server to watch your machines and satellite code. With its built-in StatsD server, you can record arbitrary performance data and instrument any process.

Check the following links to learn more about using the standalone agent:

- [StatsD reference](https://www.appsignal.com).
- [How to send custom metrics](https://docs.appsignal.com/metrics/custom.html).
- [Creating dashboards in AppSignal](https://docs.appsignal.com/metrics/dashboards.html).
- [Identifying and Resolving a Kafka Issue the Standalone Agent](https://blog.appsignal.com/2020/08/17/identifying-and-resolving-a-kafka-issue-with-appsignal.html).

*Our guest author Tomas spent 10 years working at IBM, where he did a bit of everything: development, service delivery, database administration, and cloud engineering. He's now an independent consultant and a technical writer.*