Monitoring Envoy and Ambassador on Kubernetes with the Prometheus Operator
Tags: showdev, tutorial
Published: 2018-01-18 21:09:25

---
title: Monitoring Envoy and Ambassador on Kubernetes with the Prometheus Operator
published: true
description: This tutorial discusses how to use the Prometheus Operator to monitor Envoy/Ambassador on Kubernetes.
tags: showdev, tutorial
canonical_url: https://www.datawire.io/faster/ambassador-prometheus/
---

In the Kubernetes ecosystem, one of the emerging themes is how applications can best take advantage of the various capabilities of Kubernetes. The Kubernetes community has also introduced new concepts such as [Custom Resources](https://kubernetes.io/docs/concepts/api-extension/custom-resources/) to make it easier to build Kubernetes-native software.

In late 2016, CoreOS introduced the [Operator pattern](https://coreos.com/blog/introducing-operators.html) and released the [Prometheus Operator](https://coreos.com/operators/prometheus/docs/latest/) as a working example of the pattern. The Prometheus Operator automatically creates and manages Prometheus monitoring instances.

The operator model is especially powerful for cloud-native organizations deploying multiple services. In this model, each team can deploy their own [Prometheus](https://www.prometheus.io) instance as necessary, instead of relying on a central SRE team to implement monitoring.

## Envoy, Ambassador, and Prometheus

In this tutorial, we'll show how the Prometheus Operator can be used to monitor an Envoy proxy deployed at the edge. [Envoy](https://www.envoyproxy.io) is an open source L7 proxy. One of the (many) reasons for Envoy's growing popularity is its emphasis on observability. Envoy uses [statsd as its output format](https://www.envoyproxy.io/docs/envoy/v1.5.0/intro/arch_overview/statistics.html).

Instead of using Envoy directly, we'll use [Ambassador](https://www.getambassador.io). Ambassador is a Kubernetes-native API Gateway built on Envoy. Similar to the Prometheus Operator, Ambassador configures and manages Envoy instances in Kubernetes, so that the end user doesn't need to do that work directly.

## Prerequisites

This tutorial assumes you're running Kubernetes 1.8 or later, with RBAC enabled.

Note: If you're running on Google Kubernetes Engine, you'll need to grant `cluster-admin` privileges to the account that will be installing Prometheus and Ambassador. You can do this with the commands below:

```
$ gcloud info | grep Account
Account: [username@example.org]
$ kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole=cluster-admin --user=username@example.org
```

## Deploy the Prometheus Operator

The Prometheus Operator is configured as a Kubernetes `deployment`. We'll first deploy the Prometheus operator.

{% gist https://gist.github.com/richarddli/9323bca381cf529349f074b50266419d %}


```
kubectl apply -f prom-operator.yaml
```

We'll also want to create an additional `ServiceAccount`s for the actual Prometheus instances.

{% gist https://gist.github.com/richarddli/6b564ca785daf104b9a38c6f36e64251 %}

```
kubectl apply -f prom-rbac.yaml
```

The Operator functions as your virtual SRE. At all times, the Prometheus operator insures that you have a set of Prometheus servers running with the appropriate configuration.

## Deploy Ambassador

Ambassador also functions as your virtual SRE. At all times, Ambassador insures that you have a set of Envoy proxies running the appropriate configuration.

We're going to deploy Ambassador into Kubernetes. On each Ambassador pod, we'll also deploy an additional container that runs the [Prometheus statsd exporter](https://github.com/prometheus/statsd_exporter). The exporter will collect the statsd metrics emitted by Envoy over UDP, and proxy them to Prometheus over TCP in Prometheus metrics format.

{% gist https://gist.github.com/richarddli/2b87a8b60c0842104740d824f16dade2 %}

```
kubectl apply -f ambassador-rbac.yaml
```

Ambassador is typically deployed as an API Gateway at the edge of your network. We'll deploy a service to map to the Ambassador `deployment`. Note: if you're not on AWS or GKE, you'll need to update the service below to be a `NodePort` instead of a `LoadBalancer`.

{% gist https://gist.github.com/richarddli/4437ab27f7917d384f12bb5f34d14683 %}

```
kubectl apply -f ambassador.yaml
```

You should now have a working Ambassador and StatsD/Prometheus exporter that is accessible from outside your cluster.

## Configure Prometheus

We now have Ambassador/Envoy running, along with the Prometheus Operator. How do we hook this all together? Logically, all the metrics data flows from Envoy to Prometheus in the following way:

![statsd-logical-flow](https://www.datawire.io/wp-content/uploads/2018/01/Monitoring_Envoy_and_Ambassador_on_Kubernetes_with_the_Prometheus_Operator___Faster_Guides_by_Datawire.png)

So far, we've deployed Envoy and the StatsD exporter, so now it's time to deploy the other components of this flow.

We'll first create a Kubernetes `service` that points to the StatsD exporter. We'll then create a `ServiceMonitor` that tells Prometheus to add the service as a target.

{% gist https://gist.github.com/richarddli/ed9477da044539e1c490a8a6c387efc3 %}

```
kubectl apply -f statsd-sink-svc.yaml
```

Next, we need to tell the Prometheus Operator to create a Prometheus cluster for us. The Prometheus cluster is configured to collect data from any `ServiceMonitor` with the `ambassador:monitoring` label.

{% gist https://gist.github.com/richarddli/cd2a0e5e536f10f30ac12fe94441496b %}

```
kubectl apply -f prometheus.yaml
```

Finally, we can create a service to expose Prometheus to the rest of the world. Again, if you're not on AWS or GKE, you'll want to use a `NodePort` instead.

{% gist https://gist.github.com/richarddli/3a2062d3db88f10ab45631ee52fdde1d %}

```
kubectl apply -f prom-svc.yaml
```

## Testing

We've now configured Prometheus to monitor Envoy, so now let's test this out. Get the external IP address for Prometheus.

```
$ kubectl get services
NAME                  CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
ambassador            10.11.255.93    35.221.115.102   80:32079/TCP     3h
ambassador-admin      10.11.246.117   <nodes>          8877:30366/TCP   3h
ambassador-monitor    None            <none>           9102/TCP         3h
kubernetes            10.11.240.1     <none>           443/TCP          3h
prometheus            10.11.254.180   35.191.39.173    9090:32134/TCP   3h
prometheus-operated   None            <none>           9090/TCP         3h
```

In the example above, this is `35.191.39.173`. Now, go to http://$PROM_IP:9090 to see the Prometheus UI. You should see a number of metrics automatically populate in Prometheus.

### Troubleshooting

If the above doesn't work, there are a few things to investigate:

* Make sure all your pods are running (`kubectl get pods`)
* Check the logs on the Prometheus cluster (`kubectl logs $PROM_POD prometheus`)
* Check [Ambassador diagnostics](https://www.getambassador.io/user-guide/running#diagnostics) to verify Ambassador is working correctly

## Get a service running in Envoy

The metrics so far haven't been very interesting, since we haven't routed any traffic through Envoy. We'll use Ambassador to set up a route from Envoy to the [httpbin](http://httpbin.org) service. Ambassador is configured using Kubernetes annotations, so we'll do that here.

{% gist https://gist.github.com/richarddli/c5ac730cb12965aa4810931b076f3d3c %}

```
kubectl apply -f httpbin.yaml
```

Now, if we get the external IP address of Ambassador, we can route requests through Ambassador to the httpbin service:

```
$ kubectl get services
NAME                  CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
ambassador            10.11.255.93    35.221.115.102   80:32079/TCP     3h
ambassador-admin      10.11.246.117   <nodes>          8877:30366/TCP   3h
ambassador-monitor    None            <none>           9102/TCP         3h
kubernetes            10.11.240.1     <none>           443/TCP          3h
prometheus            10.11.254.180   35.191.39.173    9090:32134/TCP   3h
prometheus-operated   None            <none>           9090/TCP         3h

$ curl http://35.221.115.102/httpbin/ip
{
  "origin": "35.214.10.110"
}
```

Run a `curl` command a few times, as shown above. Going back to the Prometheus dashboard, you'll see that a bevy of new metrics that contain `httpbin` have appeared. Pick any of these metrics to explore further. For more information on Envoy stats, Matt Klein has written a [detailed overview](https://blog.envoyproxy.io/envoy-stats-b65c7f363342) of Envoy's stats architecture.

## Conclusion

Microservices, as you know, are distributed systems. The key to scaling distributed systems is creating loose coupling between each of the components. In a microservices architecture, the most painful source of coupling is actually *organizational* and not architectural. Design patterns such as the Prometheus Operator enable teams to be more self-sufficient, and reduce organizational coupling, enabling teams to code faster.