Building Observability for Microservices
Tags: microservices, grafana, prometheus, devops
Published: 2023-09-21 12:43:18

Let’s say you distribute the work that a single highly experienced person is doing to multiple individuals, each performing a specific task. Distributing the work this way may increase throughput and eliminate the single point of failure. However, now you have to monitor many people instead of just one! Observability in microservices addresses a similar issue: how to monitor and proactively address problems in a distributed system? The solution lies in measuring the state of a system through metrics recorded for each of its services. A more software-specific definition is (ref. [Wikipedia](https://en.wikipedia.org/wiki/Observability))

> Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

This article explains setting up an observability stack for Microservices to measure the health and performance of the system, sharing our experiences from operating observability for Microservices at [NimbleWork](https://www.nimblework.com).

## The Tools

There are a lot of tools, paid and open source, in the Observability space. We at NimbleWork though prefer the following stack

1. [Prometheus](https://github.com/prometheus) - an open source toolkit for monitors and alerts built at [SoundCloud](https://soundcloud.com/)
2. [Grafana](https://grafana.com/) - a visualisation and analytics data from multiple sources, including prometheus
3. [Helm](https://helm.sh/) - a tool to manage Kubernetes applications
4. [kubectl](https://kubernetes.io/docs/reference/kubectl/) - a command line tool for working on Kubernetes clusters
5. [terraform](https://www.terraform.io/) - a tool to automate provisioning of infrastructure on clouds such as AWS.

We’re running this on Kubernetes, specifically on AWS EKS which is also the choice for running microservices at NimbleWork. Let’s dive into getting things working! The post assumes you have a working EKS cluster, you can choose AWS Fargate as described in [this](https://www.anadimisra.com/post/building-an-eks-fargate-cluster-with-terraform) blog post, or Worker Node cluster as described [here](https://www.anadimisra.com/post/eks-karpenter).

## Prometheus on EKS

Let’s look at deploying prometheus first as it serves as a data-source for Grafana.

### Preparing Volumes on EKS Node Group.

We will be using persistent storage for Prometheus data as it’s a bad to idea to keep crucial observability data in ephemeral storage. So we declare the volumes first. Since we’re using EKS NodeGroups we have to configure Persistent Volumes using EBS for the prometheus node. EFS here is a bad idea as prometheus does not support NFS well enough. While you can certainly get it running on EFS, it remains unstable with sudden restarts related to errors in the logs about NFS writes. [This link](https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects) in prometheus documentation strongly recommends against it.

Let’s configure our EKS cluster for EBS storage using Terraform, here’s a snippet you’d typically add to a `storage.tf` file in the Terraform module you’ll write for managing EKS, we’re following the same structure as in the [blog post](https://www.anadimisra.com/post/eks-karpenter) on setting up EKS via terraform that you can use for reference. You can of course do it manually but there’s a reason why IAC exits, and I like to respect it!

<script src="https://gist.github.com/740e65063a4ac13aa094cf2463ac4aa4.js?file=storage.tf"> </script>

Running the terraform script gives, amongst other things, the values of File System and its Access Point ID in the output which we’ll then use in our helm charts.

### Deploying Prometheus via Helm

We prefer the community helm chart for prometheus, available on [github](https://github.com/prometheus-community/helm-charts), install the chart using the following command

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

```

Now, let’s look at how can we configure it to work with EKS NodeGroups. Create a `volumes.yaml` file with the following contents.

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-volume-0
  labels:
    type: prometheus-volume
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: prometheus-volume-claim
    namespace: observability
  storageClassName: eks-efs-storage-class
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-XXXXXXXXXXXX::fsap-XXXXXXXXXXXX
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-volume-claim
  namespace: observability
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: eks-efs-storage-class
  volumeName: prometheus-volume-0
  resources:
    requests:
      storage: 10Gi

```

Those who’ve been working with EC2 instances or EKS Node Groups using EBS will immediately recognize what we’re doing here, the values for file system and access point ids come from the terraform snippet described above and goes into `volumeHandle: fs-XXXXXXXXXXXX::fsap-XXXXXXXXXXXX`. With the storage configuration in place, let’s configure Prometheus for use in EKS. We’re doing a default deployment with only a few changes as a detailed customisation of the `values.yaml` is beyond the scope of this blog. Refer to the `prometheus.yaml` file in this [gist](https://gist.github.com/anadimisra/e7cd377bca32deaf6ecb906fd06d63bf) for deploying prometheus to EKS. You’ll notice the job names

- kubernetes-apiservers
- kubernetes-nodes
- kubernetes-nodes-cadvisor
- kubernetes-service-endpoints
- kubernetes-service-endpoints-slow
- kubernetes-services
- kubernetes-pods
- kubernetes-pods-slow

These are prometheus jobs defined to collect metrics from the EKS itself, to monitor the health of Kube cluster too in addition to the microservices. Save the aforementioned file in the gist to a `values.yaml` file and Install the helm chart using the following command

```
helm install -i prometheus prometheus-community/prometheus -f values.yaml -n observability

```

Now let’s look at enabling collection metrics for a service written in Spring Boot. We’d simply refer to the configuration [here](https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html), the spring boot actuator endpoint can be configured in the application or bootstrap YAML as follows:

```
management:
  endpoint:
    shutdown:
      enabled: true
scrape_configs:
  - job_name: "spring"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["0.0.0.0:${server.port}"]

```

When you deploy this service you’ll notice the metrics exposed by Spring Boot Actuator are available on prometheus. You can access prometheus via the URL exposed through the ALB ingress.

## Grafana on EKS

Grafana will be deployed for visualising data collected by prometheus. Let’s see how to get it done.

### Deploying EFS volumes for Grafana

Assuming we have the EFS block configured for our EKS cluster, ref. this blog post, create a file `grafana-volumes.yaml` defining the volumes for Grafana as follows:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-volume-0
  labels:
    type: grafana-storage-volume
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: grafana-volume-0
    namespace: observability
  storageClassName: eks-efs-storage-class
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-XXXXXXXXXXXXXXX::fsap-XXXXXXXXXXXXXXX
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-volume-0
  namespace: observability
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: eks-efs-storage-class
  volumeName: grafana-volume-0
  resources:
    requests:
      storage: 10Gi

```

Run the following command to create the volumes

```
kubectl apply -f grafana-volumes.yaml -n observability

```

### Deploying Grafana via HELM chart

Download the helm chart for grafana with the command

```
helm repo add grafana https://grafana.github.io/helm-charts

```

The `grafana.yaml` file in this [gist](https://gist.github.com/anadimisra/e7cd377bca32deaf6ecb906fd06d63bf) describes a basic Grafana configuration for working with Prometheus as a data-source. We’re using mostly standard config here, adding prometheus as a data-source via the block

```
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      version: 1
      url: http://prometheus-server:80
      access: proxy

```

Save it as `values.yaml` and deploy the grafana helm chart by running the following command

```
helm install -i grafana grafana/grafana -f values.yaml -n observability

```

Here again, you can access grafana using the configuration on the ALB Exposed URL.

## Grafana Dashboards

Now that we have the infrastructure in place, let’s see how can we use it for Observability. I’ll limit examples to Spring-Boot for simplicity. Our microservices expose REST APIs and use MongoDB for data persistence. Here’s what we recommend monitoring for such services.

### Spring Boot Microservices

![JVM Dashboard](https://www.anadimisra.com/assets/img/posts/grafana.jpg)

Here’s the panels we’ve added to the Grafana dashboard for monitoring Spring Boot Microservices.

**Quick Facts**

- **Uptime Monitor:** total uptime of pods, can be filtered over namespace, pod names and containers
- **Current Memory Allocation:** current total memory allocated to pods, can be filtered over namespace, pod names and containers
- **Current Memory Usage:** average memory consumption by app containers in each pod
- CPU Load
- CPU Usage
- FATAL, ERROR and WARN logs count

**MongoDB**

- **Repository Method Execution Time:** Total time taken per method per repository, a high value indicates a performance issue
- **Repository Method Execution Count:** Number of methods executed against each of the repositories, gives an idea of traffic on each collection
- **Repository Method Avg Execution Time:** Total time taken per method per repository, a high value indicates a performance issue
- **Repository Method Execution Max Time:** Max recorded time of each MongoDB operation
- **Time in Seconds of Operations Per Collection:** The maximum time in seconds it took to execute a particular command on a MongoDB collection. This is the worst case unit. You can use it to identify the slowest operations per collection in your app.
- **Number of Commands Per collection Per Second:** The number of commands executed on each of the collections per second, helps you optimise collections sharding, and indexing to suite the read/write operations
- **Operations Average Time Per Collection:** The average time of various operations running in a collection, this is a quick view to spot slowest transactions for an app
- **Mongo Operation Max Time:** max time of operation on each of the collections, this gives a sense of what might be skewing the quick view averages in previous panel

The Dashboard config JSON with the aforementioned panels and additional ones for measuring Java Heap, Thread and GC Details can be imported from the `springboot.json` file in this [gist](https://gist.github.com/anadimisra/e7cd377bca32deaf6ecb906fd06d63bf).

## Summing it up

Setting up observability for Microservices isn’t quite intuitive but not an uphill task given the tools at our disposal. Having said that, this is pretty much the starting step of your observability journey. Once you have the infra setup as demonstrated in this blog, you have to meticulously design the graphs you will create in Grafana, which in turn depends on the kind of services you are running, and the parts of your system that you want to track. This blog aims to help fellow SRE and DevOps engineers find the steps in this blog helpful enough to be able to set up their observability stack.