Monitoring Your Spring Boot Application with Custom Metrics and Prometheus
Published: 2023-04-13 13:21:25

## 1. Background
In a microservice architecture, applications are composed of multiple independent services that work together to provide a larger functionality. While this approach offers benefits such as scalability and flexibility, it also introduces challenges in terms of monitoring and managing the application.

Integrating Prometheus with your microservices on Kubernetes can help address some of these challenges by providing the ability to monitor, alert, and visualize metrics across multiple services. This can help you identify and troubleshoot issues that span multiple services, which can be difficult in a distributed system.

--- 

## 2. Objective

- To expose metrics to integrate Prometheus by Spring Boot Actuator
- To Define custom metrics with Micrometer

We will run a demo Spring Boot application on Kubernetes with Istio.

---

## 3. Enabling Spring Boot Actuator Endpoints
Spring Boot Actuator is a great tool when deploying a Spring Boot application on Kubernetes as it provides built-in support for health checks, metrics, tracing, and security, making it a lot easier to operate and maintain the application.

Add Spring Actuator Dependencies to project
```
implementation 'org.springframework.boot:spring-boot-starter-actuator'
```

Just add following properties to your application.yml file to expose an endpoint at /actuator/prometheus to present a Prometheus scrape with the appropriate format.
```
management:
  endpoints:
    enabled-by-default: true
    web:
      exposure:
        include: '*'
  endpoint:
    health:
      show-details: always
```

Try to access the prometheus metrics endpoint (http://localhost:8080/actuator/prometheus)

Output
```
# HELP executor_active_threads The approximate number of threads that are actively executing tasks
# TYPE executor_active_threads gauge
executor_active_threads{name="applicationTaskExecutor",} 0.0
# HELP jvm_memory_usage_after_gc_percent The percentage of long-lived heap pool used after the last GC event, in the range [0..1]
# TYPE jvm_memory_usage_after_gc_percent gauge
jvm_memory_usage_after_gc_percent{area="heap",pool="long-lived",} 0.0034471343796665006
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 7787.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",} 0.101
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="G1 Evacuation Pause",} 0.0
# HELP disk_total_bytes Total space for path
# TYPE disk_total_bytes gauge
...
```
---

## 4. Creating Your Custom Prometheus Metric

Add Micrometer Prometheus Dependencies to project
```
runtimeOnly 'io.micrometer:micrometer-registry-prometheus'
```
There are 4 core metric types in Prometheus. In this article, we focus on "Counter" and "Gauge". A Counter is used to count the number of occurrences of an event, here is an example of how to define a Counter. 

```
@RestController
@RequestMapping("/api")
public class NewsController {

    private Counter counter;

    public NewsController(MeterRegistry registry) {
        // counter
        this.counter = Counter.builder("news_fetch_request_total").
                tag("version", "v1").
                description("News Fetch Count").
                register(registry);
    }

    @GetMapping("/news")
    public List<News> getNews() {
        counter.increment();
        return List.of(new News("Good News!"), new News("Bad News!"));
    }

}
```
We define a Counter named "news_fetch_request_total" and register it with the MeterRegistry. We also define an api endpoint getNews() that increments the counter each time it is called.

Try to access http://localhost:8080/api/news, it should return the following response.
```
[{"title":"Good News!"},{"title":"Bad News!"}]
```

Try to access the prometheus metrics endpoint again and search "news_fetch_request_total", and found that the value is 1.0 now.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hjnnmyya6v68pykml178.png)

And a Gauge is used to measure the value of a particular variable. We define a Gauge named "temperature" and register it with the MeterRegistry as well.

```
        Gauge.builder("temperature", () -> {
                    double min = 10, max = 30;
                    return min + new Random().nextDouble() * (max - min);
                }).
                tag("version", "v1").
                description("Temperature Record").
                register(registry);
```


---

## 5. Building and deploying a Docker image to a Kubernetes cluster

Create a Dockerfile to dockerize the Gradle project.
```
FROM openjdk:17-alpine3.14
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
```
Run the following command to build and tag the Docker image.

```
docker build --build-arg JAR_FILE=build/libs/*.jar -t myorg/myapp .
```


Add the following annotations to deployment file to tell prometheus where to scrape metrics from the service.
```
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/actuator/prometheus"    
```

Let's deploy it on Kubernetes cluster.
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v1
  labels:
    app: myapp
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: v1
  template:
    metadata:
      labels:
        app: myapp
        version: v1
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"    
    spec:
      containers:
      - name: details
        image: docker.io/myorg/myapp
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        securityContext:
          runAsUser: 1000
```

---

## 6. Result
If you are working with Istio, you can use the istioctl command to forward the port, enabling the access on localhost.
```
istioctl dashboard prometheus
```

Custom metric "temperature" is found, yeah!


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gtgsvtfnujq04lowp391.png)


With the help of the Micrometer Prometheus and the Spring Boot Actuator, implementing custom metrics in your application is really straightforward. Hope this tutorial has been helpful in getting you started with creating custom Prometheus metrics in your Spring Boot application!