How to Monitor a Spring Boot App
Tags: java, devops, tutorial
Published: 2021-03-10 18:39:44

In this blog you will learn how to monitor a Spring Boot application. You will make use of Spring Actuator, Micrometer, Prometheus and Grafana. Seems a lot of work, but this is easier as you might think!

##1. Introduction
When an application runs in production (but also your other environments), it is wise to monitor its health. You want to make sure that everything is running without any problems and the only way to know this, is to measure the health of your application. When something goes wrong, you hopefully will be notified before your customer notices the problem and maybe you can solve the problem before your customer notices anything. In this post, you will create a sample Spring Boot application which you can monitor with the help of Spring Actuator, Micrometer, Prometheus and Grafana. This is visualized in the overview below, where Spring Actuator and Micrometer are part of the Spring Boot App.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kccl9jnb3wbg8u7wt5du.png)

The purpose of the different components is explained briefly:
* **Spring Actuator**: supplies several endpoints in order to monitor and interact with your application. See [Spring Boot Actuator in Spring Boot 2.0](https://mydeveloperplanet.com/2018/04/18/spring-boot-actuator-in-spring-boot-2-0/) for more information.
* **Micrometer**: an application metrics facade that supports numerous monitoring systems, Spring Boot Actuator provides support for it.
* **Prometheus**: a timeseries database in order to collect the metrics.
* **Grafana**: a dashboard for displaying the metrics.

Every component will be covered in the next sections. The code used in this post can be found at [GitHub](https://github.com/mydeveloperplanet/myspringmetricsplanet).

##2. Create Sample App

First thing to do is to create a sample application which can be monitored. Go to [Spring Initializr](https://start.spring.io/), add dependency `Spring Boot Actuator`, `Prometheus` and `Spring Web`. The sample application will be a Spring MVC application with two dummy endpoints.

Create a `RestController` with the two endpoints. The endpoints only return a simple `String`.

```java
@RestController
public class MetricsController {
 
    @GetMapping("/endPoint1")
    public String endPoint1() {
        return "Metrics for endPoint1";
    }
 
    @GetMapping("/endPoint2")
    public String endPoint2() {
        return "Metrics for endPoint2";
    }
     
}
```

Start the application:

```bash
$ mvn spring-boot:run
```

Verify the endpoints are working:

```bash
$ curl http://localhost:8080/endPoint1
Metrics for endPoint1
$ curl http://localhost:8080/endPoint2
Metrics for endPoint2
```

Verify the Spring Actuator endpoint. The endpoint returns the information in json. In order to format the response so that it is readable, you can pipe the output of the actuator endpoint to `mjson`.

```bash
$ curl http://localhost:8080/actuator | python -mjson.tool
...
{
   "_links":{
      "self":{
         "href":"http://localhost:8080/actuator",
         "templated":false
      },
      "health":{
         "href":"http://localhost:8080/actuator/health",
         "templated":false
      },
      "health-path":{
         "href":"http://localhost:8080/actuator/health/{*path}",
         "templated":true
      },
      "info":{
         "href":"http://localhost:8080/actuator/info",
         "templated":false
      }
   }
}
```

By default, the above information is available. Much more information can be provided by Spring Actuator, but you need to enable this. In order to enable the Prometheus endpoint, you need to add the following line into the `application.properties` file.

```.properties
management.endpoints.web.exposure.include=health,info,prometheus
```

Restart the application and retrieve the data from the Prometheus endpoint. A large bunch of metrics are returned and available. Only a small part of the output is displayed because it is a really long list. The information which is available at this endpoint, will be used by Prometheus.

```bash
$ curl http://localhost:8080/actuator/prometheus
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",} 2.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",} 0.009
...
```

As mentioned before, [Micrometer](https://micrometer.io/) is also needed. Micrometer can be seen as SLF4J, but then for metrics. Spring Boot Actuator provides autoconfiguration for Micrometer. The only thing you need to do is to have a dependency on `micrometer-registry-{system}` in your runtime classpath and that is exactly what we did by adding the `prometheus` dependency when creating the Spring Boot app.

The metrics Actuator endpoint can also be accessed when you add it to the `application.properties` file.

```.properties
management.endpoints.web.exposure.include=health,info,metrics,prometheus
```

Restart the application and retrieve the data from the metrics endpoint.

```bash
$ curl http://localhost:8080/actuator/metrics | python -mjson.tool
...
{
    "names": [
        "http.server.requests",
        "jvm.buffer.count",
        "jvm.buffer.memory.used",
...
```

Each individual metric can be retrieved by adding it to the URL. E.g. the `http.server.requests` parameter can be retrieved as follows:

```bash
$ curl http://localhost:8080/actuator/metrics/http.server.requests | python -mjson.tool
...
{
    "name": "http.server.requests",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 3.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.08918682
        },
...
```

##3. Add Prometheus

[Prometheus](https://prometheus.io/) is an open source monitoring system of the Cloud Native Computing Foundation. Since you have an endpoint in your application which provides the metrics for Prometheus, you can now configure Prometheus to monitor your Spring Boot application. The Spring documentation for doing so can be found [here](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-export-prometheus).

There are several ways to install Prometheus as described in the [installation](https://prometheus.io/docs/prometheus/latest/installation/) section of the Prometheus documentation. In this section, you will run Prometheus inside a Docker container.

You need to create a [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) `prometheus.yml` file with a basic configuration to add to the Docker container. The minimal properties are:
* `scrape_interval`: how often Prometheus polls the metrics endpoint of your application
* `job_name`: just a name for the polling job
* `metrics_path`: the path to the URL where the metrics can be accessed
* `targets`: the hostname and port number. Replace `HOST` with the IP address of your host machine

```yaml
global:
  scrape_interval:     15s
 
scrape_configs:
  - job_name: 'myspringmetricsplanet'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['HOST:8080']
```

If you have difficulties finding out your IP address on Linux, you can use the following command:

```bash
$ ip -f inet -o addr show docker0 | awk '{print $4}' | cut -d '/' -f 1
```

Start the docker container and bind-mount the local `prometheus.yml` file to the one in the docker container. The above `prometheus.yml` file can be found in the git repository in directory `prometheus`.

```bash
$ docker run \
    -p 9090:9090 \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus
```

After successful startup of the Docker container, first verify whether Prometheus is able to gather the data via url [http://localhost:9090/targets](http://localhost:9090/targets).

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i6dh4efugt3cdr7dz4pi.png)

It seems that Prometheus is not able to access the Spring Boot application running on the host. An error `context deadline exceeded` is mentioned.

This error can be solved by adding the Docker container to your host network which will enable Prometheus to access the URL. Therefore, add `--network host` as a parameter. Also remove the port mapping as this has no effect when `--network` is being used. Finally, give your container a name, this will make it easier to start and stop the container. The `-d` parameter will run the container in detached mode.

```bash
$ docker run \
    --name prometheus \
    --network host \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    -d \
    prom/prometheus
```

Verify again the Prometheus targets URL, the state indicates UP which means the prerequisite of accessing the metrics endpoint is fullfilled now.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lzemji9ayvbmvablb863.png)

It is now possible to display the Prometheus metrics. Navigate to [http://localhost:9090/graph](http://localhost:9090/graph), enter `http_server_requests_seconds_max` in the search box and click the Execute button. Access a couple of times the `endPoint1` URL in order to generate some traffic. This parameter will give you the maximum execution time during a time period of a request.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kxbzi3mhhtjqowontye1.png)

##4. Add Grafana

The last component to add is [Grafana](https://grafana.com/). Although Prometheus is able to display the metrics, Grafana will allow you to show the metrics in a more fancy dashboard. Grafana also supports several ways for installing it, but you will run it in a [Docker container](https://grafana.com/docs/grafana/latest/installation/docker/), just like you did with Prometheus.

```bash
$ docker run --name grafana -d -p 3000:3000 grafana/grafana
```

Navigate to URL [http://localhost:3000/](http://localhost:3000/), the URL where Grafana is accessible.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3hw0g85swqv4lia6xrcn.png)

The default username/password is admin/admin. After clicking the **Log in** button, you need to change the default password. Google Chrome will also warn you about the default username/password.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44k5gynye62bonr4d0qm.png)

Next thing to do, is to add a **Data Source**. Click in the left sidebar the **Configuration** icon and select **Data Sources**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1fzokm7wv1z4to70ft1z.png)

Click the **Add data source** button.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5bs39i97smoafc951794.png)

Prometheus is at the top of the list, select **Prometheus**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/79k32kz9epv4p5h34g93.png)

Fill the **URL** where Prometheus can be accessed, set **HTTP Access** to **Browser** and click the **Save & Test** button at the bottom of the page.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dd662fds0yi3hqintgte.png)

When everything is OK, a green notification banner is shown indicating that the data source is working.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z8hhxa07d7fuuip5xosv.png)

Now it is time to create a dashboard. You can create one of your own, but there are also several dashboards available which you can use. A popular one for displaying the Spring Boot metrics is the [JVM dashboard](https://grafana.com/grafana/dashboards/4701).

In the left sidebar, click the + sign and choose **Import**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qgufsezxeuxrgq9jw4wm.png)

Enter the URL [https://grafana.com/grafana/dashboards/4701](https://grafana.com/grafana/dashboards/4701) where the JVM dashboard can be found and click the **Load** button.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gk3ff2bivfvjbolo0tfo.png)

Enter a meaningful name for the dashboard (e.g. **MySpringMonitoringPlanet**), select **Prometheus** as Data Source and click the **Import** button.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/15h20xc4qtcip70x77jx.png)

At this moment, you have a cool first Grafana dashboard at your disposal. Do not forget to scroll down, there are more metrics than shown in the screenshot. The default range is set to 24 hours, this is maybe a bit large when you just started the application. You can change the range in the top right corner. Change it to f.e. **Last 30 minutes**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/twoymlohl3u4cpovjaqj.png)

It is also possible to add a custom panel to the dashboard. At the top of the dashboard, click the **Add panel** icon.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1e9g33iir0gb8p8g2n7k.png)

Click **Add new panel**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5meya2dd20qjmgvgpuqe.png)

In the **Metrics** field, you enter **http_server_requests_seconds_max**, in the **Panel title** field in the right sidebar, you can enter a name for your panel.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00lh5b9bmgldnu3ojz11.png)

Finally, click the **Apply** button at the top right corner and your panel is added to the dashboard. Do not forget to save the dashboard by means of the **Save dashboard** icon next to the **Add panel** icon.

Set some load to the application and see what happens to the metrics on the dashboard.

```bash
$ watch -n 5 curl http://localhost:8080/endPoint1
$ watch -n 10 curl http://localhost:8080/endPoint2
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w715viuaxsk2mg6ngg8r.png)

##5. Conclusion

In this post, you have learnt how you can set up some basic monitoring for a Spring Boot application. It is necessary to use a combination of Spring Actuator, Micrometer, Prometheus and Grafana, but these are quite easy to set up and configure. This is of course just a starting point, but from here on, it is possible to expand and configure more specific metrics for your application.