Back2Basics: Monitoring Workloads on Amazon EKS
Tags: aws, eks, kubernetes, grafana
Published: 2024-06-26 09:34:50

## Overview
We're down to the last part of this series✨ In this part, we will explore monitoring solutions. Remember the voting app we've deployed? We will set up a basic dashboard to monitor each component's CPU and memory utilization. Additionally, we’ll test how the application would behave under load.

![Back2Basics: A Series](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hoq8clvhl7dwl8p1zxcq.jpg)

If you haven't read the second part, you can check it out here:
{% embed https://dev.to/aws-builders/back2basics-running-workloads-on-amazon-eks-5e68 %}

## Grafana & Prometheus

To start with, let’s briefly discuss the solutions we will be using. Grafana and Prometheus are the usual tandem for monitoring metrics, creating dashboards and setting up alerts. Both are open-source and can be deployed on a Kubernetes cluster - just like what we will be doing in a while.

- `Grafana` is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics, logs, and traces no matter where they are stored. It provides you with tools to turn your time-series database data into insightful graphs and visualizations. Read more: https://grafana.com/docs/grafana/latest/fundamentals/
- `Prometheus` is an open-source systems monitoring and alerting toolkit. It collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Read more: https://prometheus.io/docs/introduction/overview/

![Architecture: Grafana & Prometheus](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/02owhblm2uixahpkhm6h.png)

Alternatively, you can use an AWS native service like `Amazon CloudWatch`, or a managed service like `Amazon Managed Service for Prometheus` and `Amazon Managed Grafana`. However, in this part, we will only cover self-hosted `Prometheus` and `Grafana`, which we will host on Amazon EKS.


## Let's get our hands dirty!

Like the previous activity, we will use the [same repository](https://github.com/romarcablao/back2basics-working-with-amazon-eks). First, make sure to uncomment all commented lines in `03_eks.tf`, `04_karpenter.tf` and `05_addons.tf` to enable `Karpenter` and other addons we used in the previous activity.

Second, enable `Grafana` and `Prometheus` by adding these lines in `terraform.tfvars`:

```
enable_grafana    = true
enable_prometheus = true
```
Once updated, we have to run `tofu init`, `tofu plan` and `tofu apply`. When prompted to confirm, type `yes` to proceed with provisioning the additional resources.

### Accessing Grafana

![Grafana Login Page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/53ibkbi6sx3uw0bnu647.png)

We need credentials to access Grafana. The default username is `admin` and the auto-generated password is stored in a Kubernetes `secret`. To retrieve the password, you can use the command below:
```
kubectl -n grafana get secret grafana -o jsonpath="{.data.admin-password}" | base64 -d
```

This is what the home or landing page would look like. You have the navigation bar on the left side where you can navigate through different features of Grafana, including but not limited to `Dashboards` and `Alerting`.

![Grafana Home Page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ayrbe261ec66bnn59b8.png)

It's worth noting the `Prometheus` that we have deployed. You might be asking - Does the `Prometheus` server have a UI? Yes, it does. You can even query using `PromQL` and check the health of the targets. But we will use Grafana for the visualization instead of this.

![Prometheus Targets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j34c34rkb5lv1egyupno.png)

### Setting up our first data source

Before we can create dashboards and alerts, we first have to configure the data source.

First, expand the `Connections` menu and click `Data Sources`.

![Grafana: Data Sources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/koptcsr4rsak7qw2wemw.png)

Click `Add data source`. Then select `Prometheus`.

![Grafana: Prometheus Data Sources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vaz05c1aobrybwuawow1.png)

Set the Prometheus server URL to `http://prometheus-server.prometheus.svc.cluster.local`. Since `Prometheus` and `Grafana` reside on the same cluster, we can use the Kubernetes `service` as the endpoint.

![Grafana: Set Prometheus server URL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v0cje9uocdsqen61e55o.png)

Leave other configuration as default. Once updated, click `Save & test`.

![Grafana: Default Data Source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ezxapxq7b95jqh2a2gh.png)

Now we have our first data source! We will use this to create dashboard in the next few section.

### Grafana Dashboards

Let’s start by importing an existing dashboard. Dashboards can be searched here: https://grafana.com/grafana/dashboards/

For example, consider this dashboard - [315: Kubernetes Cluster Monitoring via Prometheus](https://grafana.com/grafana/dashboards/315-kubernetes-cluster-monitoring-via-prometheus/)

To import this dashboard, either copy the `Dashboard ID` or download the `JSON` model. For this instance, use the dashboard ID `315` and import it into our `Grafana` instance.

![Grafana: Import Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cpqsth5zp3sxq0idecrx.png)

Select the `Prometheus` data source we've configured earlier. Then click `Import`.
![Grafana: Import Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r09qpl9qfxxyn60001jf.png)

You will then be redirected to the dashboard and it should look like this:
![Grafana: Imported Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k4a8h2ncqycarwexechq.png)

Yey🎉 We now have our first dashboard!
 
### Let's Create a Custom Dashboard for our Voting App
Copy this [`JSON`](https://raw.githubusercontent.com/romarcablao/back2basics-working-with-amazon-eks/main/modules/grafana/templates/dashboard.json) model and import it into our Grafana instance. This is similar to the steps above, but this time, instead of ID, we'll use the `JSON` field to paste the copied template.

![Grafana: Import Voting App Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vdsx2vvfjmrtw1270khd.png)

Once imported, the dashboard should look like this:

![Grafana: Imported Voting App Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c0moulu2nkgd47zdqb90.png)

Here we have the visualization for basic metrics such as `cpu` and `memory` utilization for each components. Also, `replica count` and `node count` were part of the dashboard so we can check in later the behavior of vote-app component when it auto scale.

### Let's Test!

If you haven't deployed the `voting-app`, please refer to the command below:
```
helm -n voting-app upgrade --install app -f workloads/helm/values.yaml thecloudspark/vote-app --create-namespace
```
Customize the namespace `voting-app` and release name `app` as needed, but update the dashboard query accordingly. I recommend to use the command above and use the same naming: `voting-app` for namespace and `app` as the release name.

Back to our dashboard: When the `vote-app` has minimal load, it scales down to a single replica (1), as shown below.

![Grafana: Voting App Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ecr03d7gl16ik4jkkngh.png)

### **Horizontal Pod Autoscaling in Action**

The `vote-app` deployment has Horizontal Pod Autoscaler (HPA) configured with a maximum of five replicas. This means the voting app will automatically scale up to five pods to handle increased load. We can observe this behavior when we apply the `seeder` deployment. 

Now, let's test how the `vote-app` handles increased load using a `seeder` deployment.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seeder
  namespace: voting-app
spec:
  replicas: 5
...
```
The `seeder` deployment simulates real user load by bombarding the `vote-app` with vote requests. It has five replicas and allows you to specify the target endpoint using an environment variable. In this example, we'll target the Kubernetes `service` directly instead of the load balancer.

```
...
        env:
        - name: VOTE_URL
          value: "http://app-vote.voting-app.svc.cluster.local/"
...
``` 

To apply, use the command below:

```
kubectl apply -f workloads/seeder/seeder-app.yaml
```

After a few seconds, monitor your dashboard. You'll see the `vote-app` replicas increase to handle the load generated by the `seeder`.

```
D:\> kubectl -n voting-app get hpa
NAME                 REFERENCE                        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
app-vote-hpa         Deployment/app-vote              cpu: 72%/80%   1         5         5          12m
```

![Grafana: Voting App Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jqnqhqie4vbb82ywcqj1.png)

Since the `vote-app` chart's default max value for the horizontal pod autoscaler (HPA) is five, we can see that the replica for this deployment stops at five.

### Stopping the Load and Scaling Down

Once you've observed the scaling behavior, delete the `seeder` deployment to stop the simulated load:

```
kubectl delete -f workloads/seeder/seeder-app.yaml
```

Give the dashboard a few minutes and observe the `vote-app` scaling down. With no more load, the HPA will reduce replicas, down to a minimum of one. This may also lead to a node being decommissioned by `Karpenter` if pod scheduling becomes less demanding.

![Grafana: Voting App Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b6l0x0jt3w4tm3cvr32p.png)

You'll see that the vote-app eventually scales in as there is lesser load now. As you might see above, the node count also change from two to one - showing the power of Karpenter.

```
PS D:\> kubectl -n voting-app get hpa
NAME                 REFERENCE                        TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
app-vote-hpa         Deployment/app-vote              cpu: 5%/80%    1         5         2          18m
```

## Challenge: Scaling Workloads

We've successfully enabled autoscaling for the `vote-app` component using Horizontal Pod Autoscaler (HPA). This is a powerful technique to manage resource utilization in Kubernetes. But HPA isn't limited to just one component.

**Tip:** Explore the [ArtifactHub: Vote App](https://artifacthub.io/packages/helm/vote-app/vote-app) configuration in more detail. You'll find additional configurations related to HPA that you can leverage for other deployments.

## Conclusion
Yey! You've reached the end of the `Back2Basics: Amazon EKS Series`🌟🚀. This series provided a foundational understanding of deploying and managing containerized applications on Amazon EKS. We covered:

- Provisioning an EKS cluster using OpenTofu
- Deploying workloads leveraging Karpenter
- Monitoring applications using Prometheus and Grafana

While Kubernetes can have a learning curve, hopefully, this series empowered you to take your first steps. **Ready to level up?** Let me know in the comments what Kubernetes topics you'd like to explore next!