Monitor and Visualize Nginx Ingress Controller Metrics on Amazon EKS with Prometheus & Grafana
Tags: monitoring, devops, eks, kubernetes
Published: 2023-05-25 12:17:59

In today’s digital landscape, it is very necessary to figure out trends in our data and act accordingly to ensure high availability of our application. Monitoring helps us to stay on top of the game and get insights on our product environment. An Ingress Controller acts as a bridge between Kubernetes Service and the external world and can be considered as a specialized load balancer for Kubernetes. 

## Pre-Requisites 

1. A Kubernetes Cluster ([Create an EKS Cluster from scratch](https://dev.to/aws-builders/creating-an-amazon-eks-cluster-from-scratch-using-eksctl-540i)) 
2. Helm V3 ([Helm Install Docs](https://helm.sh/docs/intro/install/))

## Install Kube-Prometheus-Stack using Helm 

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
helm repo update 
helm install --create-namespace --namespace monitoring \
my-k8s-prom-stack prometheus-community/kube-prometheus-stack
```
Once you have the stack installed you should have Prometheus, Grafana and AlertManager installed onto your cluster. 

![Kube Prometheus Stack](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lki9rlbmwl7qfkrt0fc7.png)

## Install Nginx Ingress Controller  

```
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx 
helm repo update 
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.metrics.enabled=true \
--set controller.metrics.serviceMonitor.enabled=true \
--set controller.metrics.serviceMonitor.additionalLabels.release="my-k8s-prom-stack" \
--version=4.5.2 
```
> Along with installing the ingress controller we are setting the controller metrics as true which will populate an additional metrics service and creating a service monitor for that service which will feed data onto Prometheus, the additional Label would be as per the selector you have set on the Prometheus Operator, follow the later part for more details on this.

You can check the Service Monitor selector on your Prometheus Operator using:  

`kubectl get prometheus --namespace monitoring prometheus-svc-name –oyaml` (lookout for the section below on the manifest) 
 
![Service Monitor Selector on Prom Operator](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o67owwmj0k8lbwclyt5o.png)

Once Nginx controller is installed you can verify the services that are spawned using: 

`kubectl get svc --namespace ingress-nginx`

![Ingress Controller Services](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/22cupf64pg4ejovr01qn.png)

![Service Monitor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n56aepy2ev5amzvlc84w.png)

## Monitor and Visualize Nginx Controller Metrics 

To access the Prometheus and Grafana you can set ingress objects 
Sample Ingress Manifest:

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:    
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 5m  
  name: prometheus-ingress
  namespace: monitoring
spec:
  rules:
  - host: prom.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-k8s-prom-stack-kube-pro-prometheus
            port:
              number: 9090
        path: /
        pathType: ImplementationSpecific
```
Generate some load on the Ingress Controller by hitting the Ingress Endpoints.

Access Prometheus and Grafana to query and visualize the controller metrics:
![Prometheus Console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ko8zkdgdagjyy9sw5kfl.png)
Prom Query to check the number of 200 requests on ingress:
`nginx_ingress_controller_requests{status=~'2..'}`

![Prom Console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jbpn2loge7l5iakt3ifg.png)

Import Nginx Controller Dashboard on Grafana and visualize the metrics (Dashboard ID: 9614) 

![Grafana Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aj4pif05kg1msiqtebab.png)

Thank you for reading this, you can follow me for more such content, or reach out to me on [Linkedin](https://in.linkedin.com/in/rajitpaul).