Install Prometheus and Grafana on Kubernetes Cluster.
Published: 2024-01-22 20:47:08

_Prometheus is an open-source monitoring and alerting system that helps you collect and store metrics about your software systems and infrastructure and analyze that data to gain insights into their health and performance. It provides a powerful query language, a flexible data model, and a range of integrations with other tools and systems. With Prometheus, you can easily monitor metrics such as CPU usage, memory usage, network traffic, and application-specific metrics, and use that data to troubleshoot issues, optimize performance, and create alerts to notify you when things go wrong._

Install Prometheus via 2 ways.
1. Helm Charts
2. Operators

Using operators, using Operator life cycle management (OLM) one can automatically have latest updates without worrying to update manually and therefore securing cluster with zero-day vulnerabilities.  

## Install using Helm. 

### 1. Add helm repo
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
```

### 2. Update helm repo
```
helm repo update
```

### 3. Install helm
```
helm install prometheus prometheus-community/prometheus
```

```
PS C:\WINDOWS\system32> helm install prometheus prometheus-community/prometheus
NAME: prometheus
LAST DEPLOYED: Mon Jan 22 12:55:19 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.default.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster:
prometheus-alertmanager.default.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=alertmanager,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been disabled by default since    #####
######            it deprecated after k8s 1.25+. use                        #####
######            (index .Values "prometheus-node-exporter" "rbac"          #####
###### .          "pspEnabled") with (index .Values                         #####
######            "prometheus-node-exporter" "rbac" "pspAnnotations")       #####
######            in case you still need it.                                #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.default.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/
```
### - Verify Prometheus Pods Running 

```
$ kubectl get pods
NAME                                                 READY   STATUS    RESTARTS        AGE
prometheus-alertmanager-0                            1/1     Running   0               3m8s
prometheus-kube-state-metrics-745b475957-ngqm7       1/1     Running   0               3m8s
prometheus-prometheus-node-exporter-698nc            1/1     Running   0               3m8s
prometheus-prometheus-pushgateway-6ccd698d79-ld86d   1/1     Running   0               3m8s
prometheus-server-bc7ccb595-jpl7p                    2/2     Running   0               3m8s
sample-python-deployment-99fcf4b6f-rb59w             1/1     Running   2 (5m40s ago)   8d
sample-python-deployment-99fcf4b6f-vcnwt             1/1     Running   2 (5m40s ago)   8d
```

### - Verify Prometheus services via `Kubectl get svc`
```
$ kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       7m38s
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       7m38s
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       7m38s
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       7m38s
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       7m38s
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         7m38s
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
``` 

### 4. Expose Prometheus Service
This is required to access prometheus-server using your browser.

```
kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
```

- Notice below is created `prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   3s`

- Get svc
```
 kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       10m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       10m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       10m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       10m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       10m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         10m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   3s
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
```

Output: 
```
$ kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
service/prometheus-server-ext exposed
```
### Get Minikube IP
```
minikube ip
192.168.59.107
```

### Open Browser and access Prometheus 
- Go to http://192.168.59.107:31401/
- 
![Prometheus Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xn9ij3wmz3mfx6uczmlp.png)

## Install Grafana

### 1. Install using Helm


- Add helm repo 
- _Ignore is you already have help and grafana repo added _
`helm repo add grafana https://grafana.github.io/helm-charts`
```
$helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
```

### 2. Update helm repo 
- `helm repo update`
```
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "istio" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
```

### 3. Install Grafana via helm 
- `helm install grafana grafana/grafana`
```
$ helm install grafana grafana/grafana
NAME: grafana
LAST DEPLOYED: Mon Jan 22 13:20:52 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################
```

### 4. Get Grafana Login Details
- Get password: `$ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo`
```
$ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd
```
- Get Grafana services `kubectl get svc` Notice its running on Cluster IP

```
kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP         8m12s
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       33m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       33m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       33m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       33m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       33m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         33m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   23m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
```


### 5. Expose Grafana Service to Access IT
```
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
```
- Verify exposed Grafana service
```
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
service/grafana-ext exposed
```
- Get Grafana-Ext Service at port `30197`
```
 kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP         11m
grafana-ext                           NodePort       10.111.66.82     <none>        80:30197/TCP   51s
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP        8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP       37m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP       37m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP       37m
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP       37m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP       37m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP         37m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP   26m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP   8d
```

- Access via minikube IP
- http://192.168.59.107:30197/
- Login via user=`admin` and password=`XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd`
- 
![Grafana login page] (https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cz7pg5dxw2bhj49z9jk8.png)

- Grafana Dashboard

![Grafana Dashboard] (https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d02vadj1y2i63iyeihqr.png)

- Configure Prometheus as Datasource to Grafana
- Go to Data sources --> Add Prometheus --> Provide IP Address - Save and Test

![Save and Test configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/721cil65k6d6nmta8g31.png)
- 
![Configure Prometheus as Datasource to Grafana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4tkfyjloxrj942iv4m90.png)

- Import an existing dashboard

- Dashboard --> Import --> Enter ID `3662` --> Load

![Import Dashboard 3662](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/838tjkqqgjhvremnqwhq.png)


![configure Dashboard 3662](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x2cwb1w8c23z0zh7h67o.png)

- Dashboard

![Grafana Dashboard description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iqjzp78vqdddc1knljav.png)

### 6. Expose `prometheus-kube-state-metrics`
- How do we do it by following the steps below.
```
kubectl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=prometheus-kube-state-metrics-ext
```
- Verify kube-state-metrics exposed 

```
kubectl get svc
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
grafana                               ClusterIP      10.109.8.93      <none>        80/TCP           54m
grafana-ext                           NodePort       10.111.66.82     <none>        80:30197/TCP     43m
kubernetes                            ClusterIP      10.96.0.1        <none>        443/TCP          8d
prometheus-alertmanager               ClusterIP      10.100.6.213     <none>        9093/TCP         79m
prometheus-alertmanager-headless      ClusterIP      None             <none>        9093/TCP         79m
prometheus-kube-state-metrics         ClusterIP      10.103.166.42    <none>        8080/TCP         79m
prometheus-kube-state-metrics-ext     NodePort       10.100.12.19     <none>        8080:30229/TCP   15s
prometheus-prometheus-node-exporter   ClusterIP      10.104.211.162   <none>        9100/TCP         79m
prometheus-prometheus-pushgateway     ClusterIP      10.100.179.109   <none>        9091/TCP         79m
prometheus-server                     ClusterIP      10.105.170.132   <none>        80/TCP           79m
prometheus-server-ext                 NodePort       10.99.142.160    <none>        80:31401/TCP     69m
sample-python-service                 LoadBalancer   10.101.160.8     <pending>     80:30007/TCP     8d
```

- Open browser and access via port 30229
- http://192.168.59.107:30229/
- 
![Kube state metrics external service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zg39d93z4s4aikccmz2a.png)

- Kubectl get cm or configmap
```
kubectl get cm
NAME                      DATA   AGE
grafana                   1      61m
kube-root-ca.crt          1      8d
prometheus-alertmanager   1      86m
prometheus-server         6      86m
```
- Edit config map of `prometheus-server`

`kubectl edit cm prometheus-server`

change scrpe confi from local-host to `prometheus-kube-state-metrics-ext` service
- BEFORE: 
```
scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
```
- AFTER: Add config to `http://192.168.59.107:30229/` state-metrics-ext
```
scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
    - job_name: prometheus
      static_configs:
      - targets:
        - http://192.168.59.107:30229
```

**Credits:-**
Thanks to [Abhishek Veeramalla](https://www.youtube.com/@AbhishekVeeramalla)