Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles
Tags: kubernetes, aws, tutorial, devops
Published: 2020-04-26 15:19:53

---
title: Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles
published: true
date: 2020-04-26 14:44:20 UTC
tags: Kubernetes,AWS,tutorial,DevOps
canonical_url: https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/
---

![](https://rtfm.co.ua/wp-content/uploads/2017/06/kuber-1.png)The next task with our Kubernetes cluster is to set up its monitoring with Prometheus.

This task is complicated by the fact, that there is the whole bunch of resources needs to be monitored:

- from the infrastructure side – ЕС2 WokerNodes instances, their CPU, memory, network, disks, etc
- key services of Kubernetes itself – its API server stats, etcd, scheduler
- deployments, pods and containers state
- and need to collect some metrics from applications running in the cluster

To monitor all of these the following tools can be used:

- [`metrics-server`](https://github.com/kubernetes-sigs/metrics-server) – CPU, memory, file-descriptors, disks, etc of the cluster
- [`cAdvisor`](https://github.com/google/cadvisor) – a Docker daemon metrics – containers monitoring
- [`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics) –  deployments, pods, nodes
- [`node-exporter`](https://github.com/prometheus/node_exporter): EC2 instances metrics – CPU, memory, network

**What we will do in this post?**

- will spin up a Prometheus server
- spin up a Redis server and the `redis-exporter` to grab the Redis server metrics
- will add a ClusterRole for the Prometheus
- will configure Prometheus Kubernetes Service Discovery to collect metrics:
  - will take a view on the Prometheus Kubernetes  Service Discovery roles
  - will add more exporters:
    - node-exporter
    - kube-state-metrics
    - cAdvisor metrics
  - and the metrics-server

**How this will work altogether?**

The [Prometheus Federation](https://rtfm.co.ua/prometehus-obzor-federation-monitoring-docker-swarm-i-nastrojki-alertmanager/#Prometheus_crossservice_federation) will be used:

- we already have Prometheus-Grafana stack in my project which is used to monitor already existing resources – this will be our “central” Prometheus server which will PULL metrics from other Prometheus server in a Kubernetes cluster (all our AWS VPC networks are interconnected via [VPC peering](https://rtfm.co.ua/aws-nastrojka-vpc-peering/) and metrics will go via private subnetting)
- in an EKS cluster, we will spin up additional Prometheus server which will PULL metrics from the cluster and exporters and later will hand them over to the “central” Prometheus

**`helm install`**

In the 99% guides, I found during an investigation the Kubernetes and Prometheus monitoring all the installation and configuration всех встреченных гайдов по Prometheus и Kubernetes вся установка и настройка was confined to the only one “`helm install`” command.

[Helm](https://helm.sh/) it’s, of course, great – a templating, a package manager in one tool, but the problem is that it will do a lot of things under the hood, but now I want to understand what’s exactly is going on there.

By the fact, this is the following part of the [AWS Elastic Kubernetes Service: a cluster creation automation, part 1 – CloudFormation](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-a-cluster-creation-automation-part-1-cloudformation/) (_in English_) and [AWS: Elastic Kubernetes Service — автоматизация создания кластера, часть 2 — Ansible, eksctl](https://rtfm.co.ua/aws-elastic-kubernetes-service-avtomatizaciya-sozdaniya-klastera-chast-2-ansible-eksctl/) (_in Russian still_), so there will be some Ansible in this post.

Versions used here:

- Kubernetes: (AWS EKS): v1.15.11
- Prometheus: 2.17.1
- kubectl: v1.18.0

**Content**

- [kubectl context] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#kubectl_context)
- [ConfigMap Reloader] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#ConfigMap_Reloader)
- [Start Prometheus server in Kubernetes] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Start_Prometheus_server_in_Kubernetes)
- [Namespace] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Namespace)
- [The prometheus.yml ConfigMap] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#The_prometheus_yml_ConfigMap)
- [Prometheus Deployment and a LoadBalancer Service] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Prometheus_Deployment_and_a_LoadBalancer_Service)
- [Monitoring configuration] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Monitoring_configuration)
- [Redis && redis_exporter] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Redis_redisexporter)
- [Prometheus ClusterRole, ServiceAccount, and ClusterRoleBinding] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Prometheus_ClusterRole,_ServiceAccount,_and_ClusterRoleBinding)
- [Prometheus Kubernetes Service Discovery] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Prometheus_Kubernetes_Service_Discovery)
- [kubernetes_sd_config roles] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#kubernetessdconfig_roles)
- [node role] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#node_role)
- [pod role] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#pod_role)
- [node-exporter metrics] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#nodeexporter_metrics)
- [kube-state-metrics] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#kubestatemetrics)
- [cAdvisor] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#cAdvisor)
- [metrics-server] (https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#metricsserver)
- [Useful links](https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/#Useful_links)

### `kubectl` context

First, configure access to your cluster – add a new context for your `kubectl`:

```
$ aws --region eu-west-2 eks update-kubeconfig --name bttrm-eks-dev-2
Added new context arn:aws:eks:eu-west-2:534***385:cluster/bttrm-eks-dev-2 to /home/admin/.kube/config
```

Or switch to already existing if any:

```
$ kubectl config get-contexts
...
arn:aws:eks:eu-west-2:534****385:cluster/bttrm-eks-dev-1
...
kubectl config use-context arn:aws:eks:eu-west-2:534***385:cluster/bttrm-eks-dev-2
Switched to context "arn:aws:eks:eu-west-2:534***385:cluster/bttrm-eks-dev-2".
```

### `ConfigMap` Reloader

Because we will do a lot of changes to the `ConfigMap` of our future Prometheus – worth to add the [Reloader](https://rtfm.co.ua/en/kubernetes-configmaps-and-secrets-on-a-gorush-server-example/) now, so pods will apply those changes immediately without our intervention.

Create a directory:

```
$ mkdir -p roles/reloader/tasks
```

And only one task there – the Reloader installation. Will use `kubectl`, and call it with the Ansible [`command`](https://docs.ansible.com/ansible/latest/modules/command_module.html) module.

In the`roles/reloader/tasks/main.yml` add the following:

```
- name: "Install Reloader"
  command: "kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml"
```

Enough for now – I just have no time to dig into [`k8s`](https://docs.ansible.com/ansible/latest/modules/k8s_module.html) and its issues with the `python-requests` import.

Add this role to a playbook:

```
- hosts:
  - all
  become:
    true
  roles:
    - role: cloudformation
      tags: infra
    - role: eksctl
      tags: eks
    - role: reloader
      tags: reloader
```

Run:

```
$ ansible-playbook eks-cluster.yml --tags reloader
```

And check:

```
$ kubectl get po
NAME                                 READY   STATUS    RESTARTS   AGE
reloader-reloader-55448df76c-9l9j7   1/1     Running   0          3m20s
```

### Start Prometheus server in Kubernetes

First, let’s start the Prometheus itself in the cluster.

Its config file will be saved as an `ConfigMap` object.

The cluster already crated and in future everything will be managed by Ansible, so add its directories structure here

```
$ mkdir -p roles/monitoring/{tasks,templates}
```

#### Namespace

All the resources related to the monitoring will be kept in a dedicated Kubernetes namespace.

In the `roles/monitoring/templates/` directory add a config file, call it `prometheus-ns.yml.j2` for example:

```
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
```

Add a new file `roles/monitoring/tasks/main.yml` to create the namespace:

```
- name: "Create the Monitoring Namespace"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-ns.yml.j2"
```

Add the role to the playbook:

```
- hosts:
  - all
  become:
    true
  roles:
    - role: cloudformation
      tags: infra
    - role: eksctl
      tags: eks
    - role: reloader
      tags: reloader
    - role: monitoring
      tags: monitoring
```

Run to test:

```
$ ansible-playbook eks-cluster.yml --tags monitoring
```

Check:

```
$ kubectl get ns
NAME              STATUS   AGE
default           Active   24m
kube-node-lease   Active   25m
kube-public       Active   25m
kube-system       Active   25m
monitoring        Active   32s
```

#### The `prometheus.yml` ConfigMap

As already mentioned, the Prometheus config data will be kept in a `ConfigMap`.

Create a new file called `roles/monitoring/templates/prometheus-configmap.yml.j2` – this is a minimal configuration:

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      external_labels:
        monitor: 'eks-dev-monitor'
    scrape_configs:
      - job_name: 'prometheus'
        scrape_interval: 5s
        static_configs:
          - targets: ['localhost:9090']
```

Add it to the `roles/monitoring/tasks/main.yml`:

```
- name: "Create the Monitoring Namespace"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-ns.yml.j2"
- name: "Create prometheus.yml ConfigMap"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-configmap.yml.j2"
```

Now can check one more time – apply it up and check the `ConfigMap` content:

```
$ kubectl -n monitoring get configmap prometheus-config -o yaml
apiVersion: v1
data:
prometheus.yml: |
global:
scrape_interval:     15s
external_labels:
monitor: 'eks-dev-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
kind: ConfigMap
...
```

#### Prometheus Deployment and a LoadBalancer Service

Now we can start the Prometheus.

Create its deployment file – `roles/monitoring/templates/prometheus-deployment.yml.j2`:

```
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  labels:
    app: prometheus
  namespace: monitoring
  annotations:
    reloader.stakater.com/auto: "true"
#    service.beta.kubernetes.io/aws-load-balancer-internal: "true"  
spec:
  replicas: 2
  selector:
    matchLabels: 
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus-server
        image: prom/prometheus
        volumeMounts:
          - name: prometheus-config-volume
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
        ports:
        - containerPort: 9090
      volumes:
        - name: prometheus-config-volume
          configMap:
            name: prometheus-config
---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-server-alb
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9090
  type: LoadBalancer
```

Here:

1. create two pods with Prometheus
2. attach the _prometheus-config_ `ConfigMap`
3. create a `Service` with the `LoadBalancer` type to access the Prometheus serve on the 9090 port – this will create an AWS LoadBalancer with the Classic type (not Application LB) with a Listener on the 80 port

Annotations here:

- _reloader.stakater.com/auto: «true»_ – used for the Reloader service
- _service.beta.kubernetes.io/aws-load-balancer-internal: «true»_ can be commented, for now, later the  `Internal` will be used to configure [VPC peering](https://rtfm.co.ua/aws-nastrojka-vpc-peering/) and [Prometheus federation](https://rtfm.co.ua/prometehus-obzor-federation-monitoring-docker-swarm-i-nastrojki-alertmanager/), at this moment let’s use Internet-facing

Add to the  `tasks`:

```
...
- name: "Deploy Prometheus server and its LoadBalancer"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-deployment.yml.j2"
```

Run:

```
$ ansible-playbook eks-cluster.yml --tags monitoring
...
TASK [monitoring : Deploy Prometheus server and its LoadBalancer] ****
changed: [localhost]
...
```

Check pods:

```
$ kubectl -n monitoring get pod
NAME                                 READY   STATUS    RESTARTS   AGE
prometheus-server-85989544df-pgb8c   1/1     Running   0          38s
prometheus-server-85989544df-zbrsx   1/1     Running   0          38s
```

And the LoadBalancer Service:

```
$ kubectl -n monitoring get svc
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)        AGE
prometheus-server-alb   LoadBalancer   172.20.160.199   ac690710a9747460abc19cd999812af8-1463800400.eu-west-2.elb.amazonaws.com   80:30190/TCP   42s
```

Or in the AWS dashboard:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_101125.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_101125.png)

Check it – open in a browser:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_101207.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_101207.png)

if any issues with the LoadBalancer – the port-forwarding can be used here:

```
$ kubectl -n monitoring port-forward prometheus-server-85989544df-pgb8c 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
```

In this way, Prometheus will be accessible with the _localhost:9090_ URL.

### Monitoring configuration

Okay – we’ve started the Prometheus, now we can collect some metrics.

Let’s begin with the simplest task – spin up some service, an exporter for it, and configure Prometheus to collect its metrics.

#### Redis && `redis_exporter`

For this, we can use the Redis server and the [`redis_exporter`](https://github.com/oliver006/redis_exporter).

Create a new deployment file `roles/monitoring/templates/tests/redis-server-and-exporter-deployment.yml.j2`:

```
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
      - name: redis-exporter
        image: oliver006/redis_exporter:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121
---
kind: Service
apiVersion: v1
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
  - name: redis
    protocol: TCP
    port: 6379
    targetPort: 6379
  - name: prom
    protocol: TCP
    port: 9121
    targetPort: 9121
```

Annotations here:

- _prometheus.io/scrape_ – used for filters in pods and services, see the [Roles](https://rtfm.co.ua/?p=23859#kubernetessdconfig_roles) part of this post UPDATE
- _prometheus.io/port_ – non-default port can be specified here
- _prometheus.io/path_ – and an exporter’s metrics path can be changed here from the default `/metrics`

See [Per-pod Prometheus Annotations](https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations).

Pay attention, that we didn’t set a namespace here – the Redis service and its exporter will be created in the _default_ namespace, and we will see soon what will happen because of this.

Deploy it – manually, it’s a testing task, no need to add to the Ansible:

```
$ kubectl apply -f roles/monitoring/templates/tests/redis-server-and-exporter-deployment.yml.j2
deployment.extensions/redis created
service/redis created
```

Check it – it has to have two containers running.

In the default namespace find a pod:

```
$ kubectl get pod
NAME                                 READY   STATUS    RESTARTS   AGE
redis-698cd557d5-xmncv               2/2     Running   0          10s
reloader-reloader-55448df76c-9l9j7   1/1     Running   0          23m
```

And containers inside:

```
$ kubectl get pod redis-698cd557d5-xmncv -o jsonpath='{.spec.containers[*].name}'
redis redis-exporter
```

Okay.

Now, add metrics collection from this exporter  – update the `prometheus-configmap.yml.j2` ConfigMap – add a new target, _redis_:

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      external_labels:
        monitor: 'eks-dev-monitor'
    
    scrape_configs:
      
      - job_name: 'prometheus'
        scrape_interval: 5s
        static_configs:
          - targets: ['localhost:9090']
      - job_name: 'redis'
        static_configs:
          - targets: ['redis:9121']
```

Deploy it, and check the _Targets_ of the Prometheus:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_165007.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_165007.png)

Nice – a new Target appeared here.

But why it’s failed with the _ **«Get http://redis:9121/metrics: dial tcp: lookup redis on 172.20.0.10:53: no such host»** _ error?

#### Prometheus ClusterRole, ServiceAccount, and ClusterRoleBinding

So, as we remember we started our Prometheus in the _monitoring_ namespace:

```
$ kubectl get ns monitoring
NAME         STATUS   AGE
monitoring   Active   25m
```

While in the Redis deployment we didn’t set a namespace and, accordingly, its pod was created in the _default_ namespace:

```
$ kubectl -n default get pod
NAME                                 READY   STATUS    RESTARTS   AGE
redis-698cd557d5-xmncv               2/2     Running   0          12m
```

Or in this way:

```
$ kubectl get pod redis-698cd557d5-xmncv -o jsonpath='{.metadata.namespace}'
default
```

To make Prometheus available to get access to all namespace on the cluster – add a `ClusterRole`, `ServiceAccount` and `ClusterRoleBinding`, see the [Kubernetes: part 5 — RBAC authorization with a Role and RoleBinding example](https://rtfm.co.ua/en/kubernetes-part-5-rbac-authorization-with-a-role-and-rolebinding-example/) post for more details.

Also, this ServiceAccount will be used for [Prometheus Kubernetes Service Discovery](https://rtfm.co.ua/?p=23859#Prometheus_Kubernetes_Service_Discovery) EDIT.

Add a `roles/monitoring/templates/prometheus-rbac.yml.j2` file:

```
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - services
  - endpoints
  - pods
  - nodes
  - nodes/proxy
  - nodes/metrics
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---  
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
```

Add its execution after creating the ConfigMap and before deployment of the Prometheus server:

```
- name: "Create the Monitoring Namespace"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-ns.yml.j2"
- name: "Create prometheus.yml ConfigMap"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-configmap.yml.j2"
- name: "Create Prometheus ClusterRole"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-rbac.yml.j2"
- name: "Deploy Prometheus server and its LoadBalancer"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-deployment.yml.j2"
```

Update tje `prometheus-deployment.yml` – in its `spec` add the `serviceAccountName`:

```
...
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus-server
        image: prom/prometheus
...
```

Besides of this, to call for pods in another namespace we need to use an FQDN with the namespace specified, in this case, the address to access the Redis exporter will _redis.default.svc.cluster.local_, see the [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/).

Update the ConfigMap – change the Redis address:

```
...
    scrape_configs:
      - job_name: 'prometheus'
        scrape_interval: 5s
        static_configs:
          - targets: ['localhost:9090']
      - job_name: 'redis'
        static_configs:
          - targets: ['redis.default.svc.cluster.local:9121']
```

Deploy with Ansible to update everything:

```
$ ansible-playbook eks-cluster.yml --tags monitoring
```

Check the _Targets_ now:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_170406.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_170406.png)And metrics:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_170454.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200406_170454.png)

### Prometheus Kubernetes Service Discovery

`static_configs` in Prometheus is a good thing, but what if you’ll have to collect metrics from hundreds of such services?

The solution is to use the [`kubernetes_sd_config`](https://rtfm.co.ua/goto/https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#kubernetes_sd_config) feature.

#### `kubernetes_sd_config` roles

Kubernetes SD in Prometheus has a collection of so-called “roles”, which defines how to collect and display metrics.

Each such a Role has its own set of labels, see the [documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config):

- _node_: will get by one target on each cluster’s WorkerNode кластера, will collect kubelet’s metrics
- _service_: will find and return each Service and its Port
- _pod_: all pods and will return its containers as targets to grab metrics from
- _endpoints_: will create targets from each an Endpoint for each Service found in a cluster
- _ingress_: will create targets for each Path for each Ingress

The difference is only in the labels returned, and what address will be used for each such a target.

Configs examples (more at the end of this post):

- [prometheus/documentation/examples/prometheus-kubernetes.yml](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml)
- [Running Prometheus on Kubernetes](https://linuxacademy.com/blog/kubernetes/running-prometheus-on-kubernetes/)

Also, to connect to the cluster’s API server with SSL/TLS encryption need to specify a Central Authority certificate of the server to validate it, see the [Accessing the API from a Pod](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod).

And for the authorization on the API server, we will use a token from the `bearer_token_file`, which is mounted from the `serviceAccountName: prometheus`, which we’ve set in the deployment above.

##### node role

Let’s see what we will have in each such a role.

Begin with the _node_ role – add to the `scrape_configs`, can copy-paste from the [example](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L59):

```
...
      - job_name: 'kubernetes-nodes'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
```

Without relabeling for now – just run and check for targets

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112627.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112627.png)

Check the _Status > Service Discovery_ for targets and labels discovered

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112733.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112733.png)

And _kubelet_*_ metrics:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112805.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_112805.png)

Add some relabeling see the [`relabel_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) and [Life of a Label](https://www.robustperception.io/life-of-a-label).

What do they suggest there?

```
...
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics
...
```

1. collect labels ` __meta_kubernetes_node_label_alpha_eksctl_io_cluster_name`, `__ meta_kubernetes_node_label_alpha_eksctl_io_nodegroup_name` etc, select with `(.+)` – will get labels like `alpha_eksctl_io_cluster_name`, `alpha_eksctl_io_nodegroup_name`, etc
2. update the ` __address__ ` label – set the _kubernetes.default.svc:443_ value to create an address to call targets
3. get a value from the ` __meta_kubernetes_node_name`and update the `__ metrics_path__` label – set the _/api/v1/nodes/__meta_kubernetes_node_name/proxy/metrics_

As result, Prometheus will construct a request to the _kubernetes.default.svc:443/api/v1/nodes/ip-10-1-57-13.eu-west-2.compute.internal/proxy/metrics_ – and will grab metrics from this WorkerNode.

Update, check:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_115440.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_115440.png)

Nice!

##### pod role

Now, let’s see the _pod_ role example from the same resources:

```
...
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
```

Check:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_120214.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_120214.png)

All nodes were found, but why so much?

Let’s add the _prometheus.io/scrape: “true”_ annotation check:

```
...
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
...
```

Which is already added to our Redis, for example:

```
...
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
...
```

And the result:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_121841.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_121841.png)

_http://10.1.44.135:6379/metrics_ – is the Redis server without metrics.

Do you want to remove it? Add one more filter:

```
...
        - source_labels: [__meta_kubernetes_pod_container_name]
          action: keep
          regex: .*-exporter
...
```

I.e. we’ll collect only metrics with the “_-exporter_” string in the `__meta_kubernetes_pod_container_name` label.

Check:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_122337.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_122337.png)

Okay – we’ve seen how the Roles in Prometheus Kubernetes Service Discovery is working.

What we have left here?

- node-exporter
- kube-state-metrics
- cAdvisor
- metrics-server

#### node-exporter metrics

Add the [`node_exporter`](https://github.com/prometheus/node_exporter) to collect metrics from EC2 instances.

Because a pod with exporter needs to be placed on each WorkerNode – use the [`DaemonSet`](https://rtfm.co.ua/kubernetes-znakomstvo-chast-1-arxitektura-i-osnovnye-komponenty-obzor/#DaemonSet) type here.

Create a `roles/monitoring/templates/prometheus-node-exporter.yml.j2` file – this will spin up a pod on each WorkerNode in the _monitoring_ namespace, and will add a Service make  Prometheus available to grab metrics from endpoints:

```
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: node-exporter
  labels:
    name: node-exporter
  namespace: monitoring
spec:
  template:
    metadata:
      labels:
        name: node-exporter
        app: node-exporter
      annotations:
         prometheus.io/scrape: "true"
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
        - ports:
            - containerPort: 9100
              protocol: TCP
          resources:
            requests:
              cpu: 0.15
          securityContext:
            privileged: true
          image: prom/node-exporter
          args:
            - --path.procfs
            - /host/proc
            - --path.sysfs
            - /host/sys
            - --collector.filesystem.ignored-mount-points
            - '"^/(sys|proc|dev|host|etc)($|/)"'
          name: node-exporter
          volumeMounts:
            - name: dev
              mountPath: /host/dev
            - name: proc
              mountPath: /host/proc
            - name: sys
              mountPath: /host/sys
            - name: rootfs
              mountPath: /rootfs
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: dev
          hostPath:
            path: /dev
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /
---
kind: Service
apiVersion: v1
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    app: node-exporter
  ports:
  - name: node-exporter
    protocol: TCP
    port: 9100
    targetPort: 9100
```

Add its execution to the `roles/monitoring/tasks/main.yml` file:

```
...
- name: "Deploy node-exporter to WorkerNodes"
  command: "kubectl apply -f roles/monitoring/templates/prometheus-node-exporter.yml.j2"
```

And let’s think about how we can collect metrics now.

The first question is – which Role to use here? We need to specify the _9100_ port – then we can’t use the `node` role – it has no a Port value:

```
$ kubectl -n monitoring get node
NAME                                        STATUS   ROLES    AGE     VERSION
ip-10-1-47-175.eu-west-2.compute.internal   Ready    <none>   3h36m   v1.15.10-eks-bac369
ip-10-1-57-13.eu-west-2.compute.internal    Ready    <none>   3h37m   v1.15.10-eks-bac369
```

What about the Service role?

> The address will be set to the Kubernetes DNS name of the service and respective service port

Let’s see:

```
$ kubectl -n monitoring get svc
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP  PORT(S)        AGE
node-exporter           ClusterIP      172.20.242.99    <none>       9100/TCP       37m
```

Okay, what about labels for the [`service`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service) role? All good, but it has no labels for pods on the Worker Nodes – and we need to collect metrircs from each `node_exporter` pod on each WorkerNode.

Let’s go further – the `endpoints` role:

```
$ kubectl -n monitoring get endpoints
NAME                    ENDPOINTS                          AGE
node-exporter           10.1.47.175:9100,10.1.57.13:9100   44m
prometheus-server-alb   10.1.45.231:9090,10.1.53.46:9090   3h24m
```

_10.1.47.175:9100,10.1.57.13:9100_ – aha, here they are!

So – we can use the `endpoints` role which also has the  `__meta_kubernetes_endpoint_node_name` label.

Try it:

```
...
      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep
```

Check targets:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_134404.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_134404.png)

And metrics:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_134435.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_134435.png)See requests examples for `node_exporter` in the [Grafana: создание dashboard](https://rtfm.co.ua/grafana-sozdanie-dashboard/) post (_in Russian_).

#### `kube-state-metrics`

To collect metrics about Kubernetes resources we can use the  [`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics).

Add its installation to the `roles/monitoring/tasks/main.yml`:

```
...
- git:
    repo: 'https://github.com/kubernetes/kube-state-metrics.git'
    dest: /tmp/kube-state-metrics
- name: "Install kube-state-metrics"
  command: "kubectl apply -f /tmp/kube-state-metrics/examples/standard/"
```

The deployment itself can be observed in the [https://github.com/kubernetes/kube-state-metrics/blob/master/examples/standard/deployment.yaml](https://github.com/kubernetes/kube-state-metrics/blob/master/examples/standard/deployment.yaml) file.

We can skip Service Discovery here as we will have the only `kube-state-metrics` service, so use the `static_configs`:

```
...
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
```

Check targets:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142211.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142211.png)

And metrics, for example – `kube_deployment_status_replicas_available`:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142256.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142256.png)

#### cAdvisor

The [`cAdvisor`](https://github.com/google/cadvisor) is too well-known – the most widely used system to collect data about containers.

it’s already unbounded to the Kubernetes, so no need for a dedicated reporter – just grab tits metrics. An example can be found in the same [https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L102](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L102) file.

Update the `roles/monitoring/templates/prometheus-configmap.yml.j2`:

```
...
      - job_name: 'cAdvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
```

Deploy, check:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142708.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142708.png)

We have two Kubernetes WorkerNode in our cluster, and we can see metrics from both – great:

[![](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142909.png "Kubernetes: мониторинг с Prometheus")](https://rtfm.co.ua/wp-content/uploads/2020/04/Screenshot_20200407_142909.png)

#### `metrics-server`

There are exporters for it too, but I don’t need them now – just install it to make Kubernetes Limits work and to use the `kubectl top`.

Its installation was a bit more complicated earlier, see the [Kubernetes: running metrics-server in AWS EKS for a Kubernetes Pod AutoScaler](https://rtfm.co.ua/en/kubernetes-running-metrics-server-in-aws-eks-for-a-kubernetes-pod-autoscaler/), but now on the EKS it works out of the box.

Update the `roles/monitoring/tasks/main.yml`:

```
...
- git:
    repo: "https://github.com/kubernetes-sigs/metrics-server.git"
    dest: "/tmp/metrics-server"
- name: "Install metrics-server"
  command: "kubectl apply -f /tmp/metrics-server/deploy/kubernetes/"
...
```

Deploy, check pods in  the `kube-system` namespace:

```
$ kubectl -n kube-system get pod
NAME                                 READY   STATUS    RESTARTS   AGE
aws-node-s7pvq                       1/1     Running   0          4h42m
...
kube-proxy-v9lmh                     1/1     Running   0          4h42m
kube-state-metrics-6c4d4dd64-78bpb   1/1     Running   0          31m
metrics-server-7668599459-nt4pf      1/1     Running   0          44s
```

The `metrics-server` pod is here – good.

And try the `top node` in a couple of minutes:

```
$ kubectl top node
NAME                                        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-1-47-175.eu-west-2.compute.internal   47m          2%     536Mi           14%
ip-10-1-57-13.eu-west-2.compute.internal    58m          2%     581Mi           15%
```

And for pods:

```
$ kubectl top pod
NAME                                 CPU(cores)   MEMORY(bytes)
redis-6d9cf9d8cb-dfnn6               2m           5Mi
reloader-reloader-55448df76c-wsrfv   1m           7Mi
```

That’s all, in general.

### Useful links

- [Kubernetes Monitoring with Prometheus -The ultimate guide](https://sysdig.com/blog/kubernetes-monitoring-prometheus/)
- [Monitoring multiple federated clusters with Prometheus – the secure way](https://banzaicloud.com/blog/prometheus-federation/)
- [Monitoring Your Kubernetes Infrastructure with Prometheus](https://www.weave.works/blog/monitoring-kubernetes-infrastructure/)
- [Volume Monitoring in Kubernetes with Prometheus](https://blog.argoproj.io/volume-monitoring-in-kubernetes-with-prometheus-3a185e4c4035)
- [Trying Prometheus Operator with Helm + Minikube](https://medium.com/faun/trying-prometheus-operator-with-helm-minikube-b617a2dccfa3)
- [How to monitor your Kubernetes cluster with Prometheus and Grafana](https://chris-vermeulen.com/how-to-monitor-your-kubernetes-cluster-with-prometheus-and-grafana--the-whole--long--story-/)
- [Monitoring Redis with builtin Prometheus](https://kubedb.com/docs/0.12.0/guides/redis/monitoring/using-builtin-prometheus/)
- [Kubernetes in Production: The Ultimate Guide to Monitoring Resource Metrics with Prometheus](https://www.replex.io/blog/kubernetes-in-production-the-ultimate-guide-to-monitoring-resource-metrics)
- [How To Monitor Kubernetes With Prometheus](https://phoenixnap.com/kb/prometheus-kubernetes-monitoring)
- [Using Prometheus + grafana + node-exporter](https://programming.vip/docs/cluster-monitoring-using-prometheus-grafana-node-exporter.html)
- [Running Prometheus on Kubernetes](https://linuxacademy.com/blog/kubernetes/running-prometheus-on-kubernetes/)
- [Prometheus Self Discovery on Kubernetes](https://developer.sh/posts/prometheus-self-discovery)
- [Kubernetes – Endpoints](https://theithollow.com/2019/02/04/kubernetes-endpoints/)

#### Configs

- [doks-monitoring/manifest/prometheus-configmap.yaml](https://github.com/do-community/doks-monitoring/blob/master/manifest/prometheus-configmap.yaml)
- [prometheus/documentation/examples/prometheus-kubernetes.yml](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml)
- [charts/stable/prometheus/templates/server-configmap.yaml](https://github.com/helm/charts/blob/master/stable/prometheus/templates/server-configmap.yaml)

#### Misc

- [Grafana App for Kubernetes](https://grafana.com/grafana/plugins/grafana-kubernetes-app)

- [![](https://rtfm.co.ua/wp-content/plugins/share-me//assets/images/cercle/facebook.png)](http://www.facebook.com/sharer.php?u=https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/&t=Kubernetes%3A+monitoring+with+Prometheus+%26%238211%3B+exporters%2C+a+Service+Discovery%2C+and+its+roles "Share on Facebook")
- [![](https://rtfm.co.ua/wp-content/plugins/share-me//assets/images/cercle/twitter.png)](http://twitter.com/share?text=Kubernetes%3A+monitoring+with+Prometheus+%26%238211%3B+exporters%2C+a+Service+Discovery%2C+and+its+roles-&url=https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/&via=RTFM:%20Linux,%20DevOps%20and%20system%20administration "Tweet This Post")
- [![](https://rtfm.co.ua/wp-content/plugins/share-me//assets/images/cercle/googleplus.png)](https://plusone.google.com/_/+1/confirm?hl=fr-FR&url=https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/ "Share on Google+")
- [![](https://rtfm.co.ua/wp-content/plugins/share-me//assets/images/cercle/tumblr.png)](http://www.tumblr.com/share/photo?source=https%3A%2F%2Frtfm.co.ua%2Fwp-content%2Fuploads%2F2017%2F06%2Fkuber-1.png&caption=Kubernetes%3A+monitoring+with+Prometheus+%26%238211%3B+exporters%2C+a+Service+Discovery%2C+and+its+roles&clickthru=https%3A%2F%2Frtfm.co.ua%2Fen%2Fkubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles%2F "Share on Tumblr")
- [![](https://rtfm.co.ua/wp-content/plugins/share-me//assets/images/cercle/linkedin.png)](http://www.linkedin.com/shareArticle?mini=true&url=https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/&title=Kubernetes%3A+monitoring+with+Prometheus+%26%238211%3B+exporters%2C+a+Service+Discovery%2C+and+its+roles&source=RTFM:%20Linux,%20DevOps%20and%20system%20administration)

  

### Similar posts

- <small>04/08/2020</small> [Kubernetes: мониторинг с Prometheus](https://rtfm.co.ua/kubernetes-monitoring-s-prometheus/)<small> (0)</small>  
- <small>03/10/2019</small> [Prometheus: RTFM blog monitoring set up with Ansible – Grafana, Loki, and promtail](https://rtfm.co.ua/en/prometheus-rtfm-blog-monitoring-set-up-with-ansible-grafana-loki-and-promtail/)<small> (0)</small>  
- <small>03/09/2019</small> [Prometheus: мониторинг для RTFM – Grafana, Loki и promtail](https://rtfm.co.ua/prometheus-monitoring-dlya-rtfm-grafana-loki-i-promtail/)<small> (0)</small>  
- <small>04/24/2020</small> [AWS Elastic Kubernetes Service: a cluster creation automation, part 1 – CloudFormation](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-a-cluster-creation-automation-part-1-cloudformation/)<small> (0)</small>  
- <small>03/31/2020</small> [AWS Elastic Kubernetes Service: автоматизация создания кластера, часть 1 – CloudFormation](https://rtfm.co.ua/aws-elastic-kubernetes-service-avtomatizaciya-sozdaniya-klastera-chast-1-cloudformation/)<small> (0)</small>