Creating a Service Monitor in k8s
Tags: kubernetes, servicemonitor, prometheus, grafana
Published: 2022-12-18 05:49:31

Prometheus is an excellent monitoring tool developed for Kubernetes monitoring.
Many of the helm charts like Nginx, RabbitMq provides inbuilt provision for metrics exposing for Prometheus monitoring.
These metrics are generally exposed on an endpoint, say, `/metrics` and prometheus pulls down, process them on certain interval.

But the question arrises how does Prometheus knows from where to scrap metrics?
The answer to this is via **Service Monitor**.

_Note: This articles assumes the reader has basic knowledge of Prometheus, Grafana, Kubernetes, Helm Charts_

## What is Service Monitor?

Service Monitor is a [CRD](https://github.com/prometheus-operator/prometheus-operator/blob/main/README.md#customresourcedefinitions) provided by Prometheus Operator, which provides configuration on how provided services should be monitored. Or how we wish to collect metrics from different services.
Prometheus operator uses this service monitor and configures itself internally.

## Requirements

- All services which are wished to be monitored must have following annotations in their manifest configuration:

```yaml
  annotations:
    prometheus.io/port: "metrics"
    prometheus.io/scrape: "true"
```
Here, 
First line represents PORT number(We used a PORT name instead of actual port number here).
Second line represents a flag whether to scrap metrics or not.

- A service monitor which monitors which services' metrics to scrap.

## Example

Let's say we have a simple k8s deployment(service) having following manifest:

```yaml
#########################################################
# Deployment
#########################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service-deployment
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-service
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      namespace: production
      labels:
        app: my-service
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: my-service-pod
          image: myacr.azurecr.io/my_service:latest
          envFrom:
            - secretRef:
                name: prod-secrets
          ports:
            - containerPort: 4050
          resources:
            requests:
              cpu: 250m
              memory: 250Mi
            limits:
              cpu: 500m
              memory: 500Mi
---
#########################################################
# Service
#########################################################
apiVersion: v1
kind: Service
metadata:
  name: my-service-service
  namespace: production
spec:
  type: ClusterIP
  selector:
    app: my-service
  ports:
    - port: 4050
      protocol: TCP
      targetPort: 4050
      name: "metrics"
---
```

_Note: the config may look daunting, but it just has few extra optional parameters_

The above is a simple K8s Service and Deployement having type ClusterIP (not exposed as we have Nginx gateway)

If we assume, we have already setup our [prom-client]() in our NodeJs image to scrap metrics, we would have a endpoint `/metrics` where all prometheus metrics are exposed.

Now, we just need to connect this service to prometheus.

### STEP 1: 

Add required annotations in service spec template:

```yaml
  annotations:
    prometheus.io/port: "metrics"
    prometheus.io/scrape: "true"
```
and add labels to identify service:

```yaml
  labels:
    app.kubernetes.io/part-of: dms
```
_Note- Put lables according to new standard as defined in K8s [Docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/)_

Our final service manifest shall look like:

_Note: no changes to deployment maifest needed_

```yaml
#########################################################
# Service
#########################################################
apiVersion: v1
kind: Service
metadata:
  name: my-service-service
  namespace: production
  annotations:
    prometheus.io/port: "metrics"
    prometheus.io/scrape: "true"
  labels:
    app.kubernetes.io/part-of: dms
spec:
  type: ClusterIP
  selector:
    app: my-service
  ports:
    - port: 4050
      protocol: TCP
      targetPort: 4050
      name: "metrics"
---
```

Deploy this configuration.
For eg `kubectl apply -f my-service.yml`

### STEP 2:

After deployment new service manifest, create a new Service monitor as follows:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dms-service-monitor
  namespace: monitoring
spec:
  endpoints:
    - interval: 15s
      port: metrics
      scrapeTimeout: 14s
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app.kubernetes.io/part-of: dms
```
Note:
1. The **selector:** config should match with that deined in service as above.
2. Namespace can be different. You can include specifc namespaces to monitor using `namespaceSelector:`.

With this our setup is done.

### STEP 3:

Check if Prometheus is collecting your metrics.
Head over to your prometheus server (using port-forward if not exposed). On top menu select Status>Targets.

![Prometheus Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jtyvb63fyknxq0g9pe0y.png)

You can see Prometheus has discovered the new target **my-service-monitor** and working successfully.

You can even see your scrape data on home page using query.

## Troubleshoot

If you have deployed Prometheus using `kube-prometheus-stack` helm chart you might need to upgrade chart using following chart values:

`values.yml`
```yaml
prometheus:
  prometheusSpec:
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
```

For eg:
`helm upgrade prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --namespace monitoring --values values.yml`

This way we remove pre configured selectors constraints from prometheus deployement.

Also there is an excellent answer on [stackoverflow](https://stackoverflow.com/questions/52991038/how-to-create-a-servicemonitor-for-prometheus-operator) which describes the whole monitoring flow to troubleshoot.

![Monitoring](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yecvp0jqnm48nzpgpp1k.png)

### Conclusion

So we see how easily we can connect our Prometheus and Nodejs metrics on our k8s cluster.
You dont need a separate Service Monitor everytime you create a new service so long your service correctly passes constraints inside **selector** in service monitor.

**Whola! Both you and I learnt something new today. Congrats
👏 👏 👏 **

## Further Reading:

1. https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
2. https://sysdig.com/blog/kubernetes-monitoring-prometheus/
3. https://stackoverflow.com/questions/52991038/how-to-create-a-servicemonitor-for-prometheus-operator