Proactive Kubernetes Monitoring with Alerting
Tags: kubernetes, devops, grafana, prometheus
Published: 2022-06-15 01:11:47

Last year I had written about the [importance of service monitoring and observability](https://dev.to/mikeyglitz/kubernetes-service-monitoring-and-observability-3g67).
While service monitoring is important, service monitoring requires active involvement from application maintainers. When supporting applications and services which service real users, it's important to become aware of problems when they first happen.
The longer your service is down, experiencing errors or performance issues, or otherwise buggy, the more negative an experience you provide for your users.
Alerting provides application developers a passive mechanism for becoming aware of problems involving their applications.
Instead of being informed by users that a problem is occurring in an application, developers receive automated alarms which inform them a problem is occurring.

## Kubernetes Alerting

Alerting isn't natively provided in Kubernetes. Fortunately there are a few open source applications which can satisfy the alerting needs. As mentioned in my previous post, [Linkerd](linkerd.io) provides many monitoring tools out-of-the box, but some customizations need to be made in order to provide alerting.
- [Prometheus](prometheus.io) is used to provide metrics monitoring (Memory usage, CPU usage, Network usage, etc.)
- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is used to create and manage alarms using Prometheus metrics
- [Grafana] (grafana.com) A visualization application which displays graphs and charts based on data received from metrics. Grafana also provides alerting.
- [Prometheus Operator](https://prometheus-operator.dev/docs/) a project which expresses Prometheus services, rules, and alarms as Kubernetes Custom Resource Definitions (CRDs). Prometheus Operator is a Kubernetes-idiomatic way of declaring Prometheus services.

## Alerting Channels

Before going through the process of setting up alarms, it is important do decide how you want to be notified.
For this tutorial, I'll be setting up email alerts.
Simple Mail Transfer Protocol (SMTP) is the service which allows applications, in this case our monitoring services, to send emails to the people supporting the applications. By default, most Internet Service Providers (ISPs), Email Providers, and Cloud Providers block the port that SMTP sends emails on.

### Setting up a SMTP Relay

A SMTP Relay is a service which can be used as a proxy for SMTP traffic when you want to send emails external to your Local Area Network (LAN). I've researched a few SMTP relays and ultimately decided on Dynu.
[Dynu](https://www.dynu.com/) won me over through providing [easy-to-use documentation](https://www.dynu.com/Resources/Tutorials/EmailServices/EmailServerSetup/How-to-configure-postfix-with-SMTP-relay) on how to set up their SMTP relay service. Also at $9/year, the service is not unaffordable.

After opening an account with Dynu, my next step was to deploy the SMTP service onto Kubernetes.
Since I've started working with Kubernetes, I've discovered the convenience of working with Helm charts. [ArtifuctHub](artifacthub.io) is now my go-to for finding quick and easy helm charts to install onto Kubernetes.
I decided to deploy SMTP using the [bokysan postfix chart](https://artifacthub.io/packages/helm/docker-postfix/mail).
I started out with these initial `values.yml` for the helm chart:

```yaml
config:
  general:
    TZ: America/New_York
    LOG_FORMAT: json
    RELAYHOST: "{{ relay_host }}"
    ALLOWED_SENDER_DOMAINS: "cluster.local {{ domain_name }}"
secret:
  RELAYHOST_USERNAME: "{{ relay_username }}"
  RELAYHOST_PASSWORD: "{{ relay_password }}"
```

I installed the helm release using [Ansible](https://ansible.com), but you can install with the following helm commands:

```bash
helm repo add docker-postfix https://bokysan.github.io/docker-postfix/
helm install --values values.yml --namespace=mail --create-namespace sender docker-postfix/mail
```

#### Forking the Postfix Base Image

Prometheus Alertmanager is built on Golang. A limitation of the SMTP integration in Alertmanager is that [TLS is required for remote connections](https://github.com/prometheus/alertmanager/issues/275#issuecomment-206327574).
I attempted to use the `certs.create` value from the helm chart values, but the resulting certificate was not accepted by Alertmanager.
As a result, I had to create a [new image from bokysan's Postfix image](https://hub.docker.com/r/mikeyglitz/postfix).

The new image detects for the following files to be mounted in `/mnt/certs`:
- `tls.crt`
- `tls.key`
- `ca.crt`

If the files are present, they are copied into `/etc/ssl` and the postfix configuration at `/etc/postfix/main.cf` is updated to enable the SSL settings.

This set up is useful if using [cert-manager](cert-manager.io) to set up certificates using Kubernetes Secrets.

> ⚠️ When using the Postfix image, please note that the **signing algorithm for your certificates needs to be RSA**

I'm still able to use Bokysan's Helm chart with the following values:
```yaml
image:
  repository: mikeyglitz/postfix
  tag: latest
extraVolumes:
  - name: tls-cert
    secret:
      secretName: mail-tls
extraVolumeMounts:
  - name: tls-cert
    mountPath: /mnt/certs
    readOnly: true
config:
  general:
    TZ: America/New_York
    LOG_FORMAT: json
    RELAYHOST: "{{ relay_host }}"
    ALLOWED_SENDER_DOMAINS: "cluster.local {{ domain_name }}"
secret:
  RELAYHOST_USERNAME: "{{ relay_username }}"
  RELAYHOST_PASSWORD: "{{ relay_password }}"
```

## Installing Prometheus Operator

Prometheus Operator is a suite of applications leveraging the [Operator Pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/#:~:text=The%20Operator%20pattern%20aims%20to%20capture%20the%20key,and%20how%20to%20react%20if%20there%20are%20problems.) for managing Prometheus applications in a way that is idiomatic for Kubernetes.
Configurations, Prometheus instances, and Alertmanager instances are managed using [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/).

Before we can create the Prometheus Stack, we need to create a namespace where the prometheus resources will live

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  annotations:
    linkerd.io/inject: enabled
```

Installation of the Prometheus Operator is handled using the [Prometheus Operator Helm Chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack).
The Prometheus Operator Helm Chart installs the entire Prometheus Stack:

- Prometheus
- Alertmanager
- Grafana
- Prometheus-Operator
- Prometheus NodeExporter

The `values.yml` for the chart is quite extensive and will take up a lot of space in the subsequent sections. I've broken down the values that go into the helm chart so that they can be better comprehended.
With Linkerd, additional customizations need to be made to ensure that the Linkerd dashboards work by installing the correct Prometheus rules. This can be accomplished with the following `values.yaml`

### Prometheus Operator Values

Since I'm using [Linkerd](linkerd.io) as my service mesh, and have created the monitoring namespace where I will be installing the Helm Chart using the `linkerd.io/inject` annotation, I will need to set the values for the Prometheus Operator so that the webhook is skipped from linkerd injection.

```yaml
# Configure Prometheus -- we need to skip Linkerd injection or
# the operator will not install
prometheusOperator:
  admissionWebhooks:
    patch:
      podAnnotations:
        linkerd.io/inject: disabled
    certManager:
      enabled: "true"
      issuerRef:
        name: monitoring-issuer
        kind: Issuer
```

This configuration also utilizes [cert-manager](cert-manager.io) to inject certificates into the Prometheus Webhook. The certificates generated by the `monitoring-issuer` would be signed by our [Root CA](https://cert-manager.io/docs/configuration/ca/).

### Prometheus Values

The default prometheus configuration must be modified to include [custom scrapers](https://linkerd.io/2.11/tasks/external-prometheus/) so that Prometheus can export metrics to Linkerd. Without this configuration, the [Linkerd Dashboard](https://linkerd.io/2.11/features/dashboard/) will not function properly.

```yaml
prometheus:
  prometheusSpec:
    evaluationInterval: 10s
    scrapeInterval: 10s
    scrapeTimeout: 10s
    resources:
      requests:
        memory: 4Gi
    additionalScrapeConfigs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']

    - job_name: 'grafana'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ['monitoring']
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_name
        action: keep
        regex: ^grafana$
    #  Required for: https://grafana.com/grafana/dashboards/315
    - job_name: 'kubernetes-nodes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
      metric_relabel_configs:
      - source_labels: [__name__]
        regex: '(container|machine)_(cpu|memory|network|fs)_(.+)'
        action: keep
      - source_labels: [__name__]
        regex: 'container_memory_failures_total' # unneeded large metric
        action: drop

    - job_name: 'linkerd-controller'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - 'linkerd'
          - 'monitoring'
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: admin-http
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: component

    - job_name: 'linkerd-service-mirror'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: linkerd-service-mirror;admin-http$
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: component

    - job_name: 'linkerd-proxy'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_name
        - __meta_kubernetes_pod_container_port_name
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
        action: keep
        regex: ^linkerd-proxy;linkerd-admin;linkerd$
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      # special case k8s' "job" label, to not interfere with prometheus' "job"
      # label
      # __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =>
      # k8s_job=foo
      - source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
        action: replace
        target_label: k8s_job
      # drop __meta_kubernetes_pod_label_linkerd_io_proxy_job
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
      # __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =>
      # deployment=foo
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      # drop all labels that we just made copies of in the previous labelmap
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      # __meta_kubernetes_pod_label_linkerd_io_foo=bar =>
      # foo=bar
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
      # Copy all pod labels to tmp labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
        replacement: __tmp_pod_label_$1
      # Take `linkerd_io_` prefixed labels and copy them without the prefix
      - action: labelmap
        regex: __tmp_pod_label_linkerd_io_(.+)
        replacement:  __tmp_pod_label_$1
      # Drop the `linkerd_io_` originals
      - action: labeldrop
        regex: __tmp_pod_label_linkerd_io_(.+)
      # Copy tmp labels into real labels
      - action: labelmap
        regex: __tmp_pod_label_(.+)
```

### Grafana Options

> ℹ️ At the time of writing, using an external instance of Grafana
> is only supported by an upcoming release of Linkerd 2.12.
> Linkerd 2.12 can only be found on the **edge** branch, not the *stable*.
> Subsequent sections will cover how to adjust the Linkerd install
> to support an external Grafana instance in more detail.

Grafana needs to be updated to pre-install the Linkerd dashboard.
As per the [Monitoring and Observability post](https://dev.to/mikeyglitz/kubernetes-service-monitoring-and-observability-3g67), [Loki](https://grafana.com/oss/loki/) also needs to be added as a data source for Grafana.

To support [Grafana's Alerting Functionality]() Alertmanager can be added as a data source so that Alertmanager rules will appear in grafana.

> ℹ️ At the time of writing, Alertmanager is only supported in Grafana plugins alpha.
> Grafana plugins alpha can only be added in the grafana configuration via the `plugins.enable_alpha` option in `grafana.ini`.

```yaml
# Grafana options -- pre-install Linkerd dashboards
# and configure datasources
grafana:
  grafana.ini:
    server:
      root_url: '%(protocol)s://%(domain)s:/grafana/'
    auth:
      disable_login_form: false
    auth.anonymous:
      enabled: true
      org_role: Editor
    auth.basic:
      enabled: true
    analytics:
      check_for_updates: false
    panels:
      disable_sanitize_html: true
    log:
      mode: console
    log.console:
      format: text
      level: info
    plugins:
      enable_alpha: true
    smtp:
      enabled: true
      from_address: grafana@<domain>
      host: sender-mail.mail-sender:587
      skip_verify: true
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'default'
        orgId: 1
        folder: ''
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      # Logging dashboard - https://grafana.com/grafana/dashboards/7752
      logging:
        gnetId: 7752
        revision: 5
        datasource: prometheus
      # all these charts are hosted at https://grafana.com/grafana/dashboards/{id}
      top-line:
        gnetId: 15474
        revision: 3
        datasource: prometheus
      health:
        gnetId: 15486
        revision: 2
        datasource: prometheus
      kubernetes:
        gnetId: 15479
        revision: 2
        datasource: prometheus
      namespace:
        gnetId: 15478
        revision: 2
        datasource: prometheus
      deployment:
        gnetId: 15475
        revision: 5
        datasource: prometheus
      pod:
        gnetId: 15477
        revision: 2
        datasource: prometheus
      service:
        gnetId: 15480
        revision: 2
        datasource: prometheus
      route:
        gnetId: 15481
        revision: 2
        datasource: prometheus
      authority:
        gnetId: 15482
        revision: 2
        datasource: prometheus
      cronjob:
        gnetId: 15483
        revision: 2
        datasource: prometheus
      job:
        gnetId: 15487
        revision: 2
        datasource: prometheus
      daemonset:
        gnetId: 15484
        revision: 2
        datasource: prometheus
      replicaset:
        gnetId: 15491
        revision: 2
        datasource: prometheus
      statefulset:
        gnetId: 15493
        revision: 2
        datasource: prometheus
      replicationcontroller:
        gnetId: 15492
        revision: 2
        datasource: prometheus
      prometheus:
        gnetId: 15489
        revision: 2
        datasource: prometheus
      prometheus-benchmark:
        gnetId: 15490
        revision: 2
        datasource: prometheus
      multicluster:
        gnetId: 15488
        revision: 2
        datasource: prometheus
  additionalDataSources:
  - name: alertmanager
    type: alertmanager
    url: http://metrics-kube-prometheus-st-alertmanager:9093
    access: proxy
    orgId: 1
    jsonData:
      implementation: prometheus
  - name: loki   
    type: loki
    access: proxy
    default: false
    editable: true
    url: http://loki:3100
    maximumLines: "300"
    orgId: 1
    jsonData:
      manageAlerts: true
      alertmanagerUid: alertmanager
```

### Configuring Alertmanager

The final piece of the puzzle is configuring Alertmanager.
Alertmanager manages Prometheus alerts and is responsible for forwarding messages to the various alert receivers. Majority of the values below were extracted from the default configuration that the Alertmanager helm chart creates.

The notable differences are the root route. By default the root route doesn't allow a matcher. If we attached a receiver to the root route, we will be constantly notified. The constant notifications would trigger spam blockers to block our Alertmanager emails.

Under the root route, we create a nested `routes` field. The `routes` field is set up so that based on the `matchers` criteria we will receive an alert via email. In this case our alert will trigger whenever there is an alert where its severity is critical.

```yaml
alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job']
      group_wait: 15s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
        - group_by: ['alertname']
          group_wait: 15s
          group_interval: 10s
          repeat_interval: 12h
          matchers:
            - severity="critical"
          receiver: email
    receivers:
      - name: 'null'
      - name: email
        email_configs:
        - to: <recipient@mail.com>
          from: alertmanager@haus.net
          smarthost: sender-mail.mail-sender:587 
          require_tls: true
          tls_config:
            insecure_skip_verify: true
    templates:
      - '/etc/alertmanager/config/*.tmpl'
  alertmanagerSpec:
    externalUrl: https://monitoring.haus.net/alarms
    logFormat: json
    alertmanagerConfigNamespaceSelector:
      matchLabels:
        alertmanagerconfig: enabled
    alertmanagerConfigSelector:
      matchLabels:
        role: alertmanager-config
```

### Installing the chart

The chart may be installed with the following command:

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install --values values.yml --namespace monitoring metrics prometheus-community/prometheus-stack
```


## Configuring Linkerd

In [my previous post](https://dev.to/mikeyglitz/kubernetes-service-monitoring-and-observability-3g67), I had leveraged kustomize to install Grafana with an additional data source.
This time around I'll be installing the [linkerd-viz helm chart](https://github.com/linkerd/linkerd2/tree/main/viz/charts/linkerd-viz)

First I begin by installing the edge repo. The current stable branch of Linkerd does not support bringing your own Grafana instance to linkerd

> **source:** https://linkerd.io/2.11/tasks/grafana/
> These notes apply only to recent Linkerd Edge releases and the upcoming Linkerd 2.12 stable release, which have stripped off the embedded Grafana instance, recommending users to install it separately as explained below.

```bash
helm repo add linkerd-edge https://helm.linkerd.io/edge
```

Based on the [values.yml from the helm chart](https://github.com/linkerd/linkerd2/blob/main/viz/charts/linkerd-viz/values.yaml), I can set the update the grafana settings with the `grafanaUrl` parameter. I also set up Jaeger so that I can visualize application tracing.

**values.yml**
```yaml
jaegerUrl: jaeger.linkerd-jaeger:16686
prometheusUrl: http://metrics-kube-prometheus-st-prometheus.monitoring:9090
grafanaUrl: metrics-grafana.monitoring:80
# Since we're bringing our own Prometheus and Grafana instances,
# we have to disable the embedded Prometheus and Grafana instances
prometheus:
  enabled: false
grafana:
  enabled: false
```

Install the linkerd-viz helm chart with the following command:
```bash
helm install --values values.yml --namespace linkerd-viz --create-namespace linkerd-viz linkerd-edge/linkerd-viz
```

Once the install completes, you will be able to access Grafana through Linkerd.

## Alerts and Monitoring with Logging Operator

I use the [Banzaicloud Logging Operator](https://banzaicloud.com/docs/one-eye/logging-operator/) to provide idiomatic configurable logging from applications in my Kubernetes cluster to Grafana Loki. Logging Operator can provide Service Monitors and alarms for Logging instances with the following configuration:

```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: my-logger
  namespace: my-namespace
spec:
  fluentd:
    metrics:
      serviceMonitor: true
      prometheusRules: true
  fluentbit:
    metrics:
      serviceMonitor: true
      prometheusRules: true
```

The `serviceMonitor` tag sets up a Prometheus Service monitor.
The `prometheusRules` set up the Prometheus rules for alerting on certain thresholds.

> The default alerting rules trigger alerts when:
>
> - Prometheus cannot access the Fluentd node
> - Fluentd buffers are quickly filling up
> - Traffic to Fluentd is increasing at a high rate
> - The number of Fluent Bit or Fluentd errors or retries is high
> - Fluentd buffers are over 90% full

## Next Steps

Once everything has been set up, we should have a foundation for receiving alarms from our Kubernetes cluster.
From this point, we could enhance the alarms by creating additional rules, such as FluentD rules with the Logging Operator.
Perhaps you would like additional Alertmanager configurations to send alarms to messaging integrations such as Slack.

In my next post, I'll demonstrate how to set up an Alertmanager configuration to work with Discord/Guilded WebHooks.

## References

- [Kubernetes Service Monitoring and Observability](https://dev.to/mikeyglitz/kubernetes-service-monitoring-and-observability-3g67)
- [Dynu SMTP relay](https://www.dynu.com/Resources/Tutorials/EmailServices/EmailServerSetup/How-to-configure-postfix-with-SMTP-relay)
- [Bokysan SMTP Helm Chart](https://artifacthub.io/packages/helm/docker-postfix/mail)
- [Postfix Dockerhub](https://hub.docker.com/r/mikeyglitz/postfix)
- [Postfix Gitlab Repo](https://gitlab.com/mikeyGlitz/postfix/)
- [Linkerd - Bring your own Prometheus](https://linkerd.io/2.11/tasks/external-prometheus/)
- [Linkerd - Grafana](https://linkerd.io/2.11/tasks/grafana/)
- [Linkerd - Alerting on Error rates](https://naseemkullah.medium.com/linkerd-alerting-on-error-rates-33c0a30899d0)
- [Logging Operator - Monitoring](https://banzaicloud.com/docs/one-eye/logging-operator/operation/logging-operator-monitoring/)
- [Logging Operator - Alerting](https://banzaicloud.com/docs/one-eye/logging-operator/operation/alerting/)