FluxCD Integration with Prometheus for Monitoring
Published: 2024-07-01 13:39:29

FluxCD, a popular open-source tool for continuous delivery and GitOps, provides native support for Prometheus metrics to facilitate comprehensive monitoring of its components. This integration allows users to gain detailed insights into the performance and state of Flux controllers, enabling more effective management and troubleshooting. This blog post will delve into the technical aspects of FluxCD's integration with Prometheus, highlighting the key metrics, configurations, and tools involved.

### Flux Controller Metrics

By default, Flux controllers export Prometheus metrics at port `8080` in the standard `/metrics` path. These metrics provide information about the inner workings of the controllers, including:

- **Reconciliation Duration Metrics**:
  ```python
  gotk_reconcile_duration_seconds_bucket{kind, name, namespace, le}
  gotk_reconcile_duration_seconds_sum{kind, name, namespace}
  gotk_reconcile_duration_seconds_count{kind, name, namespace}
  ```

- **Cache Event Metrics**:
  ```python
  gotk_cache_events_total{event_type, name, namespace}
  ```

- **Controller CPU and Memory Usage**:
  ```python
  process_cpu_seconds_total{namespace, pod}
  container_memory_working_set_bytes{namespace, pod}
  ```

- **Kubernetes API Usage**:
  ```python
  rest_client_requests_total{namespace, pod}
  ```

- **Controller Runtime**:
  ```python
  workqueue_longest_running_processor_seconds{name}
  controller_runtime_reconcile_total{controller, result}
  ```

### Custom Resource Metrics

In addition to the default controller metrics, Flux also supports custom resource metrics using kube-state-metrics. These metrics can be configured to include custom labels, making them more informative for users who interact with Flux through custom resources. For example, metrics for GitRepositories can be labeled with department names or other relevant information.

### Monitoring Setup

To set up monitoring for Flux, the `fluxcd/flux2-monitoring-example` repository provides a comprehensive example configuration. This repository includes configurations for deploying and configuring kube-prometheus-stack, which is used to monitor Flux. The monitoring setup involves the following components:

- **kube-state-metrics**: Generates metrics about the state of Flux objects.
- **Prometheus Operator**: Manages Prometheus clusters atop Kubernetes.
- **Prometheus**: Collects and stores metrics from Flux controllers and kube-state-metrics.
- **Promtail**: Collects logs from Flux controllers.
- **Loki**: Stores logs collected by Promtail.
- **Grafana**: Displays Flux control plane resource usage, reconciliation stats, and logs.

### Example Configuration

The `fluxcd/flux2-monitoring-example` repository includes a `monitoring/` directory with configurations for deploying kube-prometheus-stack and loki-stack. The `monitoring/controllers/` directory contains the configurations for deploying these stacks.

### Flux Custom Prometheus Metrics

Flux custom Prometheus metrics can be created using kube-state-metrics. These metrics are more informative for users who interact with Flux through custom resources. For example, metrics for GitRepositories can be labeled with department names or other relevant information.

### Conclusion

[FluxCD's integration](https://platformengineers.io/blog/continuous-delivery-using-git-ops-principles-with-flux-cd/) with Prometheus provides a robust monitoring solution for its components. By leveraging Prometheus metrics and custom resource metrics using kube-state-metrics, users can gain detailed insights into the performance and state of Flux controllers. This integration is essential for effective management and troubleshooting in [Platform Engineering](www.platformengineers.io) environments.