Setting Up a Basic Prometheus Deployment with CDK8s
Published: 2024-02-19 09:34:30

I was genuinely surprised by the scarcity of CDK8s examples available. Consequently, I decided to pitch in and share what I've learned by presenting how I setup a basic Prometheus deployment using CDK8s.

## What is CDK8s i hear you ask?

CDK8s is a development framework tailored for Kubernetes. It enables users to define Kubernetes manifests using a library of constructs in their preferred programming language. In essence, CDK8s is akin to AWS CDK but designed for Kubernetes.

## Prerequisites 

Before we dive into the tutorial, make sure you have the following prerequisites:

- Basic understanding of Kubernetes and Prometheus
- CDK8s CLI installed
- kubectl installed
- A Running Kubernetes cluster (Minikube works well)

## 1. Initialise your CDK8s application

Begin by setting up your CDK8s app using the init command. While I'll be using TypeScript, there are alternative options available.

```
cdk8s init typescript-app
```

This will create a folder structure which looks something like:

```
my-cdk8s-app/
├── imports/
├── dist/
├── cdk8s.yaml
├── package.json
├── tsconfig.json
└── main.ts
```
The key file here is main.ts, this contains our CDK8s chart. Charts are just a container for your CDK8s code, they are synthesised into a single K8s manifest.

## 2. Add imports for necessary constructs

Import the required constructs at the top of your main.ts file to abstract elements like configmaps, volumes, deployments, etc.

main.ts

```
import { 
  ClusterRole, 
  ApiResource, 
  NonApiResource, 
  ServiceAccount, 
  Deployment, 
  Volume,
  ConfigMap } 
from 'cdk8s-plus-25'
```

## 3. Create a namespace for Prometheus and configure the chart

Run the following command in your terminal to create a namespace for Prometheus:
```
kubectl create namespace monitoring
```

Modify your main.ts file to ensure all API objects use this namespace:

```
new MyChart(app, 'CDK8s-Prometheus', { namespace: 'monitoring' })
```

## 4. Configure RBAC

For Prometheus to access the Kubernetes API, create a service account, cluster role, and role binding:

```
    const serviceAccount = new ServiceAccount(this, 'service-account')
    const clusterRole = new ClusterRole(this, 'cluster-role')
    
    clusterRole.allowRead(ApiResource.NODES) 
    clusterRole.allowRead(ApiResource.SERVICES)
    clusterRole.allowRead(ApiResource.ENDPOINTS)
    clusterRole.allowRead(ApiResource.PODS)
    clusterRole.allowRead(ApiResource.INGRESSES)
    clusterRole.allowGet(NonApiResource.of('/metrics'))

    
    clusterRole.bindInNamespace('monitoring', serviceAccount)
``` 

## 5. Create a file for Prometheus configuration

Add a prometheus.yml file to the root directory of your project with the following content. This is a simple setup where Prometheus scrape its own metrics

prometheus.yml

```
global:
  scrape_interval: 5s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090
```

## 6. Create volumes for storage and config

Define two volumes to be mounted in the pods later—one for Prometheus config and another for storing metrics data:

```
    const configMap = new ConfigMap(this, 'configmap')
    configMap.addFile(join(__dirname + '/prometheus.yml'))

    const configVolume = Volume.fromConfigMap(this, 'config-volume', configMap)
    const dataVolume = Volume.fromEmptyDir(this, 'data-volume', 'data')
```

## 7. Create our deployment

Specify the deployment details for Prometheus in your main.ts file:

```
const deployment = new Deployment(this, 'deployment', {

      containers: [ 
        {
          name: 'prometheus',
          image: 'prom/prometheus',
          args: ['--config.file=/etc/prometheus/prometheus.yml'],
          portNumber: 9090,
          securityContext: {ensureNonRoot: false}
          
        }
      ]
    })

    deployment.containers[0].mount('/etc/prometheus/', configVolume)
    deployment.containers[0].mount('/data/', dataVolume)
```

## 8. Deploy your application

With everything set up, deploy your Prometheus instance:

Run:
```
kubectl apply -f dist/
```

Test your Prometheus deployment by accessing the UI using port forwarding:

```
kubectl port-forward <POD NAME> 8080:9090 -n=monitoring
```

Access the UI via http://localhost:8080

Congratulations! You've successfully set up a basic Prometheus deployment using CDK8s. From here, you can explore further customisation and optimisation based on your specific needs.