Using AI to Create a Dashboard for an AWS Load Balancer
Published: 2023-12-09 18:34:21

## Stack

- K8s
- FluxCD
- AWS Load Balancer Controller
- Prometheous
- Grafana

## Problem

Prometheous has CRDs for K8s known as `ServiceMonitor` and `PodMonitor`.  I created a ServiceMonitor for AWS Load Balancer Controller, but I had no clue if Prometheous was collecting it's data.

## AI Interactions

During troubleshooting, I was starting to doubt that I even had an AWS Load balancer controller service.

I asked:

> should the aws-load-balancer-controller from eks-charts deploy a aws-load-balancer-controller that would show up when I run `kubectl get service -n kube-system`

ChatGPT: Terraform Engineer

> [no] ... AWS Load Balancer Controller primarily operates as a controller `deployment` in your Kubernetes cluster. It does not typically expose a service that you can see with `kubectl get service -n kube-system`.

Wow! ChatGPT knew better than me that there is a difference between a K8s Service-Resource and a Service referenced by a ServiceMonitor.  Now reassured that my Load Balancer Controller was deployed to my cluster just fine as a `Deployment` and not a `Service` I asked:

> how can I connect the aws-load-balancer-controller to prometheous

Along with a lot of suggestions, ChatGPT Terraform Engineer answered:

> ... Ensure that the AWS Load Balancer Controller is configured to expose metrics. This is usually done through command-line arguments in the controller's deployment configuration. Look for arguments like --metrics-bind-addr=ADDRESS where ADDRESS specifies the host and port where metrics are exposed (e.g., 0.0.0.0:8080) ...

ChatGPT had also provided a lot of details relevant to other deployment solutions, so I re-asked while giving it more context to my context.

> how can I connect the aws-load-balancer-controller to prometheous ... given I'm currently deploying with a HelmRelease in FluxCD

ChatGPT Terraform Engineer instructed:

> ... Update Your HelmRelease Resource: ... with args `- --metrics-bind-addr=0.0.0.0:8080`

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aws-load-balancer-controller
  namespace: kube-system
spec:
  chart:
    spec:
      chart: aws-load-balancer-controller
      sourceRef:
        kind: HelmRepository
        name: eks
        namespace: flux-system
      version: <chart version>
  values:
    serviceAccount:
      create: true
    clusterName: <your-cluster-name>
    region: <your-region>
    vpcId: <your-vpc-id>
    args:
      - --metrics-bind-addr=0.0.0.0:8080
```
> ... After FluxCD applies the changes, you can verify that the AWS Load Balancer Controller is exposing metrics by checking the logs of the controller pods or using kubectl port-forward 

So I deployed these updates then asked specifically>

> provide the cli command for kubectl port-forward to access the metrics endpoint locally.

ChatGPT Terraform Engineer answered:

> Find the Pod Name: ...

```bash
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller
```
> Port Forward to Access Metrics: ...

```bash
kubectl port-forward -n kube-system <pod-name> 8080:8080
```

> Access Metrics Locally ...

```bash
curl http://localhost:8080/metrics
```

Eureka! Metrics are available.  Now to work on other 1/2.  

-------

I was embolden to finish the rest on my own, and after fumbling around and getting back to ChatGPT it suggested to run:

```bash
kubectl get prometheus kube-prometheus-stack-prometheus -n monitoring -o yaml
```

Which ultimately lead to finding it's matchLabels

```yaml
 serviceMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
```

and adding them to the ServiceMonitor:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: autoscaler-monitor
  namespace: kube-system
  labels:
    release: kube-prometheus-stack
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring

    ...
```

But this still didn't quite fix it.  So it recommended looking at the prometheus logs:

```bash
kubectl logs deployment/kube-prometheus-stack-operator -n monitoring
```

Ultimately the issue was that the ServiceMonitor needed to be in the `monitoring` name space.

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: autoscaler-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring

    ...
```