Deploying Prometheus Operator via the Kubestack Catalog
Tags: gitops, kubernetes, devrel, terraform
Published: 2021-12-15 13:33:17

## Introduction
This walkthrough assumes you have already followed parts [one](https://www.kubestack.com/framework/documentation/tutorial-develop-locally), [two](https://www.kubestack.com/framework/documentation/tutorial-provision-infrastructure), and [three](https://www.kubestack.com/framework/documentation/tutorial-setup-automation) of the official Kubestack tutorial and at least have a local development cluster running via `kbst local apply`.

In this walkthrough we will explore using the Kubestack Catalog to install a [Prometheus Operator](https://www.kubestack.com/catalog/prometheus) and collect metrics from an example Go application.


> **Major Disclaimer:**
> In the interest of time and reducing technical complexity there is a very strong anti-pattern present in this walkthrough.
> 
> Best practice dictates that infrastructure and application manifests be stored in separate repositories so they can be worked on and deployed independently.
>
> In this walkthrough both the Go application and the Kubestack infrastructure manifests we create will be stored in the same repository for simplicity sake while deploying to the local development environment (see the Conclusion for an explanation of how this would look following best practices).

## 1 - Configure Local Development Environment
Before we install the Prometheus Operator, we need a few additional tools installed in our local development environment in order more easily verify that our configuration is working.


### 1.1 - Install Go Locally
Follow the instructions at https://go.dev/doc/install to install Go for your local development environment. We need this to build our example application as well as to install another tool below.


### 1.2 - Install and Configure `kubectl`
`kubectl` will be used primarily for two reasons. First, to verify which resources are deployed in our k8s cluster. Second, to forward ports and access resources inside the k8s cluster from our local development environment.
```
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(<kubectl.sha256) kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
```
At this point if `kubectl` is successfully installed you should see output similar to the following:
```
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
```
If you're having issues installing `kubectl` please refer the [offical documentation](https://kubernetes.io/docs/tasks/tools/).

### 1.3 - Install and Configure `kind`
`kind` (Kubernetes in Docker) will be used to export the cluster configuration file that `kubectl` needs to access the cluster.
```
GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.1
kind version
kind get clusters
kind export kubeconfig --name <CLUSTER_NAME>
kubectl get namespaces
```
At this point if `kind` has been installed correctly and the `kubectl config` has been exported successfully, you should see something similar to the following:
```
NAME                  STATUS   AGE
default               Active   6h27m
ingress-nginx         Active   6h26m
kube-node-lease       Active   6h27m
kube-public           Active   6h27m
kube-system           Active   6h27m
local-path-storage    Active   6h27m
```
If you're having issues installing `kind` please refer the [official documentation](https://kind.sigs.k8s.io/docs/user/quick-start/). Otherwise, congrats! You're ready to move on to the next part of the tutorial. 

> NOTE: `kind` is only needed for local development environments since that is how Kubestack deploys your environment locally. If you want to follow the rest of the tutorial using your cloud environment instead, you will need to download the `kubectl config` file from that cluster and import it locally so `kubectl` can access that cluster instead. Here are some resources about exporting that configuration file: [EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html), [AKS](https://docs.microsoft.com/en-us/azure/aks/control-kubeconfig-access), [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl).


## 2 - Deploy an Example Go Application
Now we're going to create an example application to emit some metrics for us to collect with Prometheus.


### 2.1 - Create a Go Application
Create a new GitHub repository to host your Example Go Application.

Create a new Go Module in the repository with `go mod init github.com/<account>/<repo>`.

Create a main.go file in the repository with the following content:
```
package main

import (
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func recordMetrics() {
	go func() {
		for {
			opsProcessed.Inc()
			time.Sleep(2 * time.Second)
		}
	}()
}

var (
	opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
		Name: "app_go_prom_processed_ops_total",
		Help: "The total number of processed events",
	})
)

func main() {
	println("Starting app-go-prom on port :2112")

	recordMetrics()

	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":2112", nil)
}
```
With main.go created run `go mod tidy` to create `go.mod` and `go.sum` (these files should be commited to the repo).


### 2.2 - Create a Dockerfile that Runs Go Application

Create a Dockerfile in the repository with the following content:
```
FROM golang:1.17-alpine

WORKDIR /app

COPY go.mod ./
COPY go.sum ./
RUN go mod download

COPY *.go ./
RUN go build -o /app-go-prom

EXPOSE 2112

CMD [ "/app-go-prom" ]
```


### 2.3 - Create a GitHub Action Pipeline to Build and Publish Docker Image

Create a .github/workflows/docker-publish.yml file in the repository with the following content:
```
name: Docker Publish

on:
  push:
    branches: [ main ]
    tags: [ 'v*' ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  # github.repository as <account>/<repo>
  IMAGE_NAME: ${{ github.repository }}


jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # https://github.com/docker/login-action
      - name: Log into registry ${{ env.REGISTRY }}
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # https://github.com/docker/metadata-action
      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      # https://github.com/docker/build-push-action
      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

```
This will build and publish a docker image to the GitHub Registry associated to the repository you created. This image can be used inside your k8s cluster.

In order to trigger this to build an image tagged with `latest` (rather than the current branch) you will need to tag the commit similar to the following:
```
git checkout main
git pull
git tag v0.1.1
git push origin v0.1.1
```

Once the pipeline completes you should be able verify by pulling the image with `docker pull ghcr.io/<account>/<repo>:latest`.


### 2.4 - Create an Application Manifest for Kubestack to Deploy in the Cluster
This requires two files to be created in your Kubestack IAC repository:

eks_zero_applications.tf
```
module "application_custom_manifests" {
  providers = {
    kustomization = kustomization.eks_zero
  }

  source  = "kbst.xyz/catalog/custom-manifests/kustomization"
  version = "0.1.0"

  configuration = {
    apps = {

      resources = [
        "${path.root}/manifests/applications/app-go-prom.yaml"
      ]

      common_labels = {
        "env" = terraform.workspace
      }
    }
    ops = {}
    loc = {}
  }

}
```

manifests/applications/app-go-prom.yaml
```
apiVersion: apps/v1
kind: Deployment 
metadata:
  name: app-go-prom
  namespace: default
  labels:
    app: app-go-prom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-go-prom
  template:
    metadata:
      labels:
        app: app-go-prom
    spec:
      containers:
        - image: ghcr.io/<account>/<repo>:latest
          name: app-go-prom
          ports:
            - containerPort: 2112

---

apiVersion: v1
kind: Service
metadata:
  name: app-go-prom-svc
  namespace: default
  labels:
    app: app-go-prom
spec:
  selector:
    app: app-go-prom
  ports:
    - name: metrics
      port: 2112
      targetPort: 2112
      protocol: TCP
  type: LoadBalancer
```

With these files created you may need to ^c out of the `kbst local apply` and run it again if watch doesn't pick up changes in custom manifest files.


### 2.5 - Verify that the Go Application is Running and Emitting Metrics
> Note about this section:
> If you ever destroy and re-apply the local cluster you will need to run the `kind export kubeconfig --name <CLUSTER_NAME>` command from above again to get a fresh `kubectl config` in order for `kubectl` to work. The cluster name will probably be the same, but if you need to find it you can always run `kind get clusters` to figure it out.

Once `kbst` has finished applying the changes let's verify that the pod is running and that it is emitting metrics as we expect.
```
kubectl get pods
```
You should see output similar to the following if everything is working correctly:
```
NAME                            READY   STATUS    RESTARTS   AGE
app-go-prom-6f9576879d-hvdr9    1/1     Running   0          32h
```
If STATUS is not "Running" there is an error. You can use `kubectl logs <go-app-pod-name>` to check the pod logs and fix any errors.

Once your Go application pod is running, lets forward the port from the associated service to our localhost and check for metrics:
```
kubectl get services
```
With the service name in hand run:
```
kubectl port-forward service/app-go-prom-svc 2112
curl localhost:2112/metrics
```
You should see a bunch of metrics spit out at this point, including the one we created in our Go application.
```
# HELP app_go_prom_processed_ops_total The total number of processed events
# TYPE app_go_prom_processed_ops_total counter
app_go_prom_processed_ops_total 25058
...
```
If you run the curl command a few more times you should see our metric increasing steadily as we expect.

You can stop the port forward and move on to the next section now.


## 3 - Install Prometheus Operator via the Kubestack Catalog
Now we are going to install the Prometheus Operator [following the instructions](https://www.kubestack.com/catalog/prometheus) in the Kubestack Catalog.

This consists of 3 steps:
1. Adding the Prometheus Operator module to the cluster
2. Configuring read-only access policies to monitoring targets
3. Specifying which target services to monitor

### 3.1 - Adding the Prometheus Operator module to the cluster
Create an eks_zero_services.tf file in the root of the repo with the following content:
```
module "eks_zero_prometheus" {
  providers = {
    kustomization = kustomization.eks_zero
  }

  source  = "kbst.xyz/catalog/prometheus/kustomization"
  version = "0.51.1-kbst.0"

  configuration = {
    apps = {
      additional_resources = [
        "${path.root}/manifests/services/prometheus-default-instance.yaml",
        "${path.root}/manifests/services/prometheus-service-monitors.yaml"
      ]
    }
    ops = {}
    loc = {}
  }
}
```
This file specifies to add the `eks_zero_prometheus` module to the `eks_zero` kustomization provider. If you have customized the name of your provider make sure to update that here as well.


### 3.2 - Create a Default Instance with permissions to monitor targets
Now we will create first file referenced as `additional_resources` above.

Create manifests/services/prometheus-default-instance.yaml with the following contents:
```
---

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: default-instance
  namespace: default
  labels:
    prometheus: default-instance
spec:
  serviceAccountName: prometheus-default-instance
  serviceMonitorSelector:
    matchLabels:
      prometheus-instance: default-instance
  resources:
    requests:
      memory: 2Gi

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: prometheus-default-instance
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-instance
subjects:
- kind: ServiceAccount
  name: prometheus-default-instance
  namespace: default

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-default-instance
  namespace: default
```

This file contains 3 key components:
1. The Prometheus default-instance
2. The RoleBinding permissions
3. The Service Account

The default-instance is our Prometheus server that collects the metrics and serves the Prometheus UI.
The other components are used to grant the needed permissions to read the metrics we'll specify in the next section.


### 3.3 - Specify which targets to monitor
Finally, we will create a ServiceMonitor that ties everything together.

Create the manifests/services/prometheus-service-monitors.yaml file with the following contents:
```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-go-prom-monitor
  namespace: default
  labels:
    prometheus-instance: default-instance
spec:
  selector:
    matchLabels:
      app: app-go-prom
  endpoints:
  - port: metrics
```

There are 3 important pieces here to connect everything:
1. `metadata.labels` here needs to exactly match the `spec.serviceMonitorSelector.matchLabels` from the default-instance
2. `spec.selector.matchLables` here needs to exactly match the `metadata.labels` of the Deployment from the Go application manifest
3. `spec.endpoints.port` needs to match the `spec.ports.name` of the Service from the Go application manifest

Once all those pieces are in place you can once again run `kbst local apply` to pickup the new manifests.


### 3.4 - Verify all Prometheus components are Running
> Note about this section:
> If you ever destroy and re-apply the local cluster you will need to run the `kind export kubeconfig --name <CLUSTER_NAME>` command from above again to get a fresh `kubectl config` in order for `kubectl` to work. The cluster name will probably be the same, but if you need to find it you can always run `kind get clusters` to figure it out.

Similar to how we verified our Go application was working let's check our Prometheus Operator.

Once `kbst` has finished applying the changes let's verify that the Prometheus Operator and supporting components have been successfully deployed:
```
kubectl get all --namespace operator-prometheus
```
You should see output similar to the following:
```
NAME                                       READY   STATUS    RESTARTS   AGE
pod/prometheus-operator-775545dc6b-qffng   1/1     Running   0          40h

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/prometheus-operator   ClusterIP   None         <none>        8080/TCP   40h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator   1/1     1            1           40h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-775545dc6b   1         1         1       40h
```

Now let's check that the default-instance pod is running:
```
kubectl get pods
```
You should see there is now a `prometheus-default-instance-0`:
```
NAME                            READY   STATUS    RESTARTS   AGE
app-go-prom-6f9576879d-hvdr9    1/1     Running   0          33h
prometheus-default-instance-0   2/2     Running   0          33h
```
If STATUS is not "Running" there is an error. You can use `kubectl logs prometheus-default-instance-0` to check the pod logs and fix any errors.

Lastly, let's verify that our ServiceMonitor was created:
```
kubectl get ServiceMonitors
```
You should see something similar to:
```
NAME                  AGE
app-go-prom-monitor   33h
```

If you don't have any errors, proceed to the final section and let's verify that Prometheus is correctly collecting our metrics.


## 4 - View the Metrics in the Prometheus UI
Now that we've got everything deployed and Running let's take one final step to verify that everything is working.

Like we did before with our Go application, we now need to forward the Prometheus port to access the UI:
```
kubectl port-forward prometheus-default-instance-0 9090
```

Now from your local development environment open a web browser and navigate to `localhost:9090`.

If everything has been successful to this point you should be greeted with the Prometheus dashboard.

Enter the metric name into the search box that we created in our Go application; `app_go_prom_processed_ops_total`, and click "Execute".
You will see the metric metadata and count displayed below the search box, similar to the following:
```
app_go_prom_processed_ops_total{container="app-go-prom", endpoint="metrics", instance="10.244.1.4:2112", job="app-go-prom-svc", namespace="default", pod="app-go-prom-6f9576879d-hvdr9", service="app-go-prom-svc"}  474
```


## Conclusion
Congratulations you have successfully deploy the Prometheus Operator, created an example service emitting metrics, and configured everything to collect those metrics. That is the backbone you'll need to have visibility into the metrics of your new cluster.

From here you could extend your metrics infrastructure by;
* adding additional applications / services and their associated ServiceMonitors,
* adding a Grafana deployment to create dashboards of your metrics,
* configuring an external instance of Prometheus to collect all your metrics,
* etc.

The next steps if you'd like would be to go beyond metrics and browse the [Kubestack Catalog](https://www.kubestack.com/catalog) to install additional helpful services into your cluster.

> Initial Disclaimer Explained:
>
> As mentioned in the beginning, we introduced a very strong anti-pattern in this walkthrough by placing our application and infrastructure manifests in the same repository.
> 
> This should **NEVER** be done when deploying to your real cloud infrastructure. Instead, for micro-service architectures such as this, each application would have their own code repo (in whatever language is appropriate). There would also be one additional "deployment repository" containing the Kubernetes manifests of all the applications. A service such as ArgoCD or Flux would then be configured to monitor the deployment repository and deploy changes to the Kubernetes cluster as needed when the applications are updated.
> 
> The Prometheus Operator should be deployed as part of the Kubestack infrastructure. The Prometheus Instance and ServiceMonitor (all explained in more details later) should be deployed along side each application. The only exception would be if you plan to have a single Instance monitor all your services, in that case it can be deployed as part of the Kubestack infrastructure.
> 
> For more information you can refer to the official documentation regarding [infrastructure environments](https://www.kubestack.com/framework/documentation/gitops-process#infrastructure-environments).