Remote writing your Prometheus metrics to Grafana Cloud
Tags: monitoring, kubernetes, cloudnative
Published: 2022-03-18 04:23:53

## Setup Grafana Cloud
Register to [Grafana Cloud](https://grafana.com/auth/sign-up/create-user?pg=login).

### Create API Key for Grafana Cloud
We will create an API Key to allow our local Prometheus to remote write to Grafana Cloud.

![Grafana Cloud Portal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27w223crf8lnnbph36rk.png)

- Head over to your Grafana Cloud Portal and select Send Metrics on Prometheus.


![Prometheus Configs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b7g4bisqekdla69pc8q0.png)

- If you scroll above, you should see the section for API Key. Click on `Generate now` and create an API Key with the Role `MetricsPublisher`.

- Copy the Prometheus config and save it locally as values.yaml as we will use it to install our Prometheus Helm Chart. The url and username should be unique for every user.
The password in both snippet should be filled with your API key.

## Local Setup
I used Rancher Desktop but most of Kubernetes distribution should work similarly. We will deploy Prometheus using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) with Helm.

- Add Helm repository 
```
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
```

Before installing Prometheus, we will tweak the values.yaml we created earlier. We will not directly paste the username and password(API Key) to the file. Instead, we will create a secret and reference that from Prometheus Config.


```
$ kubectl create secret generic grafanacloudsecret \ 
--from-literal=username=<YOURUSERNAME> \
--from-literal=password='<YOURPASSWORD>' 
```
Now, edit the values.yaml. Here is the edited version. Notice that the username and password is referencing the secret. Also I have added extra parameter which will attach extra labels to the metrics sent from this local cluster.

```yaml
prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
      basicAuth:
          username:
            name: grafanacloudsecret
            key: username
          password:
            name: grafanacloudsecret
            key: password
    replicaExternalLabelName: "__replica__"
    externalLabels: {cluster: "mylocalcluster"}

```

Now, let's install our Prometheus.
```
$ helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus --create-namespace -f values.yaml

```

## Check the metrics on Grafana Cloud
Go back to your Grafana Cloud and open the `Grafana Cloud Billing/Usage`. If you wait awhile, we should see some metrics being ingested in Grafana Cloud.


![Grafana Cloud Usage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4tx7oovk4uieusnws8h.png)
  
On the left pane, Go to Explore. In the Metrics browser, type in `container_memory_usage_bytes`. We can see the memory usage in my local Kubernetes cluster. Notice that there is an extra label named cluster: "mylocalcluster" which we added in our values.yaml. This can be used as an identifier for where this data is coming from.

![Prometheus metrics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v8iltprk5ol75kuo2zj5.png)

 
## Reference
- https://grafana.com/docs/grafana-cloud/kubernetes/prometheus/remote_write_helm_operator/
- https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack