Prometheus Observability Platform: Handling multiple regions
Tags: devops, kubernetes, promxy, observability
Published: 2023-09-14 10:27:13

When we have multiple regions such as the EU and US, we need to have a long-term storage solution running in both of those. If we want to combine the resources into a single query we need to use a query layer that can query both endpoints. One such component we can use is Promxy.

Promxy uses the same PromQL syntax as Prometheus and we can define server groups with multiple endpoints. In our case, we would define our EU and US long-term storage endpoints under one server group. We can then use the single Promxy endpoint to query both the EU and the US.

## Demo
This example assumes that you have completed the following steps, as the components from those are needed:
- [Prometheus Observability Platform: Prometheus](https://dev.to/polarsquad/prometheus-observability-platform-prometheus-1019)
- [Prometheus Observability Platform: Long-term storage](https://dev.to/polarsquad/prometheus-observability-platform-long-term-storage-4cbj)

We can use the Helm chart offered in the [Promxy repository](https://github.com/jacksontj/promxy/tree/master) to deploy a proxy to our Kubernetes cluster.

First we clone the repository, because the Helm chart is not published to a public registry:
```
git clone https://github.com/jacksontj/promxy.git
```

Then we navigate to the folder containing the Helm chart:
```
cd promxy/deploy/k8s/helm-charts/promxy/
```

We set up Promxy with the following `server_groups:`
```
server_groups:
  - static_configs:
    - targets:
      - vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481
      labels:
        region: eu
    scheme: http
    path_prefix: /select/0/prometheus
```

I have converted this to json so we can pass it to the helm chart:
```
{"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}
```

To install Promxy from the local Helm chart we use the following:
```
helm install promxy . --create-namespace --namespace promxy --set 'image.tag=latest' --set-json 'config.promxy={"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}'
```

We can now port-forward the Promxy service and access the web UI from http://localhost:9090:
```
kubectl port-forward -n promxy services/promxy 9090:8082
```

Here we can perform the same query for the `kube_pod_container_status_restarts_total` to verify that Promxy is able to reach the VictoriaMetrics data:

![Promxy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4nhxmxnnfh5mcl0hnku.png)

If we have more regions than just the US, we can add them under the `server_groups` and query multiple VictoriaMetrics instances from a single Promxy source.

We have now achieved setting up Promxy with `server_groups` to use for querying VictoriaMetrics instances.

Next part: [Prometheus Observability Platform: Application metrics](https://dev.to/polarsquad/prometheus-observability-platform-application-metrics-2024)