Pushing Prometheus metrics with pushgateway
Tags: python, prometheus, monitoring, analytics
Published: 2020-12-11 23:13:34

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/g60cdrr4ywgbnm1j7d3u.jpg)

# Preface
  
This post is generally about pushing metrics in [pushgateway](https://github.com/prometheus/pushgateway), however, I will warn you and admit right away that the text will contain an example - an anti-pattern of pushing metrics, since using pushgateway [is recommended ](https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway) in the case when the service does not work constantly (or the service/task being started does not have any interface at all), and therefore it is better for prometheus not to constantly knock on closed doors and not do unnecessary work.

# Introduction
  
So, [pushgateway](https://github.com/prometheus/pushgateway) is a service where you can push metrics when the standard prometheus pull-model is not applicable (in the preface, I generally described how such a situation can arise). After the metrics have entered the pushgateway, prometheus already picks them up from there, and this implies several restrictions related to pushing metrics, for example, the absence of the *up* metric, since it is generated by prometheus itself for the polled app instance, and in this case, it is only the pushgateway.
  
p.s. Although, if we talk about the *up* metric, then it is not needed if you use pushgateway in a *recomended* way.

# Bake Prometheus for poll pushgateway
  
Let's say we have a compose like this with prometheus and pushgateway:  
  
```yaml
# ....(тут какие-нибудь графаны и т.д.)   
prometheus:  
    restart: always  
    image: bitnami/prometheus:latest  
    links:  
        - pushgateway  
    volumes:  
        - ./.prom.yml:/opt/bitnami/prometheus/conf/prometheus.yml  
  
pushgateway:  
    restart: always  
    image: bitnami/pushgateway:latest  
    ports:  
        - 9091:9091  
```
  
In this case, _prom.yml_ should look something like this to collect data from pushgateway:

```yaml  
global: null
scrape_interval: 5s
scrape_timeout: 2s
evaluation_interval: 15s

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets:
          - 'pushgateway:9091'
```
  
Everything is quite clear here, they added only honor\_lables, which, if briefly resolves conflicts of label names, that is, for example, if your service has a metric with the "X" label and pushgateway has a label "X", then with honor\_lables = false you will have label "X" with pushgateway and "exported\_X" from your service that pushed metrics to pushgateway, and if true, only your service labels will be displayed (again, if there is a conflict).
  
p.s. Pushgateway security - docs recommends by default, for example, to use basic\_auth. 

# Push metrics
  
I could give a beautiful example that corresponds to normal practice, however, I thought and decided that they hadn't given me any disrespect for a long time, therefore there will be an example of pushing metrics due to the fact that the service\_discovery setting is absent (in production, of course, it is impossible).
  
So, let's say we have workers [Faust](https://faust.readthedocs.io/en/latest/) there are a lot of them and they are not in the cluster (there is no swarm, no k8s), there are also no consul and other [ ways](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file) in which prometheus is able, they just run quietly in docker-compose and multiply with the scale argument.

More horror and sedition, in addition to pushing metrics, can be done by raging ports, for example, like this:
  
```yaml
ports:  
- "9100-9200:6066"  
```

And write targets with the whole set of ports into the prometheus config.
  
Let's continue through the code. For metrics from workers, you can use an already [ready library](https://github.com/trallnag/prometheus-fastapi-instrumentator). All we will do in this case is write a small timer in order to send metrics using the standard push\_to\_gateway function.
  
```python  
async def push_metrics():  
    def auth_handler(url, method, timeout, headers, data):  
        return basic_auth_handler(url, method, timeout, headers, data, PUSHGATEWAY_USERNAME, PUSHGATEWAY_PASSWORD)  
    push_to_gateway(PUSHGATEWAY_URI, job=f"{WORKERS_APP_NAME}-{ENV}", registry=registry_metrics, handler=auth_handler)  
  
@app.timer(interval=PUSH_METRICS_INTERVAL)  
async def push_metrics_cron():  
    await push_metrics()  
  ```
  
As you can see, everything is quite simple here - we specify the job name (with a pool of metrics - this is done in the prometheus config), substitute the handler for authentication and specify the registry from which the metrics will be pushed. Well, that's all, we launch and when we open the pushgateway of the web-panel, we see that the metrics have been loaded at an interval, then the previously configured prometheus will take them from there.

# Afterword  
I decided to write a note, since I encountered a similar one in my work, I will say right away that the method from the example will not go to production, however, as using pushgateway in the absence of a service discovery, for testing it can come off.