prometheus-ecs-discoverer 3.3.5
Prometheus Service Discovery for AWS ECS
License: Apache-2.0
Requires Python: >=3.9,<4.0
Keywords: prometheus,aws,ecs,metrics

# Prometheus ECS Discoverer

[![pypi](https://img.shields.io/pypi/v/prometheus_ecs_discoverer?color=green)](https://pypi.python.org/pypi/prometheus-ecs-discoverer/)
[![downloads](https://img.shields.io/pypi/dm/prometheus-ecs-discoverer?color=green)](https://pypi.org/project/prometheus-ecs-discoverer/)
[![docker](https://img.shields.io/docker/v/trallnag/prometheus_ecs_discoverer?label=docker&color=blue)](https://hub.docker.com/repository/docker/trallnag/prometheus_ecs_discoverer)
[![pulls](https://img.shields.io/docker/pulls/trallnag/prometheus_ecs_discoverer?label=pulls&color=blue)](https://hub.docker.com/repository/docker/trallnag/prometheus_ecs_discoverer)

Based on [prometheus-ecs-sd](https://github.com/signal-ai/prometheus-ecs-sd),
this project enables your Prometheus to **dynamically scrape targets** deployed
in AWS ECS. The discoverer is perfect if you don't have
a service discovery system like Consul in-place. It provides an easy-to-use
alternative and can be used in low- to mid-sized environments with **hundreds
of tasks** running in parallel.

Once the discoverer is up and running, any task can be made visible to
Prometheus.

![diagram](https://raw.githubusercontent.com/trallnag/prometheus-ecs-discoverer/master/documents/drawio-diagram.png)

What are the advantages of using this project over [prometheus-ecs-sd](https://github.com/signal-ai/prometheus-ecs-sd
)?

* Modified throttling in combination with the already existing caching allows
    the usage in environments with hundreds of tasks.
* Every target can expose custom labels via its environment variables. This way
    you can provide more ways to aggregate metrics. For example by type of API.
* Deploy as container (provided and can be used directly) or Python package.
    No need to manually install boto3 and so on.
* Instrumented with Prometheus. Allows more insights into the discoverer. You
    can monitor how long discovery rounds take and stuff like used memory.
    Dashboard already included and ready to use.
* Extensive testing with high coverage ensures functionality.
* More configuration options. For example structured logs.

## Setup

As this project is based on
[prometheus-ecs-sd](https://github.com/signal-ai/prometheus-ecs-sd), the setup
is very similar / exactly the same. The setup consists out of three parts.
The targets must be prepared by adding environment variables. Next, PromED
itself must be deployed. And finally, the Prometheus configuration must be
updated.

### Prepare targets

Targets are setup via setting environment variables in the task definitions.

#### Mark container as target

Set `PROMETHEUS_TARGET` to `true` to make PromED consider the container. This
by itself is already enough to make it work with the configured defaults.
Alternatively you can use the `dockerLabels` in the container definition. For
this you have to set the PromED setting `MARKER_TYPE` to `dockerLabels`
instead of the default `environment`.

#### Specify metrics endpoint(s)

If your metrics are not exposed on the default `/metrics` endpoint, you can
specifiy the endpoint with `PROMETHEUS_ENDPOINT`.

You can also declare multiple endpoints and different intervals. The supported
intervals are `15s`, `30s`, `1m` and `5m`. Based on the interval, targets will
end up in different files. The default interval is "generic". Examples for this:

* `5m:/mymetrics,30s:/mymetrics2`
* `/mymetrics`
* `/mymetrics,30s:/mymetrics2`

By default, all targets will end up in a single file called `tasks.json` in the
configured directory (defaults to `/tmp`).

#### Set custom labels for container

Sometimes you might want to add additonal labels to targets to group them.
For example by the used API type (REST vs. GraphQL). This can be done by adding
environment variables to the container definition in the respective task
definition with the  `PROMETHEUS_LABEL_` prefix. For example
`PROMETHEUS_LABEL_api_type` or `PROMETHEUS_LABEL_foo`. Environment variables
set from within the container are not visible to PromED and are ignored.

```json
"environment": [
  { "name": "PROMETHEUS_LABEL_foo", "value": "bar" },
  { "name": "PROMETHEUS_LABEL_high", "value": "fi" },
]
```

There is an alternative for this using the `dockerLabels` attribute instead.
PromED always checks both locations. Unlike with the environment variables,
the additional target labels are all contained in one string. You can configure
the label PromED should look for by setting `CUSTOM_LABELS_KEY`. It works like
this:

```json
"dockerLabels": {
  "promed.custom_labels": "foo=bar, high=fi,what=ever"
}
```

The third option is to use the `WITH_DOCKER_LABELS` setting to give PromED an
array of labels it should always try to map from `dockerLabels` to target
labels. The PromED config could look like this (notice the config prefix):

```json
"environment": {
  "PROMED_WITH_DOCKER_LABELS": "['com.company.namespace', 'com.company.alias']"
}
```

A container that uses these labels:

```json
"dockerLabels": {
  "com.company.namespace": "whatever",
  "com-company-alias": "zoomzoom"
}
```

PromED translates it to `com_company_namespace` and `com_company_alias`.

#### Customize networking

Regarding networking, all network modes are supported (`bridge`, `host`
and `awsvpc`).

If `PROMETHEUS_PORT` and `PROMETHEUS_CONTAINER_PORT` are not set, the script
will pick the first port from the container definition (in awsvpc and host
network mode) or the container host network bindings in bridge mode. On
Fargate, if `PROMETHEUS_PORT` is not set, it will default to port `80`.

If `PROMETHEUS_CONTAINER_PORT` is set, it will look at the container host
network bindings, and find the entry with a matching `containerPort`. It will
then use the `hostPort` found there as target port. This is useful when the
container port is known, but the `hostPort` is randomly picked by ECS (by
setting `hostPort` to `0` in the task definition).

If your container uses multiple ports, it's recommended to specify
`PROMETHEUS_PORT` (`awsvpc`, `host`) or `PROMETHEUS_CONTAINER_PORT` (`bridge`).

*Quoted from
[prometheus-ecs-sd](https://github.com/signal-ai/prometheus-ecs-sd)
commit `ece6ca2`.*

### Deploy PromED

You can either use the Python package directly or go for the Docker image.
Please note that just running the package / image is not enough. The enviornment
must be ready for boto3. This includes credentials and approbiate rights.

#### Using package from PyPI

Available under the name `prometheus_ecs_discoverer`. To start PromED:

```sh
python -m prometheus_ecs_discoverer.run
```

To configure PromED you can either provide a settings file or use plain
environment variables.  Please see [Configuration](#configuration) for more
info. Please see [Configuration](#configuration) for more info.

#### Using Docker image

The image `trallnag/prometheus_ecs_discoverer` can be found
[here](https://hub.docker.com/r/trallnag/prometheus_ecs_discoverer).
The recommended way for configuring the image is to use environment variables.

You will probably want to run the discoverer in ECS. Here, you don't have to
provide credentials assuming everything is set up correctly. Boto3 will
automatically detect relative credentials URI and retrieve them from AWS.
Nevertheless, the region must be set by you.

#### AWS IAM

The actions that PromED is performing on the AWS API can be found in
[aws-iam-policy.json](https://github.com/trallnag/prometheus-ecs-discoverer/blob/master/documents/aws-iam-policy.json).
The allowed actions must be attached to an approbiate role. If you deploy
PromED in ECS, this should look like in [aws-iam-ecs-role.json](https://github.com/trallnag/prometheus-ecs-discoverer/blob/master/documents/aws-iam-ecs-role.json).

### Configure Prometheus

If you want all your targets to be scraped in the same interval, the following
job is enough.

```yaml
- job_name: 'ecs'
  file_sd_configs:
    - files:
        - /tmp/tasks.json
  relabel_configs:
    - source_labels: [metrics_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
```

By default, PromED exposes an `/metrics` endpoint.

```yaml
- job_name: discovery
  static_configs:
    - targets: ["discovery:8080"]
```

## Configuration

As this project is based on
[prometheus-ecs-sd](https://github.com/signal-ai/prometheus-ecs-sd),
the configuration is mostly compatible.

PromED uses [Dynconf](https://www.dynaconf.com/) for config management. There
are two main ways you can configure the application. Either by providing a
custom settings file with the env var `SETTINGS_FILES_FOR_DYNACONF` (see
[here](https://www.dynaconf.com/configuration/#on-environment-options))
or directly setting the respective values via env vars with the `PROMED_`
prefix. All supported settings together with their default values can be found
[`settings.toml` (click me)](https://github.com/trallnag/prometheus-ecs-discoverer/blob/master/prometheus_ecs_discoverer/settings.toml).

## Grafana Dashboard

You can find the Grafana dashboards for PromED in the documents folder.

![dashboard-screenshot](https://raw.githubusercontent.com/trallnag/prometheus-ecs-discoverer/master/documents/screenshot-dashboard-PromED.png)

A second dashboard contains panels with Grafana Alerts enabled.

![dashboard-screenshot](https://raw.githubusercontent.com/trallnag/prometheus-ecs-discoverer/master/documents/screenshot-dashboard-PromED-Alerting.png)

## Links

* [Docker Hub](https://hub.docker.com/repository/docker/trallnag/prometheus_ecs_discoverer)
* [GitHub](https://github.com/trallnag/prometheus-ecs-discoverer)
* [PyPI](https://pypi.org/project/prometheus-ecs-discoverer/)