Prometheus: yet-another-cloudwatch-exporter – collecting AWS CloudWatch metrics
Tags: aws, monitoring, tutorial, cloudwatch
Published: 2020-07-23 14:09:22

---
title: Prometheus: yet-another-cloudwatch-exporter – collecting AWS CloudWatch metrics
published: true
date: 2020-07-23 12:24:40 UTC
tags: aws,Monitoring,tutorial,cloudwatch
canonical_url: https://rtfm.co.ua/en/prometheus-yet-another-cloudwatch-exporter-collecting-aws-cloudwatch-metrics/
---

![](https://rtfm.co.ua/wp-content/uploads/2017/09/prometheus-logo.png)Currently, to collect metrics from the AWS CloudWatch we are using AWS’s own [`cloudwatch-exporter`](https://github.com/prometheus/cloudwatch_exporter), see the [Prometheus: CloudWatch exporter — сбор метрик из AWS и графики в Grafana](https://rtfm.co.ua/prometheus-cloudwatch-exporter-sbor-metrik-iz-aws-i-grafiki-v-grafana/) post (_in Rus_), but it has a few gaps:

- it’s written in Java, so uses CPU/memory of the monitoring host
- doesn’t scrapes AWS tags from resources
- uses the [`GetMetricStatistics`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) method to grab the data
- can collect metrics from an only one AWS region, e.g. _us-east-1_ or _eu-west-2_

To mitigate those issues – let’s try to use the [`yet-another-cloudwatch-exporter`](https://github.com/ivx/yet-another-cloudwatch-exporter) instead.

# AWS CloudWatch – gaps

## Tags

The first thing which is making the “default” exporter uncomfortable is the fact that it will not grab the tags from resources in AWS.

For example, an Application Load Balancer metric will be returned with the only such label:

[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141358-1.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141358-1.png)

However, if you’ll check it in the AWS Console you’ll find much more tags there:

[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141358-1.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141358-1.png) [![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141449.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_141449.png)

And at this moment there is no way to use those tags from Grafana.

## `GetMetricStatistics` vs `GetMetricData`

The other issue with the default exporter is the API method used to collect data – [`GetMetricStatistics`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html).

Two years ago was there created an issue [Reduce cost of api operations by using GetMetricData API instead of GetMetricStatistics API](https://github.com/prometheus/cloudwatch_exporter/issues/134) – but it still wasn’t applied.

So, the problem with the [`GetMetricStatistics`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) is how it collects metrics – for each metric your `cloudwatch-exporter` will cause a dedicated API-call.

E.g. if you have 100 EC2 instances, and each will have 10 metrics – `cloudwatch-exporter` will perform 1000 every 60 seconds which will result in big expenses for you:

[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_142250.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_142250.png)

Unlike the AWS `cloudwatch-exporter`, the `yet-another-cloudwatch-exporter` uses the [`GetMetricData`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) API call which allows us to get up to 500 metrics in the only one API-call.

# Running `yet-another-cloudwatch-exporter`

Let’s try to spin it up and to get data.

Create a new file with AWS credentials for the exporter, let’s name it _alb-cred_:

```
[default]
aws_region = us-east-2
aws_access_key_id = AKI***D4Q
aws_secret_access_key = QUC***BTI
```

Or, you can use the [AWS EC2 Instance Profile](https://rtfm.co.ua/en/aws-iam-users-keys-rotation-ec2-iam-roles-and-jenkins/) but its policy has to have the following permissions:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchExporterPolicy",
            "Effect": "Allow",
            "Action": [
                "tag:GetResources",
                "cloudwatch:ListTagsForResource",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}
```

Now, create a file with the exporter’s settings:

```
discovery:
  jobs:
  - regions:
      - us-east-2
    type: elb
    enableMetricData: true
    metrics:
      - name: ActiveConnectionCount
        statistics:
        - Sum
        period: 300
        length: 600
```

And run it with Docker:

```
$ docker run -ti -p 5000:5000 -v /home/setevoy/Temp/alb-cred:/exporter/.aws/credentials -v /home/setevoy/Temp/config.yaml:/tmp/config.yml quay.io/invisionag/yet-another-cloudwatch-exporter:v0.19.1-alpha

{"level":"info","msg":"Parse config..","time":"2020-07-21T10:59:49Z"}

{"level":"info","msg":"Startup completed","time":"2020-07-21T10:59:49Z"}
```

Check the metrics:

```
$ curl localhost:5000/metrics
#HELP aws\_elb\_info Help is not implemented yet.
#TYPE aws\_elb\_info gauge
aws\_elb\_info{name="arn:aws:elasticloadbalancing:us-east-2:534\*\*\*385:loadbalancer/app/fea06ba9-eksstage1mealplan-a584/\*\*\*",tag\_Env="",tag\_Name="",tag\_Stack="",tag\_ingress\_k8s\_aws\_cluster="bttrm-eks-stage-1",tag\_ingress\_k8s\_

aws\_resource="LoadBalancer",tag\_ingress\_k8s\_aws\_stack="eks-stage-1-mealplan-api-ns/mealplan-api-ingress",tag\_kubernetes\_io\_cluster\_bttrm\_eks\_dev\_0="",tag\_kubernetes\_io\_cluster\_bttrm\_eks\_dev\_1="",tag\_kubernetes\_io\_cluster\_bttrm\_eks\_prod\_0="",tag\_kubernetes\_io\_cluster\_bttrm\_eks\_prod\_1="",tag\_kubernetes\_io\_cluster\_bttrm\_eks\_stage\_1="owned",tag\_kubernetes\_io\_cluster\_eksctl\_bttrm\_eks\_production\_1="",tag\_kubernetes\_io\_ingress\_name="mealplan-api-ingress",tag\_kubernetes\_io\_namespace="eks-stage-1-mealplan-api-ns",tag\_kubernetes\_io\_service\_name=""} 0
...
```

And you can see now that every metric has also all the AWS Tags attached.

## `yet-another-cloudwatch-exporter` configuration

### `exportedTagsOnMetrics`

If you don’t need all the tags – you can limit which tags will be attached to the metrics using the `exportedTagsOnMetrics`.

For example, let’s leave the only following tags:

```
discovery:
  exportedTagsOnMetrics:
    alb:
      - Name
      - kubernetes.io/service-name
      - ingress.k8s.aws/cluster
      - kubernetes.io/namespace
...
```

### `SeacrhTags`

Also, you can set a limit to specify from which resources the exporter will grab metrics – analog of the [`tag_selections`](https://github.com/prometheus/cloudwatch_exporter#configuration) in the AWS `cloudwatch-exporter`.

Let’s say, we want to get metrics from the only “_bttrm-eks-prod-1_” EKScluster.

Then you can specify the _“ingress.k8s.aws/cluster_” as the TAG key, and the “_bttrm-eks-prod-1_” as its value:

```
discovery:
  exportedTagsOnMetrics:
    alb:
      - Name
      - kubernetes.io/service-name
      - ingress.k8s.aws/cluster
      - kubernetes.io/namespace
  jobs:
  - type: alb
    regions:
      - us-east-2
    searchTags:
      - Key: ingress.k8s.aws/cluster
        Value: bttrm-eks-prod-1
    metrics:
      - name: UnHealthyHostCount
        statistics: [Maximum]
        period: 60
        length: 600
      - name: ActiveConnectionCount
        statistics: [Sum] 
        period: 300
        length: 600
```

Restart and check it:

```
$ curl localhost:5000/metrics
#HELP aws\_alb\_active\_connection\_count\_sum Help is not implemented yet.
#TYPE aws\_alb\_active\_connection\_count\_sum gauge

aws\_alb\_active\_connection\_count\_sum{dimension\_LoadBalancer="app/bcf678a9-eksprod1bttrmapps-447a/\*\*\*",name="arn:aws:elasticloadbalancing:us-east-2:534\*\*\*385:loadbalancer/app/bcf678a9-eksprod1bttrmapps-\*\*\*",region="us-east-2",tag\_Name="",tag\_ingress\_k8s\_aws\_cluster="bttrm-eks-prod-1",tag\_kubernetes\_io\_namespace="eks-prod-1-bttrm-apps-ns",tag\_kubernetes\_io\_service\_name=""} 112
...
aws\_alb\_tg\_un\_healthy\_host\_count\_maximum{dimension\_LoadBalancer="app/bcf678a9-eksprod1bttrmapps-447a/\*\*\*",dimension\_TargetGroup="targetgroup/bcf678a9-9b32ce4accea2525b4d/e0f341421a33a453",name="arn:aws:elasticloadbalancing:us-east-2:534\*\*\*385:targetgroup/bcf678a9-9b32ce4accea2525b4d/e0f341421a33a453",region="us-east-2",tag\_Name="",tag\_ingress\_k8s\_aws\_cluster="bttrm-eks-prod-1",tag\_kubernetes\_io\_namespace="eks-prod-1-bttrm-apps-ns",tag\_kubernetes\_io\_service\_name="bttrm-apps-backend-svc"} 0
...
```

Now we are getting metrics from the only one EKS cluster and metrics have only selected tags.

## Running with Prometheus

We have our Prometheus stack running under Docker Compose, so let’s add the YACE exporter:

```
...
  yace-clouwatch-exporter:
    image: quay.io/invisionag/yet-another-cloudwatch-exporter:v0.19.1-alpha
    networks:
      - prometheus
    ports:
      - 5000:5000
    volumes:
      - /etc/prometheus/prometheus-yace-cloudwatch-exporter.yaml:/tmp/config.yml:ro
    restart: unless-stopped
```

And now add a new target to the Prometheus server configuration:

```
...
scrape_configs:
  - job_name: 'yace-clouwatch-exporter'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['yace-clouwatch-exporter:5000']
...
```

Restart Prometheus and check the target:

[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_161559.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_161559.png)

Metrics:

# [![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_161639.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_161639.png)

# Grafana graphs

Now we are able to use those metrics in Grafana with an ability to chose, for example, an environment – _dev_, _stage_, _prod_ (see the [Grafana: создание dashboard](https://rtfm.co.ua/grafana-sozdanie-dashboard/), _Rus_):

[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_172213.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_172213.png)

The `$env` variable here is created from our label  `ekscluster`, which is attached to each cluster during its creation from the CloudFormation (see the [AWS Elastic Kubernetes Service: a cluster creation automation, part 1 – CloudFormation](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-a-cluster-creation-automation-part-1-cloudformation/)):[![](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_172317.png "Prometheus: yet-another-cloudwatch-exporter - сбор метрик AWS CloudWatch")](https://rtfm.co.ua/wp-content/uploads/2020/07/Screenshot_20200721_172317.png)

Done.

# Useful links

- [Improving the Prometheus exporter for Amazon CloudWatch](https://sysdig.com/blog/improving-prometheus-cloudwatch-exporter/)
- [Monitoring AWS Lambda with Prometheus and Sysdig](https://sysdig.com/blog/monitor-aws-lambda-prometheus/)

### Similar posts
 
- <small>06/21/2018</small> [Prometheus: CloudWatch exporter – сбор метрик из AWS и графики в Grafana](https://rtfm.co.ua/prometheus-cloudwatch-exporter-sbor-metrik-iz-aws-i-grafiki-v-grafana/)<small> (0)</small>  
- <small>07/18/2018</small> [Grafana: добавление CloudWatch data-sorce](https://rtfm.co.ua/grafana-dobavlenie-cloudwatch-data-sorce/)<small> (0)</small>  
- <small>03/05/2020</small> [Prometehus: мониторинг AWS Billing](https://rtfm.co.ua/prometehus-monitoring-aws-billing/)<small> (0)</small>  
- <small>03/10/2019</small> [Prometheus: RTFM blog monitoring set up with Ansible – Grafana, Loki, and promtail](https://rtfm.co.ua/en/prometheus-rtfm-blog-monitoring-set-up-with-ansible-grafana-loki-and-promtail/)<small> (0)</small>