o11y: OpenTelemetry, Prometheus, Loki e Tempo no EKS [Lab Session]
Tags: community, eks, o11y, opensource
Published: 2024-02-01 13:20:27


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dvh4snj4gzchw7hu0f3r.png)

## Loki overview

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5wgb7fuombqsyze2h8z0.png)
[Grafana Loki documentation](https://grafana.com/docs/loki/latest/get-started/overview/)

## Tempo architecture

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ktszf9wl7rz1b6iur29.png)
[Tempo documentation](https://grafana.com/docs/tempo/latest/operations/architecture/)

* [Repositório de infraestrutura (terraform) - EKS-Istio-Karpenter](https://github.com/paulofponciano/EKS-Istio-Karpenter.git)
* [Repositório stack o11y](https://github.com/paulofponciano/o11y-lab-prom-otel-loki-tempo.git)
* [Repositório K6](https://github.com/paulofponciano/k6.git)

---

## Lab architecture 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f12o9vx9h9h74yfrzvyn.png)

---

- **Deploy do EKS:**

`tofu init`
`tofu plan --var-file variables.tfvars`
`tofu apply --var-file variables.tfvars`

- **IAM Role para Loki e Tempo:**

Para que o Loki e Tempo consigam gravar os dados coletados em um bucket S3, é necessário criar uma _role_ contendo a _policy_ com as permissões necessárias. Essa _role_ será acionada via _AssumeRole_ através de uma _annotation_ que definimos nos arquivos _values.yaml_ do loki e do tempo no _deploy_ via helm chart.

**Policy**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLokiandTempoBucketonK8s",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObjectVersion",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::<loki-bucket-name>/*",
                "arn:aws:s3:::<loki-bucket-name>",
                "arn:aws:s3:::<tempo-bucket-name>/*",
                "arn:aws:s3:::<tempo-bucket-name>"
            ]
        }
    ]
}
```

**Coletando OIDC provider para utilizar na role**

`aws iam list-open-id-connect-providers`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1g28rewd04932rjpo1gq.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cl322nnj7m67pzmrr17e.png)

**Role**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<ACCOUNT>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<ID>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.<REGION>.amazonaws.com/id/<ID>:sub": [
                        "system:serviceaccount:o11y:loki-sa",
                        "system:serviceaccount:o11y:tempo-sa"
                    ]
                }
            }
        }
    ]
}
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cl1axzqvd8njrx19jhsd.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o6wmjc0x9042q0gf20si.png)

- **Bucket S3:**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7tjegbz0ib8s1yg5d3i3.png)

- **Conectar no EKS:**

`aws eks update-kubeconfig --region us-east-2 --name pegasus`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3t02v8lvomwehc6o5wwr.png)

Agora no repositório da _stack_ de o11y, podemos iniciar os _deploys_ com Helm.

- **Criar namespace:**

`kubectl create namespace o11y`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8iuyy4id32t3s51cxbuw.png)

- **Deploy promtail:**

`cd promtail`
`helm repo add grafana https://grafana.github.io/helm-charts`
`helm repo update`
`helm upgrade --install promtail grafana/promtail  --values values.yaml -n o11y`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m4moww6q54cp7zeijsa6.png)

- **Deploy Loki:**

`cd loki`
`helm upgrade --install loki grafana/loki-distributed --values values.yaml -n o11y`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y9ivx03isdr8xua1kzcx.png)

- **Deploy Tempo:**

`cd tempo`
`helm upgrade --install tempo grafana/tempo-distributed --values values.yaml -n o11y`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/intgqam89q0srs91u57a.png)

- **Deploy Prometheus:**

`cd prometheus-grafana`
`helm repo add prometheus-community https://prometheus-community.github.io/helm-charts`
`helm repo update`
`helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack --values values.yaml -n o11y`
`kubectl apply -f istio-ingress-prometheus.yaml`
`kubectl apply -f istio-ingress-grafana.yaml`

Neste caso, criamos também o _ingress_ para o Prometheus e o Grafana.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4c9y94z6gmotbmq6wttc.png)

Temos agora tudo rodando no _cluster_:

`kubectl get nodes`
`kubectl get pods -n o11y`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wyrfak3bxmtpd850god4.png)

- **Registros no DNS (Route 53):**

No DNS externo, criamos alguns registros direcionando para o NLB publico, que utilizamos como entrada. Para o Prometheus, Grafana e os _apps_ que vamos utilizar para gerar dados.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3nlbweqepb8fb0qgaja0.png)

- **Deploy API de exemplo:**

`cd fastapi-observability`
`kubectl apply -f mono_manifest.yaml`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/usid0jeiwsg1ht7ht3tx.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f1lunurxblvyox8tzx49.png)

Essa API está utilizando a biblioteca do [OpenTelemetry](https://opentelemetry.io/) para gerar os _spans_ das operações que compõe os _traces_.

- **Prometheus targets:**

Podemos ver na interface do prometheus que ele já está coletando as métricas que a API de exemplo expõe no _**/metrics**_. Lembrando que temos a _annotation_ no manifesto de _deployment_ da API:

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "8000"
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vphzc7j2p7kxd8j3aq0l.png)

- **Grafana:**

Acessando o grafana, fazemos o _import_ do _dashboard_ que iremos utilizar. `user: admin`, `password: prom-operator`. O _json_ do _dashboard_ está em: _**fastapi-observability/etc/fastapi-dashboard.json**_

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/796arpl6zrk8sv6e32gw.png)

**Dashboard importado**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yxa9ifjklpb19asjvqud.png)

**Data sources**

Esses _data sources_ são criados no momento do _deploy_ da _stack_ prometheus, os valores são informados no _values.yaml_ do helm chart, em 'additionalDataSources'.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfzzb18h5nj7b08qt16a.png)

- **Gerando dados com K6:**

Utilizando o K6, podemos gerar _requests_ na API de exemplo para que os dados sejam gerados e métricas capturadas para visualizarmos.

_request.js_

```js
import { check } from 'k6';
import http from 'k6/http';

export const options = {
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 100,
      timeUnit: '1s', // 100 iterations per second, i.e. 100 RPS
      duration: '120s',
      preAllocatedVUs: 100, // how large the initial pool of VUs would be
      maxVUs: 100, // if the preAllocatedVUs are not enough, we can initialize more
    },
  },
};

export function test(params) {
  const res = http.get('https://app-a.pauloponciano.digital');
  check(res, {
    'is status 200': (r) => r.status === 200,
  });
}

export default function () {
  test();
}
```

`k6 run request.js --vus=200 --duration=60m`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lj0ld2o5fqlo5edtlp97.png)

- **Visualização:**

Ainda no grafana, no _dashboard_ [**FastAPI Observability**](https://grafana.com/grafana/dashboards/16110-fastapi-observability/) vamos visualizar as métricas, traces e logs da API de exemplo.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1mflxuma8dgwtbtffwpr.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rzrjvowzqmo1c0av0h75.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cf0idcynqs3k51a0tc71.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/78mb4qvypzzjbmmxgh89.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3lsg5wwxhpmmacvac0qd.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/74r1apcctta95iyi10cw.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2htclba690rx9mtu1e3v.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e2e3on6bhgwyrsybrzxq.png)

---

References:

[Grafana Labs - FastAPI Observability](https://grafana.com/grafana/dashboards/16110-fastapi-observability/)

https://github.com/blueswen/fastapi-observability

{% embed https://www.youtube.com/live/dXR8WNm5uos?si=8vn4d2dH9pUk2h37 %}

---

Happy building!