Kubernetes: tracing requests with AWS X-Ray, and Grafana data source
Tags: kubernetes, aws, monitoring, devops
Published: 2024-03-07 15:26:39

---
title: Kubernetes: tracing requests with AWS X-Ray, and Grafana data source
published: true
date: 2024-03-02 13:04:07 UTC
tags: kubernetes,aws,monitoring,devops
canonical_url: https://rtfm.co.ua/en/kubernetes-tracing-requests-with-aws-x-ray-and-grafana-data-source/
---

![](https://cdn-images-1.medium.com/max/1024/1*wOMtZQ4cb_gxIZ1HkiKOfw.png)

Tracing allows you to track requests between components, that is, for example, when using AWS and Kubernetes we can trace the entire path of a request from AWS Load Balancer to Kubernetes Pod and to DynamoDB or RDS.

This helps us both to track performance issues — where and which requests are taking a long time to execute — and to have additional information when problems arise, for example, when our API returns 500 errors to clients, and we need to find out which component of the system is causing the problem.

AWS has a service for tracing called [X-Ray](https://aws.amazon.com/xray/), where we can send data using [AWS X-Ray SDK for Python](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python.html) or [AWS Distro for OpenTelemetry Python](https://docs.aws.amazon.com/xray/latest/devguide/xray-python-opentel-sdk.html) (or other languages, but we’ll talk about Python here).

AWS X-Ray adds a unique X-Ray ID to each request and allows you to build a picture of the full “route” of the request.

Also, in Kubernetes we can trace with tools like [Jaeger](https://www.jaegertracing.io/) or [Zipkin](https://zipkin.io/), and then build the picture in [Grafana Tempo](https://grafana.com/oss/tempo/).

Another way is to use the [X-Ray Daemon](https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html), which we can run in Kubernetes, and add the [X-Ray plugin to Grafana](https://grafana.com/grafana/plugins/grafana-x-ray-datasource/). See [Introducing the AWS X-Ray integration with Grafana](https://grafana.com/blog/2020/10/05/introducing-the-aws-x-ray-integration-with-grafana/) for examples.

Additionally, AWS Distro for OpenTelemetry also works with AWS X-Ray-compliant Trace IDs — see [AWS Distro for OpenTelemetry and AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-services-adot.html) and [Collecting traces from EKS with ADOT](https://edransit.medium.com/collecting-traces-from-eks-with-adot-bfae2c0123aa).

Today, however, we will be adding an X-Ray collector that will create a Kubernetes DaemonSet and a Kubernetes Service to which Kubernetes Pods can send data that we can then see either in the AWS X-Ray Console or in Grafana.

### AWS IAM

### IAM Policy

To access AWS API from X-Ray daemon Pods, we need to create an IAM Role, which we will then use in the ServiceAccount for X-Ray.

We still use the old way of adding IAM Role via ServiceAccounts, see [Kubernetes: ServiceAccount from AWS IAM Role for Kubernetes Pod](https://rtfm.co.ua/en/kubernetes-serviceaccount-z-aws-iam-role-dlya-kubernetes-pod-2/), although AWS recently announced the Amazon EKS Pod Identity Agent add-on — see [AWS: EKS Pod Identities — a replacement for IRSA? Simplifying IAM access management](https://rtfm.co.ua/en/aws-eks-pod-identities-a-replacement-for-irsa-simplifying-iam-access-management/).

So, create an IAM Policy with permissions to write to X-Ray:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
```

Save it:

![](https://cdn-images-1.medium.com/max/1024/0*26py46oID5j7KZZj.png)

### IAM Role

Next, add an IAM Role that the Kubernetes ServiceAccount can use.

Find the _Identity provider_ of our EKS cluster:

![](https://cdn-images-1.medium.com/max/1024/0*Gmy0ZEJMOJkQLtZG.png)

Go to the IAM Roles, add a new role.

In the _Trusted entity type_, select _Web Identity_, and in Web identity select the Identity provider of our EKS, and in the Audience field — set the AWS STS endpoint:

![](https://cdn-images-1.medium.com/max/1024/0*JFc-xoPMuOR4qKib.png)

Attach the IAM Policy created above:

![](https://cdn-images-1.medium.com/max/714/0*dgrII3fvKgRXeVjh.png)

Save it:

![](https://cdn-images-1.medium.com/max/1024/0*sAIP3MQAUL0zELuK.png)

### Running X-Ray Daemon in Kubernetes

Let’s use the [okgolove/aws-xray](https://github.com/okgolove/helm-charts/tree/master/charts/aws-xray) Helm chart.

Create `x-ray-values.yaml` file, see the default values in [`values.yaml`](https://github.com/okgolove/helm-charts/blob/master/charts/aws-xray/values.yaml):

```
serviceAccount:
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::492***148:role/XRayAccessRole-test
xray:
  region: us-east-1
  loglevel: prod
```

Add a repository:

```
$ helm repo add okgolove https://okgolove.github.io/helm-charts/
```

Install the chart into the cluster, this will create a DaemonSet and a Service:

```
$ helm -n ops-monitoring-ns install aws-xray okgolove/aws-xray -f x-ray-values.yaml
```

Check the Pods:

```
$ kk get pod -l app.kubernetes.io/name=aws-xray
NAME READY STATUS RESTARTS AGE
aws-xray-5n2kt 0/1 Pending 0 41s
aws-xray-6cwwf 1/1 Running 0 41s
aws-xray-7dk67 1/1 Running 0 41s
aws-xray-cq7xc 1/1 Running 0 41s
aws-xray-cs54v 1/1 Running 0 41s
aws-xray-mjxlm 0/1 Pending 0 41s
aws-xray-rzcsz 1/1 Running 0 41s
aws-xray-x5kb4 1/1 Running 0 41s
aws-xray-xm9fk 1/1 Running 0 41s
```

And Kubernetes Service:

```
$ kk get svc -l app.kubernetes.io/name=aws-xray
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
aws-xray ClusterIP None <none> 2000/UDP,2000/TCP 77s
```

### Checking and working with X-Ray

### Create a Python Flask HTTP App with X-Ray

Let’s create a service on [Python Flask](https://flask.palletsprojects.com/en/3.0.x/) that will respond to HTTP requests and log X-ray IDs ( **ChatGPT promt**  — “_Create a simple Python App with AWS X-Ray SDK for Python to run in Kubernetes. Add X-Ray ID output to requests_”):

```
from flask import Flask
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
import logging

app = Flask( __name__ )

# Configure AWS X-Ray
xray_recorder.configure(service='SimpleApp')
XRayMiddleware(app, xray_recorder)

# Set up basic logging
logging.basicConfig(level=logging.INFO)

@app.route('/')
def hello():
    # Retrieve the current X-Ray segment
    segment = xray_recorder.current_segment()
    # Get the trace ID from the current segment
    trace_id = segment.trace_id if segment else 'No segment'
    # Log the trace ID
    logging.info(f"Responding to request with X-Ray trace ID: {trace_id}")
    
    return f"Hello, X-Ray! Trace ID: {trace_id}\n"

if __name__ == ' __main__':
    app.run(debug=True, host='0.0.0.0', port=5000)
```

Create `requirements.txt`:

```
flask==2.0.1
werkzeug==2.0.0
aws-xray-sdk==2.7.0
```

Add Dockerfile:

```
FROM python:3.8-slim

COPY requirements.txt .
RUN pip install --force-reinstall -r requirements.txt

COPY app.py .

CMD ["python", "app.py"]
```

Build a Docker image — here we use a repository in AWS ECR:

```
$ docker build -t 492***148.dkr.ecr.us-east-1.amazonaws.com/x-ray-test .
```

Log in to the ECR:

```
$ aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 492***148.dkr.ecr.us-east-1.amazonaws.com
```

Push the image:

```
$ docker push 492***148.dkr.ecr.us-east-1.amazonaws.com/x-ray-test
```

### Run Flask App in Kubernetes

Create a manifest with Kubernetes Deployment, Service, and Ingress.

For Ingress, enable logging into an AWS S3 bucket — logs will be collected from it to Grafana Loki, see [Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda](https://rtfm.co.ua/en/grafana-loki-collecting-aws-loadbalancer-logs-from-s3-with-promtail-lambda/).

For Deployment, set the `AWS_XRAY_DAEMON_ADDRESS` environment variable, with the URL of the Kubernetes Service of our X-Ray Daemon:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - name: flask-app
        image: 492***148.dkr.ecr.us-east-1.amazonaws.com/x-ray-test
        ports:
        - containerPort: 5000
        env: 
          - name: AWS_XRAY_DAEMON_ADDRESS
            value: "aws-xray.ops-monitoring-ns.svc.cluster.local:2000"
          - name: AWS_REGION
            value: "us-east-1"
---
apiVersion: v1 
kind: Service
metadata:
  name: flask-app-service
spec:
  selector:
    app: flask-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: flask-app-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/target-type: "ip"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=ops-1-28-devops-monitoring-ops-alb-logs
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: flask-app-service
            port:
              number: 80
```

Deploy it and check Ingress/ALB:

```
$ kk get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
flask-app-ingress alb * k8s-default-flaskapp-25042181e0-298318111.us-east-1.elb.amazonaws.com 80 10m
```

Make a request to the endpoint:

```
$ curl k8s-default-flaskapp-25042181e0-298318111.us-east-1.elb.amazonaws.com
Hello, X-Ray! Trace ID: 1-65e1d287-5fc6f0f34b4fb2120da8bbec
```

And we see the X-Ray ID.

We can also see it in the Load Balancer Access Logs:

![](https://cdn-images-1.medium.com/max/1024/0*_irxH4wZV60WD17P.png)

And in the X-Ray itself:

![](https://cdn-images-1.medium.com/max/1024/0*4nQ21AB59HzU_8mC.png)

Although, I expected the Load Balancer to be in the request map too, but it wasn’t.

### Grafana X-Ray data source

Add a new Data source:

![](https://cdn-images-1.medium.com/max/777/0*8_Z80oi61ap_-mPC.png)

Configure access to AWS — here it’s simple with ACCESS and SECRET keys (see [X-Ray documentation](https://grafana.com/grafana/plugins/grafana-x-ray-datasource/)):

![](https://cdn-images-1.medium.com/max/1024/0*tMfXOmAH3HXQvaOU.png)

And now we have a new data source in Explore:

![](https://cdn-images-1.medium.com/max/1024/0*ksvnkLOv8VZi-m8X.png)

And a new type of visualization — Traces:

![](https://cdn-images-1.medium.com/max/1024/0*XuEPn-_62MrBQSE2.png)

And somewhere in another post I will probably describe the creation of a real dashboard with X-Ray.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/kubernetes-tracing-requests-with-aws-x-ray-and-grafana-data-source/)_._

* * *