Distributed tracing: how to propagate the context with Redis
Tags: tracing, jaeger, observability, redis
Published: 2020-05-17 13:34:22

Photo by [Milestoned](https://www.flickr.com/photos/baccharus/) on [flicker.com](https://www.flickr.com/photos/baccharus/5817342671/).

## Introduction

During an application-monitoring workshop, I was introduced to the distributed tracing. I was immediately interested in that and I understood the potentiality that this method can offer to monitor a distributed system in production.

So, I started to learn about [Jaeger](https://www.jaegertracing.io/)¹, [OpenTracing](https://opentracing.io/)²,  traces, spans, tags, …
Then I instrumented my first APIs using HTTP headers for the transport layer and everything was fine. I was able to see the tracing in action! 

But then I wondered: how can I instrument services that don’t expose APIs and that don’t speak each other through HTTP? I’m speaking about services that are part of the same pipeline, that work on the same data in an event-driven architecture or that use direct requests to communicate, but in which I can’t use an HTTP header to propagate down the context information.

## Distributed tracing: a very brief description 

> *Distributed tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.³*

That means you can easily understand and monitor your services in production in a visual way. Tracing adds observability. 

Thanks to that, the troubleshooting teams can analyze issues and debug a system in all its parts. It simplifies and reduces the time for the root cause discovery. In addition, it can be useful to the developers to better understand how to develop a new feature or introduce an improvement in the system.

![alt text](https://cdn-images-1.medium.com/max/2148/1*QRpDqUaLtHCVhGhW1QM_JA.jpeg "Example of trace and spans")

Since the scope of this article is not an introduction about distributed tracing I don’t deepen into this. To understand better what tracing is, I posted below some useful articles.

## Redis as a message queue

Referring to my previous evaluation, in case you can’t propagate the information context through HTTP Headers, you can use a *message queue*.

So, I created the example application [*Distributed Tracing with Redis*](https://github.com/mattiacapitanio/distributed-tracing-with-redis)⁴ in which I simulated a pipeline composed of 3 different apps: one *main service* and two different *workers* that work on the same data. 

In this example, the *main service* starts the two *workers* by executing a shell command. The *main service* passes a parameter — that contains the *job id* — to the apps.

{% gist https://gist.github.com/mattiacapitanio/bbd1fbf1b4ae80e79122c3385198f882 %}

### The application’s transaction cycle

In the application, the *main service* starts a new tracing span and then propagates the span context saving it in Redis, our message queue. The context is saved using the *job id* as the key. Then it starts the execution of the two *workers*, just providing the *job id*. 

When the two *workers* start, it is their responsibility to retrieve the context from Redis, create a new span, and establish a relationship with the main span.

The two *workers* are executed in a sequential way, so only when the *first worker* is completed, the second will start. All the apps simulate the execution of internal tasks, sending some tracing spans, tags, logs to the Jeager Agent.

The following image can be useful to describe better what these apps do.

![alt text](https://cdn-images-1.medium.com/max/2000/1*8-f-T2GJyBFSAfm3KWO0jw.jpeg "The application's transaction cycle")

### The architecture

The application’s architecture is mainly composed of:

* Apps module

* Redis, to propagate the span context

* the Jaeger stack: Agent, Collector, Query/UI

* Elasticsearch, to save the traces

The image below shows the tracing architecture.

![alt text](https://cdn-images-1.medium.com/max/2008/1*h49fpKFCDuBZM9VziE1dPg.jpeg "Tracing architecture")

## Let’s jump into the code!

I instrumented the components using OpenTracing. So, I added in each of them the code to send the spans to the Jaeger Agent. In order to do that, I used the  [Jaeger clients](https://www.jaegertracing.io/docs/1.17/client-libraries/) and the [OpenTracing libraries](https://opentracing.io/guides/) for NodeJS and Python.

In the following sections, I briefly described how I instrumented the code to propagate the context. 

### The main service

In the following snippets of code, the *main service* creates the new *main span*, saves the context information, and then starts up the two *workers* providing the *job id*. 

{% gist https://gist.github.com/mattiacapitanio/8d664796675c8dae19042429e4fe1193 %}

As shown in the code, the context span is saved in Redis using a key-value pair, providing the *job id* as the key. The value is instead filled with the context span in the [TEXT_MAP format](https://doc.esdoc.org/github.com/opentracing/opentracing-javascript/variable/index.html#static-variable-FORMAT_TEXT_MAP) provided by OpenTracing.

{% gist 
https://gist.github.com/mattiacapitanio/3a795b5ab1610ec2023bd94dd80a9f53
%}

### The worker apps

Here below you can find the snippet of code developed for the two apps to read the span context from Redis. 

When the *worker* apps start, the *job id* is provided as an input parameter. The *worker* creates a *continuation span* that will be attached to the main span before being extracted from Redis using the *job id* key.

Then some tasks are executed internally by the *worker*. 

{% gist 
https://gist.github.com/mattiacapitanio/ba16a77239b784090da34b9e599d47b6
%}

During the execution of the internal tasks, new *child spans* are created. All of them are created establishing a *follows from* reference to the propagated parent span context. All the *child spans* will be displayed in the same trace, as children of the main process span.

{% gist 
https://gist.github.com/mattiacapitanio/8219f111d60f10e3c0ba79e86ac9a171
%}

A very similar Python version of the code is displayed below for the *second worker*.

{% gist 
https://gist.github.com/mattiacapitanio/ad4679c23df5613b01b44fe5251f726c
%}

## Visualizing traces

If you deploy and start all the components and you open a browser on [http://localhost:16686/](http://localhost:16686/), you will be able to see all the traces sent by the three apps through the Jaeger UI.

![alt text](https://cdn-images-1.medium.com/max/3040/1*pYgtT_-i3AFFr21bmQNoZA.png "Jaeger UI")

If you select a trace, you can see the details and the time spent from each component, also when they execute their internal tasks. 

In the example below, we can see: 

* the main span created by the *main service* (in blue)

* the spans created around the tasks executed by the *first worker* (in yellow)

* the spans created and attached by the *second worker* (in brown)

All the spans are part of the same trace.

![alt text](https://cdn-images-1.medium.com/max/3072/1*vR6KGFdStQt5zWcTK8ZW7g.png "Application's trace and spans")

## Conclusion

In the article, I presented a possible solution to instrument apps for distributed tracing when you have a system in which the services are triggered using direct requests, but not through Rest APIs. So, HTTP headers are not available to propagate the context and you don’t want to introduce them in your system.

The aim can be reached by instrumenting the apps using Jeager, OpenTracing, and adding a message queue in the architecture for the transportation layer.

The presented solution can be also modified and used in an event-driven architecture, in which the services are using a message queue to share the process information. You can simply add the context spans information to the messages in the queue.

***

### References

Some very useful Medium articles that helped me:

{% medium https://medium.com/swlh/microservices-observability-with-distributed-tracing-32ae467bb72a %}

{% medium https://medium.com/@codeboten/redis-tributed-distributed-tracing-through-redis-9b671187da47 %}

{% medium https://medium.com/jaegertracing/weave-trimmed-troubleshooting-fat-cut-api-response-time-from-seconds-to-milliseconds-with-jaeger-ab594c22c921 %}

***

### Footnotes

[1] *Jaeger*: an open-source software for tracing transactions between distributed services.

[2] *OpenTracing*: a specification and standard way that helps developers to easily instrument their code base for distributed tracing.

[3] *What is tracing*: [https://opentracing.io/docs/overview/what-is-tracing/](https://opentracing.io/docs/overview/what-is-tracing/)

[4] *GitHub project*: [https://github.com/mattiacapitanio/distributed-tracing-with-redis](https://github.com/mattiacapitanio/distributed-tracing-with-redis)