Distributed Tracing for Kafka with OpenTelemetry in Node
Published: 2022-08-30 09:10:17

![Distributed Tracing for Kafka with OpenTelemetry in Node](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0un92gsqkc65n95yykuy.png)
 

In this guide, you will learn how to run OpenTelemetry in Node to generate spans for different Kafka operations (e.g., consume and produce) and ultimately visualize your traces. We will also touch on the basics of Kafka, OpenTelemetry, distributed tracing, and how they all play out together.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the  [practical](https://www.aspecto.io/blog/distributed-tracing-for-kafka-with-opentelemetry-in-node/#How-to-use-Kafka-with-OpenTelemetry-in-Node)  part of this guide.

## What to Expect

-   [What is Apache Kafka](#What-is-Apache-Kafka)
-   [What is OpenTelemetry](#What is OpenTelemetry)
-   [What is Distributed Tracing](#What-is-Distributed-Tracing)
-   [What is OpenTelemetry Instrumentation](#What-is-OpenTelemetry-Instrumentation)
-   [How to use Kafka with OpenTelemetry in Node](#How-to-use-Kafka-with-OpenTelemetry-in-Node)
-   [Visualizing OpenTelemetry Traces](#Visualizing-OpenTelemetry-Traces)
-   [TroubleShooting OpenTelemetry Node Installation Issues](#TroubleShooting-OpenTelemetry-Node-Installation-Issues)
-   [Additional OpenTelemetry Resources](#Additional-OpenTelemetry-Resources)

## What is Apache Kafka? <a name="What-is-Apache-Kafka"/>

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka  [here](https://kafka.apache.org/intro).

## What is OpenTelemetry? <a name="What is OpenTelemetry"/>

OpenTelemetry is an open-source project, a collection of APIs and SDKs. Led by the CNCF (Cloud Native Computing Foundation, same foundation responsible for Kubernetes), it allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

In a distributed software world, messaging systems (like Kafka) are key to enabling communication between independent microservices, making it easier to build complex systems.

But understanding the path each message has gone through, where it goes, why, and when is as critical as it is complicated.

Users need a way to visualize and troubleshoot the complexity of their distributed architecture.

OpenTelemetry is our tool to collect data from the different transactions within our services and components (including various message brokers like Kafka) and understand our software’s performance and behavior.

For this specific guide, we will focus on distributed traces, and you can take a deeper dive into OpenTelemetry  [in this short guide](https://www.aspecto.io/blog/what-is-opentelemetry-the-infinitive-guide/).

## What is Distributed Tracing? <a name="What-is-Distributed-Tracing"/>

Distributed tracing defines spans and traces. A trace tracks the progression of requests across the services, message brokers, and other components in our system.

A trace is a tree of spans. A span is a characterization of an activity/process that occurred between our services. For example, a call to a database, external service, sending messages to a Kafka topic, etc.

Traces inform us of the length of each request, which components and services it interacted with, the latency introduced during each step, and other valuable information about the nature of that activity.

![OpenTelemetry Tracing Spans and Traces](https://www.aspecto.io/wp-content/uploads/2022/03/Spans-Diagram-edited-1024x682.jpg)

### What is OpenTelemetry Instrumentation? <a name="What-is-OpenTelemetry-Instrumentation"/>

**Instrumentation**  is a piece of code responsible for generating spans (which make up traces) and sending them to a dedicated backend of your choice. Later, depending on the tool you’re using, the end goal is to visualize them and leverage that visualization for troubleshooting our system.

We do that by using the SDKs provided by OpenTelemetry, which bundles useful instrumentations.

OpenTelemetry includes instrumentations for various libraries in different languages, including a Kafka instrumentation responsible for creating Kafka spans.

## How to use Kafka with OpenTelemetry in Node <a name="How-to-use-Kafka-with-OpenTelemetry-in-Node"/>

By now you should have all the theory you need, so let’s start writing some code. A note before we begin – This guide assumes you are running Kafka in your local machine – If you don’t,  [visit this quick guide](https://www.aspecto.io/blog/how-to-achieve-end-to-end-microservices-visibility-in-asyn-messaging-with-opentelemetry/)  (it shows how to run Kafka with docker in the beginning).

### The Practical Part

First, let’s create a brand new Express app & install relevant packages:

```typescript
npx express-generator kafkaotel
npm install
npm install kafkajs
npm install @opentelemetry/exporter-collector
npm install @opentelemetry/auto-instrumentations-node
npm install @opentelemetry/sdk-node @opentelemetry/api opentelemetry-instrumentation-kafkajs
```

Now create a tracing file that would create spans and traces for Kafka operations (like produce and consume). For this we use the opentelemetry kafkajs  [instrumentation](https://www.npmjs.com/package/opentelemetry-instrumentation-kafkajs):

```typescript
/* tracing.js */
const { KafkaJsInstrumentation } = require('opentelemetry-instrumentation-kafkajs');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const tracerProvider = new NodeTracerProvider({
 // be sure to disable old plugin
 plugins: {
   kafkajs: { enabled: false, path: 'opentelemetry-plugin-kafkajs' }
 }
});
const sdk = new opentelemetry.NodeSDK({
 tracerProvider,
 traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
 instrumentations: [
   new KafkaJsInstrumentation({
     // see kafkajs instrumentation docs for available configuration
   })
 ]
});
sdk.start()
```

Let’s update package.json to require the tracing file so that opentelemetry would be able to instrument:

```typescript
"scripts": {
 "start": "node --require './tracing.js' ./bin/www"
},
```

Now let’s update our main endpoint to produce and consume Kafka messages:

```typescript
const express = require('express');
const router = express.Router();
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092'],
})
const sendKafkaMessage = async () => {
 const producer = kafka.producer();
 await producer.connect();
 await producer.send({
   topic: 'test-topic',
   messages: [
     { value: 'Hello KafkaJS user!' },
   ],
 })
 await producer.disconnect();
}
const consumeMessages = async () => {
 const consumer = kafka.consumer({ groupId: 'test-group' });
 await consumer.connect();
 await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       value: message.value.toString(),
     })
   },
 })
}
/* GET home page. */
router.get('/', async function(req, res, next) {
 await sendKafkaMessage();
 await consumeMessages();
 res.render('index', { title: 'Express' });
});
module.exports = router;
```

Now you can run npm start and run go to localhost:3000 in your favorite browser.

Then you can see the producer span in the console like this one:

```typescript
{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: undefined,
  name: 'test-topic',
  id: 'c83af592be261547',
  kind: 3,
  timestamp: 1646386704369183,
  duration: 28633,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic'
  },
  status: { code: 0 },
  events: []
}

```

And also the consumer span:

```typescript
{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: 'c83af592be261547',
  name: 'test-topic',
  id: '875ea3c9e2b0dbdf',
  kind: 4,
  timestamp: 1646386726625784,
  duration: 562,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic',
    'messaging.operation': 'process'
  },
  status: { code: 0 },
  events: []
}
```

And that’s that! You now know how to use Kafka and OpenTelemetry together, and can create spans for Kafka operations that happen in your microservices.

So far we only printed outputs to our console. But OpenTelemetry has a lot to offer in terms of visualization, so let’s see how you can achieve that.

## Visualizing OpenTelemetry Traces in Node <a name="Visualizing-OpenTelemetry-Traces"/>

For the purposes of this guide, I chose to use  [Aspecto](https://www.aspecto.io/)  as my visualization tool. This is because Aspecto provides built-in support for visualizing messaging systems like Kafka (and, of course, any other part of our microservice architectures).

Before jumping in, you can try it yourself with the [free-forever plan](https://www.aspecto.io/pricing/) that has no limited features.

Give this [Live Playground](https://app.aspecto.io/play/search) a try to get a better idea.

Here’s how it would look at the end of the process:

![Kafka message with OpenTelemetry in Node](https://www.aspecto.io/wp-content/uploads/2022/03/image1-1.png)

You can also click on a specific node and see tracing data:

![Aspecto UI seeing OpenTelemetry tracing data in Node](https://www.aspecto.io/wp-content/uploads/2022/03/image4-1.png)

You can follow along by  [creating a free account](https://www.aspecto.io/pricing/), then, obtain your Aspecto API key, and modify your tracing.js file:

```typescript
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   // add other instrumentations here for packages your app uses
   new KafkaJsInstrumentation({})
 ],
});
```

Then run the app with your Aspecto api key, like this:

```typescript
ASPECTO_API_KEY=YOUR_API_KEY node --require './tracing.js' ./bin/www
```

When you go to localhost:3000 with a browser, it would produce and consume a Kafka message and send corresponding spans to aspecto. Allow a minute or 2 and you could see it under the trace search in aspecto, just like the above picture shows.

If you want to instrument libraries other than Kafka, update your tracing file like this(using the node auto instrumentations library that enables a  [variety of instrumentations](https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node)  by default):

```typescript
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   getNodeAutoInstrumentations(),
   new KafkaJsInstrumentation({})
 ],
});
```

But note: for Node, it’s recommended to use the  [Aspecto SDK](https://docs.aspecto.io/v1/send-tracing-data-to-aspecto/aspecto-sdk/nodejs)  which uses OpenTelemetry under the hood but also supports collecting actual payloads.

If you take the exact same service from scratch – without all the OpenTelemetry installations (make sure uninstall them to avoid any version conflicts), and run:

```typescript
npm install @aspecto/opentelemetry
```

Then add the following code at the top of your app.js file:

```typescript
require('@aspecto/opentelemetry')({
 aspectoAuth: process.env.ASPECTO_API_KEY
});
```

And now you can even see the payload of the Kafka message:

![Payload of the Kafka message with OpenTelemetry in Node](https://www.aspecto.io/wp-content/uploads/2022/03/image3-1-1024x476.png)

P.S. If you prefer to use an open-source to visualize your traces, you can try Jaeger Tracing. You can follow this  [guide on deploying Jaeger on AWS](https://www.aspecto.io/blog/how-to-deploy-jaeger-on-aws-a-comprehensive-step-by-step-guide/)  and make sure to scroll to the part on sending traces to Jaeger with the OpenTelemetry SDK (In NodeJS). Keep in mind that it may take a bit more time to set up the required environment.

If you’re not familiar with Jaeger and get a deeper understanding of it, we recommend visiting this  [guide](https://www.aspecto.io/blog/jaeger-tracing-the-ultimate-guide/).

I hope this has been a useful guide for you to learn how to use OpenTelemetry in Node to generate traces for Kafka operations. Feel free to  [reach out](https://twitter.com/thetomzach)  if you have any questions!

## TroubleShooting OpenTelemetry Node Installation Issues <a name="TroubleShooting-OpenTelemetry-Node-Installation-Issues"/>

It is not uncommon to encounter several issues when trying to install OpenTelemetry. More specifically, trying to install OpenTelemetry in your Node application and not seeing any traces or some expected spans are missing.

To help you avoid these common pitfalls, we recommend going over this  [in-depth checklist of issues and solutions](https://www.aspecto.io/blog/checklist-for-troubleshooting-opentelemetry-nodejs-tracing-issues/). We cover the most common mistakes and show you how to solve them.

## Additional OpenTelemetry Resources <a name="Additional-OpenTelemetry-Resources"/>

If you want to go deeper into learning about OpenTelemetry, visit  [The OpenTelemetry Bootcamp video library](https://www.aspecto.io/opentelemetry-bootcamp/). It’s a free and vendor-neutral, in-depth course that will help you master OpenTelemetry. We cover everything from the very basics to security and deploying in production.