Distributed Tracing and OpenTelemetry Guide
Tags: opentelemetry, backend, microservices, node
Published: 2023-09-29 06:26:07

---
title: Distributed Tracing and OpenTelemetry Guide
description: Dive into the essentials of distributed tracing in microservices with OpenTelemetry, amplifying system observability in the Node.js ecosystem.
published: true
tags: opentelemetry,backend,microservices,nodejs
canonical_url: https://amplication.com/blog/distributed-tracing-and-open-telemetry-guide
cover_image: https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/hero.png
---

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/0.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Microservices have become popular for modern web applications since they provide many benefits over traditional monolithic architectures. However, microservices are not a silver bullet; they also have a fair share of challenges. For example, debugging and troubleshooting errors in microservices can be challenging since tracking the request flow across multiple services is difficult.

That's where distributed tracing and [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) come in. OpenTelemetry is an Observability framework designed to create and manage telemetry data like traces, metrics, and logs from distributed systems. So, in this article, I will take you through the steps of using OpenTelemetry within a Node.js ecosystem to trace your microservices applications effectively.

## What is distributed tracing

The complexity of microservices makes it difficult to track the request path through multiple microservices. Distributed tracing is an observability technique used to track these requests across microservices. In other words, we can define it as a flashlight that helps you understand the request flow across your system.

Distributed tracing is beneficial for developers in many scenarios. For example, there can be a single microservice with a slow response time, slowing down the whole application. Tracing data lets you pinpoint the exact origin and easily troubleshoot the issue.

**Benefits of using distributed tracing:**

*   Identify performance bottlenecks.
*   Provides a comprehensive view of the system.
*   Provides insights into the dependencies between different services.
*   Identify potential security vulnerabilities.
*   Support both synchronous (gRPC, REST, GraphQL) and asynchronous (event sourcing, pub-sub) application architectures.

## Components of distributed tracing?


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/1.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

A typical distributed tracing system is built up with the below components:

*   **Trace**: End-to-end path of a single user request as it moves through various services.
*   **Span**: A single operation or unit of work within a distributed system. It captures information like start time, end time, metadata, or annotation that might be useful to understand what is happening.
*   **Context Propagation**: Passing contextual information between different services within a distributed system. It is essential for connecting spans to construct a complete trace of a request.

Since you now have a brief idea of what distributed tracing is, let's see how to implement distributed tracing with Node.js.

## Instrumenting Node.js app with OpenTelemetry

In this example, I will create 3 Node.js services (shipping, notification, and courier) using [Amplication](https://amplication.com/), add traces to all services, and show how to analyze trace data using [Jaeger](https://www.jaegertracing.io/).

### Step 1: Generating services using Amplication

As the first step, you must create the Node.js services with [Amplication](https://app.amplication.com/login). In this example, I will be using three already created Prisma schemas. You can find those schemas in [this GitHub repository](https://github.com/overbit/otel-workshop).

Once you are ready with schemas, go to the Amplication dashboard and create a new Project.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/2.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Then, select the project from the dashboard and connect the GitHub repository with Prisma schemas to that project.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/3.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Now, you can start creating services. For that, return to the Amplication dashboard and click the **Add Resources** button.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/4.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Then, enter the necessary information to make the service. In this case, I have named the services as "**courier gateway service**" with the below settings:


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/5.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

*   **Git Repository**: I've used the GitHub repo, which I connected earlier.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/6.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

*   **REST or GraphQL**: I've enabled both options to show the file structure generated by Amplication.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/7.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />


*   **Repo type**: Monorepo.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/8.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

*   **Database**: PostgreSQL


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/9.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

*   **Authentication**: Included


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/10.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

It will take a few seconds to generate the service.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/11.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

After that, you need to modify a few database settings to avoid collision between databases when sharing the same Docker service. For that, navigate to **the Plugins** tab, select **the PostgreSQL DB** plugin, and click the **Settings** button.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/12.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

There, you will see a JSON file like below, and you need to update the **dbName** property. Here, I have renamed it to **courier**.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/13.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Then, go back to **the Entities** tab and import the courier Prisma schema to generate the entities related to the courier service.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/14.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Once the schema is imported, you will see 2 new entities named **Parcel** and **Quote** in the **Entities** tab.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/15.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Now, perform the same steps again for the other two services.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/16.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

### Step 2: Adding a Kafka integration

In this example, I will use a Message Broker to communicate between these services. You can easily generate a Message Broker through Amplication by clicking the **Add Resource** button and selecting **the Message Broker** option.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/17.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Then, go back to the shipping service and install the Kafka plugin to allow the shipping service to use the Message Broker.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/18.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Then, go to the **Connections** tab and select the **Message pattern** as **Send** to allow the shipping service to send messages.


<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/19.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Similarly, go to the notification service and select **Message pattern** as **Receive** to subscribe to the Message Broker.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/20.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

### Step 3: Building the application

Click the **Commit change & build** button to finalize the changes. It will start the build process, generate the new files in the Git repo, and create a pull request.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/21.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

**_Note:_** _Make sure to merge the pull request to the main branch to get the latest updates._ 

### Step 4: Configuring Docker compose

Each service generated by Amplication contains a separate Docker compose file. But, in this example, I want to share the same database with all services. Hence, I created a [new Docker compose file](https://github.com/overbit/otel-workshop/blob/main/docker-compose.yml) by coping the content of the docker-compose files generated by amplication.



```
version: "3"
name: otel-workshop
services:
  # Shared DB for all services
  db:
    image: postgres:12
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: admin
    volumes:
      - postgres:/var/lib/postgresql/data

  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268"
      - "14250"
  # Collector
  collector-gateway:
    image: otel/opentelemetry-collector:latest
    volumes:
      - ./collector-gateway.yml:/etc/collector-gateway.yaml
    command: ["--config=/etc/collector-gateway.yaml"]
    ports:
      - "1888:1888" # pprof extension
      - "13133:13133" # health_check extension
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP HTTP receiver
      - "55670:55679" # zpages extension
    depends_on:
      - jaeger-all-in-one

  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8080:8080"
    depends_on:
      - zookeeper
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
      KAFKA_CLUSTERS_0_JMXPORT: 9997

  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.3.1
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9997:9997"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_MESSAGE_MAX_BYTES: 10485760
      JMX_PORT: 9997
      KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka -Dcom.sun.management.jmxremote.rmi.port=9997
    healthcheck:
      test: nc -z localhost 9092 || exit -1
      start_period: 15s
      interval: 30s
      timeout: 10s
      retries: 10

volumes:
  postgres: ~
```

I don't need to specify a collector getaway in the above configuration since I'm using **jaeger-all-in-one**. However, I have specified a [collector getaway](https://github.com/ChameeraD/otel-example/blob/main/collector-gateway.yml) to highlight the receiver's components and ports.

Run `docker-compose up --detach` command to start Docker.

### Step 5: Configuring services for local development

Now, you need to set up all 3 services for local development. For that, you just need to follow the instructions given in the **README.md** file.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/22.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

**_Note:_**_: You don't need to run `_npm run docker:dev_` command since Docker is already running_

Once all the databases are initialized and dependencies are installed, start each service using `npm run start:watch` command and the courier-gateway-service-admin using `npm run start` command.

**_Note_**_: You need to update the ports of each service from the_ `_.env_` _files to avoid clashes between the services._

## Step 6: Creating a Parcel through admin view

You can easily create a new Parcel by logging into the admin view.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/23.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />
.
<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/24.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

### Step 7: Connecting the services

First, navigate to the shipping service and install axios using `npm install axios` command. Then, add the below code to the **shipping-service/src/shipment/shipment.service.ts** file to get parcel details. 

```ts
import { Injectable } from "@nestjs/common";
import { PrismaService } from "../prisma/prisma.service";
import { ShipmentServiceBase } from "./base/shipment.service.base";
import { Prisma, Shipment } from "@prisma/client";
import axios from "axios";
import { KafkaProducerService } from "../kafka/kafka.producer.service";
import { ShippingEvent } from "./shipping.event";
import { MyMessageBrokerTopics } from "../kafka/topics";

@Injectable()
export class ShipmentService extends ShipmentServiceBase {
  constructor(
    protected readonly prisma: PrismaService,
    private readonly kafkaProducerService: KafkaProducerService
  ) {
    super(prisma);
  }

  async create<T extends Prisma.ShipmentCreateArgs>(
    args: Prisma.SelectSubset<T, Prisma.ShipmentCreateArgs>
  ): Promise<Shipment> {
    const {
      data: { accessToken },
    } = await axios.post(http://localhost:3002/api/login , {
      username: "admin",
      password: "admin",
    });

    const { data: parcels } = await axios.get(
      http://localhost:3002/api/parcels ,
      {
        params: {},
        headers: {
          Authorization: Bearer ${accessToken} ,
        },
      }
    );

    const randomParcel = Math.floor(Math.random() * parcels.length);

    const shipment = await super.create<T>({
      ...args,
      data: {
        ...args.data,
        price: parcels[randomParcel].price,
      },
    });

    const event: ShippingEvent = {
      Message: Shipment id: ${shipment.id} ,
      CustomerId: "1b2c",
    };

    await this.kafkaProducerService.emitMessage(
      MyMessageBrokerTopics.ShipmentCreateV1,
      {
        key: shipment.id,
        value: event,
      }
    );

    return shipment;
  }
}

```
### Step 8: Creating a client app

Before starting instrumenting, let's create a client application to get shipment data. This can be a simple Node.js project with a **main.js** file containing the code to fetch shipment data. 

```js
// main.js

"use strict";
const axios = require("axios");

const url = "http://localhost:3004/api/shipments";
const numberOfRequests = 5;

const makeRequest = async (requestId) => {
  const result = await axios.post(url);
  return result;
};

const main = async () => {
  for (let i = 0; i < numberOfRequests; i++) {
    const res = await makeRequest(i);
    console.log("Response", res.data);
  }
};

main();

```



### Step 9: Adding tracing

Create a new file named **tracing.js** in the same directory where you created the **main.js** file to fetch shipment data. Then, install all the OpenTelemetry dependencies using the below command:

```
ls
npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/resources\
  @opentelemetry/semantic-conventions \
  @opentelemetry/instrumentation-http

```

Add the below code to the **tracing.js** file.

```
const {
  BasicTracerProvider,
  SimpleSpanProcessor,
} = require("@opentelemetry/sdk-trace-base");
const { Resource } = require("@opentelemetry/resources");
const {
  SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");
const { trace } = require("@opentelemetry/api");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-http");
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { B3Propagator } = require("@opentelemetry/propagator-b3");

const exporter = new OTLPTraceExporter({});

const getTracer = () => {
  return trace.getTracer("default");
};

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "fake-client-app",
    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.0",
  }),
  spanProcessor: new SimpleSpanProcessor(exporter),
  traceExporter: exporter,
  instrumentations: [new HttpInstrumentation()],
  textMapPropagator: new B3Propagator(),
});

sdk.start();

module.exports = { getTracer };

```

To start tracing, you need to import the above file to the **main.js** file and do some modifications. The updated **main.js** file will look like below:

```
js
"use strict";
const { getTracer } = require("./tracing");
const axios = require("axios");
const { trace } = require("@opentelemetry/api");

const tracer = getTracer("fake-client");

const url = "http://localhost:3004/api/shipments";
const numberOfRequests = 1;

const makeRequest = async (requestId) => {
  return tracer.startActiveSpan("makeRequests", async (span) => {
    span.updateName(makeRequests-${requestId} );
    const result = await axios.post(url);
    span.end();
    return result;
  });
};

tracer.startActiveSpan("main", async (span) => {
  for (let i = 0; i < numberOfRequests; i++) {
    const res = await makeRequest(i);
    console.log("Response", res.data);
  }
  span.end();
});
```

Now, you can run the client application with the `node main.js` command and monitor the trace data with Jeager.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/25.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />
.
<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/26.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

That's it. You successfully created a Node. js-based microservices application using Amplication, configuring tracing, and monitoring trace data through Jeager. You can find the complete code example in [GitHub](https://github.com/amplication/otel-workshop/tree/main) and watch the video below to understand the code used for tracing.

### Step 10: Adding tracing to the generated services

As Amplication now supports OpenTelemetry through a plugin, we will leverage the plugin to integrate all the services without much effort.

Go to each service starting from the shipping service and install the OpenTelemetry plugin.

<div>
<img src="https://static-assets.amplication.com/blog/distributed-tracing-and-open-telemetry-guide/27.png" alt="" style="margin-left: auto; margin-right: auto;" height="943" width="1910">
</div>
<br />

Click the **Commit change & build** button to finalize the changes. It will start the build process again, generate the new files and update existing ones in the Git repo, and create/update a pull request.

Now try to perform new requests as before and observe the tracing data in Jaeger!

## Watch Webinar

I took a live workshop few weeks ago on Distributed Tracing and Open Telemetry. You can watch it here: [https://www.youtube.com/watch?v=Pu-HiD2QksI](https://www.youtube.com/watch?v=Pu-HiD2QksI)

## Best practices to follow

*   Prioritize critical paths and high-impact services.
*   Use consistent and meaningful naming conventions for spans, and services.
*   Ensure that trace context is propagated across service boundaries. This typically involves adding trace headers to HTTP requests or message headers.
*   Use tags and annotations to [add additional metadata](https://youtu.be/Pu-HiD2QksI?si=1WsJNHO5PWUCrB2V&t=3056) to spans.
*   Implement adaptive sampling strategies that adjust the sampling rate based on the service's load, and error rates.
*   Automatically capture and log errors.
*   Retain trace data for an appropriate period.

## Conclusion

This guide provided an overview of implementing tracing for Node.js-based microservices applications. As you can see, enabling tracing for your application requires little effort. But it can save you a whole lot of troubleshooting and debugging time. Thank you for reading.