The Mechanics of Distributed Tracing in OpenTelemetry
Tags: distributedsystems, programming, opentelemetry, monitoring
Published: 2024-05-30 15:19:08

## Introduction

[OpenTelemetry](https://opentelemetry.io/) is an open-source observability framework that provides mechanisms for creating and sending traces, metrics, and logs. It consists of various elements such as protocols for transmission and SDKs for different programming languages. In this article, we will explore how OpenTelemetry achieves distributed tracing.

## What is Distributed Tracing?

Distributed tracing is a technique for tracking and monitoring traces across multiple servers, like microservices. It helps to visualize and understand the flow of a request as it traverses through different services.

### Key Components of Distributed Tracing

- **Trace**: A collection of spans representing a single request or transaction.
- **Span**: A single unit of work within a trace, representing a specific operation.

A trace is a tree structure composed of multiple spans. Here's a visual representation:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t8zp5c7ztufadedpa0ea.png)

_Explain Like I'm 5 explanation about Distributed Tracing_ _(for [LinkedIn users](https://www.linkedin.com/posts/siddhantkhare24_distributedtracing-microservices-techexplained-activity-7200167422656409601-I636?utm_source=share&utm_medium=member_desktop), for [Twitter users](https://x.com/Siddhant_K_code/status/1794395782335926364))_

## Understanding Trace from Span

To achieve distributed tracing, it is essential to understand the relationship between traces and spans. Each span includes the following elements:

- **TraceId**: The ID of the trace to which the span belongs.
- **SpanId**: A unique ID for the span within the trace.
- **ParentSpanId**: The ID of the parent span.

These elements are specified in the span using Protocol Buffers.


![Opentelemetry proto](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/31bx29oa50sx10ri2j0x.png)


[**_Code snippet ref_**](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto#L83-L110)



### Example of Span Elements in a Trace

Consider the following Go code example:

```go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

func CreateTrace() {
    tracer := otel.Tracer("example-tracer")
    ctx, parentSpan := tracer.Start(context.Background(), "parent-span")
    defer parentSpan.End()

    ctx, childSpan := tracer.Start(ctx, "child-span")
    defer childSpan.End()
}
```

If you print this to stdout, it will output something like this:

```json
{
  "Name": "child-span",
  "SpanContext": {
    "TraceID": "9023c11c3272a955da5f499faa9afa71",
    "SpanID": "ca44f59e13b40d44"
  },
  "Parent": {
    "TraceID": "9023c11c3272a955da5f499faa9afa71",
    "SpanID": "70e471ef5735034d"
  }
}
{
  "Name": "parent-span",
  "SpanContext": {
    "TraceID": "9023c11c3272a955da5f499faa9afa71",
    "SpanID": "70e471ef5735034d"
  },
  "Parent": {
    "TraceID": "00000000000000000000000000000000",
    "SpanID": "0000000000000000"
  }
}
```

In this example, the `TraceId` is the same for both spans, indicating they belong to the same trace. The `ParentSpanId` of the child span matches the `SpanId` of the parent span, establishing a parent-child relationship.


![example diagram of parent, child span](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/emnpubvw9hrtjtcal9ef.png)



## Propagation of Trace Context

To enable distributed tracing across multiple services, the trace context needs to be propagated. This is achieved by passing the `TraceId` and `SpanId` through headers in HTTP requests.

### W3C Trace Context

The W3C Trace Context specification standardizes how trace context information is passed. The `traceparent` header is used in HTTP requests with the format: `${version}-${trace-id}-${parent-id}-${trace-flags}`.

Example using `curl`:

```bash
curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" localhost
```

### Propagation in Go

Here's an example of a server and client in Go that demonstrates trace context propagation:

#### Server Code

```go
package main

import (
    "fmt"
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/trace"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func RunServer() {
    exp, _ := stdouttrace.New()
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
    )

    otelHandler := otelhttp.NewHandler(http.HandlerFunc(handler), "handle-request", otelhttp.WithTracerProvider(tp))
    http.Handle("/", otelHandler)
    http.ListenAndServe(":9002", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Println("handled")
}
```

#### Client Code

```go
package main

import (
    "context"
    "io"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
)

func CreatePropagationTrace() {
    tracer := otel.Tracer("example-tracer")
    ctx, span := tracer.Start(context.Background(), "hello-span")
    defer span.End()

    req, _ := otelhttp.Get(ctx, "http://localhost:9002")
    io.ReadAll(req.Body)
}
```

### Output

When you run the server and client, the output will show the trace propagation:

#### Server Trace

```json
{
  "Name": "handle-request",
  "SpanContext": {
    "TraceID": "817f4043c5837f2bbb44562f3683f274",
    "SpanID": "3bba3b994e029bfc"
  },
  "Parent": {
    "TraceID": "817f4043c5837f2bbb44562f3683f274",
    "SpanID": "892d624c6f0c01a6"
  }
}
```

#### Client Trace

```json
{
  "Name": "HTTP GET",
  "SpanContext": {
    "TraceID": "817f4043c5837f2bbb44562f3683f274",
    "SpanID": "892d624c6f0c01a6"
  },
  "Parent": {
    "TraceID": "817f4043c5837f2bbb44562f3683f274",
    "SpanID": "1f312e90fb65c0e3"
  }
}
{
  "Name": "hello-span",
  "SpanContext": {
    "TraceID": "817f4043c5837f2bbb44562f3683f274",
    "SpanID": "1f312e90fb65c0e3"
  },
  "Parent": {
    "TraceID": "00000000000000000000000000000000",
    "SpanID": "0000000000000000"
  }
}
```


![Overall process](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a8f1328d02r4bjbfixqe.png)



## Conclusion

Distributed tracing with OpenTelemetry enables us to track and monitor requests across multiple services by passing trace context through headers. By understanding and implementing the elements of `TraceId`, `SpanId`, and `ParentSpanId`, we can visualize the flow of a request and diagnose issues more effectively.

With the standardized W3C Trace Context, trace context propagation becomes consistent and interoperable across different services and platforms.

This article has covered the basics of how OpenTelemetry achieves distributed tracing, providing code examples and visualizations to illustrate the concepts. Happy tracing!

For more details, visit the [OpenTelemetry Documentation](https://opentelemetry.io/docs/).

---

For more tips and insights on monitoring and tech, follow me on Twitter [@Siddhant_K_code](https://twitter.com/Siddhant_K_code) and stay updated with the latest & detailed tech content like this. Happy coding!