Learning Observability (w/ OpenTelemetry and Jaeger)
Tags: devops, todayilearned, go, monitoring
Published: 2021-12-16 23:41:03

Hey folks, there has been a long time since I have last posted anything here and I wanted to start writing again.

A quick disclaimer, as always, I am learning as I write this article, so I may get something wrong, I would love some feedback if you catch anything

My idea here is to make some projects to teach myself how to create highly observable services

### First Project: Fibonacci

To start, I've decided to follow the golang  [OpenTelemetry tutorial](https://opentelemetry.io/docs/instrumentation/go/getting-started/).

And used the [jaeger example](https://github.com/open-telemetry/opentelemetry-go/blob/main/example/jaeger/main.go) to create the tracer provider.

The result for this project is a simple program that calculates Fibonacci based on the input you provide and sends the data about each individual operation to jaeger

It looks like this in the jaeger UI: 

![Trace example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/34cbt6xci9zsz96968gv.png)

The important concepts behind this project are traces, spans, and tracer providers.

#### Traces 

Traces are an abstraction for an operation or call inside your system, and they are composed by spans.

Traces can track the life-cycle of a operation through many processes (I will probably get there in the next project).

In the case of this program, this is the trace strucutre:

![trace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5m0joieptucq5nlk0gaw.png)

Specifically in golang, the context library is used to create this structure, so you will need a context for each separate trace.

#### Spans

Spans are the subdivisions of a trace, the way I see it, they are meant to represent each function execution inside a given trace.

Each span can contain attributes, that are key/value pairs that should represent the state of that function when they're called.

#### Tracer Provider

Tracer providers are configurations for the project about where to send the tracing data and how to send it.

In the case of this project, that would be jaeger.

### Second Project: Square

The second project was customized, I did not write the steps for it in as much detail as these documentations, but it's fairly simple code (the link for the repositories is at the end of the post).

The idea was to create two services and a trace that would have spans in both services.

So I have created this two services:

* API: has an http server that receives requests and calls the other service
* Square: squares the number sent to it

This is how a request looks like for the user >

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ezwajb3bkksd9lh3wnq2.png)

And this is how it looks on jaeger >

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i9zpho8cnjt6er7hx6ep.png)

And the structure for the trace >

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfkmm0rl54wtozzsmgti.png)

And as a nice side effect, you also get this graph that shows the dependencies for your services and the number of requests sent from a service to another >


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01y97fc41rlpsiifyn26.png)

And for this second project there was a single new relevant concept: propagators 

#### Propagators

Propagators are the tool used to share context;

The metadata is injected in the headers for the request, then the server receiving the request extracts it and adds to it's own spans.

The result is this graphic where you know how much time the trace spent in each service and in each operation.

### Final Words

So, I covered the basics of observability, and I can see how this would be relevant on onboarding members to your team and also to debugging

Next I will probably try to learn other observability tools like prometheus and grafana

### Repositories

{% github ciochetta/learn-jaeger no-readme %}

{% github ciochetta/go-square no-readme %}

### Useful links

https://opentelemetry.io/docs/concepts/what-is-opentelemetry/

https://www.jaegertracing.io/docs/1.29/getting-started/

https://github.com/open-telemetry/opentelemetry-go

https://github.com/open-telemetry/opentelemetry-go/blob/main/example/jaeger/main.go