Ways of effortless instrumentation.
Tags: observability, monitoring, ebpf, opensource
Published: 2023-09-18 06:38:24

Understanding the inner workings of our services and applications on production is a great and powerful thing but there is one big upfront cost - code  instrumentation. Not that long ago the default option to run applications was to buy racks and servers and set them up in the server room or in the co-located data center, now in addition to that we have a landscape of cloud IaaS and serverless offerings, essentially computing power on demand where the upfront cost is significantly smaller. We need a similar transformation of cost and effort distribution in the observability world especially in the times when the number of possible architectures is growing and architectures like microservices allow full team ownership over a set of services including tech stack choices which can result in a vast range of different techniques to monitor these stacks. 

_This article is a part of formatted notes after my search for effortless yet reliable observability stack instrumentation techniques. Hopefully you will find them useful and insightful._

## Table of contents
1. [Intro](#introduction) 
2. [.NET Startup Hooks.](#net-startup-hooks)
3. [Monkey-Patching python example.](#monkeypatching-python-example)
4. [New(old)eBPF.](#newoldebpf)
5. [Is autoinstrumentation enough?](#is-autoinstrumentation-enough)
6. [Autoinstrumentation at scale.](#autoinstrumentation-at-scale)

## Resources
1. [Opentelemetry project](https://github.com/open-telemetry)
2. [Cilium Hubble](https://github.com/cilium/hubble)
3. [Odigos](https://github.com/keyval-dev/odigos)
4. [Autoinstrumentation-playground](https://github.com/SzymonSt/autoinstrumentation-playground)

## Introduction
There are many techniques of instrumentation, each is adjusted to a specific programming language or set of languages. I've tested a few implemented by maintainers of the opentelemetry project. Cloning [autoinstrumentation-playground repo](https://github.com/SzymonSt/autoinstrumentation-playground) and running services with sample usage of instrumentation techniques while going through this article  is highly recommended to understand nuances of following implementations.

## .NET Startup Hooks
This specific type of instrumentation is dedicated for .net clr virtual machine.
Basic set up is fairly simple as it should be and thanks to opentelemetry standard even with only one service(root) instrumented we are able to get some meaningful information about downstream services as well. That can be a proof of value in the project that needs to be instrumented.
`curl -X PUT -H "Content-Type: application/json" -d '{"Name": "alpha","Email": "alpha@alpha.com"}' http://localhost:8088/set_profile`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l330n57m5ggkmsf8tb4k.png)



In order to enable instrumentation in c#/.net application we need to provide opentelemetry details starting with OTEL_ prefix, clr and general dotnet details.

```
CORECLR_ENABLE_PROFILING="1"
CORECLR_PROFILER='{918728DD-259F-4A6A-AC2B-B85E1B658318}'
CORECLR_PROFILER_PATH="/otel-dotnet-auto/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so"
DOTNET_ADDITIONAL_DEPS="/otel-dotnet-auto/AdditionalDeps"
DOTNET_SHARED_STORE="/otel-dotnet-auto/store"

#Openeteletry default STARTUP HOOK an dotnet specific opentelemetry settings
DOTNET_STARTUP_HOOKS="/otel-dotnet-auto/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll"
OTEL_DOTNET_AUTO_HOME="/otel-dotnet-auto"
OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES: dummy-gateway

#Opentelemetry configuration details
OTEL_SERVICE_NAME: dummy-gateway
OTEL_EXPORTER_OTLP_ENDPOINT: http://autoinstrumentation-playground-otel-collector-1:4318
OTEL_PROPAGATORS: tracecontext,baggage
```

Startup hooks allow us to inject code before application startup that's why we need to provide the path to the dll file which will inject [Opentelemetry .NET SDK](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#opentelemetry-net-sdk) to our application using `DOTNET_STARTUP_HOOKS="/otel-dotnet-auto/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll"` env variable. In autoinstrumentation-playground this file is downloaded and set up using script provided by maintainers of dotnet autoinstrumentation components.

_[Dockerfile](https://github.com/SzymonSt/autoinstrumentation-playground/blob/main/dummy-gateway/Dockerfile)_
```
##OTEL DOTNET AUTOINSTRUMENTATION
ARG OTEL_VERSION=0.7.0
ADD https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v${OTEL_VERSION}/otel-dotnet-auto-install.sh otel-dotnet-auto-install.sh
RUN apt-get update && apt-get install -y unzip && \
    OTEL_DOTNET_AUTO_HOME="/otel-dotnet-auto" sh otel-dotnet-auto-install.sh
##
```

Another ineteresing piece of configuration is `OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES: dummy-gateway` env variable by providing the name of our binary we are able to extend auto generated traces with custom ones according to the needs. Similar variable exists for metrics, more about it [here](https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/manual-instrumentation.md).

## Monkey-Patching python example
Monkey-patching is a techinque that allows instrumenting library to intercept oryginal call, produce necessary telemetry data and then calls underlying library. It is useful for interpreted languages where we cannot inject code like in a previous example. 
In order to instrument our python app we need to use wrapper script:
_[Dockerfile](https://github.com/SzymonSt/autoinstrumentation-playground/blob/main/dummy-profile-service/Dockerfile)_

```
# With opentelemetry instrumentation
CMD [ \
"opentelemetry-instrument", \
"--traces_exporter", "otlp", \
"--metrics_exporter", "none", \
 "python", "-u", "main.py" ]
```
...and set up basic env variables for opentelemetry trace exporter:

```
OTEL_SERVICE_NAME=dummy-profile-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://autoinstrumentation-playground-otel-collector-1:4318
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
```
Let's call /set_profile gateway endpoint once again:
`curl -X PUT -H "Content-Type: application/json" -d '{"Name": "alpha","Email": "alpha@alpha.com"}' http://localhost:8088/set_profile`


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bagk6mb5nugj9ddsff0w.png)

Span produced by python instrumentation lib is now a child span to dummy-gateway spans thanks to [tracecontext standard](https://www.w3.org/TR/trace-context/). After a bit of playing around I've even managed to add custom attribute by getting current tracer:

[server.py](https://github.com/SzymonSt/autoinstrumentation-playground/blob/main/dummy-profile-service/profileservice_server/server.py)
```
def __init__(self, db_conn) -> None:
   self.db_conn = db_conn
   self.tracer = trace.get_tracer(__name__)
                           ......
trace.get_current_span().set_attribute("api-version","v1")
```

This doesn't seem to be possible while working with .net startup hooks(at least I didn't manage to reproduce it) same for eBPF instrumentation. Any suggestions in comments on how it could be approached in .net and eBPF instrumentation types are most welcome.


## New(old)eBPF
Piece of linux technology that seem to experience some type of rebirth recently, even [microsoft is trying to incorporate this feature in their OSes](https://github.com/microsoft/ebpf-for-windows).
eBPF is a feature of linux kernel which can do a lot more than pulling insights about apps running on host but we will focus on just this use case here. Thanks to eBPF programs running in kernel not userspace all of the syscalls sourcing from our application process can be monitored. For those who are interested in eBPF technology I recommend book by Liz Rice "Learning eBPF".

In opposition to previous two examples this instrumentation type doesn't inject any code, we need to run a sidecar though, see [docker-compose.yaml](https://github.com/SzymonSt/autoinstrumentation-playground/blob/main/dummy-score-service/docker-compose.yaml).

```
  dummy-score-instrumentation:
    image: 322456/otel-go-autoinstrumentation:latest
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://autoinstrumentation-playground-otel-collector-1:4317
      OTEL_SERVICE_NAME: dummy-score-service
      OTEL_GO_AUTO_TARGET_EXE: /app/bin/scoreservice
      OTEL_PROPAGATORS: tracecontext,baggage
      CGO_ENABLED: 1
    privileged: true
    pid: host
    volumes:
      - shared-instrumentation-data:/shared-data
      - /proc:/host/proc
```
In addition to that as you can see in the mentioned docker compose file we need to assign pid from host to our container process and mount /proc host volume to both `dummy-score-service` and `dummy-score-instrumentation`.

Let's call /submit_score gateway endpoint:
`curl -X PUT -H "Content-Type: application/json" -d '{"UserProfile": "alpha","Score": 1}' http://localhost:8088/submit_score`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pt1nmedqj44sy535esuj.png)

## Is autoinstrumentation enough?
While it all looks very promising and taking into account how immature yet how powerful frameworks for this techniques are now one can be very optimistic. However autoinstrumentation certainly has some limitations. The right question to ask here would be how we can combine autoinstrumentation and minimalistic manual instrumentation, correlate it to acheive truly reliable and customizable instrumenation with positive change of cost and effort distribution to achieve state of the art observability.

## Autoinstrumentation at scale.
Previous example on dummy application shows that in different tech stacks autoinstrumentation can be approached differently. My first thought about that was that it would be super hard and tidious to enforce it on a multiple teams level or even whole organization level. While trying to find a solution for that problem I learnt about few projects. I've created an obsrervablity stack with three projects: Odigos, Opentelemetry and Jaeger Tracing. Let's explore it a bit.
Odigos essentialy orchestrates different types of instrumentation and leverages eBPF technology so you do not need to.
After deploying dummy app and observability stack to k8s - [guide](https://github.com/SzymonSt/autoinstrumentation-playground#running-k8s-cluster-wide-instrumentation-with-otel-collector-jaeger-and-odigos)

We need to set up odigos with opentelemetry this time only traces but it can be done for metrics and logs as well. Autoinstrumentation-playground uses helm chart to deploy odigos however odigos cli seems to be recommended approach for instalation and managing odigos. More about odigos and opentelemetry project in the Resources section of the article.
1.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y6eqau17bkddz7z6l91x.png)
2.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g7cpjgzwptl192lmdwc5.png)

After generating some traffic we can already see generated traces with just a two steps configuration.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/99pa591mbozldm2r0m1b.png)



##Contributing
Observablity landscape evolves very quickly and a lot of people are working on these opensource projects becuase of that all of the contributions to [autoinstrumentation-playground](https://github.com/SzymonSt/autoinstrumentation-playground) project to keep it up to date are most welcome.