OTel operator: to simplify observability on kubernetes
Tags: opentelemetry, kubernetes, observability, instrumentation
Published: 2022-11-14 13:30:03

OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for understanding, analyzing and debugging purposes.

OTel also promotes the concept of Bring Your Own (BYO) storage backend (eg: elastic stack, AppDynamics, Dynatrace ..) and BYO query (eg: prometheus) by providing common intermediate framework for instrumentation and data collection/transport/transform with OTLP.

#### OTel kubernetes operator
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/39wxz9asr7pjdsit3i81.png)
This operator provides 2 custom resources:
1. OTel collector for collection/transform and transport telemetry data
2. Auto-instrumentation through init container pattern for supported languages (Python, NodeJS and Java at the moment)

#### Modes of deployment
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mj8tnohl3ncdmzbml8p1.png)
The operator(collector CR) can be deployed as:
1. Deployment (default)
2. DaemonSet
3. SideCar injection
Note: Multiple modes can co-exist based on the architecture (eg: side-car collectors pushing to a central Deployment aggregator)
[Official documentation] (https://github.com/open-telemetry/opentelemetry-operator/blob/main/README.md#deployment-modes)

##### Installation
```bash
# Pre-req: kubernetes cluster with cert-manager enabled

# Deployment based
$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
namespace/opentelemetry-operator-system created
customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created
...
deployment.apps/opentelemetry-operator-controller-manager created

$ kubectl get crds | grep opentel
instrumentations.opentelemetry.io          2022-11-14T06:14:32Z
opentelemetrycollectors.opentelemetry.io   2022-11-14T06:14:32Z

$ kubectl get deployments.apps -n opentelemetry-operator-system
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
opentelemetry-operator-controller-manager   1/1     1            1           56s

# Create a collector like below
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-col
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
EOF
opentelemetrycollector.opentelemetry.io/otel-col created
```

---
#### Auto-instrumentation
1. Create an instrumentation object
```bash
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
EOF
instrumentation.opentelemetry.io/my-instrumentation created

$ kubectl get otelinst
NAME                 AGE   ENDPOINT                     SAMPLER                    SAMPLER ARG
my-instrumentation   35s   http://otel-collector:4317   parentbased_traceidratio   0.25

ku
```

2. Enabling instrumentation is done through annotations
```yaml
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-python: "true" # or inject-java or inject-nodejs
```
Possible values are `true`, `false`(no instrumentation), `my-instrumentation` (custom instrumentation) or `my-namespace/my-instrumentation` (instrumentation defined in `my-namespace`).

Further tuning on which containers to instrument can be done with annotation `instrumentation.opentelemetry.io/container-names: a,b,c...`

When side-car based instrumentation is to be done, use the annotation `sidecar.opentelemetry.io/inject: "true"`
To use 3rd-party instrumentation, check this [link](instrumentation.opentelemetry.io/container-names)

----
#### Demo
My partially working gist so-far is [here](https://gist.github.com/ashok-an/02b6aa8413f719c0aba1cdc472569510)
```bash
$ kubectl apply -f .
opentelemetrycollector.opentelemetry.io/sidecar-collector created
deployment.apps/flask-example created
instrumentation.opentelemetry.io/python-instrumentation created
service/flask-example created

$ kubectl get opentelemetrycollectors.opentelemetry.io,instrumentations.opentelemetry.io
NAME                                                        MODE      VERSION   AGE
opentelemetrycollector.opentelemetry.io/sidecar-collector   sidecar   0.63.1    80s

NAME                                                      AGE   ENDPOINT                SAMPLER                    SAMPLER ARG
instrumentation.opentelemetry.io/python-instrumentation   79s   http://localhost:4317   parentbased_traceidratio   0.25

$ kubectl get deployments.apps
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
flask-example   1/1     1            1           49s

$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].image}"
ashoka007/flask-example:0.1 ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.63.1%

$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].name}"
flask-example otc-container%
# otc-container is the local side-car collector
```

Here is a full-fledged working [demo](https://github.com/isItObservable/Otel-Collector-Observek8s) and a detailed blog - https://isitobservable.io/open-telemetry/how-to-observe-your-kubernetes-cluster-with-opentelemetry