Open-telemetry collector: the powerful recipe for observability pipelines
Tags: opentelemetry, kubernetes, observability
Published: 2022-11-09 14:20:22

OTel collector is a component used to receive, process and export telemetry data (signals) from sources to observability backends like elasticsearch, cassandra, Datadog, NewRelic ...

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hd53qi9sjeewvwv5rt5p.png) 

#### Advantages
1. Avoids resource contentions due to scalable nature and variety of deployment modes
2. Customizable at multiple levels, no need for constant reboot/reload of the pipeline
3. Tolerant to network partitions to the most part

#### Use a collector when you
- need a common ingestion point for variety of signals like metrics and traces
- need to collect signals from multiple sources like application, infra, cluster, framework, databases
- apply transformation to the signal data before storing it in the backend
- enrich signals with additional meta-data
- filter-out signals based on various predefined criteria
- send signal data to multiple observability backends
- build a loosely coupled, scalable pipeline for signal data flow

--- 

#### Installation
1. Create a configuration file, say `config.yaml`; details [here](https://opentelemetry.io/docs/collector/configuration/) 
2. Run the collector
```bash
$ docker pull otel/opentelemetry-collector:latest
$ docker run [-d] -v $(pwd)/config.yaml:/etc/otelcol/config.yaml [port-config] otel/opentelemetry-collector:latest
```
```yaml
# port configuration
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver
      - "55679:55679" # zpages extension
```
#### Configuration
Has 3 (or 4) components all of which needs to be enabled in `service` section
Note: Atleast one `service > pipeline` is mandatory
1. **Receivers** - describe modes of how collector gets the data IN and can be PUSH or PULL based (eg: host metrics, application metrics, zipkin traces).
2. **Processors** - run on the data being transported and optionally massage, transform and filter-out data (eg: filter, batch, samplers)
3. **Exporters** - specify how data is sent out to one/more configured backends, can be PUSH or PULL based (eg: file, jaeger, prometheus). They generally involve details of authentication in production environments.
4. **Extensions(optional)** - provide additional capabilities to the collector, but not requiring direct access to signal data (eg: health_check, pprof)

More info: [configuration wiki](https://opentelemetry.io/docs/collector/configuration/)

##### Deployment modes
1. Agent: A collector instance running on the same node as the application (binary, side-car or daemonset)
2. Gateway: One or more instances collectively running centrally as a standalone service. It can often offer advanced capabilities like simple load-balancing, tail based sampling, independent scaling .. generally acting as a receiver for the agents.

---

#### Demo
[Gist to working demo](https://gist.github.com/ashok-an/39c155a15fc6b6a5eb1a742405efc85f)
Note: metrics exporter does not seem to work based on official documentation

#### OTel Deployment patterns 
*Source of all the below information* [CNCF presentation](https://youtu.be/WhRrwSHDBFs)

**Basic - instrument and send to a collector**
Used when application is instrumented with OTel SDK and signals are sent to a predefined collector

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxgi9qaeqd0zonxca3pg.png)

**Basic - fanout**
Used when signals are (processed and) sent to multiple destinations. Useful in situations where multiple views/perspectives of same data is to be generated (eg: one from Jaeger one from Datadog)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tc4hr0qg102eusl9h8v2.png)

**Normalizer**
Collector works as an intermediate proxy and massages the data before passing on to destination; used when common processors are to be applied on the incoming signals

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5rho4wq2ko2nqwddu1of.png)

**Kubernetes sidecar**
Workloads send signals to a OTel collector sidecar which is sent over to a collector residing in a central namespace (which processes and sends it to destination). Advantages of this pattern is decoupled central collector, easily customizable side-car and implicit load balancing.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cgys8ghudxr1hmxz2vyp.png)

**Kubernetes daemonset**
Collector is deployed as a daemonset; while it eases management, multi-tenancy and scaling requirements are hard to customize.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g4okcps4bwki64nfqdjc.png)

**Loadbalanced collector**
A central load-balancing collector is used to aggregate all signals from a given source to a given backend collector (like how session affinity is handled). The idea behind the implementation is that any given collector should provide full picture of the source application independently.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uv7m2sliv8n66k6lxt5g.png)

**Multicluster**
A common Otel collector is deployed on a central cluster which acts as the final stop before writing to destinations. It is useful in regulatory scenarios where common point of control need be established

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7bgev5fldkujjbn7mzv.png)

**Multi-tenant**
Multiple destinations generally are involved and Otel collector processes and sends to multiple destinations based on filtering tags

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2nhpzandxnxlbrye6zpj.png)

**Per signal**
An otel collector per signal type (eg: one for metrics, one for traces ..). Useful to establish saperate observability pipelines per signal. Note: A PUSH based collector can be scaled easily while PULL based (prometheus) is not straight-forward given the idempotency semantics.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j8kaa8cszh8d6hksw3e9.png)