Observability Mythbusters: How hard is it to get started with OpenTelemetry?
Tags: opentelemetry, opensource, observability, tutorial
Published: 2022-08-29 15:31:00

So you're [new to Observability](https://medium.com/@adri-v/list/unpacking-observability-be1835c6dd23) *and* [OpenTelemetry](opentelemetry.io) *and* Lightstep and you want to send traces to Lightstep to get a feel for the flow of things. But you're not sure how to get started. There's a LOT to learn and take in, and things can get pretty overwhelming, pretty fast, *amirite*? I feel ya. I've been there before. That was me last year. Solidarity!!

But, not to worry, my friend, because I am here to walk you through things. If you're looking to do a quickie setup to learn how to send OpenTelemetry data over to Lightstep, then you, my friend, have come to the right place!

The [OpenTelemetry Demo](https://github.com/open-telemetry/opentelemetry-demo) repo was created to help lower the barrier to entry into OpenTelemetry and to illustrate its awesomeness. It features a fully-functioning online store made up of multiple micro-services, written in different languages that are supported by OpenTelemetry, that can send traces to an Observability back end, like Lightstep. (OMG that was a mouthful!) Most importantly, I found it superly duperly easy to set up and get going!

In today's tutorial, I will show you how to get [Traces](https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces) produced by the [OpenTelemetry Demo](https://github.com/open-telemetry/opentelemetry-demo) into Lightstep. The example uses the [OpenTelemetry (OTel) Collector](https://opentelemetry.io/docs/collector) to ingest the Traces and send them to an Observability back-end, which in our case, will be Lightstep. 

A few important notes:

* I am assuming that you have a basic understanding of OpenTelemetry and Observability. If not, check out this great [OpenTelemetry overview](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/) and [Observability oberview](https://opentelemetry.io/docs/concepts/observability-primer/).
* I will not be covering application instrumentation with OpenTelemetry - our goal is merely to get [Trace](https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces) data into Lightstep via the OTel Collactor.

Before we start the tutorial, let's do a quick OTel Collector overview, so that you can understand *how* you're getting data into Lightstep!

### OTel Collector 101

The OpenTelemetry (OTel) Collector is a vendor-neutral service that ingests, transforms, and exports data to one or more Observability back-ends.

![OTel Collector](//images.ctfassets.net/ne67e1btg495/2GFtT5Q0LC71IBWa7YN9RV/b4326af7c0d99d0d89dcfbf5111f24f3/otel-collector.png "The OpenTelemetry Collector is made up of receivers, processtors, and exporters")

The OTel Collector consists of 3 main components:

* **[Receivers](https://opentelemetry.io/docs/collector/configuration/#receivers)** ingest data. Example data sources include: Kubernetes clusters, Prometheus, and application code.
* **[Processors](https://opentelemetry.io/docs/collector/configuration/#processors)** transform data. This can include adding/removing data, modifying data (e.g. data masking), and even filtering data.
* **[Exporters](https://opentelemetry.io/docs/collector/configuration/#exporters)** send data to an Observability back-end. In our case, Lightstep. Lightstep ingests OTel data natively, using the [OpenTelemetry Protocol (OTLP)](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md).

[Pipelines](https://opentelemetry.io/docs/collector/configuration/#service) are used to define how data flows in and out of the OTel Collector.

For more on the OTel Collector, please see the [official OTel docs site](https://opentelemetry.io/docs/collector/).


> __NOTE:__ Although your instrumented code could *technically* send data directly to an Observability back-end (e.g. Lightstep) without a Collector, it's not really the right way to go about it. You should always funnel all OTel data (regardless of the soure) to your back-end by way of the OTel Collector.

#### Running the OTel Collector

There are [several ways to deploy the OTel Collector](https://opentelemetry.io/docs/collector/getting-started); however, we're keeping things simple and local today, using [Docker Compose](https://docs.docker.com/get-started/08_using_compose), which runs [Docker](https://www.docker.com/get-started/).

#### Configuration

The OTel Collector is [configured via YAML](https://opentelemetry.io/docs/collector/configuration/).

The YAML file is used to define [receivers](https://opentelemetry.io/docs/collector/configuration/#receivers), [processors](https://opentelemetry.io/docs/collector/configuration/#processors), [exporters](https://opentelemetry.io/docs/collector/configuration/#exporters), and [pipelines](https://opentelemetry.io/docs/collector/configuration/#service).

## Tutorial

Ah, now the moment y'all have been waiting for - the tutorial! Let's do this!

For this tutorial, we will be using a [modified version of the OpenTelemetry Demo](https://github.com/lightstep/opentelemetry-demo) repo. This repo has been modified to demonstrate how to send data to Lightstep using the OTel Collector. While the [upstream repo](https://github.com/open-telemetry/opentelemetry-demo) contains configs to send data to a local installation of Jaeger, the modified version contains configs to send data to Lightstep. We will be keeping the Lightstep repo in sync with the upstream repo.

### Pre-Requisites

* [Docker](https://www.docker.com/get-started/) and [Docker Compose](https://docs.docker.com/get-started/08_using_compose)
* A [Lightstep account](https://app.lightstep.com/signup)
* A [Lightstep Access Token](https://docs.lightstep.com/docs/create-and-manage-access-tokens#create-an-access-token)

### Running the Demo App Locally

#### 1- Clone the repo

```bash
git clone https://github.com/lightstep/opentelemetry-demo.git
```

#### 2- Edit the [OTel Collector Config file](https://github.com/lightstep/opentelemetry-demo/blob/main/src/otelcollector/otelcol-config-extras.yml)

This file allows you to configure the OpenTelemetry Collector with your own Observability back-end. This case, otelcol-config-extras.yml is already configured to use Lightstep as the back-end. All you need to do is add your Lightstep Access Token to the file, to be able to send traces to Lightstep.

```bash
cd opentelemetry-demo
```

Open [`src/otelcollector/otelcol-config-extras.yml`](https://github.com/lightstep/opentelemetry-demo/blob/main/src/otelcollector/otelcol-config-extras.yml) for editing using your favourite editor. The file looks like this:

```yaml
# extra settings to be merged into OpenTelemetry Collector configuration
# do not delete this file
exporters:
  logging:
    logLevel: debug
  otlp/ls:
    endpoint: "ingest.lightstep.com:443"
    headers:
      "lightstep-access-token": <lightstep_access_token>

service:
   pipelines:
     traces:
       receivers: [otlp]
       processors: [batch]
       exporters: [logging, otlp/ls]
     metrics:
       receivers: [otlp]
       processors: [batch]
       exporters: [prometheus, logging, otlp/ls]
```

Replace `<lightstep_access_token>` with your own [Lightstep Access Token](https://lightstep.com/docs/create-and-manage-access-tokens#create-an-access-token), and save the file. The Access Token tells what [Lightstep project](https://docs.lightstep.com/docs/create-projects-for-your-environments) to send your telemetry data to.

A few noteworthy items:
* Lightstep ingests data in native [OpenTelemetry Protocol (OTLP)](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md) format, so we will use the [OTLP Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter). The exporter can be called either `otlp` or follow the naming format `otlp/<something>`. We could call it `otlp/bob` if we wanted to. We're calling our exporter `otlp/ls` to signal to us that we are using the OTLP exporter to send the data to Lightstep.
* Though not mandatory, we are also using a [Logging Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/loggingexporter). This is helpful, as it prints our traces to the Collector's `stdout`. As you might imagine, this is great for debugging, especially if you see traces sent to `stdout`, but not to your back-end.
* We must define a pipeline in the `service.pipelines` section of the YAML config. Specifically, we need to define a pipeline for our taces. The pipeline tells the Collector:
     * Where it's getting Trace data from (it's being sent via OTLP)
     * If there's any processing that needs to be done (this is optional)
     * Where to send data to. In our case, it's to `stdout` (via the [Logging Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/loggingexporter)) and to Lightstep (via [OTLP Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter))
* The Collector can ingest data using both HTTP *and* gRPC. The `receivers` configuration may *appear* to be empty; however, it actually means that we are using the defult values for the `receivers` config. It is actually the same as saying:

```yaml
receivers:
   otlp:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317
       http:
         endpoint: 0.0.0.0:4318
```

>__TIP:__ If you find that you can't see data in Lightstep, make sure that you've defined a pipeline, and that the pipeline's exporter lists your `otlp/ls` exporter. I can tell you that twice this week I forgot to add my exporter to the pipeline, and was sitting in front of my computer in sheer panic trying to figure out why I could see my traces in `stdout` but not in Lightstep. 😱

#### 3- Launch the sample app

If you're not already in the repo root, make sure you go there:

```bash
cd opentelemetry-demo
```

Run Docker compose. If you’re on an x86-64 machine (i.e. Intel or AMD-based, non-Apple Silicon), run the command below. If you’re running the Demo App for the first time, it will download all of the Docker images first. This may take several minutes.

```bash
docker compose up --no-build
```

If you’re on an Apple Silicon machine, you will need to build all images from source using the command below, as the images hosted in the GitHub Container Registry (GHCR) are built and optimized for x86-64 machines. This can take upwards of 20 minutes.

```bash
docker compose build
docker compose up
```

Once the app is up and running, you will see text continuously scrolling down the screen. *This is expected!!* The sample output looks something like this:

![otel-webstore-app-output](//images.ctfassets.net/ne67e1btg495/3XnQjkivYR2sIikMELiK2y/7dcb7a0d091a1f5117caa90559d9cb94/otel-webstore-app-output.png "OTel Demo App output - console")

>**ASIDE:** After the app started up, I kept waiting for the scrolling text to stop, only to realize that, "Oh, this thing is running. Guess I can access the URL now!" 🙄

Since we added our [Logging Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/loggingexporter) to our pipeline definition, you'll also see that zoom by past you in the terminal:

![otel-collector-stdout](//images.ctfassets.net/ne67e1btg495/WeufpJvJyGJIK60q3c81m/963972b703358b8ed489af70319c7664/otel-collector-stdout.png "OTel Demo App output - OTel Collector logging exporter")

>**NOTE:** Obviously the Logging Exporter is more useful when it's not running with a bunch of other services competing for your attention on your terminal. This will be more useful if you're running it in [Kubernetes](https://opentelemetry.io/docs/collector/getting-started/#kubernetes) or [Nomad](https://opentelemetry.io/docs/collector/getting-started/#nomad), or running standalone with [Docker](https://opentelemetry.io/docs/collector/getting-started/#docker).

You can now access the Demo app: http://localhost:8080

![Screen capture of the OTel Demo App UI](https://images.ctfassets.net/ne67e1btg495/-988184819/205a25bf691865b4bd73b9cf5a8a3a38/otel-demo-ui.png_en-US?w=3024&h=1876&q=50&fm=webp "Screen capture of OTel Demo App UI")

Be sure to play around. Browse. Add items to your cart. Remove items from your cart. Checkout.

#### 4- See the Traces in Lightstep

![Screen capture of traces in Lightstep](https://images.ctfassets.net/ne67e1btg495/5qsIxT75tcxS1urLNExPSb/e208c54dd1ab636818024d693139e5d6/ls-trace-view.png?w=3014&h=1646&q=50&fm=webp "Screen capture of Traces in Lightstep")

To view traces in your [Lightstep Observability](https://app.lightstep.com) project, check out [Lightstep Observability Notebooks](http://localhost:4000/docs/use-notebooks).

And that's it! Didn't I tell you? Pretty straightforward!

## Conclusion

Today we saw how easy it was to get Trace data into Lightstep using the [modified version of the OpenTelemetry Demo Webstore](https://github.com/lightstep/opentelemetry-demo), which uses our good friend, the OTel Collector.

Now that you've been able to get Trace data into Lightstep, I encourage you to play around with it. If you have any questions about Observability or OTel, pop into the [Lightstep Community Discord](https://ltstp.run/discord), or [get in touch by e-mail](mailto:devrel@lightstep.com).

And to reward you for your work today, please enjoy this cartoon drawn by my daughter.

![The Coffee Break by Hannah Maxwell](//images.ctfassets.net/ne67e1btg495/5rdyd3h2uXtYMd0zJRyQsq/f6cf6fe05d5919d5a7b588df18f97651/coffee-break-Hannah-maxwell.jpg "The Coffee Break by Hannah Maxwell")

Peace, love, and code. 🦄 🌈  💫 

***

The [OpenTelemetry Demo Webstore App](https://github.com/open-telemetry/opentelemetry-demo) is always looking for feedback and contributors. Please consider [joining the OTel Community](https://github.com/open-telemetry/community/blob/main/community-membership.md#member) to help make OpenTelemetry AWESOME!