Jaeger Tracing Tutorial: Get Going From Scratch
Tags: tracing, programming, coding
Published: 2019-09-17 14:49:12

The Jaeger tracing system is an open-source tracing system for microservices, and it supports the <a href="https://opentracing.io">OpenTracing</a> standard. We talked about <a href="https://scalyr.com/blog/what-is-opentracing/">OpenTracing and why it's essential</a> in a previous post. So now, let's talk more about Jaeger.

Jaeger was initially published as open source by Uber Technologies and has evolved since then. The system gives you distributing tracing, root cause analysis, service dependency analysis, and more.

We're going get started with Jaeger tracing by installing it and using it to examine some RESTful API calls to a single microservice. To do this, we'll need to build a small service with tracing enabled. Jaeger has tooling for Go, Java, JavaScript (Node.js,) Python, and C++. We'll use Java for this tutorial, but the concepts we cover here will apply to any supported platform.

<img class="aligncenter wp-image-3011 size-full" src="https://library.scalyr.com/2019/02/20121919/Jaeger_tracing_image_showing_Jeager_mascot_in_scalyr_colors.png" alt="Jaeger_tracing_image_showing_Jeager_mascot_in_scalyr_colors" width="600" height="600">

<!--more-->
<h2>Installation and Setup</h2>
<h3>Docker</h3>
The preferred way to install and run Jaeger tracing is with Docker. It's also the easiest. So if you're not running Docker yet, take a look at the installation process for your platform <a href="https://docs.docker.com/install/">here</a>. The Community Edition is more than adequate for this tutorial.
<h3>Install Jaeger</h3>
Jaeger is a set of distributed components for collecting, storing, and displaying trace information. But it also ships as an "all-in-one" image that runs the entire system. We'll use that to keep the install simple for this tutorial. There are instructions for getting started <a href="https://www.jaegertracing.io/docs/1.9/getting-started/">here</a>, but I'll cover a condensed version in this post.

Docker will download the image for you when you try to start a container. I'll use a shorter command line than the one in Jaeger's instructions because we're only going to use one of the system's tracing modes.
<pre class="lang:sh decode:true">docker run -d --name jaeger -p 16686:16686 -p 6831:6831/udp jaegertracing/all-in-one:1.9</pre>
So, when you run the container, your command should look like this.

<img class="size-full wp-image-2993 aligncenter" src="https://library.scalyr.com/2019/02/20120923/012.png" alt="" width="750" height="542">

When the command finishes, check to see if the server is running with <strong>docker ps -a</strong>

<img class="size-full wp-image-2994 aligncenter" src="https://library.scalyr.com/2019/02/20120948/022.png" alt="" width="680" height="247">

You should see the container name, <strong>jaeger</strong>, with <strong>up</strong> in the status column. You'll also see a lot of information about service ports.

Now, you can connect to the Jaeger console at http://localhost:16686

<img class="size-full wp-image-2995 aligncenter" src="https://library.scalyr.com/2019/02/20121019/032.png" alt="" width="750" height="417">

We see the Jaeger user interface. It's running!
<h3>Java Microservice</h3>
We'll use a simple Spring Boot service to create some traces. The code for the project is on <a href="https://github.com/egoebelbecker/jaeger-tutorial">GitHub.</a>

The project has scripts for running the service via script or in a container. If you want to run both the service and Jaeger in containers, you'll need to know how to get them to connect over UDP using Docker networking. That's beyond the scope of this tutorial.
<h2>Jaeger and Open Tracing Concepts</h2>
Before we start our service, we can take a look at Jaeger's interface and review some basic open tracing concepts. The user interface service reports its queries so that we can see examples of a few basic traces.

<img class="size-full wp-image-2996 aligncenter" src="https://library.scalyr.com/2019/02/20121047/042.png" alt="" width="386" height="791">

Look at the box on the left-hand side of the page labeled <strong>Find Traces.</strong> The first control, a chooser, lists the services available for tracing. The count should show one. (If it doesn't, try refreshing the page.) Now, click the chooser and you'll see <strong>jaeger-query</strong> listed as the only service.

<img class="aligncenter wp-image-2997 " src="https://library.scalyr.com/2019/02/20121118/052.png" alt="" width="572" height="171">

A service is an application that's registered itself to Jaeger. We'll see how to register our application below.

Next, with <strong>jaeger-query</strong> selected, click the <strong>Find Traces</strong> button on the bottom of the form.

<img class="size-full wp-image-2998 aligncenter" src="https://library.scalyr.com/2019/02/20121203/061.png" alt="" width="750" height="532">

A list of traces will appear on the right-hand side of the screen. The traces have titles that correspond to the <strong>Operation</strong> selector on the search form. So, select <strong>/api/services</strong> in the <strong>Operation</strong> box and click the <strong>Find</strong> button again. Depending on how many times you reloaded the page, you'll see a few operations.

<img class="size-full wp-image-2999 aligncenter" src="https://library.scalyr.com/2019/02/20121245/07.png" alt="" width="750" height="416">

Now click on one of the traces.

<img class="size-full wp-image-3000 aligncenter" src="https://library.scalyr.com/2019/02/20121312/08.png" alt="" width="750" height="133">

This trace has one operation in it. It took 0.14 ms. There's not much to look at here. But we can look at what the service sent to the Jaeger Tracing server. So click on the box in the upper right-hand side of the page.

<img class=" wp-image-3001 aligncenter" src="https://library.scalyr.com/2019/02/20121354/09.png" alt="" width="278" height="221">
<h3>Jaeger Tracing Tags</h3>
Next, let's look at the JSON.
<pre class="lang:js decode:true">{
  "data": [
    {
      "traceID": "3b8496f91e044c34",
      "spans": [
        {
          "traceID": "3b8496f91e044c34",
          "spanID": "3b8496f91e044c34",
          "flags": 1,
          "operationName": "/api/traces",
          "references": [],
          "startTime": 1549827709524283,
          "duration": 142,
          "tags": [
            {
              "key": "sampler.type",
              "type": "string",
              "value": "const"
            },
            {
              "key": "sampler.param",
              "type": "bool",
              "value": true
            },
            {
              "key": "span.kind",
              "type": "string",
              "value": "server"
            },
            {
              "key": "http.method",
              "type": "string",
              "value": "GET"
            },
            {
              "key": "http.url",
              "type": "string",
              "value": "/api/traces?end=1549827709522000\u0026limit=20\u0026lookback=1h\u0026maxDuration\u0026minDuration\u0026service=jaeger-query\u0026start=1549824109522000\u0026tags=%7B%22http.status_code%22%3A%22404%22%7D"
            },
            {
              "key": "component",
              "type": "string",
              "value": "net/http"
            },
            {
              "key": "http.status_code",
              "type": "int64",
              "value": 200
            }
          ],
          "logs": [],
          "processID": "p1",
          "warnings": null
        }
      ],
      "processes": {
        "p1": {
          "serviceName": "jaeger-query",
          "tags": [
            {
              "key": "client-uuid",
              "type": "string",
              "value": "6550fb460c8ee430"
            },
            {
              "key": "hostname",
              "type": "string",
              "value": "9f77a41dfd0c"
            },
            {
              "key": "ip",
              "type": "string",
              "value": "172.17.0.2"
            },
            {
              "key": "jaeger.version",
              "type": "string",
              "value": "Go-2.15.1dev"
            }
          ]
        }
      },
      "warnings": null
    }
  ],
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": null
}</pre>
There's a lot of information here. Toward the top of the JSON, you see an array of <strong>spans</strong>. This trace only has one. A <strong>trace</strong> consists of one or more spans. A span is, as you might guess, an interval of time that contains one or more operations. We'll take a closer look at spans when we add some code to the Java service.

Inside the span, there's an array of <strong>tags.</strong> Tags are attributes an application adds to traces. Here are two:
<pre class="lang:js decode:true">{
    "key": "http.method",
    "type": "string",
    "value": "GET"
},
{
    "key": "http.status_code",
    "type": "int64",
    "value": 200
}
</pre>
We'll see how to add these tags to our spans below. For now, let's go back to the main page and use tags to search.

Now enter <strong>http.method=get</strong> in the <strong>Tags</strong> field and click the find button again.

<img class="size-full wp-image-3002 aligncenter" src="https://library.scalyr.com/2019/02/20121429/10.png" alt="" width="750" height="473">

You'll see a list of traces. Most of the operations in the Jaeger UI are GETS, which makes sense.

That's the basics of the Jaeger interface. Let's connect a service.
<h2>Tracing a Service</h2>
The Jaeger tutorial application contains a create-read-update-delete (CRUD) API for managing employee records. The records are stored in a local hashmap. We're going to add a trace with two spans to the application.
<h3>Creating a Tracer</h3>
To add tracing to an application, you need a <strong>Tracer.</strong> We'll create one and use Spring to supply it to the microservice's service and controller classes.

Here's the method for creating the tracer:
<pre class="lang:java decode:true">@Bean
public static JaegerTracer getTracer() {
    Configuration.SamplerConfiguration samplerConfig = Configuration.SamplerConfiguration.fromEnv().withType("const").withParam(1);
    Configuration.ReporterConfiguration reporterConfig = Configuration.ReporterConfiguration.fromEnv().withLogSpans(true);
    Configuration config = new Configuration("jaeger tutorial").withSampler(samplerConfig).withReporter(reporterConfig);
    return config.getTracer();
}
</pre>
The first step is constructing configuration classes. You use them to create the <strong>Tracer.</strong> Jaeger has an extensive set of tools for configuration. We're accepting the default settings and naming our tracer <strong>jaeger tutorial</strong>.

This method is in the class with the service's <strong>main</strong> method. We're treating it like a Spring <strong>Bean</strong> and injecting into the constructors of the controller and service classes. If you don't understand Spring dependency injection, you can assume that the controller and service methods have access to a tracer.

You can learn more about Jaeger configuration <a href="https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md">here</a> and <a href="https://www.jaegertracing.io/docs/1.6/client-features/">here.</a>
<h3>Tracing a REST API Call</h3>
Let's start with adding a single span to a POST method. Here's our code for adding a new employee.
<pre class="lang:java decode:true">@ApiOperation(value = "Create Employee ", response = ResponseEntity.class)
@RequestMapping(value = "/api/tutorial/1.0/employees", method = RequestMethod.POST)
public ResponseEntity createEmployee(@RequestBody Employee employee) {

    // Create a span
    Span span = tracer.buildSpan("create employee").start();
        
    HttpStatus status = HttpStatus.FORBIDDEN;

    log.info("Receive Request to add employee {}", employee);
    if (employeeService.addEmployee(employee)) {
        status = HttpStatus.CREATED;
            
        // Set http status code
        span.setTag("http.status_code", 201);
    } else {
        span.setTag("http.status_code", 403);
    }
        
    // Close the span
    span.finish();
    return new ResponseEntity(null, status);
}
</pre>
We create a <strong>Span</strong> at the start of the method, using our <strong>Tracer</strong> instance. Then we set a tag corresponding to the HTTP status code of the request. This should make out trace look a lot like the Jaeger query service. The service has a Swagger interface, so we can use it to add an employee.

<img class="size-full wp-image-3003 aligncenter" src="https://library.scalyr.com/2019/02/20121505/11.png" alt="" width="750" height="581">

Fill out details for an employee and then click the <strong>Try it out!</strong> button twice. The first request will succeed. The second will fail because the service will not accept a new employee with an existing ID.

Now, take a look at the Jaeger search page. Select <strong>jaeger tutorial</strong> in the service selector and <strong>create employee</strong> in the operation selector and click the find button.

<img class="size-full wp-image-3004 aligncenter" src="https://library.scalyr.com/2019/02/20121534/12.png" alt="" width="750" height="488">

We see two traces, but we know one failed and one succeeded. Let's refine the search. Enter <strong>http.status_code=403</strong> in the <strong>Tags</strong> text box.

<img class=" wp-image-3005 aligncenter" src="https://library.scalyr.com/2019/02/20121610/13.png" alt="" width="661" height="732">

Now, click the find button again. You'll see only one trace. Tags are useful for filtering traces and looking at specific criteria.
<h3>Multiple Spans and Log Messages</h3>
Let's finish up by adding a second span to a trace, along with log messages.

Here is the controller's delete method:
<pre class="lang:java decode:true">@ApiOperation(value = "Delete Employee ", response = ResponseEntity.class)
@RequestMapping(value = "/api/tutorial/1.0/employees/{id}", method = RequestMethod.DELETE)
public ResponseEntity deleteEmployee(@PathVariable("id") String idString) {

    Span span = tracer.buildSpan("delete employee").start();

    HttpStatus status = HttpStatus.NO_CONTENT;

    try {
        int id = Integer.parseInt(idString);
        log.info("Received Request to delete employee {}", id);
        span.log(ImmutableMap.of("event", "delete-request", "value", idString));
        if (employeeService.deleteEmployee(id, span)) {
            span.log(ImmutableMap.of("event", "delete-success", "value", idString));
            span.setTag("http.status_code", 200);
            status = HttpStatus.OK;
        } else {
            span.log(ImmutableMap.of("event", "delete-fail", "value", "does not exist"));
            span.setTag("http.status_code", 204);
        }
    } catch (NumberFormatException | NoSuchElementException nfe) {
        span.log(ImmutableMap.of("event", "delete-fail", "value", idString));
        span.setTag("http.status_code", 204);
    }

    span.finish();
    return new ResponseEntity(null, status);
 }
</pre>
Like the add method, we're opening a span at the start of the method. We're also setting the status code tag based on the result of the delete request. Also, the code has log messages based on the outcome of the query.

We're also passing our <strong>Span</strong> object to the service. Let's look at why. Here is the delete method in the service:
<pre class="lang:java decode:true">public boolean deleteEmployee(int id, Span rootSpan) {

    Span span = tracer.buildSpan("service delete employee").asChildOf(rootSpan).start();

    boolean result = false;
    if (employeeMap.containsKey(id)) {
        employeeMap.remove(id);
        result = true;
    }
    span.finish();
    return result;
}</pre>
We're creating a new span inside the method, setting it as a child of the span that was passed in.

Run the service and try to delete a valid employee and then an invalid one.

Now, select <strong>delete employee</strong> in the operation control and click the find button.

<img class=" wp-image-3006 aligncenter" src="https://library.scalyr.com/2019/02/20121645/14.png" alt="" width="497" height="386">

You should see two operations with two spans each.

<img class="size-full wp-image-3007 aligncenter" src="https://library.scalyr.com/2019/02/20121719/15.png" alt="" width="750" height="314">

Inspect each trace, and you'll see a few new things. Here's the successful trace, with both spans displayed:

<img class="size-full wp-image-3008 aligncenter" src="https://library.scalyr.com/2019/02/20121746/16.png" alt="" width="750" height="361">

You can see the second trace and the log messages. The failed delete has different log messages.

<img class="size-full wp-image-3009 aligncenter" src="https://library.scalyr.com/2019/02/20121814/17.png" alt="" width="750" height="301">

So, with a few lines of code, we can see how long operations take and get an idea of why!
<h2>Jaeger Tracing for Microservices</h2>
Jaeger tracing is an open-source implementation of the OpenTracing standard. In just a few minutes we installed the system and used it to trace a REST microservice. Tracing is an essential strategy for managing your services and monitoring your users' experience, so enjoy the fruits of this new knowledge!