Tracing with OpenTelemetry
Tags: opentelemetry, googlecloud, node
Published: 2022-04-01 14:42:11

<p><img src="https://inthepocket.dev/img/2022-02-04-tracing-with-opentelemetry/opentelemetry.png" alt="image" title="opentelemetry logo" />
Tracing gives you great insights into certain bottlenecks within your application. I'll go over the steps needed to enable <a href="https://opentelemetry.io/">OpenTelemetry</a> within a Fastify application on Google Cloud Run. I got inspired by the <a href="https://www.nearform.com/">Nearform</a> webinar from <a href="https://daily.dev/">Daily.Dev</a>.</p>
<p>https://www.youtube.com/watch?v=UKaJDmwIIpE</p>
<p>By default Cloud Run already has tracing enabled on requests that enter your cloud run instances. So if you go to <a href="https://console.cloud.google.com/traces">traces</a> in the console, you should see some traces for every HTTP endpoint of your application. You can see an example below:</p>
<p><img src="https://inthepocket.dev/img/2022-02-04-tracing-with-opentelemetry/cloud_run_default.png" alt="image" /></p>
<p>Now we can start adding additional instrumentation for our application. There are multiple libraries proposed by Google:</p>
<ul>
<li><a href="https://cloud.google.com/trace/docs/setup/nodejs-ot">OpenTelemetry</a>: <a href="https://cloud.google.com/trace/docs/setup">recommended by Google</a></li>
<li><a href="https://opencensus.io/language-support/">OpenCensus</a>: only alpha support and NodeJS support not even mentioned by Google</li>
<li><a href="https://cloud.google.com/trace/docs/setup/nodejs">Google Client library</a></li>
</ul>
<h2>OpenTelemetry</h2>
<p>As described on the website of <a href="https://opentelemetry.io/">OpenTelemetry</a></p>
<blockquote>
<p>OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.</p>
</blockquote>
<p>It's still in beta, but general availability should arrive soon.</p>
<h3>Installing</h3>
<p>Make sure to initialize all OpenTelemetry packages at the start of your application so that it can patch the necessary calls to underlying libraries. In this example, we add HTTP, TypeORM and Postgress instrumentation with the following instrumentation packages:</p>
<ul>
<li><a href="https://www.npmjs.com/package/@opentelemetry/instrumentation-http">@opentelemetry/instrumentation-http</a></li>
<li><a href="https://www.npmjs.com/package/opentelemetry-instrumentation-typeorm">@opentelemetry-instrumentation-typeorm</a></li>
<li><a href="https://www.npmjs.com/package/@opentelemetry/instrumentation-pg">@opentelemetry/instrumentation-pg</a></li>
</ul>
<pre class="language-ts"><code class="language-ts"><span class="token keyword">import</span> opentelemetry<span class="token punctuation">,</span> <span class="token punctuation">{</span> DiagConsoleLogger<span class="token punctuation">,</span> DiagLogLevel <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/api'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> NodeTracerProvider <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/sdk-trace-node'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> SimpleSpanProcessor <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/sdk-trace-base'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> TraceExporter <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@google-cloud/opentelemetry-cloud-trace-exporter'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> PgInstrumentation <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/instrumentation-pg'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> HttpInstrumentation <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/instrumentation-http'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> registerInstrumentations <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@opentelemetry/instrumentation'</span><span class="token punctuation">;</span><br /><span class="token keyword">import</span> <span class="token punctuation">{</span> TypeormInstrumentation <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'opentelemetry-instrumentation-typeorm'</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">export</span> <span class="token keyword">function</span> <span class="token function">tracing</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br />  <span class="token comment">// Enable OpenTelemetry exporters to export traces to Google Cloud Trace.</span><br />  <span class="token comment">// Exporters use Application Default Credentials (ADCs) to authenticate.</span><br />  <span class="token comment">// See https://developers.google.com/identity/protocols/application-default-credentials</span><br />  <span class="token comment">// for more details.</span><br />  <span class="token keyword">const</span> provider <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">NodeTracerProvider</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br />  <span class="token comment">// Initialize the exporter. When your application is running on Google Cloud,</span><br />  <span class="token comment">// you don't need to provide auth credentials or a project id.</span><br />  <span class="token keyword">const</span> exporter <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">TraceExporter</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br />  <span class="token comment">// Configure the span processor to send spans to the exporter</span><br />  provider<span class="token punctuation">.</span><span class="token function">addSpanProcessor</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">SimpleSpanProcessor</span><span class="token punctuation">(</span>exporter<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />  provider<span class="token punctuation">.</span><span class="token function">register</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />  opentelemetry<span class="token punctuation">.</span>trace<span class="token punctuation">.</span><span class="token function">setGlobalTracerProvider</span><span class="token punctuation">(</span>provider<span class="token punctuation">)</span><span class="token punctuation">;</span><br />  <span class="token function">registerInstrumentations</span><span class="token punctuation">(</span><span class="token punctuation">{</span><br />    instrumentations<span class="token operator">:</span> <span class="token punctuation">[</span><br />      <span class="token keyword">new</span> <span class="token class-name">HttpInstrumentation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br />      <span class="token keyword">new</span> <span class="token class-name">PgInstrumentation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br />      <span class="token keyword">new</span> <span class="token class-name">TypeormInstrumentation</span><span class="token punctuation">(</span><span class="token punctuation">{</span><br />        <span class="token comment">// see under for available configuration</span><br />      <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br />    <span class="token punctuation">]</span><span class="token punctuation">,</span><br />  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />  <span class="token keyword">return</span> <span class="token punctuation">{</span> provider <span class="token punctuation">}</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /></code></pre>
<p>The <a href="https://inthepocket.dev/posts/2022-02-04-tracing-with-opentelemetry/@google-cloud/opentelemetry-cloud-trace-exporter">Google Cloud trace exporter</a> automatically uses the Cloud Run service account, so make sure this service account has access to utilize the Tracing API. You can configure or verify this in the <a href="https://console.cloud.google.com/iam-admin">IAM Console</a>.</p>
<p>Once you have this added to your application and deployed on Cloud Run, you should be able to see more in-depth traces. The trace ID can always be found in the response header of your request: <strong>'X-Cloud-Trace-Context'</strong>.</p>
<p><img src="https://inthepocket.dev/img/2022-02-04-tracing-with-opentelemetry/cloud_run_enhanced.png" alt="image" /></p>
<h2>Google Coud - Daily analysis reports</h2>
<p>If you feed Google Cloud with this tracing data, you will also get more detailed insights. An example where we had a significant impact on specific API calls is also noticed automatically by Google Cloud.</p>
<p><img src="https://inthepocket.dev/img/2022-02-04-tracing-with-opentelemetry/graphql_performance.png" alt="image" /></p>
<h2>Summary</h2>
<p>Easy to implement and can give you more insights over time. The Google Cloud Trace dashboard is quite simple and easy to use, so the entry barrier is relatively low.</p>