How to use Structured Logs in your Go Application
Tags: go, logging, elk, microservices
Published: 2020-06-13 05:23:12

---
title: How to use Structured Logs in your Go Application
published: true
description: A tutorial on emitting structured logs in a golang application
tags: go, logging, elk, microservices
series: Integrating your Go service with ELK
cover_image: https://i1.wp.com/pmihaylov.com/wp-content/uploads/2020/06/structured-logs.png
canonical_url: https://pmihaylov.com/go-structured-logs
---
<!-- wp:paragraph -->
<p>The <a href="https://www.elastic.co/elastic-stack">Elastic stack</a> (also referred to as ELK) can bring a lot of value to your production services. But it is not that much of value if you don't use structured logs in your services.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In <a href="https://pmihaylov.com/analysing-logs-elk/">one of my latest posts</a>, I wrote about what ELK is and why you should care. I also wrote a tutorial about <a href="https://pmihaylov.com/go-service-with-elk/">how to integrate ELK with your Go app</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In this article, I will walk you through how to integrate structured logging in your Go services. We will use a sample HTTP service with a few basic endpoints and we'll use the <a href="https://github.com/uber-go/zap">zap library</a> to emit logs on error/success, which would also include some domain-specific info.</p>
<!-- /wp:paragraph -->

<!-- wp:more -->
<p><!--more--></p>
<!-- /wp:more -->

<!-- wp:heading -->
<h2>The Starting Point</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Download this tutorial’s repository:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/7b5037a45644f078c2c1f0063eabe72c.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>Currently, the repo has a <strong>main.go</strong> file which has http handlers for three different endpoints:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul>
<li>/payments/execute</li>
<li>/payments/list</li>
<li>/payments/authhold</li>
</ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>The handlers themselves don’t do anything special, they simply return a <strong>status OK</strong> when invoked.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Additionally, you have a logging middleware which you’ll be working in.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Currently, what it does is it wraps the standard <strong>http.ResponseWriter</strong> into a <strong>statusWriter</strong>, which records the returned http status code.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We will use it to determine if the http response was successful or not.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Setting up the REST Client</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In addition to the Go application, you have a file <strong>go-elk-exercise-reqs.json</strong> in the branch’s root. It is an exported set of requests you can import in a REST client of your choice.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The one I use, and recommend, is <a href="https://insomnia.rest/">Insomnia</a>. If you want to use that one, simply import the file in your insomnia workspace like this:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/lNgfShNKmOTvGmAho_eTppgHsuj-wid7CRz22moC-wo7Qlv95E5vuxze8mE8NNzsR9-yNmBHvDPtjdC8kDQ4c_3y3CVsJH2fcNSknWoA6EXC_T4gPTpNoZPzc5Q_gRDBU64raY7H" alt="How to import your requests JSON file" /></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Next, choose import data from file &amp; import the file you’re provided:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh6.googleusercontent.com/WRDiXp3LKRDVJJT-i2kzvFI5wVEuItfHIebaKNYUksv6wh8bbwpmt9Ee5l6yR2DuB8hryzKOPIQd7k_EaRNeGh2unE8-mUI0lpUjaPTWyfOwmQTyFQZLXdEJRydAclXol-mUlOSk" alt="How to import requests from a file" /></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>If everything was successful, you’ll get all necessary HTTP requests imported in you workspace with all necessary data already setup (e.g. headers).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Alternatively, if the import doesn’t work for your rest client, here is what the requests are, so that you can add them manually:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Execute Payment:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh6.googleusercontent.com/54GhOM9cr2M3iLYPTqh6e8VI4L1etzVZlvMcNt_JDlxh_y7Khzh52SY3LvekpOoZMLy3KJalaviw_jRZj17IqUPxAlpAYHByintnmi1CYmPanrkDkc8KVrXCPpj9GErpGqLNd6-r" alt="payments/execute endpoint parameters" /></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>List Payments:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/rsPa0k_dVQAKBdRGoahI5hBPTy3IhYXO7ezBfpYz7sgM7YDf84t8hdfz4nldcC6ANX32g04gO3sXlaeHSvlswEmAt6INAnk_lat1C0iT0PkZU5V6TJV9P5ZPNT0HNgrbNQAEBdne" alt="payments/list endpoint parameters" /></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Auth Hold:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/55Q3xbTUa6ofrBZ-J0aw9ZdEnj7w8ky9kdvaUe0YPfLRL70kL6cU4G5hOEGqIGbfzrGB1XALvlZGLY4tUHFRlLiERV7dHgSuwdG9Y5RgfQ0AfeIdapa6qwNUJElJqZxQqy5DaFAP" alt="payments/authhold endpoint parameters" /></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>What We'll be Doing</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In this tutorial, we'll complete the loggingMiddleware function by emitting structured logs on success (status code is less than 400) or error (status code is greater than or equal to 400).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The structured logs should contain the following keys:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul>
<li><strong>endpoint</strong> - the invoked endpoint</li>
<li><strong>method</strong> - the HTTP method used</li>
<li><strong>countryISO2, userID, paymentMethod, userType</strong> - domain-specific keys for this application</li>
</ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>In case of a success, we'll write <strong>“Inbound request succeeded”</strong> via <strong>logger.Info</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In case of a bad status code, we'll write <strong>“Inbound request failed with status [returned status code]” </strong>via <strong>logger.Error </strong>in the default<strong> error key</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is an example log line we'll emit by the end of the tutorial:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/60dccd36c6e3d64cbed7ffe7363dabc2.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>In the following section, I will walk you through how to achieve this step by step.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The Walkthrough</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In this section, we will go through the application &amp; implement structured logging step by step.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>A noteworthy thing to notice is how we are plugging in logging as a middleware. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This makes the developers writing the domain-specific logic unaware of the implementation details of how their http invocations are logged.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Additionally, it allows us to reuse the logging middleware on each endpoint we want instrumented, rather than copy-pasting the code in each handler.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>And finally, this application is kept simple on purpose. In a real application, you might want to include this middleware as a <a href="https://pmihaylov.com/shared-components-go-microservices/">Fx module</a> which you include as part of your baseline Fx modules for each of your microservices.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This gives all your microservices some baseline logging capabilities out of the box.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s get started.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Log on success/failure</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We’ll start by simply emitting log messages on each successful or failed http call.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, create an empty <strong>zap.Field</strong> array:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/283da3b2e800cb839a640f0bfc4f013a.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>We’ll use this array in the later steps to populate it with the specific keys we want to emit in our logs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Next, use <strong>logger.Info</strong> and <strong>logger.Error</strong> in the appropriate spots with their respective success/error messages. Also include all logging fields which are currently empty, but will be filled in in the next steps.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We also encapsulate the error message in a <strong>zap.Error</strong> field. This will propagate the error message in a separate <strong>error</strong> field and will also include the stack trace.</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/578ba63bf07fac432a3c824e53ed2aab.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>Test that everything works correctly by running your application using<br /><strong>go run main.go</strong> and sending an http request from your rest client.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If everything went well, here is a sample log line you should see on your screen:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/b3c57a7f50063d6e48eb243ae1fb4fcb.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>And here’s how an error log should look like: </p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/b451d38ecce864752e52ff0da8c0fcef.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>Test this by returning an <strong>http.StatusBadRequest</strong> in any of the endpoints and invoking it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>These logs will also be available in <strong>logs/go.log</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s move on with adding the rest of the keys we’ll need.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Include the endpoint and method in your logs</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To include the invoked endpoint and the HTTP method in the structured logs, you will have to take them from the <strong>*http.Request</strong> object.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Add the respective fields in the <strong>fields</strong> array we introduced previously:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/e3b13d5ca20d9b7535d3e0ce804d8d0e.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>Make a similar test as in the previous section and see that your logs now include the endpoint and http method:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/8e25b71abfec59deb96cc4ec565baf6a.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>Your error logs should also include the new keys.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Include the domain-specific fields in the logs</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This step is pretty similar to the previous one and is hence, straightforward.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here, we extract the headers from the http request and add them to our logs:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/59e9f3f75ade63f934ad354e9a2917cc.js %}</p>
<!-- /wp:html -->

<!-- wp:paragraph -->
<p>After you add this, we should be able to see the log as expected from the beginning of this exercise:</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<p>{% gist https://gist.github.com/preslavmihaylov/41c185b9bb0f48d5388f1529f48cebef.js %}</p>
<!-- /wp:html -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you got this far, well done!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You should now have a good grasp of what structured logs are and how to use them in your Go services.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you want to learn more about how to leverage it, check out some of the other articles I've linked at the beginning of the post.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Stay tuned for more upcoming Go/Microservice tutorials!</p>
<!-- /wp:paragraph -->