Using Kibana to Debug Production Issues
Tags: logging, go, microservices, kibana
Published: 2020-08-16 16:52:01

---
title: Using Kibana to Debug Production Issues
published: true
description: Learn how to use Kibana for debugging issues on production
tags: logging,go,microservices,kibana
series: Integrating your Go service with ELK
cover_image: https://i2.wp.com/pmihaylov.com/wp-content/uploads/2020/08/pexels-lukas-669610-scaled.jpg?w=1352
canonical_url: https://pmihaylov.com/kibana-debugging-tutorial/
---
<!-- wp:paragraph -->
<p>In the <a href="https://pmihaylov.com/kibana-dashboard-tutorial/">last chapter</a> of these series, we covered how to create your first Kibana dashboard for a sample web application.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Although that will help you setup Kibana for your production services, what you'll be doing 99% of the time would be to actually use the dashboard to monitor &amp; debug issues you observe on production.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is the final post in the series for monitoring Go applications \w Kibana. It will give you some practice using your newly created dashboard to identify problems with a sample web application. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So if you've setup Kibana for the first time or you've just joined your new team &amp; seen your Kibana dashboards without a clue what's going on, this post is for you!</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The Starting Point</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>First, make sure you have <strong>docker</strong> and <strong>docker-compose</strong> installed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For <strong>docker</strong>, follow the installation instructions for your platform <a href="https://docs.docker.com/get-docker/">here</a>.<br>For <strong>docker-compose</strong>, follow the installation instructions <a href="https://docs.docker.com/compose/install/">here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Afterwards, download this exercise’s repository on the <strong>kibana-debugging-tutorial</strong> branch:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>git clone --branch kibana-debugging-tutorial https://github.com/preslavmihaylov/tutorials</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Boot up all docker containers:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>docker-compose up</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Run this script to import the dashboard to Kibana from the project’s root:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>./import-kibana-dashboard.sh</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This is the exact same dashboard as the one we create in <a href="https://pmihaylov.com/kibana-dashboard-tutorial/">this post</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now, you should be good to go.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Verify everything is working properly by going to <a href="http://localhost:5601/app/kibana#">http://localhost:5601/app/kibana#</a>, opening the <strong>Dashboard</strong> tab and making sure you see some incoming traffic with occasional errors:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/PPou4oZIokrgp2SJ5fIefQ73ZkjWlACSK-WUjSPeNtbxeDjlMc3ghYLPUWCfKsCEogkmLGoIiU2ZIhU5BH4iYuwEjqMXQX0hYJ10A1aW9BV1ekx_dr5YcU868f8xwc4gb9Nm1KuM" alt="starting dashboard"/></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Your Goal</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In this tutorial, you have a black box application, which has the following HTTP API:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>/payments/execute
/payments/list
/payments/authhold</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>In addition to these endpoints, what you'll see in the dashboard are some additional properties:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>method - the HTTP method invoked
countryISO2 - the ISO2 of a user's country (e.g. US, FR, UK)
userID - the ID of the user making the request
paymentMethod - the payment method used for making a transaction
userType - the type of user (e.g. trial, individual, business)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Inspect the panels in the dashboard to see some sample data for these properties.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This API is the same as the one seen in the previous tutorials from the series. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The difference is that this time, the application fails under certain hardcoded conditions.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Example:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The application fails when <strong>the country is “US”</strong> and<br><strong>the endpoint is “/payments/list”</strong> and <strong>the method is “PUT”</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is just an example, of course, it doesn’t necessarily mean that the application returns an error under these exact conditions (or does it?).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, the failure conditions will be of that kind.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Your task is to use the Kibana dashboard you are given, apply some filters to narrow down the errors and figure out what are the exact conditions which cause an error.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are <strong>5 error conditions</strong> which you have to find out.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here are the hints:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>The application fails when ________ and ________</li><li>The application fails when ________ and ________</li><li>The application fails when ________</li><li>The application fails when ________ and ________ and ________ and ________</li><li>The application fails when ________ and ________ and ________ and ________</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>The hints are ordered by the difficulty of finding the given error condition - The first condition is the easiest to find.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One caveat to have in mind is that <strong>you won’t be 100% sure that a given error condition you think you’ve found is correct</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Instead, focus on finding <strong>the first two error conditions first</strong>, write them down and then consult the walkthrough for the first two issues to verify if your assumptions are correct.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Then, continue with the third and final two error conditions and check the walkthrough for each of those steps again.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>The Walkthrough</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before you go through this, I urge you to attempt the exercise on your own.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Use this only if you get stuck or if you want to check out some alternative ways to debug the issues.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Well, without further ado, let’s get started.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Finding Issue #1</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Issue #1 is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when ________ and ________</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>First, go to your Errors table and filter the dashboard to only show the errors:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/atRBgwp15fQJFsJizAX9ufhC3P7YB-tWpvnBF8qDTz9oTaxkrRyhcMRk2WCHUmk4UKd5Q_6B8DFv0OQ2hH00gA2J1mzQlkU3y1SBbR8zHjaxzXz8M7cFaQ_q31PM7HlM-vJ7V2XY" alt="issue #1"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This will only filter the errors in the view, which will hide the irrelevant details we aren’t interested in.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><br>After we do this, this is an example of what your dashboard should look like:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh6.googleusercontent.com/bZxKYPxAfOx48ZdJMdCwK311zTFcx9_Ul6CquF-EF3Q4r0vSS9Nj_9DYgjwKmxogkWd0vsCkICDfHORTykgSx564cJYPfiru4UCgf8Q7XRCetj5mmFek9s-yHGGxLXeoZ9E6Qt6W" alt="issue #1 errors"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Notice the applied filter at the top-left. That’s the bar which shows all filters applied to the current dashboard. This is one of the main mechanisms you’ll use when debugging production issues.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Next, notice how the greatest amount of errors appears to happen on the <strong>/payments/authhold</strong> endpoint.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Apply that to the current filter as well:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/mAjF9tPWsd7-HnfoLB-n68fYiExfWpuWD5SYCy4SdQIyJJAVd08mgOUU3-isZ1hqW_K0nfeys60ILPyB040jo8s632wbu0LHml9XEiSKyW1B4dcq3iHUCeeu--rmcu-ovgm6WBMC" alt="filtering payments/authhold"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>After we apply this filter, take a look at your tables:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/CDsukU-GuLg7NvP9I7hEixqNtiBUqpMPIJm43c7orgy8oSL2W86-js3XhvbdJl0-hCfz4zRwq-OJa0qK37KqbTN9UGyGM0RXeoGrdiG9TOx8KV9k4ENRvscXMK0H0aNsTcbHiR_C" alt="filtered /payments/authhold dashboard"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>It seems that the errors per value are evenly distributed across all tables <strong>except for the HTTP Method</strong>. There is an abnormal amount of errors which occur when the <strong>HTTP method is POST</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Apply a filter on the POST method and let’s see the results:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/VApNgjSnc5cDd-IyThmmt1OaEx3_a0glPxLJxtJcSv0TghKsJp5TfAO_aw7YSXlTOP0x7uunLXsOSgt1VaI-WVfluvZjDpNulyU7_yXC7onuFbvVJV5CSqJm2OIwzSsM-zvtSCiT" alt="filtering POST method"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Notice that the rest of the errors per value remain evenly distributed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This means that our first error condition is:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>The application fails when <strong>the endpoint is /payments/authhold</strong> and <strong>the HTTP method is POST</strong></li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>That’s a good start.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One important thing to do before you move on, however, is to exclude the current error conditions from the rest.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will allow you to focus on the errors that are left, rather than rediscover the errors which you’ve already found.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It is enough to exclude one of the error conditions (as both of them have to match).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To do this, clear all current filters except for the error filter:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/XvzJrpRmcvc-m-FqjAlTPW2aRxouVXISi7HQKYjzqtUEyZq4tF-gDxueF1Cj2tGyB8C4pQNcvvsTbv2ROGEtpqxIFUoJRGfiew7CuYo1Yev7PUg6hGIV4W1lqb7Vg6k5EL79cLZC" alt="filters for issue #1"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Next, exclude the endpoint <strong>/payments/authhold</strong>:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/0ZWPD6iUm85kq2GQM23L5zAu5a3Tte21MdR6z-OB3Mgt3ZkODGb7pkPooz2e_sAVcukmUkhVY7NVOC6xd-d4DmJndO0C4bWdqzi-t17EpsQqOCJU88OmY7BtWs9svQ6Zhxv-09Ba" alt="exclude payments/authhold"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This is how your filters bar should look like:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/YWKK0fkw85iimKcl6fm21FwqAYVsMZldcUEsD4NN1PBLPdEk_F6sdZLLZ4i4X2plUnShWZE8rsAB9COJLuQMVe_cDPPQFCzRnC-u1BXWYdwVldqJD6my0QHLUKm3mI2GI2zlmNsQ" alt="filters after excluding payments/authhold"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Of course, it is possible (and is the case) that there are other error conditions that aren’t the one we found for the endpoint we excluded.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, we will have to do some heuristics like this to narrow down the rest of the errors until there is no more room to “zoom in on”.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>At that point, we will have to relax our filters a bit until something we are yet to discover pops up.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Typically, when working in a production environment, you would approach this kind of issue by finding a few problems at a time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Only after you’ve fixed the most outstanding issues, will you narrow down on the rest of the issues in your application which aren’t so pervasive.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, in this context, it would be quite shallow to only get a single error condition to discover. This is why you’ll have to discover all 5 at once.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But don’t worry, I’ve made sure the error conditions are quite distinct to one another in order to enable you to more easily discover them.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s move on to the next issue.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Finding Issue #2</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Issue #2 is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when ________ and ________</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>To find this one, we’ll take a very similar approach to the previous one. Take a good look at the errors distribution at a high level.</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/0dhfoLDruNog0g_5akQuc52oLVfvdsQcLFXc5JxWibA9qombuC2U2kn1HM7DspbOEk6BZ5eVl9pxOhRaZrLgEmR-zstC6-q2UemGVOAZmVDVkETJxUEKyRZjkDnRhW_bc1K6EcrA" alt="issue #2 dashboard"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Notice the outstanding amount of errors for the <strong>/payments/list</strong> endpoint.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Filter by that value and take a look at the dashboard again:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/PpaqEmSDKtvfi0ARkNDVNnHeriDfBh2SjaIWYx-O1naxAInK8c4ZkFqRPJKSCIt8F5kpJgMhKPhY4HVjepoh3pEAiHBwRqmeksNF0x4cYmekIfC49ewSRZDO3Ivynl67nApIROYX" alt="filtering /payments/list"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As previously, there is one outstanding value which catches the eye. it is the <strong>ES country</strong>.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Filter by that value now:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/9jJoyDRcEzW63LbRa1PkkB-KeBw5OuusHcFjUAOc0izBE4l_AFyeIhpagNvtYkMlugFfVUHaBhyFOhIAImOtnzmUvwpX3iJF3DBB_WtY6lLJ7zB_wOOfVJLaDhAnvRV9uA4S0qBV" alt="filtering ES country"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As you see, it looks like all other values are evenly distributed when we filter by endpoint <strong>/payments/list</strong> and <strong>country ES</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This means, that the second error condition our application has is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when the endpoint is <strong>/payments/list</strong> and <strong>the country is ES</strong></li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Nicely done. Now remove the filters we added in this step and exclude one of the error conditions we just discovered.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In this case, I would choose to exclude the <strong>ES country</strong> as excluding another endpoint will make the resulting view way too narrow.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is what your filters bar should look like:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/eTDWVeoWylnGpKxLH8PtTJz8ecL1xRBmeO4RT--u6Co8l7CFggJBmXQFuYJqSv3rcSqFotz8OvRvB5fp2bgQ458k_Qv5tnIBNZURPU2wL0GIqTsdvKuPAohKiUjS7Zb1g9yNlqov" alt="filters after issue #2"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Let’s move on.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Finding Issue #3</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Issue #3 is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when ________</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Taking a look at the view we have now, there is nothing that immediately pops out:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/c2ny4kRecQA2YONgihQTcxfmYo-OmhLBZfBZi8IdvHTOeEoSTLm9oZ6mU2ma903c0P0edZs_zcPACl-ABrPTjsyCNlLp7dfSf9pe0NXstgfOd3m5HrAdjyTL5AlUO01s07IeVTRc" alt="issue #3 dashboard"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>So it's hard to narrow down in any direction from here.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, there is one view on our dashboard which we’ve neglected thus far.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It is the detailed logs view at the bottom of the dashboard.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It will be the one which will guide us in the right direction:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/RMBpk7Ib0HtRfn_oDGeVrTo3dtPAOMngdRM4voH77X_kbVlmjQwU9o68IQinWcPV0ZduIm6IF61z_TZt9kfeEzbot7yWvGZsCVijg5cAexgDvWkkxCWXMA3vP44UePWugjCmQoG7" alt="issue #3 user panel"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Do you notice something abnormal?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The amount of errors for <strong>user ID 68</strong> tends to be quite high in comparison with other user IDs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Perhaps there is something wrong with that user. Apply a filter on his user ID and see what the result is:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh6.googleusercontent.com/3hes2nM6Slot0Gm5ibq27hASfDd7gmXD26XuWSMdg4S_IRPopXDZyO0eh-r0flOVlCTFnJNsaKhliZcLlqWf-r9C0XO_aivA-RzbVuB7KqittyqEuVZ-LPM_YGuwm9shKLVyEyyQ" alt="problems for user 68"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Notice that after we filter user ID 68, there is nothing extraordinary which pops out. Perhaps the application unconditionally fails for this user every time?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To test this hypothesis remove all filters except for the user ID. This is what the filters bar should look like:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/qzjKpTl0NobE9UyPNVJrK5j2UbRYIiz2GHW3vnvsEzGC2Oj2cgv3GtL9OBPtxX2b2DFG7odYN8ystFzFHst1fmGmIT-sCbKno2GXemK_M7PwN0Jub5dGm0SqaUCX04TzOwZ8Q4d_" alt="filtering by user 68"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>After we do this, notice that <strong>there isn’t a single successful request for this user</strong>:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/pxxW7UhEjxlq7NrB_13Wit1AOA5X_E_ifNFnfNW07xJNlgUM2l00n9KRc82J_l0opq9Bl3IWBWI5WRGMrh3LLEgu9SKUwxYcJHc42nw8O6Vo10qfl6Zbrc3BA5KLo9Etg6JMvaX7" alt="errors for user 68"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This is our third error condition:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when <strong>the user ID is 68</strong></li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Now that we’ve discovered this issue, there are still two more to go.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finding them now will be significantly easier than at the beginning because we’re going to filter all other error conditions so that only those remaining will pop up.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To do this, remove all filters and exclude the erroneous values from the previous issues along with the user ID we just found:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/1_J6UyUgmYQVkvQq9cg9kZcgVHfTQ2XsNfFK5DJTtPiFDsxhlztOJmh-ws6KvP6buuWYtueBiePwCtyvSLFW2517YRXVnjBhu9wOtlPGlgtouZxJs_Yhv6UNFs_unnPYjAnd5Jca" alt="exludes after issue #3"/></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>Finding Issues #4 and #5</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Issues #4 and #5 are:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when ________ and ________ and ________ and ________</li><li>The application fails when ________ and ________ and ________ and ________</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>After applying all filters for the rest of the errors, you should have a very small number of errors remaining.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is normal as these two errors are the hardest to hit due to the many input values which cause them.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finding them first at the beginning is extremely difficult because of that.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, now that we’ve narrowed down our errors so much, seeing the root cause for these issues is a lot easier.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Take a look at the dashboard:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh3.googleusercontent.com/A546YpG_0QMh94g3VyS11w4n0-tCDyVHHR95YAFHCOrJtC9H5qWWSs9RePmI4vlKjrmRwrLGYR3vpAkvz9uV624V2VoY19fjKuTwNosyzqR9tymUQ7BaI_SYPyzaC-K7TBx-P_Af" alt="issue #4 and #5 dashboard"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>It seems that all errors are caused by the <strong>/payments/execute</strong> endpoint. This first part of the error condition is obvious and shared by both issues.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The rest of the values are not so obvious but they are all present in the current view.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We simply have to find out what combination of each value will match issue #4 and, respectively, issue #5.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>At this point, we can simply guess until the solution pops out.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s start by filtering the <strong>GET method</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The solution to any of the issues isn’t obvious yet:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/R1to2riWCuRBT4fkb8blD7HeASvqFYztkuhOv2KCBfq-MqTVHRqoqgiYqvUeT35NqcL7WUFdnWFBXIjx5mAVEMUU4cTZcwjp8jQN_95qTcMArKS-AxreAIkQE85OsDb9NM9hqAfr" alt="filtering GET method"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><br>Let’s continue by filtering the <strong>AMEX</strong> <strong>payment method</strong>:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/BJWMzvbXhOrT3b5yVi5mcsUKONRWNMj49Q92joIKgpplp9quWPRiGSQx_WvkTidzcDejFEos28TnQLnjkjqVzxwS18Y_gf2b1BiJyXwxtC1X4b-uqQGLRgxXkZGjHAZ2SdGdN8vI" alt="filtering AMEX"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>It appears we’ve found one of our issues.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, there is one problem - it currently appears that the error condition consists of <strong>five different inputs</strong>, while both issues consist of <strong>four inputs</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now, we have to find the odd one out.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We can do this by removing the filter on any of the parameters on-screen.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After several iterations of trial-and-error, you will figure out that when we remove the filter on the <strong>GET method</strong>, it can be seen that the error happens on other methods as well:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh6.googleusercontent.com/FmcafsycUG5YUe7rCC3ZXh9ui-EoIsDNCsz8ji_olBNz_OyZJxTs1JN-DPhjn-M1cP5U9cLNCJp1xLWN6JGRW4iKbHJJdUs3mxcJWhLwxYDEU00xowEjH7jsxkEfU4iYIkucr3SH" alt="result after filtering AMEX"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>This leads us to the conclusion that issue #4 is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when <strong>the endpoint is /payments/execute</strong> and <strong>the payment method is AMEX</strong> and <strong>the country is AU</strong> and <strong>the user type is corporate</strong></li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Now, exclude one of those parameters, e.g. <strong>the AMEX payment method </strong>and the final issue is evident:</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://lh5.googleusercontent.com/E10fn0P7aF5-KRXqLphkk4Srw4CWLIt92aNebXaxsBmdXNUpBiNFpyXutyJbGubZnEqjZ_PYwKJcPNsNifjC3WJgYbsa2IVr7gdzrjytNURqiN8feSUIuqWnON_1ibqapVJRcklh" alt="issue #5 root cause"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Issue #5 is:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The application fails when <strong>the endpoint is /payments/execute</strong> and <strong>the payment method is Cash</strong> and <strong>the country is BG</strong> and <strong>the user type is trial</strong></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Congratulations. 👏👏👏</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You’ve successfully completed the tutorial. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You should now have a good practice using Kibana to analyze real issues with your services.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This concludes the series on <strong>Integrating your Go service with the ELK Stack</strong>. You should now have a decent understanding of:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>How to install &amp; setup ELK</li><li>How to use structured logging in your services</li><li>How to create Kibana dashboards</li><li>How to analyse production issues in the Kibana dashboard</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Well Done &amp; happy logging. 👌</p>
<!-- /wp:paragraph -->