Ever miss the forest for the trees?
Tags: cli, linux, tools
Published: 2019-01-18 13:19:39

---
title: Ever miss the forest for the trees?
published: true
description: Ever miss the forest for the trees?
tags: #cli, #linux, #tools
cover_image: https://blog.davidjeddy.com/wp-content/uploads/2019/01/forst_in_estonia.jpg
---

<!-- wp:paragraph -->
<p>A couple of days ago while working on an automation project I found myself logging progress to the Linux logging sub-system. This worked well enough as it gave me a central place to dump messages without having to install yet _another_ system dependency. This also provides a place that for everyone to look when debugging an issue. 'What do the logs say' is one of the first things I say to myself when something does not work as expected. But what about when the logs stop saying...anything?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So there I was Trucking along <code>in the zone</code>, everything was great. Knocking out task after task. Then my trust log reading command stopped working.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":1459,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://blog.davidjeddy.com/wp-content/uploads/2019/01/2017_01_17_1.png" alt="" class="wp-image-1459"/><figcaption>Useful little command I picked up years ago.</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>'OK' I think. No problem, so I started the troubleshooting process. Maybe the application is not installing? Nope, it is. Maybe credentials expired? Nope, they are ok. CLI authentication works. Ok, so I roll back a couple commits and <strong>boom</strong> the messages are back. Time to break this down into the atomic parts. So for the next hour I incremented commits one at a time to see where the issue first appears. (This would be a nice time to use Git Bisect, but I did not :(. )</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One commit after another, checking the process. Each time using the following to check the system logs.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":1460,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://blog.davidjeddy.com/wp-content/uploads/2019/01/2017_01_17_2.png" alt="" class="wp-image-1460"/><figcaption>I do indeed know how to exit Vim :D.</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>From two hours back all the way back to the current commit; everything worked as expected. At this point I was reaching the 2 hour 'I am stuck' limit so I reached out to a colleague and asked for a rubber duck session. They agreed and came to sit with me.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I explain What the end goal is, how I was formulating the solution, the start of the issue, what I have tried, and showed the current state. <code>Vim</code> viewing log works, but <code>tail ... | grep...</code>'ing the logs...AARRGRGG! As soon as I said the words I knew what the problem was.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Tail</strong> only prints the last <strong>10 lines to standard out</strong>. So grep only had 10 lines to work with. As my process length increased the logging messages got pushed passed the 10 line mark. To confirm this I tried:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":1461,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://blog.davidjeddy.com/wp-content/uploads/2019/01/2017_01_17_3.png" alt="" class="wp-image-1461"/><figcaption>Neat terminal screenshot tool provided by <a href="https://carbon.now.sh/">https://carbon.now.sh/</a></figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Sure enough, the messages were there. :| Just one of those thing I guess.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>How about you? Every get so deep into something you missed the obvious?</p>
<!-- /wp:paragraph -->