How to Merge Log Files
Tags: logging
Published: 2020-03-24 14:20:15

<!-- wp:paragraph -->
<p>You have log files from two or more applications, and you need to see them together. Viewing the data together in proper sequence will make it easier to correlate events, and listing them side-by-side in windows or tabs isn’t cutting it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You need to <strong>merge log files</strong> by timestamps.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But just merging them by timestamp isn’t the only thing you need. Many log files have entries with more than one line, and not all of those lines have timestamps on them.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2203,"align":"center","width":552,"height":713} -->
<div class="wp-block-image">
<figure class="aligncenter is-resized"><img class="aligncenter  wp-image-1907" src="https://library.scalyr.com/2018/11/13003159/Merge_sign_in_Scalyr_colors_for_merge_log_files.png" alt="" width="310" height="400" /></figure>
</div>
<!-- /wp:image -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:heading -->
<h2>Merge Log Files by Timestamp</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Let’s take a look at the simple case. We have two files from Linux's syslog daemon. One is the <strong>messages</strong> file and the other is the <strong>crontab</strong> log.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here are four lines from the <strong>messages</strong> file:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Sep 4 00:00:08 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 120000ms.
Sep 4 00:02:08 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 124910ms.
Sep 4 00:04:13 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 109850ms.
Sep 4 00:06:03 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 112380ms.
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>And here are five lines from<strong> cron</strong>:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Sep 4 00:01:01 ip–10–97–55–50 CROND[18843]: (root) CMD (run-parts /etc/cron.hourly)
Sep 4 00:01:01 ip–10–97–55–50 run-parts(/etc/cron.hourly)[18843]: starting 0anacron
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Anacron started on 2018–09–04
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Jobs will be executed sequentially&lt;
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Normal exit (0 jobs run)
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>When we’re only dealing with ten lines of logs, it’s easy to see where the merge belongs. The five lines in the <strong>cron</strong> log belong between the first and second lines of the <strong>messages</strong> log.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But with a bigger dataset, we need a tool that can merge these two files on the date and the time. The good news is that Linux has a tool for this already.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Merge Log Files With Sort</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The <strong>sort</strong> command can, as its name implies, sort input. We can stream both log files into <strong>sort</strong> and give it a hint on how to sort the two logs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s give it a try.</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">cat messages.log cron.log |sort –key=1,2 &gt; merge.log
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>This creates a new file named <strong>merge.log.</strong> Here’s what it looks like:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Sep 4 00:00:08 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 120000ms.
Sep 4 00:01:01 ip–10–97–55–50 CROND[18843]: (root) CMD (run-parts /etc/cron.hourly)
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Anacron started on 2018–09–04
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Jobs will be executed sequentially&lt;
Sep 4 00:01:01 ip–10–97–55–50 anacron[18853]: Normal exit (0 jobs run)
Sep 4 00:01:01 ip–10–97–55–50 run-parts(/etc/cron.hourly)[18843]: starting 0anacron
Sep 4 00:02:08 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 124910ms.
Sep 4 00:04:13 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 109850ms.
Sep 4 00:06:03 ip–10–97–55–50 dhclient[2588]: XMT: Solicit on eth0, interval 112380ms.
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>It worked!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s dissect that command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>cat messages.log cron.log |</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Cat</strong> concatenates files. We used it to send both logs to standard output. In this case, it sent <strong>messages.log</strong> first and then <strong>cron.log.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The pipe <strong>|</strong> is what it sounds like. It’s a pipe between two programs. It sends the contents of the two files to the next part of the command. As we’ll see below, <strong>sort</strong> can accept a single filename on the command line. When we want to sort more than one file, we use a pipe to send the files on standard input.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>sort –key=2,3 &gt; merge.log</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Sort</strong> receives the contents of two files and sorts them. Its output goes to the <strong>&gt;</strong> redirect operator, which creates the new file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The most important part of this command is <strong>–key=2,3</strong>. We used this to tell <strong>sort</strong> to sort its input using two fields and three of the files. For some reason, <strong>sort</strong> starts counting fields at one instead of zero.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So <strong>sort</strong> was able to merge the two files using the day of the month and the timestamp.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is our easy case. These log files both had single line entries, and our dataset was for less than thirty days. So we don't have to worry about sorting by months.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s look at something that’s a little more complicated.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Merge Log Files With Multiline Entries</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here are a couple of Java application logs that we would like to merge.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here’s the first:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">2018-09-06 15:20:40,980 [INFO] Heimdall main:26 [main] 

Fix Engine is starting.


2018-09-06 15:20:45,639 [ERROR] AcceptorFactory createSessionSettings:92 [main] 

Session settings: [default]
SocketAcceptPort=7000
ConnectionType=acceptor
ValidateUserDefinedFields=N
ValidateLengthAndChecksum=N
ValidateFieldsOutOfOrder=N


2018-09-06 15:20:50,645 [ERROR] AcceptorFactory getSessionSettings:123 [main]

Second Session settings: [default]
SocketAcceptPort=7000
ConnectionType=acceptor
ValidateUserDefinedFields=N
ValidateLengthAndChecksum=N
ValidateFieldsOutOfOrder=N


2018-09-06 15:21:45,653 [INFO] ThreadedSocketAcceptor startSessionTimer:291 [main] SessionTimer started
2018-09-06 15:21:47,711 [INFO] NetworkingOptions logOption:119 [main] Socket option: SocketTcpNoDelay=true
2018-09-06 15:21:59,919 [INFO] SendMessageToSolace addSession:51 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02] Adding session: FIX.4.2:FOOU-&gt;TEST02
2018-09-06 15:22:59,920 [INFO] MessageClient openTopic:422 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02]
Opening FOO/DEV/AMER/FixEngine/Admin/*/TEST02
2018-09-06 15:23:59,937 [ERROR] ConsumerNodeStatusHandler setStateUp:186 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02] Setting State up: TEST02
2018-09-06 15:24:03,962 [INFO] MessageClient openTopic:422 [stateHeartbeat]

Opening FOO/DEV/AMER/State/Admin/Events
2018-09-06 15:25:00,536 [INFO] incoming messageReceived:146 [NioProcessor-2] FIX.4.2:FOOU-&gt;TEST02: 8=FIX.4.29=6235=149=TEST0256=FOOU34=252=20180906-15:21:00.528112=TEST10=198
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>This log has a lot of whitespace and entries that span multiple lines.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here’s the other:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">2018-09-06 15:20:43:031 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-10-5] Adding session: TEST02 at 1536243961031
2018-09-06 15:20:46:031 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-13-4] Adding session: TEST02 at 1536243961031
2018-09-06 15:23:15:032 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-7-5] Adding session: TEST02 at 1536243961032
2018-09-06 15:24:35:257 [INFO] com.foobar.atr.rest.controller.StatusController getSessionStatus():67 [http-nio-8010-exec-4] Received request a fix session, senderCompId:RBSG2
2018-09-06 15:27:30:691 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-10-5] Adding session: PLOP02 at 1536244050691
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>This log is more uniform, with entries that only span a single line.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>When we merge these two files, we want the multiline log message to remain together. So, <strong>sort's</strong> numeric sorting won’t work. We need a tool that's capable of associating the lines without timestamps with the last line that has one.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Unfortunately, no command line tool does this. We’re going to have to write some code.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>A Merging Algorithm</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here’s an algorithm for merging log files that have multiline entries.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, we need to preprocess the log files.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Scan the log file line by line until we reach the end.</li><li>If a line has a timestamp, save it and print the last saved line to a new file.</li><li>If a line has no timestamp, append it to the saved line, after replacing the new line with a special character</li><li>Continue with step #1.</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>We could do this in memory, but what happens when we’re dealing with huge log files? We’ll save the preprocessed log entries to disk so that this tool will work on huge log files.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After we perform this on both files, we have a new one that is full of single line entries. We’ll use the <strong>sort</strong> command to sort it for us, rather than reinventing the wheel. Then, we’ll replace the special characters with new lines, and we have a merged log file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>And we’re done!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let's do it.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Merge Log Files With Python</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We’ll use python. It’s available on all systems, and it’s easy to write a cross-platform tool that manipulates text files. I wrote the code for this article with version 2.7.14. You can find the entire script here on <a href="https://github.com/egoebelbecker/mergelogs" target="_blank" rel="noopener noreferrer">Github</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, we need to process our input files.</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">parser = argparse.ArgumentParser(description="Process input and output file names")
parser.add_argument("-f", "--files", help="list of input files", required=True, nargs='+')
parser.add_argument("-o", "--output", help="output file", required=True, type=argparse.FileType('w'))
args = parser.parse_args()

line_regex = re.compile("^[^0-90-90-90-9\-0-90-9\-0-90-9]")

with open("tmp.log", "w") as out_file:
    for filename in args.files:
        lastline = ""
        with open(filename, "r") as in_file:
            for line in in_file:
                if line_regex.search(line):
                    lastline = lastline.rstrip('\n')
                    lastline += '\1'
                    lastline += line
                else:
                    out_file.write(lastline)
                    lastline = line
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>We'll start by processing command line arguments. This script accepts two:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong>-f</strong> is a comma-separated list of input files</li><li><strong>-o</strong> is the name of the file to write the output to</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong>Argparse</strong> gives us a list from the arguments passed to <strong>-f </strong>and opens the output file for us, as we’ll see below.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Python Regular Expressions</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Then we'll create a regular expression. Let’s take a close look at it since this is what you’ll need to change if your logs are formatted differently.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here’s the whole expression:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>^[^0-90-90-90-9\-0-90-9\-0-90-9]</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The expression starts with a caret <strong>^</strong>. This means the beginning of a line.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But then we have this: <strong>[^ ]</strong> with some characters in the middle. Square brackets with a caret at the beginning mean <strong>not.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So the expression means "if this is not at the beginning of the line."</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The pattern we're matching is inside the brackets.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>0–90–90–90–9\-0–90–9\-0–90–9</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Each <strong>0–9</strong> corresponds to a numeral. Each <strong>\- </strong>is a dash.&nbsp;So it could be read like this:&nbsp;<strong>NNNN-NN-NN.</strong> It’s a pattern for the date we see at the beginning of each log entry.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So in English, the expression means “if the line does not begin with a date.”</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you need to process logs with a different format, you'll need to change this. There's a guide to python regular expressions <a href="https://docs.python.org/3.4/library/re.html" target="_blank" rel="noopener noreferrer">here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Sorting the Results</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now, we'll start the real work.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Open a temporary file.</li><li>Open the first log file.</li><li>Join lines with no timestamp to their predecessors, as described above.</li><li>Repeat this for each file passed on the command line.</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>For the third step, we'll chop the newline&nbsp;<strong>'\n'&nbsp;</strong>from the end of the last line we saved. Then we'll add an&nbsp;<strong>SOH ('\1')&nbsp;</strong>character and concatenate the lines. (I could've done this in one line, but I spelled it out to make it clear.)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We're replacing newlines&nbsp;<strong>'\n'&nbsp;</strong>with the&nbsp;<strong>SOH</strong> character instead of <strong>NULLs ('\0')&nbsp;</strong>because nulls would confuse python's string processing libraries and we'd lose data.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finally, the result of this code is a file named <strong>tmp.log</strong> that contains the log files preprocessed to be one line per entry.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s finish the job.</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">sorted_logs = check_output(["/usr/bin/sort", "--key=1,2", "tmp.log"])

os.remove("tmp.log")

lines = sorted_logs.split('\n')
for line in lines:
    newline = line.replace('\1', '\n')
    args.output.write(newline + "\n")
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p><strong>Check_output</strong> executes an external command and captures the output.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So we'll use it to run <strong>sort</strong> on our temporary file and return the results to us as a string.&nbsp;Then, we'll remove the temporary file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We wouldn’t want to capture the result in memory with a large file, but to keep this post short, I cheated. An alternative is to send the output of <strong>sort</strong> to a file with the <strong>-o</strong> option and then open that file and remove the special characters.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Next, we'll split the output on the new lines into an array. Then we'll process that array and undo the special characters. We'll write each line to the file opened for us by <strong>argparse.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We’re done!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let's run this script on two files:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>./mergelogs.py -f foo.log bar.log -o output.log</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>And we'll see this.</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">2018-09-06 15:20:40,980 [INFO] Heimdall main:26 [main] 

Fix Engine is starting.


2018-09-06 15:20:43:031 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-10-5] Adding session: TEST02 at 1536243961031
2018-09-06 15:20:45,639 [ERROR] AcceptorFactory createSessionSettings:92 [main] 

Session settings: [default]
SocketAcceptPort=7000
ConnectionType=acceptor
ValidateUserDefinedFields=N
ValidateLengthAndChecksum=N
ValidateFieldsOutOfOrder=N


2018-09-06 15:20:46:031 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-13-4] Adding session: TEST02 at 1536243961031
2018-09-06 15:20:50,645 [ERROR] AcceptorFactory getSessionSettings:123 [main]

Second Session settings: [default]
SocketAcceptPort=7000
ConnectionType=acceptor
ValidateUserDefinedFields=N
ValidateLengthAndChecksum=N
ValidateFieldsOutOfOrder=N


2018-09-06 15:21:45,653 [INFO] ThreadedSocketAcceptor startSessionTimer:291 [main] SessionTimer started
2018-09-06 15:21:47,711 [INFO] NetworkingOptions logOption:119 [main] Socket option: SocketTcpNoDelay=true
2018-09-06 15:21:59,919 [INFO] SendMessageToSolace addSession:51 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02] Adding session: FIX.4.2:FOOU-&gt;TEST02
2018-09-06 15:22:59,920 [INFO] MessageClient openTopic:422 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02]

Opening FOO/DEV/AMER/FixEngine/Admin/*/TEST02
2018-09-06 15:23:15:032 [INFO] com.foobar.atr.rest.controller.SessionStatusCache addSessionStatus():28 [lettuce-nioEventLoop-7-5] Adding session: TEST02 at 1536243961032
2018-09-06 15:23:59,937 [ERROR] ConsumerNodeStatusHandler setStateUp:186 [QF/J Session dispatcher: FIX.4.2:FOOU-&gt;TEST02] Setting State up: TEST02
2018-09-06 15:24:03,962 [INFO] MessageClient openTopic:422 [stateHeartbeat]

Opening FOO/DEV/AMER/State/Admin/Events
2018-09-06 15:24:35:257 [INFO] com.foobar.atr.rest.controller.StatusController getSessionStatus():67 [http-nio-8010-exec-4] Received request a fix session, senderCompId:RBSG2

</pre>
<!-- /wp:preformatted -->

<!-- wp:heading -->
<h2>Log Files, Merged</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In this tutorial, we covered how to merge <a href="https://www.scalyr.com/blog/common-ways-people-destroy-their-log-files/">log files</a>, looking at a straightforward case and then a more complicated situation. The code for this is available on <a href="https://github.com/egoebelbecker/mergelogs" target="_blank" rel="noopener noreferrer">Github</a>, and you're free to download and modify it for your individual needs.</p>
<!-- /wp:paragraph -->