How to analyze OpenBSD's httpd access.log with a shell script
Tags: openbsd, httpd
Published: 2018-10-24 19:39:33

---
title: How to analyze OpenBSD's httpd access.log with a shell script
published: true
tags: OpenBSD, httpd
canonical_url: https://wildauer.io/httpd-analyze_access_log/
cover_image: https://thepracticaldev.s3.amazonaws.com/i/zuv9l8hfkb5kthuf5vml.png
---

<p>I wanted a shell script to analyze my access logs from httpd. 
I found <a href="https://sysadmins.co.za/bash-script-to-parse-and-analyze-nginx-access-logs/">this</a> from <a href="https://twitter.com/ruanbekker" target="_new">@ruanbekker</a>, 
adopted it and modified it for httpd.</p>

<p>You can see</p>
<ul>
  <li>Top requests by IP</li>
  <li>Request methods like <code class="highlighter-rouge">GET</code> or <code class="highlighter-rouge">HEAD</code></li>
  <li>Response codes like <code class="highlighter-rouge">200</code>, <code class="highlighter-rouge">301</code>, <code class="highlighter-rouge">404</code></li>
  <li>Top requested pages like <code class="highlighter-rouge">/atom.xml</code></li>
  <li>Top requested pages with a 404 response</li>
</ul>

<p>You can download the script <a href="https://wildauer.io/bin/analyze_access_log">here</a></p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">#!/bin/sh</span>

<span class="nv">LOGFILE</span><span class="o">=</span><span class="s2">"/var/www/logs/access.log"</span>
<span class="nv">RESPONSE_CODE</span><span class="o">=</span><span class="s2">"200"</span>

filters<span class="o">()</span> <span class="o">{</span>
<span class="nb">grep</span> <span class="nv">$RESPONSE_CODE</span> <span class="se">\</span>
| <span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"&lt;UNKNOWN&gt;"</span> <span class="se">\</span>
| <span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"favicon.ico"</span> <span class="se">\</span>
| <span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"logfile turned over"</span>
<span class="o">}</span>

filter_response_codes<span class="o">()</span>
<span class="o">{</span>
<span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"&lt;UNKNOWN&gt;"</span> <span class="se">\</span>
| <span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"logfile turned over"</span> <span class="se">\</span>
| <span class="nb">awk</span> <span class="s1">'{print $10}'</span>
<span class="o">}</span>

filter_404_response<span class="o">()</span> <span class="o">{</span>
<span class="nb">grep</span> <span class="s2">"404"</span>
<span class="o">}</span>

ips<span class="o">()</span> <span class="o">{</span>
<span class="nb">awk</span> <span class="s1">'{print $2}'</span>
<span class="o">}</span>

pages<span class="o">()</span> <span class="o">{</span>
<span class="nb">awk</span> <span class="s1">'{print $8}'</span>
<span class="o">}</span>

domain<span class="o">()</span> <span class="o">{</span>
<span class="nb">awk</span> <span class="s1">'{print $1}'</span>
<span class="o">}</span>

methods<span class="o">()</span> <span class="o">{</span>
<span class="nb">awk</span> <span class="s1">'{print $7}'</span> | <span class="nb">cut</span> <span class="nt">-d</span><span class="s1">'"'</span> <span class="nt">-f2</span>
<span class="o">}</span>

sort_count<span class="o">()</span> <span class="o">{</span>
<span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span>
<span class="o">}</span>

sort_desc<span class="o">()</span> <span class="o">{</span>
<span class="nb">sort</span> <span class="nt">-rn</span>
<span class="o">}</span>

top_ten<span class="o">()</span> <span class="o">{</span>
<span class="nb">head</span> <span class="nt">-10</span>
<span class="o">}</span>

sep<span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"=================================================="</span>
<span class="o">}</span>

<span class="c">##</span>
<span class="c"># Actions</span>
<span class="c">##</span>
action_request_ips<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">""</span>
	<span class="nb">echo</span> <span class="s2">"Top requests from IPs"</span>
	sep
	<span class="nb">cat</span> <span class="nv">$LOGFILE</span> <span class="se">\</span>
		| filters <span class="se">\</span>
		| ips <span class="se">\</span>
		| sort_count <span class="se">\</span>
		| sort_desc <span class="se">\</span>
		| top_ten
	<span class="nb">echo</span> <span class="s2">""</span>
<span class="o">}</span>

action_request_methods<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">""</span>
	<span class="nb">echo</span> <span class="s2">"Count requests methods"</span>
	sep
	<span class="nb">cat</span> <span class="nv">$LOGFILE</span> <span class="se">\</span>
		| filters <span class="se">\</span>
		| methods <span class="se">\</span>
		| sort_count
	<span class="nb">echo</span> <span class="s2">""</span>
<span class="o">}</span>

action_pages<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">""</span>
	<span class="nb">echo</span> <span class="s2">"Top requested pages"</span>
	sep
	<span class="nb">cat</span> <span class="nv">$LOGFILE</span> <span class="se">\</span>
		| filters <span class="se">\</span>
		| pages <span class="se">\</span>
		| sort_count <span class="se">\</span>
		| sort_desc <span class="se">\</span>
		| top_ten
	<span class="nb">echo</span> <span class="s2">""</span>
<span class="o">}</span>

action_404<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">""</span>
	<span class="nb">echo</span> <span class="s2">"Top requests 404"</span>
	sep
	<span class="nb">cat</span> <span class="nv">$LOGFILE</span> <span class="se">\</span>
		| filter_404_response <span class="se">\</span>
		| pages <span class="se">\</span>
		| sort_count <span class="se">\</span>
		| sort_desc <span class="se">\</span>
		| top_ten
	<span class="nb">echo</span> <span class="s2">""</span>
<span class="o">}</span>

action_response_codes<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">""</span>
	<span class="nb">echo</span> <span class="s2">"Response code"</span>
	sep
	<span class="nb">cat</span> <span class="nv">$LOGFILE</span> <span class="se">\</span>
		| filter_response_codes <span class="se">\</span>
		| sort_count <span class="se">\</span>
		| sort_desc
	<span class="nb">echo</span> <span class="s2">""</span>
<span class="o">}</span>

action_request_ips
action_request_methods
action_response_codes
action_pages
action_404</code></pre></figure>

<h2 id="output">Output</h2>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ analyze_access_log

Top requests from IPs
==================================================
13 1.2.3.4
 8 1.2.3.5
 8 1.2.3.6
 4 1.2.3.7
 4 1.2.3.8
 2 1.2.3.9
 2 1.2.3.10
 2 1.2.3.11
 2 1.2.3.12
 2 1.2.3.13


Count requests methods
==================================================
1146 GET
  10 HEAD


Response code
==================================================
1190 200
 792 304
 615 301
  80 404
   8 400
   5 403
   1 405
   1 206
   1 0


Top requested pages
==================================================
 694 /atom.xml
 136 /
  92 /xxx
  46 /yyy
  23 /zzz
  20 /aaa
  17 /eee/fff
  16 /humans.txt
  14 /foo/bar
  12 /something/else


Top requests 404
==================================================
  61 /robots.txt
   5 /css_.php
   2 /admin
   2 /.git/config
   1 /wp-login.php?action=register
   1 /login
   1 /user/register
   1 /index.php?option=com_user%2526task=register
   1 /etc/passwd
   1 /console
</code></pre></div></div>

<p>You can download the script <a href="https://wildauer.io/bin/analyze_access_log">here</a></p>