A good old-fashioned Perl log analyzer
Tags: perl, awk, webdev, apache
Published: 2021-09-14 14:44:00

---
title: A good old-fashioned Perl log analyzer
published: true
date: 2021-09-14 14:00:00 UTC
tags: Perl,AWK,webdev,Apache
canonical_url: https://phoenixtrap.com/2021/09/14/a-good-old-fashioned-perl-log-analyzer/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ykjq9w4b4pcis56ru8pq.jpeg
---

A [recent Lobsters post](https://lobste.rs/s/ztofac/awk_power_promise_40_year_old_language) lauding the virtues of [AWK](https://en.wikipedia.org/wiki/AWK) reminded me that although the language is powerful and lightning-fast, I usually find myself exceeding its capabilities and reaching for Perl instead. One such application is analyzing voluminous log files such as the ones generated by [this blog](https://phoenixtrap.com). Yes, [WordPress has stats](https://wordpress.com/support/stats/), but I’ve never let reinvention of the wheel get in the way of a good programming exercise.

So I whipped this script up on Sunday night while watching _[RuPaul’s Drag Race](https://www.vh1.com/shows/rupauls-drag-race)_ reruns. It parses my [Apache web server log files](https://httpd.apache.org/docs/2.4/logs.html) and reports on hits from week to week.

```perl
#!/usr/bin/env perl

use strict;
use warnings;
use Syntax::Construct 'operator-double-diamond';
use Regexp::Log::Common;
use DateTime::Format::HTTP;
use List::Util 1.33 'any';
use Number::Format 'format_number';

my $parser = Regexp::Log::Common->new(
    format  => ':extended',
    capture => [qw<req ts status>],
);
my @fields      = $parser->capture;
my $compiled_re = $parser->regexp;

my @skip_uri_patterns = qw<
  ^/+robots\.txt
  [-\w]*sitemap[-\w]*\.xml
  ^/+wp-
  /feed/?$
  ^/+\?rest_route=
>;

my ( %count, %week_of );
while (<<>>) {
    my %log;
    @log{@fields} = /$compiled_re/;

    # only interested in successful or cached requests
    next unless $log{status} =~ /^2/ or $log{status} == 304;

    my ( $method, $uri, $protocol ) = split ' ', $log{req};
    next unless $method eq 'GET';
    next if any { $uri =~ $_ } @skip_uri_patterns;

    my $dt  = DateTime::Format::HTTP->parse_datetime( $log{ts} );
    my $key = sprintf '%u-%02u', $dt->week;
    $week_of{$key} ||= $dt->date;    # get first date of each week
    $count{$key}++;
}

printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
  for sort keys %count;
```

Here’s some sample output:

```
Week of 2021-07-31: 2,672
Week of 2021-08-02: 16,222
Week of 2021-08-09: 12,609
Week of 2021-08-16: 17,714
Week of 2021-08-23: 14,462
Week of 2021-08-30: 11,758
Week of 2021-09-06: 14,811
Week of 2021-09-13: 407
```

I first started prototyping this on the command line as if it were an `awk` one-liner by using the [`perl -n`](https://perldoc.pl/perlrun#-n) and [`-a`](https://perldoc.pl/perlrun#-a) flags. The former wraps code in a [`while`](https://perldoc.perl.org/perlsyn#Compound-Statements) loop over the [`<>` “diamond operator”](https://perldoc.perl.org/perlop#I/O-Operators), processing each line from standard input or files passed as arguments. The latter splits the fields of the line into an array named [`@F`](https://perldoc.perl.org/perlvar#@F). It looked something like this while I was listing URIs (locations on the website):

```shell
gunzip -c ~/logs/phoenixtrap.com-ssl_log-*.gz | 
perl -anE 'say $F[6]' 
```

But once I realized I’d need to filter out a bunch of URI patterns and do some aggregation by date, I turned it into a script and turned to [CPAN](https://www.cpan.org/).

There I found [Regexp::Log::Common](https://metacpan.org/pod/Regexp::Log::Common) and [DateTime::Format::HTTP](https://metacpan.org/pod/DateTime::Format::HTTP), which let me pull apart the Apache log format and its timestamp strings without having to write even more complicated regular expressions myself. (As noted above, this was already a wheel-reinvention exercise; no need to compound that further.)

Regexp::Log::Common builds a compiled regular expression based on the log format and fields you’re interested in, so that’s the constructor on lines 11 through 14. The expression then returns those fields as a list, which I’m assigning to a hash slice with those field names as keys in line 29. I then skip over requests that aren’t successful or browser cache hits, skip over requests that don’t `GET` web pages or other assets (e.g., `POST`s to forms or updating other resources), and skip over the URI patterns mentioned earlier.

(Those patterns are worth a mention: they include the [`robots.txt`](https://www.robotstxt.org/) and [sitemap XML](https://www.sitemaps.org/) files used by search engine indexers, WordPress administration pages, files used by [RSS](https://en.wikipedia.org/wiki/RSS) newsreaders subscribed to my blog, and routes used by the [Jetpack](https://jetpack.com/) WordPress add-on. If you’re adapting this for your site you might need to customize this list based on what software you use to run it.)

Lines 38 and 39 parse the timestamp from the log into a [DateTime](https://metacpan.org/pod/DateTime) object using [DateTime::Format::HTTP](https://metacpan.org/pod/DateTime::Format::HTTP) and then build the key used to store the per-week hit count. The last lines of the loop then grab the first date of each new week (assuming the log is in chronological order) and increment the count. Once finished, lines 46 and 47 provide a report sorted by week, displaying it as a friendly “Week of _date_” and the hit counts aligned to the right with [`sprintf`](https://perldoc.perl.org/functions/sprintf). [Number::Format’s `format_number`](https://metacpan.org/pod/Number::Format#format_number(%24number,-%24precision,-%24trailing_zeroes)) function displays the totals with thousands separators.

**Update:** After this was ini­tial­ly pub­lished. astute read­er Chris McGowan not­ed that I had a bug where `$log{status}` was assigned the val­ue 304 with the `=` oper­a­tor rather than com­pared with `==`. He also sug­gest­ed I use the [double-​diamond `<<>>` oper­a­tor](https://perldoc.perl.org/perl5220delta#New-double-diamond-operator) intro­duced in Perl v5.22.0 to avoid maliciously-​named files. Thanks, Chris!

## Room for improvement

DateTime is a very powerful module but this comes at a price of speed and memory. Something simpler like [Date::WeekNumber](https://metacpan.org/pod/Date::WeekNumber) should yield performance improvements, especially as my logs grow (here’s hoping). It requires a bit more manual massaging of the log dates to convert them into something the module can use, though:

```perl
#!/usr/bin/env perl

use strict;
use warnings;
use Syntax::Construct qw<
  operator-double-diamond
  regex-named-capture-group
>;
use Regexp::Log::Common;
use Date::WeekNumber 'iso_week_number';
use List::Util 1.33 'any';
use Number::Format 'format_number';

my $parser = Regexp::Log::Common->new(
    format  => ':extended',
    capture => [qw<req ts status>],
);
my @fields      = $parser->capture;
my $compiled_re = $parser->regexp;

my @skip_uri_patterns = qw<
  ^/+robots\.txt
  [-\w]*sitemap[-\w]*\.xml
  ^/+wp-
  /feed/?$
  ^/+\?rest_route=
>;

my %month = (
    Jan => '01', Feb => '02', Mar => '03', Apr => '04',
    May => '05', Jun => '06', Jul => '07', Aug => '08',
    Sep => '09', Oct => '10', Nov => '11', Dec => '12',
);

my ( %count, %week_of );
while (<<>>) {
    my %log;
    @log{@fields} = /$compiled_re/;

    # only interested in successful or cached requests
    next unless $log{status} =~ /^2/ or $log{status} == 304;

    my ( $method, $uri, $protocol ) = split ' ', $log{req};
    next unless $method eq 'GET';
    next if any { $uri =~ $_ } @skip_uri_patterns;

    # convert log timestamp to YYYY-MM-DD for Date::WeekNumber
    $log{ts} =~ m!^ (?<day>\d\d) / (?<month>...) / (?<year>\d{4}) : !x;
    my $date = "$+{year}-$month{ $+{month} }-$+{day}";

    my $week = iso_week_number($date);
    $week_of{$week} ||= $date;
    $count{$week}++;
}

printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
  for sort keys %count;
```

It looks almost the same as the first ver­sion, with the addi­tion of a hash to con­vert month names to num­bers and the actu­al con­ver­sion (using named [regular expression capture groups](https://perldoc.perl.org/perlre#Capture-groups) for read­abil­i­ty, using [Syntax::Construct to check for that feature](https://metacpan.org/pod/Syntax::Construct#?%3C%3E)). On my serv­er, this results in a ten- to eleven-​second sav­ings when pro­cess­ing two months of com­pressed logs.

What’s next? Pretty graphs? Drilling down to specific blog posts? Database storage for further queries and analysis? Perl and CPAN make it possible to go far beyond what you can do with AWK. What would you add or change? Let me know in the comments.