Super flexible infrastructure audit with Outthentic
Tags: perl, linux, testing, devops
Published: 2018-10-04 14:37:09

---
title: Super flexible infrastructure audit with Outthentic
published: true
description: How to monitor server infrastructure using Outthenitc
tags:  perl, linux, testing, devops
---

![https://raw.githubusercontent.com/melezhik/outthentic-dev.to/master/images/outthentic-example.png](https://raw.githubusercontent.com/melezhik/outthentic-dev.to/master/images/outthentic-example.png)

Bringing devops to our daily life makes creation of new servers is really simple. Cloud providers makes it possible to create new virtual machines in just one click. So the importance of the server infrastructure audit tools become more and more important. Tools like [goss](https://github.com/aelsabbahy/goss) and [inspec](https://www.inspec.io/) do the job.


[Outthentic](https://github.com/melezhik/outthentic), being a universal script runner can do that too, but in ***very different way***

Instead of providing declarative DSL to check server state, Outthentic supplies us with effective primitives to parse command output and validate server resources and configurations.


In some real life examples I will show you how, and if you find this approach attractive, welcome to our camp (;



# Install Outthentic

Before creating any test let's install Outthentic.


```
$ cpanm https://github.com/melezhik/outthentic.git
```
 
Now let's write tests

# Check if a package is installed

Let's start with something simple, yet useful example. Here we just run bash command and check that exit code is `0`.


```
$ nano story.bash

yum list python 2>/dev/null | tail -n 1
```

```
$ strun

2018-10-01 20:53:25 :  [path] /
python.x86_64                      2.7.5-69.el7_5                       @updates
ok	scenario succeeded
STATUS	SUCCEED

```

Checking of minimal package version would take just an adding a few lines to a *check file*:

```
$ nano story.check

regexp: python.x86_64\s+(\d+)\.(\d+)\.(\d+)-
generator: <<CODE
  [
    "assert: ".( capture()->[0] >= 2   || 0  )." major version is more or equal 2",
    "assert: ".( capture()->[1] >= 7   || 0  )." minor version is more or equal 7"
  ]
CODE

$ strun

2018-10-01 21:06:40 :  [path] /
python.x86_64                      2.7.5-69.el7_5                       @updates
ok	scenario succeeded
ok	text match /python.x86_64\s+(\d+)\.(\d+)\.(\d+)-/
ok	major version is more or equal 2
ok	minor version is more or equal 7
STATUS	SUCCEED
```

Any other test case could be defined the same way. The approach looks like:

1. Execute command(s)
2. Check status code (optional)
3. Validate commands output using check files

Check files contains static **[validation rules](https://github.com/melezhik/outthentic-dsl)** and **pieces of code** ( written on Perl, Bash, Ruby or Python ) to **dynamically** generate new rules.  


Here more complicated example.


# Monitoring a group of processes:


```
$ nano story.bash

sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &

ps aux | grep sleep | grep -v grep

```

```
$ nano story.check

begin:
sleep 5
sleep 5
sleep 5
sleep 5
sleep 5
end:

```

```
$ strun

2018-10-02 22:26:05 :  [path] /
root      48180  0.0  0.0   7772   664 pts/2    S+   22:26   0:00 sleep 5
root      48181  0.0  0.0   7772   748 pts/2    S+   22:26   0:00 sleep 5
root      48182  0.0  0.0   7772   640 pts/2    S+   22:26   0:00 sleep 5
root      48183  0.0  0.0   7772   712 pts/2    S+   22:26   0:00 sleep 5
root      48184  0.0  0.0   7772   748 pts/2    S+   22:26   0:00 sleep 5
ok      scenario succeeded
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
STATUS  SUCCEED

```

The same as above, but for **N** number of processes:


```
$ nano story.check

generator: [ "begin:", (map { "sleep 5" } (1 .. config()->{N})), "end:" ]

$ strun --param N=5

```

Following examples of various checks happening in daily operations life, some of them are trivial, some are not, some are shamelessly "stolen" from the goss issues [list](https://github.com/aelsabbahy/goss/issues) to show that almost nothing is impossible with Outthentic check tool!

# Check if an http response has a http header


```
$ nano story.bash

curl -s -k -o /dev/null -D - http://headers.jsontest.com/ | grep Content-Type:

$ nano story.check

Content-Type: application/json

```


```
$ strun

2018-10-03 18:33:33 :  [path] /
Content-Type: application/json; charset=ISO-8859-1
ok      scenario succeeded
ok      text has 'Content-Type: application/json'
STATUS  SUCCEED
```

# Check if user running process

```

$ nano story.bash

ps -o command -u root|grep /usr/lib/systemd/systemd-logind|grep -v grep


$ nano story.check

regexp: ^/usr/lib/systemd/systemd-logind$

```


```
$ strun

2018-10-03 18:50:24 :  [path] /
/usr/lib/systemd/systemd-logind
ok      scenario succeeded
ok      text match /^/usr/lib/systemd/systemd-logind$/
STATUS  SUCCEED
```

# Check if a kernel parameter greater than

```
$ nano story.bash

sysctl -a 2>/dev/null | grep net.core.somaxconn


$ nano story.check

regexp: net.core.somaxconn = (\d+)

generator: <<CODE 
[ 
  "assert: ".( 
     capture()->[0] >= 128 ? 1 : 0 
   )." net.core.somaxconn is greater or equal then 128" 
]
CODE

```

```
$ strun

2018-10-03 19:52:01 :  [path] /
net.core.somaxconn = 128
ok      scenario succeeded
ok      text match /net.core.somaxconn = (\d+)/
ok      net.core.somaxconn is greater or equal then 128
STATUS  SUCCEED

```

# Basic auth for http/https checks

```
$ nano story.bash

curl -u guest:guest -f -o /dev/null -s https://jigsaw.w3.org/HTTP/Basic/
```

```
$ strun

2018-10-03 21:53:38 :  [path] /
ok      scenario succeeded
STATUS  SUCCEED
```

# Check if a http port from a range is accessible


```

$ nano story.bash

for i in 83 82 81 80; do
  curl http://sparrowhub.org:$i -o /dev/null -s -f --connect-timeout 3 && echo connected to $i
done


$ nano story.check

regexp: connected to \d+

```

```
$ strun

2018-10-03 22:06:03 :  [path] /
connected to 80
ok      scenario succeeded
ok      text match /connected to \d+/
STATUS  SUCCEED
```

# Check files by Glob/Regex names

Let's check that configuration files in `/etc/` directory has `root` owner and `644` mode:


```

$ nano story.bash 

stat -c %U' '%a' '%n /etc/*.conf


$ nano story.check

regexp: (\S+)\s(\d\d\d)\s(\S+)

generator: <<CODE
  my @out;
  for my $f (@{captures()}){
    my $fname = $f->[2];
    push @out, "assert: ".( $f->[0] eq 'root' ? 1 : 0 )." $fname has a root owner";
    push @out, "assert: ".( $f->[1] eq '644' ? 1 : 0 )." $fname has a 644 mode";
  }
  [@out]
CODE


```

```

$ strun

2018-10-03 22:45:49 :  [path] /
root 644 /etc/dracut.conf
root 644 /etc/e2fsck.conf
root 644 /etc/host.conf
root 644 /etc/krb5.conf
root 644 /etc/ld.so.conf
root 640 /etc/libaudit.conf
root 644 /etc/libuser.conf
root 644 /etc/locale.conf
root 644 /etc/man_db.conf
root 644 /etc/mke2fs.conf
root 644 /etc/nsswitch.conf
root 644 /etc/resolv.conf
root 644 /etc/rsyncd.conf
root 644 /etc/sysctl.conf
root 644 /etc/vconsole.conf
root 644 /etc/yum.conf
ok      scenario succeeded
ok      text match /(\S+)\s(\d\d\d)\s(\S+)/
ok      /etc/dracut.conf has a root owner
ok      /etc/dracut.conf has a 644 mode
ok      /etc/e2fsck.conf has a root owner
ok      /etc/e2fsck.conf has a 644 mode
ok      /etc/host.conf has a root owner
ok      /etc/host.conf has a 644 mode
ok      /etc/krb5.conf has a root owner
ok      /etc/krb5.conf has a 644 mode
ok      /etc/ld.so.conf has a root owner
ok      /etc/ld.so.conf has a 644 mode
ok      /etc/libaudit.conf has a root owner
not ok  /etc/libaudit.conf has a 644 mode
ok      /etc/libuser.conf has a root owner
ok      /etc/libuser.conf has a 644 mode
ok      /etc/locale.conf has a root owner
ok      /etc/locale.conf has a 644 mode
ok      /etc/man_db.conf has a root owner
ok      /etc/man_db.conf has a 644 mode
ok      /etc/mke2fs.conf has a root owner
ok      /etc/mke2fs.conf has a 644 mode
ok      /etc/nsswitch.conf has a root owner
ok      /etc/nsswitch.conf has a 644 mode
ok      /etc/resolv.conf has a root owner
ok      /etc/resolv.conf has a 644 mode
ok      /etc/rsyncd.conf has a root owner
ok      /etc/rsyncd.conf has a 644 mode
ok      /etc/sysctl.conf has a root owner
ok      /etc/sysctl.conf has a 644 mode
ok      /etc/vconsole.conf has a root owner
ok      /etc/vconsole.conf has a 644 mode
ok      /etc/yum.conf has a root owner
ok      /etc/yum.conf has a 644 mode
STATUS  FAILED (2)

```

On my docker machine `/etc/libaudit.conf` has a `640` mode which caused the test failure.


# Check if a TCP address/port is being listened by a process


Let's validate that `22` port is listened by `/usr/sbin/sshd`

```
$ mkdir -p modules/file-proc

$ mkdir modules/netstat

$ nano modules/netstat/story.bash

netstat -nlpW|grep ':22'

$ nano modules/netstat/story.check

regexp: tcp\s+\d+\s+\d+\s+\S+?:22.*LISTEN\s+(\d+)

code: <<CODE
  our @pids;
  for my $f (@{captures()}){
    push @pids, $f->[0];
  }
CODE


$ nano modules/file-proc/story.bash

file /proc/$pid/exe

$ nano modules/file-proc/story.check

symbolic link to `/usr/sbin/sshd'


$ nano hook.pl

run_story "netstat";

for my $pid (our @pids){
  run_story "file-proc", { pid => $pid }
}

$ strun

2018-10-04 14:06:28 :  [path] /modules/netstat/
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      69/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      69/sshd
ok      scenario succeeded
ok      text match /tcp\s+\d+\s+\d+\s+\S+?:22.*LISTEN\s+(\d+)/
2018-10-04 14:06:28 :  [path] /modules/file-proc/ [params] pid:69
/proc/69/exe: symbolic link to `/usr/sbin/sshd'
ok      scenario succeeded
ok      text has 'symbolic link to `/usr/sbin/sshd''
STATUS  SUCCEED
```


Finally we can a whole [bunch](https://github.com/melezhik/outthentic-dev.to/tree/master/paper02-audit) of tests we have written, recursively:


```
$ strun --recurse --format=production
2018-10-04 14:14:13 :  [path] /kernel-param/
2018-10-04 14:14:14 :  [path] /http-basic-auth/
2018-10-04 14:14:14 :  [path] /modules/netstat/
2018-10-04 14:14:14 :  [path] /modules/file-proc/ [params] pid:69
2018-10-04 14:14:14 :  [path] /http-header-response/
2018-10-04 14:14:15 :  [path] /package-exists/
2018-10-04 14:14:15 :  [path] /file-by-regexp/
not ok  /etc/libaudit.conf has a 644 mode
root 644 /etc/dracut.conf
root 644 /etc/e2fsck.conf
root 644 /etc/host.conf
root 644 /etc/krb5.conf
root 644 /etc/ld.so.conf
root 640 /etc/libaudit.conf
root 644 /etc/libuser.conf
root 644 /etc/locale.conf
root 644 /etc/man_db.conf
root 644 /etc/mke2fs.conf
root 644 /etc/nsswitch.conf
root 644 /etc/resolv.conf
root 644 /etc/rsyncd.conf
root 644 /etc/sysctl.conf
root 644 /etc/vconsole.conf
root 644 /etc/yum.conf
STATUS  FAILED (2)
2018-10-04 14:14:15 :  [path] /multiple-processes/
2018-10-04 14:14:16 :  [path] /process-run-user/
2018-10-04 14:14:16 :  [path] /range-of-ports/
STATUS  FAILED (254)
stories failed: /file-by-regexp
```

We use `production` output format, which only shows details for failed tests (`/file-by-regexp`)


# Tests reuse


We just have to package the test case we want to reuse into sparrow plugin:

```

$ cd kernel-param/

$ nano story.bash

param=$(config name)

sysctl  2>/dev/null -a | grep $param


$ nano story.check

generator: <<CODE

my $value = config()->{value}; 
my $param = config()->{name};
 
[ 
  'regexp: '.$param.' = (\d+)',
 
  "assert: ".( 
     capture()->[0] >= $value ? 1 : 0 
   )." $param is greater or equal then $value" 
]
CODE



$ nano sparrow.json
{

    "name": "kernel-param-check",
    "version": "0.0.1",
    "category" : "audit", 
    "description" : "check kernel parameter",
}

$ sparrow plg upload

```

Then:

```
$ ssh host

$ sparrow index update

$ sparrow plg install kernel-param-check

$ sparrow plg run kernel-param-check \
--param name=net.core.somaxconn \
--param value=128
```


# Conclusion


[Outthentic](https://github.com/melezhik/outthentic) is flexible and efficient tool to get job done, when you need infrastructure tests. Rather then providing declarative DSL as tools like goss and inspec does it provides powerful tool-set to  run arbitrary command and parse output. Sparrow plugin gives you a decent level of code reuse, when you can distribute independent small test sets across your environments.
 
It would be interesting to hear readers opinion on comparison this approach against existing ones ( goss, inspec ).