Ssh-bulk-check - Super flexible check of group of nodes by ssh.
Tags: monitoring, devops, perl6, bash
Published: 2019-04-29 21:23:32

---
title: Ssh-bulk-check - Super flexible check of group of nodes by ssh.
published: true
description: Ssh-bulk-check - super flexible plugin to check many ssh nodes states
tags: monitoring, devops, perl6, Bash
---

Sometimes we need to monitor that a _group_ of hosts are in right state. Writting tests take tame. Existing tools like goss or inspec are good but they lack of flexibility. There is always something that are not covered by provided API.


New [Sparrow6](https://github.com/melezhik/sparrow-plugins/tree/master/Sparrow6) plugin [ssh-bulk-check](https://github.com/melezhik/sparrow-plugins/tree/master/ssh-bulk-check) allows to write check scripts to validate states of group of ssh hosts in unlimitted manner.

It's extremely flexible and comprehensive because relies on [Sparrow6 Task Check DSL](https://github.com/melezhik/Sparrow6/blob/master/documentation/taskchecks.md) and plain Bash scripts  that are able to cover almost any use cases.


Let me show how it works.


# Install Sparrow6


`zef install https://github.com/melezhik/Sparrow6.git`


# Create Sparrow6 Repository


`export SP6_REPO=file:///tmp/repo`
`mkdir -p /tmp/repo`
`s6 --repo_init /tmp/repo`
`git clone https://github.com/melezhik/sparrow-plugins.git`
`cd sparrow-plugins && find -maxdepth 2 -mindepth 2 -name sparrow.json -execdir s6 --upload \;`


# Ssh hosts state


Say, we have a group of nodes, where every node has to have the same state: 


* directory `/var/data` exists

* size of `/tmp` is no more then 1GB 

* nginx runs with no more 2 workers.


This is quite imaginative example but you've probably caught the idea.


Let's create shell script that run _test_ commands first:



# Cmd.sh


`mkdir files`

`nano files/cmd.sh`

```bash
echo '==='

echo "check /var/data dir"
  ls -d /var/data && echo "/var/data is a directory"
echo "end check"

echo '==='

echo "check /tmp/ dir size"
  sudo du -sh /tmp/
echo "end check"


echo '==='

echo "check if nginx is alive"
  ps uax| grep nginx| grep -v grep
echo "end check"

echo '==='

```


Now let's define check rules, that _analyze_ output of `cmd.sh` script. I use Sparrow6 TaskCheck DSL with range expressions, so to tell one check from another:


# State.check

```
between: { 'check /var/data dir' } { end \s+ check }

  /var/data is a directory

end:

note: ===

between: { '/tmp/ dir size' } { end \s+ check }

  regexp: ^^ \d+(\w+) \s+ '/tmp/'

  generator: <<HERE
  !perl

    if (@{matched()}){
      my $order = capture()->[0];
      print "assert: ", ( $order eq 'G' ? 0 : 1 ), " the size of /tmp dir is less
      then 1 GB\n";
    }

  HERE

end:

note: ===

between: { 'check if nginx is alive' } { end \s+ check }

  /usr/sbin/nginx -g daemon on; master_process on;

  regexp: ^^ 'www-data' \s+ .* \s+ worker \s+ process $$

  generator: <<HERE
  !perl

    if (my $cnt = @{matched()}){
      print "assert: ", ( $cnt <= 2 ? 1 : 0  ), " no more 2 nginx worker
      launched\n";
    }

  HERE

end:

```

We are almost set, now let's run all our checks against ssh hosts, we're gonna use Sparrowdo Task runner in `--localhost` mode, because we run tests from localhost using `ssh/sshpass` utilities:


# Sparrowdo scenario

sparrowfile


```perl
#!perl6

task-run "check my hosts", "ssh-bulk-check", %(
  cmd => "files/cmd.sh",
  state => "files/state.check",
  hosts => [ "192.168.0.1" ],
);
```


To run test just say:



`sparrowdo --localhost`


Here is the result:


```
20:01:46 04/29/2019 [check my hosts] check host [192.168.0.1]
20:01:46 04/29/2019 [check my hosts] ===
20:01:46 04/29/2019 [check my hosts] check /var/data dir
20:01:46 04/29/2019 [check my hosts] /var/data
20:01:46 04/29/2019 [check my hosts] /var/data is a directory
20:01:46 04/29/2019 [check my hosts] end check
20:01:46 04/29/2019 [check my hosts] ===
20:01:46 04/29/2019 [check my hosts] check /tmp/ dir size
20:01:46 04/29/2019 [check my hosts] 40K        /tmp/
20:01:46 04/29/2019 [check my hosts] end check
20:01:46 04/29/2019 [check my hosts] ===
20:01:46 04/29/2019 [check my hosts] check if nginx is alive
20:01:46 04/29/2019 [check my hosts] root      1243  0.0  0.0 140628  1500 ?        Ss   18:32   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
20:01:46 04/29/2019 [check my hosts] www-data  1244  0.0  0.0 143300  6264 ?        S    18:32   0:00 nginx: worker process
20:01:46 04/29/2019 [check my hosts] www-data  1245  0.0  0.0 143300  6264 ?        S    18:32   0:00 nginx: worker process
20:01:46 04/29/2019 [check my hosts] end check
20:01:46 04/29/2019 [check my hosts] ===
20:01:46 04/29/2019 [check my hosts] end check host [192.168.0.1]
[task check] ====================================================
[task check] check results
[task check] ====================================================
[task check] stdout match (r) </var/data is a directory> True
[task check] ===
[task check] stdout match (r) <^^ \d+(\w+) \s+ '/tmp/'> True
[task check] <the size of /tmp dir is less then 1 GB> True
[task check] ===
[task check] stdout match (r) </usr/sbin/nginx -g daemon on; master_process on;> True
[task check] stdout match (r) <^^ 'www-data' \s+ .* \s+ worker \s+ process $$> True
[task check] <no more 2 nginx worker launched> True
```


For this example I use only one host - `192.168.0.1` but we can add as much hosts as we need, the plugin will check them all in one transaction.


It's dead easy to write almost any check with that plugin. Please share your use cases and I'd be glad to help you how to use `ssh-bulk-check` for them


---