Command line application tests made easy with Outthentic
Tags: testing, cli, perl, tdd
Published: 2018-09-26 21:22:04

---
title: Command line application tests made easy with Outthentic
published: true
description: How to write tests for command line utilities using Outthentic
tags: testing, cli, perl, tdd
---



Writing tests is an essential part of software development. There are a plenty of frameworks to do that. In this post I'd like to introduce [Outthentic](https://github.com/melezhik/outthentic) - an universal script engine with _embedded_ testing facilities, allow one to create BDD/Integration tests for command line applications easy.



# Install

We will need the the cutting edge of Outthentic to see some recent features created especially for test development. 

```
$ cpanm https://github.com/melezhik/outthentic.git
```


# Application to test

Say, we have an application script - **application.bash** - which does all useful job. 

Our goal to test this script properly:

```
$ nano /usr/local/bin/application.bash

#!bash

echo "Hello world"

```

Let's outline our simple testing plan:

* Ensure that script exit code is `0` 
* Ensure that script output certain string to STDOUT
* Skip test for a certain environment
* Abort test if script is not installed in the system


## Ensure that script exit code is `0` 

This exit code is checked _by default_ when external script gets through Outthentic, we just need to add a thin layer to enable this test:


```
$ mkdir check-exit-code

$ nano check-exit-code/story.bash

#!/bash

application.bash
 
```

Now let's run out first test and see the output:


```
$ strun --story check-exit-code


2018-09-26 16:35:47 :  [path] /check-exit-code/
/root/projects/outthentic-dev.to//check-exit-code/story.bash: line 1: /usr/local/bin/application.bash: Permission denied
not ok  scenario succeeded
STATUS  FAILED (2)
```

Well, the test has failed. And we see the reason. We forgot to set execution bit, it's easy to fix and re-run test again. This is what we have write our tests for - to see things that do not work, rather then see things that are fine.


```
$ chmod a+x /usr/local/bin/application.bash

2018-09-26 20:07:27 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
STATUS  SUCCEED

```

A few words about status code emitted by Outthentic. As you might have noticed the first failed test produces status code `2`, which generally means _test failures_, there are 3 exit codes provided by Outthentic:


* `0` tests are passed, everything is ok 
* `2` tests are failed, something _definitely_ went wrong
* `1` some tests are passed, some are not, there is _something wrong_ or there are some _warnings_

The last case makes it possible to use Outthentic tests as [Consul check scripts](https://www.consul.io/docs/agent/checks.html#check-scripts)


# Ensure that script outputs some string to STDOUT 


Before going into details, let's think _why_ we would need it? 

A few reasons ( I wonder if a reader would provide more ) :

* Some external scripts do not provide a sane exit code, the only reasonable thing we can test is an output

* None zero exit code _might not_ means that program works _unexpectedly_. Again we can check output rather then exit code

* Program emits zero exit code ( finishes successfully ) but makes different output messages when being called with different parameters. So to check various test cases we need to run the same program differently and check different output


As our application is just an example, the check for "Hello world" string is trivial:


```
$ nano check-exit-code/story.check

Hello World
```


Now run:


```
$ strun --story check-exit-code

2018-09-26 17:23:29 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED
```
 
There are [more things](https://github.com/melezhik/outthentic#check-files-syntax) you can check with Outthentic. Imagine we want to check the program's output consists of `sequential numbers from 1 to 10`:


```
$ nano check-exit-code/story.check

begin:
generator: <<CODE
[ map { "regexp: ^\\d+$_" } (1..10) ]
CODE
end:

```

But let's go further.


# Splitting tests for different cases

In real projects we can eventually have many test cases mapped to many test scenarios. Outthentic is flexible enough to adopt for such a scheme, running multiples tests as a suite.

Let's reorgonize our test structure a bit:


```
$ mkdir check-exit-code

$ nano check-exit-code/story.bash

#!/bash

application.bash


$ mkdir check-stdout 

$ nano check-exit-code/story.bash

#!/bash

application.bash


$ nano check-stdout/story.check

Hello World

```

Now we have two test scenarios. To check exit code and to check returned output.

These tests overlaps by the fact they both runs `application.bash` however for our purposes it is not critical, we just create different tests to emphasize what we what to test.



```
$ tree 

.
├── check-exit-code
│   └── story.bash
└── check-stdout
    ├── story.bash
    └── story.check

```


Finally Outthentic allows us to run all our tests _recursively_: 


```

$ strun --recurse

2018-09-26 17:34:52 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
STATUS  SUCCEED
2018-09-26 17:34:52 :  [path] /check-stdout/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED
STATUS  SUCCEED

```

As we have and more and more tests, we might need to narrow down the output, it is achievable by `--format` option of Outthentic test runner:


```
$ strun --recurse --format=production
2018-09-26 17:36:14 :  [path] /check-exit-code/
2018-09-26 17:36:14 :  [path] /check-stdout/
```


Now let's go to the two last points of our testing plan. 

Sometimes we need to skip our tests for some reasons or even raise exception if _some_ preliminary conditions are not met, so we don't waste our time and run tests polluting console with _unnecessary_ messages, as we already know that we _might not_ run test suite.


# Quit test for some environment


Say, we don't want run tests for production environment which is _defined_ by passing `environment` variable:


```
$ export environment=production
```


Outthentic allows to *immediately * quit test execution phase by using `quit` function, let's see an example of it:


```

$ nano check-exit-code/hook.bash

#!bash
if test "$environment" = "production"; then
  quite "production tests are disabled, please use dev environment"
fi

```

```
$ strun --story check-exit-code

2018-09-26 18:10:54 :  [path] /check-exit-code/
? forcefully exit:  production tests are disabled, please use dev environment
STATUS  SUCCEED

```

The opposite idea is to let your tests fail immediately upon a certain condition. You should choose `outthentic_die` function  for this:

```
$ nano check-exit-code/hook.bash

#!bash

which application.bash 2>/dev/null || \
  outthentic_die "application.bash is not installed. You should install it to run tests"


```

```
$ unlink /usr/local/bin/application.bash
$ strun --story check-exit-code

2018-09-26 18:16:49 :  [path] /check-exit-code/
!! forcefully die:  application.bash is not installed. You should install it to run tests
STATUS  FAILED (2)
```

Now our project structure looks like this:

```
.
├── check-exit-code
│   ├── hook.bash
│   └── story.bash
└── check-stdout
    ├── story.bash
    └── story.check

```

`Hook.bash` is an example Outthentic hooks - small script gets run _before_ main test run, you can read more about [hooks](https://github.com/melezhik/outthentic#hooks) on Outthentic documentation pages.



## Decentralized or centralized testing model

As you could have noticed, we would have to add hook.bash to every test where we want to ensure preliminary conditions are met. In this scheme every test is treated as independent unit, and this follows the pattern we can see in many testing frameworks. While it seems easier to implement, it also results in duplication of code. Now we have hook.bash scripts doing _the same job_ for every story.

Alternatively we might check those preliminary conditions ( like environment and application being installed ) **once**, *in the very beginning*, before we run *any* test. 

This approach leads us to the opposite scheme, where all test become deepened on some "main" entry point where we can:

* perform initialization steps ( preliminary conditions check )

* call tests as functions in order


In Outthentic this type of design could be easily implemented through so called "story modules". Laterally when you call tests as a functions.

Let's slightly refactor our project to implement the idea:


```

$ nano hook.bash

#!bash

# this a main entrypoint 

if test "$environment" = "production"; then
  quite "production tests are disabled, please use dev environment"
fi

which application.bash 2>/dev/null || \
  outthentic_die "application.bash is not installed. You should install it to run tests"

run_story "check-exit-code"
run_story "check-stdout"


$ # we don't need hook.bash per story anymore

$ rm check-exit-code/hook.bash

# we make our tests - modules or functions, just when we copy those ones to modules/ folder

$ mkdir modules

$ mv check-exit-code check-stdout modules/

```

So we end up with this structure with one main entrypoint ( hook.bash ) and two dependable tests ( check-exit-code/story.bash , check-stdout/story.bash ), also notice that we run preliminary conditions check inside main entrypoint and control *how* and *what* tests to run.


```
.
├── hook.bash
└── modules
    ├── check-exit-code
    │   └── story.bash
    └── check-stdout
        ├── story.bash
        └── story.check

```



Now we are ready to run tests, note that we don't need `--recurse` option anymore, because all the tests sequence is defined through the hook.bash file.


```
$ strun

2018-09-26 20:24:20 :  [path] /modules/check-exit-code/
Hello world
ok      scenario succeeded
2018-09-26 20:24:20 :  [path] /modules/check-stdout/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED
```


# The end

This was just a brief introduction into testing capabilities provided by Outthentic framework. If you like it - go to [GH pages](https://github.com/melezhik/outthentic) to see all the details.

Feel free to share you opinions here in comments.