Don’t Interpret Me Wrong: Improvising Tests for an Interpreter
Tags: showdev, shellscript, testing, book
Published: 2018-08-07 15:14:12

---
title: Don’t Interpret Me Wrong: Improvising Tests for an Interpreter
published: true
tags: showdev, shell script, testing, book
canonical_url: http://suspensao.blog.br/disbelief/dont-interpret-me-wrong-improvising-tests-for-an-interpreter/
---

I’m in love with the [_Crafting Interpreters_](http://craftinginterpreters.com/) book. In it, [Bob Nystrom](http://journal.stuffwithstuff.com/) teach us how to writer an interpreter by implementing a little programming language called [Lox](http://craftinginterpreters.com/the-lox-language.html). It was a long time since I had so much fun programming! Besides being well-written, the book is funny and teach way more than I would expect. But I have a problem.

The snippets in the bug are written in a way we can copy and paste them. However, the book has challenges at the end of each chapter, these challenges have no source code and sometime they force us to change the interpreter a lot. I do every one of these exercises and as a result my interpreter diverges too much from the source in the book. Consequently, I often break some part of my interpreter.

How to solve that?

Unity tests would be brittle since the code structure changes frequently. [End-to-end tests](https://www.techopedia.com/definition/7035/end-to-end-test) seem more practical in this case. So, for each new feature of the language, I wrote a little program. For example, my interpreter should create [closures](http://craftinginterpreters.com/functions.html#local-functions-and-closures), and to ensure that I copied the Lox program below to the file [`counter.lox`](https://bitbucket.org/brandizzi/jlox/src/default/examples/counter.lox?at=default&fileviewer=file-view-default):

```kotlin
fun makeCounter() {
  var i = 0;
  fun count() {
    i = i + 1;
    print i;
  }

  return count;
}

var counter = makeCounter();
counter(); // "1".
counter(); // "2".
```

This program result should be the numbers 1 and 2 printed in different lines. So I put these values in a file called [`counter.lox.out`](https://bitbucket.org/brandizzi/jlox/src/default/examples/counter.lox.out). The program cannot fail either, so I created an empty file called [`counter.lox.err`](https://bitbucket.org/brandizzi/jlox/src/default/examples/counter.lox.out). (In some cases, it is necessary to ensure the Lox program will fail. In these cases, the file `.lox.err` should have content.)

Well, I wrote programs and output files for various examples; now I need to compare the programs’ results to the expected outputs. I decided to use the tool that [helps me the most in urgent times](https://dev.to/adambrandizzi/exchanging-sticker-cards-with-the-terminal-2ki9): shell script. I did a Bash script with a `for` iterating over all examples:

```bash
for l in examples/*.lox
do

done
```

For each example, I executed the Lox program, redirecting the outputs to temporary files:  

```bash
for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err
done
```

Now, we compare the real output with the expected output through [`diff`](https://www.gnu.org/software/diffutils/). When it compares two files, [`diff` returns 0 if there is no difference, 1 if there exists a difference or 2 in case of error](https://stackoverflow.com/questions/6971284/what-are-the-error-exit-values-for-diff). Since in Bash the conditional `if` considers 0 as true, we just check the negation of `diff`‘s exit code.

If the program prints something in standard output that is different from what is in its `.lox.out` file, we have a failure:

```bash
for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi
done
```

We also check the standard error and the `.lox.err` file:

```bash
for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi
done
```

Finally, I check if there was some failure and report the result:  


```bash
for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done
```

Not all of my Lox programs can be checked, though. For example, there is [a program which times loop execution](https://bitbucket.org/brandizzi/jlox/src/default/examples/timeit.lox)s, it is impossible to anticipate the value it will print. Because of that, I added the possibility to jump some programs: we need just to create [a file with the `.lox.skip` extension](https://bitbucket.org/brandizzi/jlox/src/default/examples/timeit.lox.skip):

```bash
for l in examples/*.lox
do
  if [-e $l.skip]
  then
    echo SKIP $l
    continue
  fi

  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done
```

If, however, I have a Lox example and it does not have expected output files (nor the `.lox.skip` file) then I have a problem and the entire script fails:

```bash
for l in examples/*.lox
do
  if [-e $l.skip]
  then
    echo SKIP $l
    continue
  elif [! -e $l.out ] || [! -e $l.err ]
  then
    echo missing $l.out or $l.err
    exit 1
  fi

  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done
```

With that, my test script is done. Let us see how it behaves:

```
$ ./lcheck.sh
PASS examples/attr.lox
PASS examples/bacon.lox
PASS examples/badfun.lox
PASS examples/badret.lox
PASS examples/bagel.lox
PASS examples/bostoncream.lox
PASS examples/cake.lox
PASS examples/checkuse.lox
PASS examples/circle2.lox
PASS examples/circle.lox
1d0
< 3
1c1
<
---
> [line 1] Error at ',': Expect ')' after expression.
FAIL examples/comma.lox
PASS examples/counter.lox
PASS examples/devonshinecream.lox
PASS examples/eclair.lox
PASS examples/fibonacci2.lox
PASS examples/fibonacci.lox
PASS examples/func.lox
PASS examples/funexprstmt.lox
PASS examples/hello2.lox
PASS examples/hello3.lox
PASS examples/hello.lox
PASS examples/math.lox
PASS examples/notaclass.lox
PASS examples/noteveninaclass.lox
PASS examples/point.lox
PASS examples/retthis.lox
PASS examples/scope1.lox
PASS examples/scope.lox
PASS examples/supersuper.lox
PASS examples/thisout.lox
PASS examples/thrice.lox
SKIP examples/timeit.lox
PASS examples/twovars.lox
PASS examples/usethis.lox
PASS examples/varparam.lox
```

Oops, apparently I removed the support for the [comma operator](https://en.wikipedia.org/wiki/Comma_operator) by accident. Good thing I wrote this script, right?

I hope this post was minimally interesting! Now, I am going to repair my comma operator and keep reading [this wonderful book](http://craftinginterpreters.com/).

_(This post is a translation of [Não me Interprete Mal: Improvisando Testes para um Interpretador](http://suspensao.blog.br/descrenca/nao-me-interprete-mal-improvisando-testes-para-um-interpretador/).)_