Untestable: testing compile-safe strategies via `dynamic`
Tags: testing, csharp, logic
Published: 2020-11-16 08:42:06

If we fix a bug by making a C# program more type safe, how do we test this?

Say for example you arrange for invalid code to not compile anymore?

## The case

[Active Logic](https://github.com/active-logic/activelogic-cs) uses a modified Kleene logic to implement Behavior Trees using the short-circuiting operators `&&` and `||`. In addition of a three-valued type, we have restricted types to represent ahead-of-runtime knowledge. Anyway this is handy because it avoids logical errors (we love uncertainty, in moderation).

So here's an example of an operation that we're disallowing:

```cs
impending x;
status y;
var z = x && y;
```

*impending* represents a task that's either running or failing; whereas *status* (typical of BT) is either *running*, *failing* or *done*. In the above example, y (which normally is an expression, not a pre-assigned value) would never evaluate. 

So this is something we can enforce at compile time, and the question then becomes: how do you put this under test.

Initially, our test looked like this:

```cs
[Test] public void Impending_AND_Status(){
    impending x = impending_fail;
    status    y = done;
    var z = x && y;  // CS217
}
```

With our implementation this raises CS217 at compile-time, invalidating the `(impending) && (status)` combination.

Expected behavior; also, who broke my test suite?

The [dynamic](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/types/using-type-dynamic) keyword lets you disable compile checks:

```cs
[Test] public void Impending_AND_Status(){
    dynamic x = impending.fail, y = status.done;
    var z = x && y;
    Print($"{x} && {y} => {z}");
}
```

...and you'd expect a runtime error to assert against. However (in our case) things get a little more complicated because the above DOES run and produce an output. 

The compiler sees an invalid logical AND, but the runtime takes a more incremental approach:

1) Check the left hand for "falsehood". If the left hand is false return the left hand.
2) Type-check the right hand. If there is an applicable `&` operation of the form lh & rh, evaluate the right hand.
3) Invoke the (user defined) `&` operator on the resulting operands.

With this in mind, I then rewrote the `false` operator for the impending type:

```cs
public static bool operator false(impending s)
=> throw new InvOp("Cannot test falsehood (impending)");
```

NOTE: *In C#, `true` and `false` work in pairs. You can't implement one and not another. `pending` should just never allow `&&`. But this, ultimately, is not something we can enforce at compile time.*

And our test then looks like this:

```cs
[Test] public void Impending_AND_Status(){
    dynamic x = impending_fail, y = done;
    Assert.Throws<InvOp> ( () => z = x && y  );
}
```

Downside of course, the compiler and runtime aren't checking the same thing. Strictly we'd have to forgo NUnit, and write a script that:

1) Runs a build on invalid code
2) Verifies every error issued by the compiler

This approach, however, is heavy-handed, so I decided to go ahead with runtime checking and here's the final version of the test, as it will look in the next commit:

```cs
// o(x, y) // assert x equals y
// s(x)    // new status from an int
// i(x)    // new 'impending' value from int
[Test] public void Impending_x_Status([Range(-1, 0)] int x,
                                      [Range(-1, 1)] int y){
    AND_CS217_InvOp(i(x), s(y));
    o( i(x) || s(y), s(x) || s(y) );
}
```

`AND_CS217_InvOp` encapsulates the NUnit call (which is a bit long winded when you have (7 x 7 x 2) cases to cover). The compile error is still no more than a label, still useful for documenting our APIs.

## Cooler talk

With runtime testing, compile-safe techniques are in a blind spot. This doesn't sit too well with me because testing is supposed to guide you towards *better code*. Is compile time safety not a good thing, then?

When you fix a bug or add a feature by tweaking type safety, if you don't have a test you also can't ensure your bug fix/feature won't be removed accidentally. You can roll your own tools, and `dynamic` is a poor man's version of doing just that. Oh, and it helps understanding how the C# runtime even works.

<span>Photo by <a href="https://unsplash.com/@aliisarvari?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ali Sarvari</a> on <a href="https://unsplash.com/s/photos/tile?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></span>