Conformance Testing in Rust
Tags: rust, testing, showdev
Published: 2019-10-04 21:20:28

I'm excited to announce [`conformance::tests`](https://crates.io/crates/conformance), a new crate I've published as part of building the Tiny-C compiler for the [Crafting IDE-Ready-Compilers](https://dev.to/cad97/crafting-ide-ready-compilers-500o) series.

The idea of this crate is to make it simpler to test any API which takes the shape of a `&str -> impl serde::Serialize + serde::Deserialize` function. It works with any serde compatible data format, though I find that [`serde_yaml`](https://lib.rs/crates/serde_yaml) or [`ron`](https://lib.rs/crates/ron) give the most readable diffs for a failed test.

Using the Tiny-C lexer and YAML for serialization, here's what a test looks like:

```yaml
simple_test
===
a=b=c=2<3;
---
- Identifier: 1
- EqualsSign: 1
- Identifier: 1
- EqualsSign: 1
- Identifier: 1
- EqualsSign: 1
- Integer: 1
- LessThanSign: 1
- Integer: 1
- Semicolon: 1
...
```

and to hook it up to the standard Rust test runner, all that has to be written is:

```rust
#[conformance::tests(exact,
    ser = serde_yaml::to_string,
    de = serde_yaml::from_str,
    file = "tests/yaml.test")]
fn lex_tokens(s: &str) -> Vec<Token> {
    tokenize(s).collect()
}
```

A test in the test file is represented by a test name (which needs to be a valid rust identifier continuation), terminated by a `===` on its own line, the test input string, terminated by a `---` on its own line, and the serialized expected output, terminated by a `...` on its own line. A test file contains any number of these tests. (The extension of the file does not matter, but idiomatically should be `format.test`.) The file name (excluding the extension) is also required to be a valid Rust identifier (or a `.` is also allowed).

The procedural macro is applied to the function that you want to test. The function must have the shape of taking a single parameter of type `&str` and returning a type that can be de/serialized by `serde`. The arguments to the macro itself are:

- `exact`: the produced value must exactly match the expected value. In the future, a separate mode where the produced value is just required to be a superset of the expected value is planned.
- `ser`: a path to a function of shape `fn<T: serde::Serialize>(&T) -> String`.
- `de`: a path to a function of shape `fn<T: serde::Deserialize>(&str) -> Result<T, impl std::error::Error>`.
- `file`: a file path string relative to the cargo manifest directory to the intended test file.

Of course, multiple `conformance::tests` attributes can be applied to the same function to run the tests in multiple files.

If all you want to do is use the library, you now know everything you need to know to use it. But I find the details of how it works interesting to discuss, so read on to see.

-----

The Rust [Custom Test Frameworks feature](https://github.com/rust-lang/rust/issues/50297) is still experimental and unstable. But, while `#[test_case]` might simplify the implementation of `conformance`, it's not needed and by using just `#[test]` we work on stable.

In order to integrate natively into the standard `#[test]` test runner, a unique `#[test]` function is emitted for each test in the test file.

To do the actual testing work, we emit a single function:

```rust
#[cfg(test)]
fn #testing_fn(expected: &str, actual: &str) -> Result<(), Box<dyn ::std::error::Error>> {
    const _: &str = include_str!(#filepath);
    let actual = #ser(&#fn_name(actual))?;
    let expected = #ser(&#de::<#tested_type>(expected)?)?; // normalize
    assert_eq!(actual, expected);
    Ok(())
}
```

We call the function by the file name, such that multiple files can be tested in the same Rust namespace. The file path is `include_str!`'d to add a data dependency onto the test file so changing it will recompile the crate. (Hopefully Rust doesn't get smart enough to just recompile the `const` until a real API for reading files in a procedural macro in a principled way is stabilized.)

For each function in the test file, we emit a function that delegates to `#testing_fn`:

```rust
#[test]
fn #test_name() -> Result<(), Box<dyn ::std::error::Error>> {
    #testing_fn(#output, #input)
}
```

The function name is assembled as `{filename}_{testname}`, thus the requirements on naming for the filename and test name. The output and input to the test are sliced out of the test file and included as string literals in the generated token stream.

Because the comparison is just done by `assert_eq`, that's the whole macro. Complete with all the formalities, it's just under 200 lines (plus `syn{features=[full]}`)! The test failure is best viewed with [IntelliJ Rust](https://intellij-rust.github.io/) since they provide a diff view of `assert_eq!` failures, or with some other `assert_eq` patch like [pretty-assertions](https://lib.rs/crates/pretty_assertions) to provide an in-terminal diff.

There's a few features I'd like to add to `conformance::tests` that I'd love to have community assistance in designing.

- "Includes"/"superset" mode, as opposed to the current "strict" mode. The produced value includes at least the specified data, but is allowed to include more data. This is mostly useful for testing part of a structure, omitting data not relevant to the test.
- `serde =` shortcut: give a path to a `serde_json`/`serde_yaml`-like module/crate, that exposes a `to_string` and `from_str` appropriate for `ser` and `de`.
- A smarter diff algorithm. While comparing the string serialized version of data structures is nicely minimal and utilitarian, I can't help but think we can do better and use some sort of "minimizing diff" algorithm (like Longest Common Sub-sequence is for sequences) that can generally apply to the [serde data model](https://serde.rs/data-model.html). This might even be required for the "superset" mode.
- Remove the `Deserialize` requirement. It's currently just used to normalize the serialized format; removing it means many more things can be tested through `conformance::tests`, even if they can't deserialize well.

If you're interested, check out [the repository](https://github.com/CAD97/tinyc/tree/master/crates/conformance), and get a bonus sneak peek at the content of next week's post for the Tiny-C lexer.