Javascript Composable Decoders with Validation
Tags: monads, functional, typescript, javascript
Published: 2019-07-10 05:54:45

I'm a suspicious person. If I don't find my slippers by my bed, I suspect. If I see a cat wearing a coat and looking at my direction, I suspect. The same way, when I receive data from an external source, I suspect.

Doesn't matter what the Swagger or the specs say, if you are receiving data from an API or any other external source, it is always good to know that it fulfills a format and any business restrictions. This is where TypeScript types stop helping you. Even if there are nice libraries that help cover this gap, like [io-ts](https://github.com/gcanti/io-ts), you might still miss a good way to trace what failures happened and where.

Here I'll describe how I implemented a Javascript decoder that accumulates the errors of the received data while keeping all needed data after the decoding process. You can see the full snippet [here](https://runkit.com/kwirke/validation-decoder).

## The Problem

To illustrate the example, I'll be receiving a list of videogames data, like this:

```javascript
const videogames = [
  {id: 1, name: 'Doom', genre: 'FPS', rating: 7},
  {id: 2, name: 'NieR: Automata', genre: 'Action RPG', rating: 100},
  {id: 3, name: 'Dead Cells', genre: 'Rogue-like', rating: 8},
]
```

We will have some restrictions as well, namely:
- The dataset must be an array of videogames.
- A videogame must have an `id`, a `name`, a `genre` and a `rating`.
- `name` can't be empty
- `genre` must be one of our recognised genres (FPS, RPG, Simulator, Strategy, and Platforms).
- `rating` must be between 1 and 10.

If you are a keen observer, you'll see the example dataset already breaks some of those restrictions. Excellent.

What we want is to be able to parse this kind of datasets, know all the errors that happened so we can inform the user or developer, and keep or throw away invalid data at our convenience.

## The Tool to Solve It

In order to do so, I'll use my library [Validation](https://www.npmjs.com/package/@rexform/validation). Validation is a monad. A monad is a software pattern for a type/class that has certain functions and certain restrictions. Being a monad means two things:
- It is a wrapper for something (in our case, values) and can be constructed using `Validation.of(value)` (this is called Applicative).
- It implements "Mappable" or has a `map` function (this is called Functor), and "Chainable", or a `chain` function (and this, Chain).

Mapping a monad means applying a function to its values without modifying the container, as if it was an array: `[1, 2].map(x => x + 1) === [2, 3]`

Chaining a monad means applying a function to its values and changing the container for the one returned by the function. It is also called flatMap because, if you map a function that returns another container and don't flatten the result, you end up with a container inside a container: 
`[1, 2].map(x => [x, 0]) === [[1, 0], [2, 0]]`, but
`[1, 2].flatMap(x => [x, 0]) === [1, 0, 2, 0]`

`Validation<E, V>` is a monad that can be of two types: `Valid` and `Invalid`. `E` and `V` here are generic types for the two values that a Validation wraps: Its errors and its value.

A `Valid` type only stores some data of type `V`, and affirms it is "valid" so far. It can be constructed with `Validation.of`, but also using `Validation.valid`

An `Invalid` type stores two values: Some invalid data of type `V`, and a list of errors of type `E`. It can be constructed using `Validation.invalid`.

Having all that we can validate a specific rating like this:

```javascript
const ratingDecoder = rating => (
  isBetween(1, 10)(rating)
    ? Validation.valid(rating)
    : Validation.invalid(rating, `Rating must be between 1 and 10, but received ${rating}`)
)
```

Here we're returning a `Valid(rating)` in case rating fulfills the restriction, and `Invalid(rating, errorMessage)` when rating does not fulfill it.

The unicorn stuff we have here that other validation libraries don't offer is that _we're keeping the rating value_, even if we know it is invalid, because we may need this information later.

## The Plan

Ok, so what's the plan? How are we going to use this to decode all the data?

First, we want to do it recursively, and secondly, with composable decoders that also describe our data shape. For example, our `videogameDecoder` will be something like this:

```javascript
const videogameDecoder = videogame => doSomeStuff({ // We'll complete this later
  id: idDecoder,
  name: nameDecoder,
  genre: genreDecoder,
  rating: ratingDecoder,
}, videogame)
```

This way, `videogameDecoder` is serving two purposes:
- It is a decoder function that returns a validated videogame.
- It is a declaration of the videogame type shape, like `PropTypes`. This is also a good documentation when we don't have TypeScript.

We will do this with all levels, all shapes and types. In our case, this is our four attributes (`id`, `name`, `genre`, and `rating`), our `videogame` type, and our `videogameArray` type. Let's begin:

## Decoding For Fun

We'll begin with the `nameDecoder` function. Assuming we have a function `isFilled(str)` that tells us if a name is non-empty, we can do something similar to the `ratingDecoder` before:

```javascript
const nameDecoder = name => (
  isFilled(name)
    ? valid(name)
    : invalid(name, 'name can not be empty')
)
```

So we'll have to do this with all the attributes. Isn't it a little boilerplate-y? Luckily, Validation comes with several helpers, one of which is `fromPredicateOr(errorFn, predicate)`. It can be used to create a function that will receive our value and return a Validation of it. Let's look at how can we use it:

```javascript
const nameDecoder = fromPredicateOr(() => 'name can not be empty', isFilled)
```

Much better! Now, when we call nameDecoder, it will check isFilled and return a Valid or Invalid depending on its truthiness.

Moreover, if another type that is not a videogame needs to have a name that can not be empty, we can reuse this decoder!

We now have all the attributes decoded:

```javascript
const idDecoder = valid
const nameDecoder = fromPredicateOr(() => 'name can not be empty', isFilled)
const genreDecoder = fromPredicateOr(() => 'genre must be in validGenres', flip(includes)(validGenres))
const ratingDecoder = fromPredicateOr(() => 'rating must be between 1 and 10', isBetween(1, 10))
```

What about `idDecoder`? It doesn't have any restriction, so it will be valid always, but we still need to provide a Validation out of it, so we will use the valid constructor directly.

## The Videogame Type

The function `videogameDecoder` that we want will receive a videogame object, validate each one of its attributes, and then group (or _reduce_) all validations into one single validation:

```javascript
// Videogame
{
  id: 3,
  name: 'Dead Cells',
  genre: 'Rogue-like',
  rating: 8
}

// Videogame with validated attributes
{
  id: valid(3),
  name: valid('Dead Cells'),
  genre: invalid('Rogue-like', ['genre is not in validGenres']),
  rating: valid(8),
}

// Validated videogame without invalid properties
invalid({
  id: 3,
  name: 'Dead Cells',
  rating: 8,
}, ['genre is not in validGenres'])
```

Note that, in the last step, we are choosing not to keep the invalid values. It doesn't need to be like that, we can choose to keep them, but we won't need them any longer in this example.

In order to do the first step, we could use the [`evolve`](https://ramdajs.com/docs/#evolve) method from Ramda, but we won't because it doesn't apply any function to missing attributes, and we want to detect a missing attribute to say it is invalid.

We could also iterate over the object properties:

```javascript
const validations = {id: idDecoder, name: nameDecoder, /* ... */}

Object.keys(validations).reduce(
  (acc, k) => ({
    ...acc,
    [k]: property(k, videogame).chain(val => validations[k](val)),
  }),
  {}
)
```

Note how, in the fourth line, we are using the `Validation.property` method that returns a Valid if it finds that attribute, and an Invalid otherwise. Then, we chain it to a function that will return a Valid if the validation for that attribute passes, or an Invalid otherwise.

How does that work?

When we chain Validations, it remembers the errors that we had detected previously, and adds them to any new errors. It will behave like this:

```javascript
valid(1).chain(n => valid(2)) === valid(2)
valid(1).chain(n => invalid(2, ['error'])) === invalid(2, ['error'])
invalid(1, ['error1']).chain(n => invalid(2, ['error2']) === invalid(2, ['error1', 'error2'])
```

This way, information about the errors is preserved.

Instead of doing it this way, we will use another Validation helper: `validateProperties`. It does exactly what we wanted:

```javascript
const videogameWithValidatedProperties = validateProperties({
  id: idDecoder,
  name: nameDecoder,
  /* ... */
}, videogame)
```

On to the second and last step, we need to iterate over the object properties and add only the properties that are valid. We can check this using `Validation.isValid()` method, and access the value inside with `Validation.value`:

```javascript
const allProperties = obj => (
  Object.keys(obj).reduce((validatedObj, k) => (
    validatedObj.chain(validObj => obj[k].isValid()
      ? Validation.of({...validObj, [k]: obj[k].value})
      : obj[k].map(() => validObj)
    )),
    valid({})
  )
)
```

However, this is a complex function, and a common enough one to have its own helper as well, `Validation.allProperties`, so we'll use that.

At the end, we will have our `videogameDecoder` pretty terse thanks to the helpers:

```javascript
const videogameDecoder = videogame => {
    const videogameWithValidatedProperties = Validation.validateProperties({
        id: idDecoder,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }, videogame)
    return Validation.allProperties(videogameWithValidatedProperties)
}
```

We can improve that if we refactor it using point-free style with the help of the [pipe](https://ramdajs.com/docs/#pipe) function from Ramda:

```javascript
const videogameDecoder = pipe(
    Validation.validateProperties({
        id: idDecoder,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
)
```

## Validation Arrays

Just as Validation has some helpers to deal with objects, it has other ones to deal with arrays.

As it turns out, these operations are well defined in the functional programming world, because FP loves lists. Enter the **Monoid**.

A monoid is, just like a monad, another programming pattern (although they don't have much more in common, even if the names look similar). A type is a monoid if it is "Concatenable" and has an "empty" function that returns an empty element.

Therefore, a monoid will always have two functions:
- `empty` returns the empty element. With arrays, that would be `[]`.
- `concat` concatenates the values of two monoids and returns another monoid. With arrays, that would be `Array.concat`.

This means JS arrays are a monoid, and if they had an `.empty()` method that returned `[]`, they would even be [Static Land](https://github.com/fantasyland/static-land) compliant. But they don't.

Validation, as it turns out, is conveniently a Static Land compliant monoid when the wrapped value is an array (when it is not, it is casted to an array when concatenating). This means we have the full power of the monoids in the palm of our hands.

## The List Type

Now for the last function: `videogameArrayDecoder`. It receives an array of videogames, and returns a Validation of the array.

We can do that in two steps just like before: Validate each of the videogames, then accumulate (_reduce_) the Validations into a single Validation.

```javascript
// Array of videogames
[vg1, vg2, vg3]

// Array of videogame Validations
[valid(vg1), invalid(vg2, err2), invalid(vg3, err3)]

// Validation of array of videogames
invalid([vg1], [...err2, ...err3])
```

Note that, just like before, in the last step we will be dropping the invalid videogames off the list because we want to.

To validate each of the videogames, we can do that with a conventional `Array.map` like this:

```javascript
const validatedVideogames = videogames.map(videogameDecoder)
```

Eezy-peezy. For the second step, we want to reduce the array of validations to a validation of arrays. As we know, Validation acts as a monoid when the values are arrays, so let's map them to one-element arrays:

```javascript
const toArrayValidation = Validation.map(x => [x])
const videogameArrayValidations = validatedVideogames.map(toArrayValidation)
```

Now we are ready to concat them, because they contain arrays. `Validation.concat` method concatenates the valid values, and drops the invalid values, just like we want. This means we can reduce the list like follows:

```javascript
const videogamesValidation = videogameArrayValidations
    .reduce(Validation.concat, Validation.empty())
```

This looks awesome because it is the very definition of generating a list with a monoid. It is so awesome it has its own function in the library:

```javascript
const videogamesValidation = Validation.sequence(videogameArrayValidations)
```

If we instead wanted to keep the invalid values, we would have to do it another way:

```javascript
const losslessSequence = l => l.reduce((valList, val) => (
  valList.chain(list => val.map(x => [...list, ...x]))
), Validation.empty())

const videogamesValidation = losslessSequence(videogameArrayValidations)
```

By using `map` inside `chain`, what we are doing is concatenating all values inside the new validation in each iteration, and then chaining it to the original one to keep the errors, because the `chain` function preserves all the errors.

So how will the decoder look like?

```javascript
const videogameArrayDecoder = videogames => {
    const validatedVideogames = videogames.map(videogameDecoder)
    return Validation.sequence(validatedVideogames)
}
```

If we refactor it using point-free style and Ramda, we get this:

```javascript
const videogameArrayDecoder = pipe(map(videogameDecoder), Validation.sequence)
```

## The Result

Finally, this is the complete code of our whole decoder:

```javascript
const {Validation, valid, invalid} = require("@rexform/validation")
const {isNil, isEmpty, complement, either, includes, flip, both, lte, gte, pipe, map} = require('ramda')

const videogames = [
    {id: 1, name: 'Doom', genre: 'FPS', rating: 7},
    {id: 2, name: 'NieR: Automata', genre: 'Action RPG', rating: 100},
    {id: 3, name: 'Dead Cells', genre: 'Rogue-like', rating: 8},
]

const validGenres = ['FPS', 'Platforms', 'RPG', 'Strategy', 'Simulator']

const isFilled = complement(either(isNil, isEmpty))
const isBetween = (a, b) => both(flip(lte)(b), flip(gte)(a))

const nameDecoder = Validation.fromPredicateOr(() => 'name can not be empty', isFilled)
const genreDecoder = Validation.fromPredicateOr(() => 'genre must be in validGenres', flip(includes)(validGenres))
const ratingDecoder = Validation.fromPredicateOr(() => 'rating must be between 1 and 10', isBetween(1, 10))

const videogameDecoder = pipe(
    Validation.validateProperties({
        id: valid,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
)

const videogameArrayDecoder = pipe(map(videogameDecoder), Validation.sequence)

videogameArrayDecoder(videogames)
```

And this is the result:

```javascript
Invalid(
  [{id: 1, name: 'Doom', genre: 'FPS', rating: 7}],
  [
    "genre must be in validGenres",
    "rating must be between 1 and 10",
    "genre must be in validGenres",
  ]
)
```

The only missing problem is that, when we see the errors, we don't know what videogame produced them. We can fix that if we come back to our videogameDecoder and add the videogame `id` in the error message (or, instead of the id, the whole videogame object stringified, if we want).

We can use the function `mapError` to add the `id` to the error message. The function `mapError` works like `map`, but for the wrapped error array instead of the wrapped value. It will only modify each of the errors without changing the Validation:

```javascript
const videogameDecoder = pipe(
    Validation.validateProperties({
        id: valid,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
    videogame => videogame.mapError(e => `In ID=${videogame.value.id}: ${e}`),
)
```

That's it, now the result will have much more meaning:

```javascript
Invalid(
  [{id: 1, name: 'Doom', genre: 'FPS', rating: 7}],
  [
    "In ID=2: genre must be in validGenres",
    "In ID=2: rating must be between 1 and 10",
    "In ID=3: genre must be in validGenres",
  ]
)
```

We finished our new videogame decoder, hurray! :smile:

### Thanks

If you reached this point, thank you! This is my first article, and I welcome any suggestions or feedback. I hope you learned something out of it, but if you didn't, maybe you can teach me something!

Also, if you liked it, give [Validation](https://www.npmjs.com/package/@rexform/validation) a try :wink: