Validating Data in Python with Cerberus
Tags: python, adventofcode, cerberus, validation
Published: 2021-01-04 12:53:46

---
title: Validating Data in Python with Cerberus
published: true
date: 2020-12-29 00:00:00 UTC
tags: python, adventofcode, Cerberus, validation
canonical_url: https://hector.dev/2020/12/29/validating-data-in-python-with-cerberus/
---

This year was my first participating in [Advent of Code](https://adventofcode.com)—and I’m glad I did, because solving one of the [challenges](https://adventofcode.com/2020/day/4) exposed me to an excellent data validation library for Python named [Cerberus](https://docs.python-cerberus.org/en/stable/).

## What’s in a valid passport

Below are some excerpts from the challenge, along with specific field level validation rules:

> You arrive at the airport only to realize that you grabbed your North Pole Credentials instead of your passport. While these documents are extremely similar, North Pole Credentials aren’t issued by a country and therefore aren’t actually valid documentation for travel in most of the world.
> 
> It seems like you’re not the only one having problems, though; a very long line has formed for the automatic passport scanners, and the delay could upset your travel itinerary.
> 
> …
> 
> The line is moving more quickly now, but you overhear airport security talking about how passports with invalid data are getting through. Better add some data validation, quick!
> 
> You can continue to ignore the `cid` field, but each other field has strict rules about what values are valid for automatic validation:
> 
> - `byr` (Birth Year) - four digits; at least `1920` and at most `2002`.
> - `iyr` (Issue Year) - four digits; at least `2010` and at most `2020`.
> - `eyr` (Expiration Year) - four digits; at least `2020` and at most `2030`.
> - `hgt` (Height) - a number followed by either `cm` or `in`: 
>   - If `cm`, the number must be at least `150` and at most `193`.
>   - If `in`, the number must be at least `59` and at most `76`.
> - `hcl` (Hair Color) - a # followed by exactly six characters `0-9` or `a-f`.
> - `ecl` (Eye Color) - exactly one of: `amb` `blu` `brn` `gry` `grn` `hzl` `oth`.
> - `pid` (Passport ID) - a nine-digit number, including leading zeroes.
> - `cid` (Country ID) - ignored, missing or not.
> 
> Your job is to count the passports where all required fields are both **present** and **valid** according to the above rules.

For completeness, here are some invalid passports (delimited by `\n\n`):

```
eyr:1972 cid:100
hcl:#18171d ecl:amb hgt:170 pid:186cm iyr:2018 byr:1926

iyr:2019
hcl:#602927 eyr:1967 hgt:170cm
ecl:grn pid:012533040 byr:1946

hcl:dab227 iyr:2012
ecl:brn hgt:182cm pid:021572410 eyr:2020 byr:1992 cid:277
```

And, some valid passports:

```
pid:087499704 hgt:74in ecl:grn iyr:2012 eyr:2030 byr:1980
hcl:#623a2f

eyr:2029 ecl:blu cid:129 byr:1989
iyr:2014 pid:896056539 hcl:#a97842 hgt:165cm

hcl:#888785
hgt:164cm byr:2001 iyr:2015 cid:88
pid:545766238 ecl:hzl
eyr:2022
```

Most of the validation rules look straightforward in isolation, but less so when you think about composing them all together.

## Validating passports with Cerberus

Step one involved getting familiar with Cerberus [validation rules](https://docs.python-cerberus.org/en/stable/validation-rules.html). The library supports rules like the following:

- `contains` - This rule validates that the a container object contains all of the defined items.

```python
>>> document = {"states": ["peace", "love", "inity"]}

>>> schema = {"states": {"contains": "peace"}}
>>> v.validate(document, schema)
True
```

- `regex` - The validation will fail if the field’s value does not match the provided regular expression.

```python
>>> schema = {
... "email": {
... "type": "string",
... "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
... }
... }
>>> document = {"email": "john@example.com"}
>>> v.validate(document, schema)
True
```

- `required` - If `True` the field is mandatory. Validation will fail when it is missing.

```python
>>> v.schema = {"name": {"required": True, "type": "string"}, "age": {"type": "integer"}}
>>> document = {"age": 10}
>>> v.validate(document)
False
```

Step two involved converting the passports into Cerberus documents. This was mostly an exercise in parsing uniquely assembled text into Python dictionaries.

```python
# Split the batch file records by double newline.
for record in batch_file.read().split("\n\n"):
    # Split the fields within a record by a space or newline.
    record_field_list = [
        tuple(field.split(":")) for field in re.compile(r"\s").split(record.strip())
    ]
```

That leaves `record_field_list` looking like:

```python
>>> record_field_list
[('ecl', 'gry'),
 ('pid', '860033327'),
 ('eyr', '2020'),
 ('hcl', '#fffffd'),
 ('byr', '1937'),
 ('iyr', '2017'),
 ('cid', '147'),
 ('hgt', '183cm')]
```

From there, `dict` converts the list of tuples into a proper Cerberus document:

```python
>>> document = dict(record_field_list)
>>> document
{'byr': '1937',
 'cid': '147',
 'ecl': 'gry',
 'eyr': '2020',
 'hcl': '#fffffd',
 'hgt': '183cm',
 'iyr': '2017',
 'pid': '860033327'}
```

## Putting it all together

Equipped with a better understanding of what’s possible with Cerberus, and a list of Python dictionaries representing passports, below is the schema I put together to enforce the passport validation rules of the challenge. Only one of the rules (`hgt`) required a custom function (`compare_hgt_with_units`).

```python
SCHEMA = {
    "byr": {"min": "1920", "max": "2002"},
    "iyr": {"min": "2010", "max": "2020"},
    "eyr": {"min": "2020", "max": "2030"},
    "hgt": {
        "anyof": [
            {"allof": [{"regex": "[0-9]+cm"}, {"check_with": compare_hgt_with_units}]},
            {"allof": [{"regex": "[0-9]+in"}, {"check_with": compare_hgt_with_units}]},
        ]
    },
    "hcl": {"regex": "#[0-9a-f]{6}"},
    "ecl": {"allowed": ["amb", "blu", "brn", "gry", "grn", "hzl", "oth"]},
    "pid": {"regex": "[0-9]{9}"},
    "cid": {"required": False},
}

# Provide a custom field validation function for a height with units.
def compare_hgt_with_units(field: str, value: str, error: Callable[..., str]) -> None:
    if value.endswith("cm"):
        if not (150 <= int(value.rstrip("cm")) <= 193):
            error(field, "out of range")
    elif value.endswith("in"):
        if not (59 <= int(value.rstrip("in")) <= 76):
            error(field, "out of range")
    else:
        error(field, "missing units")
```

With a schema in place, all that’s left to do is instantiate a `Validator` and validate each document:

```python
>>> v = Validator(SCHEMA, require_all=True)
>>> v.validate(document)
True
```

Thanks, Cerberus!