Python: Data validation
Tags: python, data, validation
Published: 2023-01-20 19:31:16

**NOTE:** This was written in late 2022 with py3.9 in mind, so visitors of the future should take the follow with a pinch of salt.

When presented with the challenge to validate data as part of python script (when dealing with JSON message as part of an API), you need to validate the data incoming, and sometimes you might want to validate the data outgoing.

The present post will try to present several different ways to do this. First of all we need to define what the structure of the data looks like. In a typical business context it could look like the following:

* **User** object containing: first name, last name, date of birth (must be 18 years old or older), contact (optional) and address (optional)
* **Contact** object contains: email address (optional — must be valid) and phone number (optional — must be a valid UK phone number)
* **Address** object contains: street address (should include house number, city, postcode (must be a valid UK postcode) and country (must be one of England, Wales, Scotland & Northern Ireland)

Before we start try to validate anything we need to generate some data and try to leverage some existing library to validate things like postcode, email address and phone numbers; Doing that by hand reliably using regex is notoriously challenging.

# Some preliminary checks

Let’s also use the [Faker](https://faker.readthedocs.io/en/master/) library which is a powerful way to generate all sorts of data:

```python
from dateutil.relativedelta import relativedelta
from datetime import date, datetime
from typing import Union
from faker import Faker

fake = Faker("en_GB")
```
## Testing some of the logic we will reuse for validation

### Validating a date of birth as a string

```python
def is_underaged(date_of_birth: str, years: int = 18) -> Union[bool, None]:
    dob = datetime.strptime(date_of_birth, "%Y-%m-%d").date()
    eighteen_years_ago = date.today() - relativedelta(years=years)
    return dob > eighteen_years_ago
  
is_underaged("1992-04-01")
```
Output:
```sh
False
```

### Validating a phone number using [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers)

```python
import phonenumbers

my_number = fake.phone_number()

def validate_uk_phone_number(phone_number: str) -> bool:
    try:
        valid = phonenumbers.parse(phone_number, 'GB')
        return phonenumbers.is_valid_number(valid)
    except:
        return False

print(my_number, validate_uk_phone_number(my_number))
```
Output:
```sh
+44808 1570733 True
```

### Validating an email address using [validate_email](https://github.com/syrusakbary/validate_email)

```python
from validate_email import validate_email

email_address = fake.free_email()
is_valid = validate_email(email_address=email_address)

print(email_address, is_valid)
```
Output:
```sh
eur.olc.protection.outlook.com resolve error: The DNS response does not contain an answer to the question: eur.olc.protection.outlook.com. IN AAAA
patrickwright@hotmail.co.uk False
```

### Validating an email address using [email_validator](https://github.com/JoshData/python-email-validator)

```python
from email_validator import validate_email, EmailNotValidError

def validate_email_address(email_address: str, is_new_account: bool = True) -> str:
    try:
      validation = validate_email(email_address, check_deliverability=is_new_account)

      if validation.email:
          return True

      return False
    except EmailNotValidError as e:
      raise
    
valid_email_address = fake.free_email()

print(valid_email_address, validate_email_address(valid_email_address))
```
Output:
```sh
rgreen@hotmail.co.uk True
```

### Validating postcode using [postcode_validator_uk](https://github.com/guimunarolo/postcode-validator-uk)

```python
from postcode_validator_uk.validators import UKPostcode

postcode = fake.postcode()

validate_postcode = True if UKPostcode(postcode) else False

print(postcode, validate_postcode)
```
Output:
```sh
W3 8WP True
```
## Refresher about dates manipulation
```python
from datetime import datetime

dob = "1995-11-28"
datetime.strptime(dob, "%Y-%m-%d").date()
```

Output:
```sh
datetime.date(1995, 11, 28)
```

### Faking a bunch of valid date of birth

```python
[
  fake.date_between_dates(
    date_start=date(1970, 1, 1),
    date_end=(date.today() - relativedelta(years=18))
  ).strftime("%Y-%m-%d") for _ in range(10)
]
```
Output:
```sh
['1981-04-11',
 '1971-02-23',
 '1995-03-23',
 '1993-01-18',
 '2001-12-03',
 '1975-03-24',
 '1996-06-01',
 '1972-05-29',
 '1978-04-01',
 '1988-09-22']
 ```
 
### Creating a faker provider for the date of birth and the UK country

We don't want to repeat that logic too many times, so let's create a faker provider to hold that for us.

```python
from faker.providers import BaseProvider

class CustomProvider(BaseProvider):
  def date_of_birth(self, is_underaged: bool = False):
    start_date = date.today() - relativedelta(years=18) if is_underaged else date(1970, 1, 1)
    end_date = date.today() if is_underaged else date.today() - relativedelta(years=18)
    dob = self.generator.date_between_dates(date_start=start_date, date_end=end_date)
    return dob.strftime("%Y-%m-%d")
  
  def uk_home_country(self):
    return self.generator.random_element(
      elements=["England", "Wales", "Scotland", "Northern Ireland"]
    )
  
fake.add_provider(CustomProvider)

print(fake.date_of_birth())
print(fake.date_of_birth(is_underaged=True))
print(fake.uk_home_country())
```
Output:
```sh
1989-09-01
2009-06-07
Scotland
```

## Common data

Let's create some data to test all the different solutions
```python
data = [
    {
      "first_name": fake.first_name(),
      "last_name": fake.last_name(),
      "dob": fake.date_of_birth()
    },
    {
      "first_name": fake.first_name(),
      "last_name": fake.last_name(),
      "dob": fake.date_of_birth(),
      "contact": {}
    },
    {
      "first_name": fake.first_name(),
      "last_name": fake.last_name(),
      "dob": fake.date_of_birth(),
      "contact": {
        "phone_number": fake.phone_number()
      },
      "address": {
        "street": fake.street_address(),
        "city": fake.city(),
        "country": fake.uk_home_country(),
        "postcode": fake.postcode()
      }
    },
    {
      "first_name": fake.first_name(),
      "last_name": fake.last_name(),
      "dob": fake.date_of_birth(),
      "contact": {
        "email_address": fake.free_email()
      },
      "address": {
        "street": fake.street_address(),
        "city": fake.city(),
        "country": fake.uk_home_country(),
        "postcode": fake.postcode()
      }
    },
]

data
```
Output:
```sh
[{'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'},
 {'first_name': 'Lydia',
  'last_name': 'Lawrence',
  'dob': '1973-12-25',
  'contact': {}},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419'},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```

# Validating using the standard library

## Create a helper class to dump the data
```python
import json

class Serialiser:
  def __items(self):
    props = {}
    for k, v in self.__dict__.items():
      if k.startswith("_"):
        k = k[1:]
      props[k] = v

    return props

  def __repr__(self):
    return json.dumps(self, default=lambda o: o.__items())

class Example(Serialiser):
  def __init__(self, id, name):
    self.id = id
    self.name = name
    
ex = Example(1, "foo")
ex
```
Output:
```sh
{"id": 1, "name": "foo"}
```

## Using validation functions

This is probably the most basic solution out there
```python
from typing import Optional
import json
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

class Contact(Serialiser):
  def __init__(self, phone_number: Optional[str] = None, email_address: Optional[str] = None):
    if phone_number:
      self.phone_number = self._is_valid_uk_phone_number(phone_number)
    if email_address:
      self.email_address = self._is_valid_email_address(email_address)

  def _is_valid_uk_phone_number(self, value):
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
      raise ValueError("Invalid UK phone number.")
    return value
  
  def _is_valid_email_address(self, value):
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    return value

class Address(Serialiser):
  def __init__(self, street: str, postcode: str, country: str, city: str):
    self.street = street
    self.postcode = self._is_valid_uk_postcode(postcode)
    self.country = self._is_valid_home_country(country)
    self.city = city
    
  def _is_valid_uk_postcode(self, value):
    if not UKPostcode(value):
      raise ValueError("Invalid UK postcode.")
    return value
  
  def _is_valid_home_country(self, value):
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")
    return value

class Person(Serialiser):
  def __init__(
    self,
    first_name: str,
    last_name: str,
    dob: str,
    contact: Optional[Contact] = None,
    address: Optional[Address] = None
  ):
    self.first_name = first_name
    self.last_name = last_name
    self.dob = self._legal_age(dob)
    if contact:
      self.contact = contact
    if address:
      self.address = address
    
  def _legal_age(self, value: str) -> str:
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage.")
    return value
  
[Person(**d) for d in data]
```
Output:
```sh
[{"first_name": "Gregory", "last_name": "Brooks", "dob": "1996-11-18"},
 {"first_name": "Lydia", "last_name": "Lawrence", "dob": "1973-12-25"},
 {"first_name": "Timothy", "last_name": "Hilton", "dob": "2002-03-10", "contact": {"phone_number": "+44191 496 0419"}, "address": {"street": "Studio 5\nTom union", "city": "North Carolineton", "country": "Wales", "postcode": "AL2R 0BL"}},
 {"first_name": "Kerry", "last_name": "Brown", "dob": "1979-09-12", "contact": {"email_address": "angelaturner@hotmail.co.uk"}, "address": {"street": "Flat 81\nSimmons courts", "city": "Marcton", "country": "England", "postcode": "IG6 9NH"}}]
```

### Using [property](https://docs.python.org/3/library/functions.html#property)

NOTE: only the properties where validation is required are needed

```python
from typing import Optional
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

class Contact(Serialiser):
  def __init__(self, phone_number: Optional[str] = None, email_address: Optional[str] = None):
    if phone_number:
      self._phone_number = phone_number
    if email_address:
      self._email_address = email_address

  @property
  def phone_number(self):
    return self._phone_number
  
  @phone_number.setter
  def phone_number(self, value):
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
      raise ValueError("Invalid UK phone number.")
    self._phone_number = value
  
  @property
  def email_address(self):
    return self._email_address
  
  @email_address.setter
  def email_address(self, value):
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    self._email_address = value

class Address(Serialiser):
  def __init__(self, street: str, postcode: str, country: str, city: str):
    self._street = street
    self._postcode = postcode
    self._country = country
    self._city = city
  
  @property
  def street(self):
    return self._street
  
  @street.setter
  def street(self, value):
    self._street = value
  
  @property
  def postcode(self):
    return self._postcode
  
  @postcode.setter
  def postcode(self, value):
    if not UKPostcode(value):
      raise ValueError("Invalid UK postcode.")
    self._postcode = value
  
  @property
  def country(self):
    return self._country
  
  @country.setter
  def country(self, value):
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")
    self._country = value
  
  @property
  def city(self):
    return self._city
  
  @city.setter
  def city(self, value):
    self._city = value

class Person(Serialiser):
  def __init__(
    self,
    first_name: str,
    last_name: str,
    dob: str,
    contact: Optional[Contact] = None,
    address: Optional[Address] = None
  ):
    self._first_name = first_name
    self._last_name = last_name
    self._dob = dob
    if contact:
      self.contact = contact
    if address:
      self.address = address
  
  @property
  def first_name(self):
    return self._first_name
  
  @first_name.setter
  def first_name(self, value):
    self._first_name = value
    
  @property
  def last_name(self):
    return self._last_name
  
  @last_name.setter
  def last_name(self, value):
    self._last_name = value
    
  @property
  def dob(self):
    return self._dob
  
  @dob.setter
  def dob(self, value):
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage.")
    self._dob = value

[Person(**d) for d in data]
```
Output
```sh
[{"first_name": "Gregory", "last_name": "Brooks", "dob": "1996-11-18"},
 {"first_name": "Lydia", "last_name": "Lawrence", "dob": "1973-12-25"},
 {"first_name": "Timothy", "last_name": "Hilton", "dob": "2002-03-10", "contact": {"phone_number": "+44191 496 0419"}, "address": {"street": "Studio 5\nTom union", "city": "North Carolineton", "country": "Wales", "postcode": "AL2R 0BL"}},
 {"first_name": "Kerry", "last_name": "Brown", "dob": "1979-09-12", "contact": {"email_address": "angelaturner@hotmail.co.uk"}, "address": {"street": "Flat 81\nSimmons courts", "city": "Marcton", "country": "England", "postcode": "IG6 9NH"}}]
 ```

### Using [Descriptors](https://docs.python.org/3/howto/descriptor.html#descriptor-howto-guide)

```python
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

class EmailAddress:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    self.value = value

class UKPhoneNumber:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
      raise ValueError("Invalid UK phone number.")
    self.value = value

class Contact(Serialiser):
  email_address = EmailAddress()
  phone_number = UKPhoneNumber()
  
  def __init__(self, phone_number: str = None, email_address: str = None):
    self.phone_number = phone_number
    self.email_address = email_address
    
class UKPostcode:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not UKPostcode(value):
      raise ValueError("Invalid UK postcode.")
    self.value = value
    
class UKHomeCountry:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")
    self.value = value
    
class Address(Serialiser):
  postcode = UKPostcode()
  country = UKHomeCountry()
  
  def __init__(self, street: str, postcode: str, country: str, city: str):
    self.street = street
    self.postcode = postcode
    self.city = city
    self.country = country
    
class LegalAge:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage.")
    self.value = value

class Person(Serialiser):
  dob = LegalAge()
  
  def __init__(
    self,
    first_name: str,
    last_name: str,
    dob: str,
    contact: Optional[Contact] = None,
    address: Optional[Address] = None
  ):
    self.first_name = first_name
    self.last_name = last_name
    self.dob = dob
    if contact:
      self.contact = contact
    if address:
      self.address = address
    
[Person(**d) for d in data]
```
Output:
```sh
[{"first_name": "Gregory", "last_name": "Brooks"},
 {"first_name": "Lydia", "last_name": "Lawrence"},
 {"first_name": "Timothy", "last_name": "Hilton", "contact": {"phone_number": "+44191 496 0419"}, "address": {"street": "Studio 5\nTom union", "city": "North Carolineton", "country": "Wales", "postcode": "AL2R 0BL"}},
 {"first_name": "Kerry", "last_name": "Brown", "contact": {"email_address": "angelaturner@hotmail.co.uk"}, "address": {"street": "Flat 81\nSimmons courts", "city": "Marcton", "country": "England", "postcode": "IG6 9NH"}}]
 ```
 
 ### Using Decorator and Descriptor
 
 ```python
 import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

class EmailAddress:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    self.value = value

def email_address(attr: str):
  def decorator(cls):
    setattr(cls, attr, EmailAddress())
    return cls
  return decorator

class UKPhoneNumber:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
      raise ValueError("Invalid UK phone number.")
    self.value = value

def uk_phone_number(attr: str):
  def decorator(cls):
    setattr(cls, attr, UKPhoneNumber())
    return cls
  return decorator

@email_address("email_address")
@uk_phone_number("phone_number")
class Contact(Serialiser):
  def __init__(self, phone_number: str = None, email_address: str = None):
    self.phone_number = phone_number
    self.email_address = email_address
    
class UKPostcode:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not UKPostcode(value):
      raise ValueError("Invalid UK postcode.")
    self.value = value

def uk_postcode(attr: str):
  def decorator(cls):
    setattr(cls, attr, UKPostcode())
    return cls
  return decorator
    
class UKHomeCountry:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")
    self.value = value

def uk_home_country(attr: str):
  def decorator(cls):
    setattr(cls, attr, UKHomeCountry())
    return cls
  return decorator

@uk_postcode("postcode")
@uk_home_country("country")
class Address(Serialiser):
  def __init__(self, street: str, postcode: str, country: str, city: str):
    self.street = street
    self.postcode = postcode
    self.city = city
    self.country = country
    
class LegalAge:
  def __get__(self, obj, objtype=None):
    return self.value

  def __set__(self, obj, value):
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage.")
    self.value = value


def legal_age(attr: str):
  def decorator(cls):
    setattr(cls, attr, LegalAge())
    return cls
  return decorator

@legal_age("dob")
class Person(Serialiser):
  def __init__(
    self,
    first_name: str,
    last_name: str,
    dob: str,
    contact: Optional[Contact] = None,
    address: Optional[Address] = None
  ):
    self.first_name = first_name
    self.last_name = last_name
    self.dob = dob
    if contact:
      self.contact = contact
    if address:
      self.address = address
    
[Person(**d) for d in data]
```
Output:
```sh
[{"first_name": "Gregory", "last_name": "Brooks"},
 {"first_name": "Lydia", "last_name": "Lawrence"},
 {"first_name": "Timothy", "last_name": "Hilton", "contact": {"phone_number": "+44191 496 0419"}, "address": {"street": "Studio 5\nTom union", "city": "North Carolineton", "country": "Wales", "postcode": "AL2R 0BL"}},
 {"first_name": "Kerry", "last_name": "Brown", "contact": {"email_address": "angelaturner@hotmail.co.uk"}, "address": {"street": "Flat 81\nSimmons courts", "city": "Marcton", "country": "England", "postcode": "IG6 9NH"}}]
```

### Using [@dataclass](https://docs.python.org/3/library/dataclasses.html) and [dataclass_json](https://lidatong.github.io/dataclasses-json/reference/dataclasses_json/api/)

```python
from datetime import date
from typing import Optional
from dataclasses import dataclass, field
from dataclasses_json import dataclass_json, config
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

def exclude_optional_dict(value):
  return value is None or not value

@dataclass_json
@dataclass
class Contact:
  email_address: Optional[str] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  phone_number: Optional[str] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  
  def __post_init__(self):
    if not validate_email(self.email_address, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    if not phonenumbers.is_valid_number(phonenumbers.parse(self.phone_number, "GB")):
      raise ValueError("Invalid UK phone number.")

@dataclass_json
@dataclass
class Address:
  street: str
  postcode: str
  city: str
  country: str
  
  def __post_init__(self):
    if not UKPostcode(self.postcode):
      raise ValueError("Invalid UK postcode.")
    if self.country not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")

@dataclass_json
@dataclass
class Person:
  first_name: str
  last_name: str
  dob: str
  contact: Optional[Contact] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  address: Optional[Address] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  
  def __post_init__(self):
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(self.dob, "%Y-%m-%d").date():
      raise ValueError("Underage.")

[Person(**d).to_dict() for d in data]
```
Output:
```sh
[{'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'},
 {'first_name': 'Lydia', 'last_name': 'Lawrence', 'dob': '1973-12-25'},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419'},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```

### Using @dataclass with custom Validation class

https://gist.github.com/rochacbruno/978405e4839142e409f8402eece505e8

```python
from datetime import date
from typing import Optional
from dataclasses import dataclass, field
from dataclasses_json import dataclass_json, config
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

def exclude_optional_dict(value):
  return value is None or not value


class Validations:
    def __post_init__(self):
        """Run validation methods if declared.
        The validation method can be a simple check
        that raises ValueError or a transformation to
        the field value.
        The validation is performed by calling a function named:
            `validate_<field_name>(self, value, field) -> field.type`
        """
        for name, field in self.__dataclass_fields__.items():
            if (method := getattr(self, f"validate_{name}", None)):
                setattr(self, name, method(getattr(self, name), field=field))

@dataclass_json
@dataclass
class Contact(Validations):
  email_address: Optional[str] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  phone_number: Optional[str] = field(metadata=config(exclude=exclude_optional_dict), default=None)

  def validate_email_address(self, value, **_) -> str:
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid Email address.")
    return value
  
  def validate_phone_number(self, value, **_) -> str:
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
      raise ValueError("Invalid UK phone number.")
    return value

@dataclass_json
@dataclass
class Address(Validations):
  street: str
  postcode: str
  city: str
  country: str
  
  def validate_postcode(self, value, **_) -> str:
    if not UKPostcode(value):
      raise ValueError("Invalid UK postcode.")
    return value
    
  def validate_country(self, value, **_) -> str:
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Invalid home country")
    return value

@dataclass_json
@dataclass
class Person:
  first_name: str
  last_name: str
  dob: str
  contact: Optional[Contact] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  address: Optional[Address] = field(metadata=config(exclude=exclude_optional_dict), default=None)
  
  def validate_dob(self, value, **_) -> str:
    if not (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage.")
    return value

[Person(**d).to_dict() for d in data]
```
Output:
```sh
[{'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'},
 {'first_name': 'Lydia', 'last_name': 'Lawrence', 'dob': '1973-12-25'},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419'},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```

## Validation using [Schema](https://github.com/keleshev/schema)

```python
from schema import Schema, And, Use, Optional
from datetime import date
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

legal_age = lambda d: (date.today() - relativedelta(years=18)) > datetime.strptime(d, "%Y-%m-%d").date()
valid_uk_phonenumber = lambda d: phonenumbers.is_valid_number(phonenumbers.parse(d, "GB"))
valid_email_address = lambda d: validate_email(d, check_deliverability=True).email
valid_postcode = lambda d: UKPostcode(d)

schema = Schema(
  [
    {
      "first_name": And(str, len),
      "last_name": And(str, len),
      "dob": And(Use(str), legal_age),
      Optional("contact"): {
        Optional("phone_number"): And(Use(str), valid_uk_phonenumber),
        Optional("email_address"): And(Use(str), valid_email_address),
      },
      Optional("address"): {
        "street": And(Use(str), len),
        "postcode": And(Use(str), valid_postcode),
        "country": And(Use(str), lambda d: d in ["England", "Wales", "Scotland", "Northern Ireland"]),
        "city": And(Use(str), len),
      }
    }
  ]
)

persons = schema.validate(data)
persons
```
Output:
```sh
[{'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'},
 {'first_name': 'Lydia',
  'last_name': 'Lawrence',
  'dob': '1973-12-25',
  'contact': {}},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419'},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```

## Validation using [Cerberus](https://docs.python-cerberus.org/en/stable/)

```python
from cerberus import Validator
from datetime import date
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

def legal_age(field, value, error):
  try:
    if not bool(
      (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date()
    ):
      error(field, "Underage")
  except TypeError:
      error(field, "Underage")

def valid_uk_phonenumber(field, value, error):
  try:
    if not bool(phonenumbers.is_valid_number(phonenumbers.parse(value, "GB"))):
      error(field, "Invalid phone number.")
  except TypeError:
      error(field, "Invalid phone number.")

def valid_email_address(field, value, error):
  try:
    if not bool(validate_email(value, check_deliverability=True).email):
      error(field, "Invalid email address")
  except TypeError:
      error(field, "Invalid email address")

def valid_postcode(field, value, error):
  try:
    if not bool(UKPostcode(value)):
      error(field, "Invalid Postcode")
  except TypeError:
      error(field, "Invalid Postcode")
      
def valid_country(field, value, error):
  try:
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      error(field, "Invalid Postcode")
  except TypeError:
      error(field, "Invalid Postcode")
    

schema = {
  "first_name": {"type": "string", "required": True},
  "last_name": {"type": "string", "required": True},
  "dob": {"validator": legal_age, "required": True},
  "contact": {
    "type": "dict",
    "schema": {
      "phone_number": {"validator": valid_uk_phonenumber},
      "email_address": {"validator": valid_email_address},
    }
  },
  "address": {
    "type": "dict",
    "schema": {
      "street": {"type": "string", "required": True},
      "postcode": {"validator": valid_postcode, "required": True},
      "country": {"validator": valid_country, "required": True},
      "city": {"type": "string", "required": True},
    }
  }
}

v = Validator(schema)

for d in data:
  if not v.validate(d):
    print(v.errors)
  else:
    print("Valid Schema", d)
```
Output:
```sh
Valid Schema {'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'}
Valid Schema {'first_name': 'Lydia', 'last_name': 'Lawrence', 'dob': '1973-12-25', 'contact': {}}
Valid Schema {'first_name': 'Timothy', 'last_name': 'Hilton', 'dob': '2002-03-10', 'contact': {'phone_number': '+44191 496 0419'}, 'address': {'street': 'Studio 5\nTom union', 'city': 'North Carolineton', 'country': 'Wales', 'postcode': 'AL2R 0BL'}}
Valid Schema {'first_name': 'Kerry', 'last_name': 'Brown', 'dob': '1979-09-12', 'contact': {'email_address': 'angelaturner@hotmail.co.uk'}, 'address': {'street': 'Flat 81\nSimmons courts', 'city': 'Marcton', 'country': 'England', 'postcode': 'IG6 9NH'}}
```

### Cerberus Validation with class

```python
from cerberus import Validator
from datetime import date
import phonenumbers
from email_validator import validate_email
from postcode_validator_uk.validators import UKPostcode

class CustomValidator(Validator):
  def _validate_type_legal_age(self, value):
    try:
        if (date.today() - relativedelta(years=18)) > datetime.strptime(value, "%Y-%m-%d").date():
          return True
    except TypeError:
        self._error(self._field, "Underage")

  def _validate_type_uk_phonenumber(self, value):
    try:
      if phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
        return True
    except TypeError:
        self._error(self._field, "Invalid phone number.")

  def _validate_type_email_address(self, value):
    try:
      if validate_email(value, check_deliverability=True).email:
        return True
    except TypeError:
        self._error(self._field, "Invalid email address")

  def _validate_type_postcode(self, value):
    try:
      if UKPostcode(value):
          return True
    except TypeError:
        self._error(self._field, "Invalid Postcode")
        
  def _validate_type_country(self, value):
    try:
      if value in ["England", "Wales", "Scotland", "Northern Ireland"]:
        return True
    except TypeError:
        self._error(self._field, "Invalid Postcode")

schema = {
  "first_name": {"type": "string", "required": True},
  "last_name": {"type": "string", "required": True},
  "dob": {"type": "legal_age", "required": True},
  "contact": {
    "type": "dict",
    "schema": {
      "phone_number": {"type": "uk_phonenumber"},
      "email_address": {"type": "email_address"},
    }
  },
  "address": {
    "type": "dict",
    "schema": {
      "street": {"type": "string", "required": True},
      "postcode": {"type": "postcode", "required": True},
      "country": {"type": "country", "required": True},
      "city": {"type": "string", "required": True},
    }
  }
}

v = CustomValidator(schema)

for d in data:
  if not v.validate(d):
    print(v.errors)
  else:
    print("Valid Schema", d)
```
Output:
```sh
Valid Schema {'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'}
Valid Schema {'first_name': 'Lydia', 'last_name': 'Lawrence', 'dob': '1973-12-25', 'contact': {}}
Valid Schema {'first_name': 'Timothy', 'last_name': 'Hilton', 'dob': '2002-03-10', 'contact': {'phone_number': '+44191 496 0419'}, 'address': {'street': 'Studio 5\nTom union', 'city': 'North Carolineton', 'country': 'Wales', 'postcode': 'AL2R 0BL'}}
Valid Schema {'first_name': 'Kerry', 'last_name': 'Brown', 'dob': '1979-09-12', 'contact': {'email_address': 'angelaturner@hotmail.co.uk'}, 'address': {'street': 'Flat 81\nSimmons courts', 'city': 'Marcton', 'country': 'England', 'postcode': 'IG6 9NH'}}
```

## Validation using [Pydantic](https://pydantic-docs.helpmanual.io/)

```python
from typing import Optional
from pydantic import BaseModel, validator, Field

class Address(BaseModel):
  street: str
  city: str
  country: str
  postcode: str

  @validator("postcode")
  def postcode_is_valid(cls, value):
    if not UKPostcode(value):
      raise ValueError("Must be a valid UK Postcode.")
    return value
  
  @validator("country")
  def country_of_the_uk(cls, value):
    if value not in ["England", "Wales", "Scotland", "Northern Ireland"]:
      raise ValueError("Must be a country from the UK.")
    return value

class Contact(BaseModel):
  phone_number: Optional[str] = None
  email_address: Optional[str] = None

  @validator("phone_number")
  def valid_uk_phonenumber(cls, value):
    if not phonenumbers.is_valid_number(phonenumbers.parse(value, "GB")):
        raise ValueError("Must be a valid UK phone number.")
    return value

  @validator("email_address")
  def _validate_type_email_address(cls, value):
    if not validate_email(value, check_deliverability=True).email:
      raise ValueError("Invalid email address")
    return value
        
class Person(BaseModel):
  first_name: str
  last_name: str
  dob: str
  contact: Optional[Contact] = None
  address: Optional[Address] = None
  
  @validator("dob")
  def legal_age(cls, value):
    if (date.today() - relativedelta(years=18)) < datetime.strptime(value, "%Y-%m-%d").date():
      raise ValueError("Underage")
    return value

persons = [Person(**d).dict() for d in data]
persons
```
Output:
```sh
[{'first_name': 'Gregory',
  'last_name': 'Brooks',
  'dob': '1996-11-18',
  'contact': None,
  'address': None},
 {'first_name': 'Lydia',
  'last_name': 'Lawrence',
  'dob': '1973-12-25',
  'contact': {'phone_number': None, 'email_address': None},
  'address': None},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419', 'email_address': None},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'phone_number': None,
   'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```

### Pydantic [excluding None values](https://stackoverflow.com/questions/65362524/in-json-created-from-a-pydantic-basemodel-exclude-optional-if-not-set) recipe

```python
def union(source, destination):
    for key, value in source.items():
        if isinstance(value, dict):
            node = destination.setdefault(key, {})
            union(value, node)
        else:
            destination[key] = value

    return destination

def exclude_optional_dict(model: BaseModel):
    return union(model.dict(exclude_unset=True), model.dict(exclude_none=True))
  
persons = [exclude_optional_dict(Person(**d)) for d in data]
persons
```
Output:
```sh
[{'first_name': 'Gregory', 'last_name': 'Brooks', 'dob': '1996-11-18'},
 {'first_name': 'Lydia',
  'last_name': 'Lawrence',
  'dob': '1973-12-25',
  'contact': {}},
 {'first_name': 'Timothy',
  'last_name': 'Hilton',
  'dob': '2002-03-10',
  'contact': {'phone_number': '+44191 496 0419'},
  'address': {'street': 'Studio 5\nTom union',
   'city': 'North Carolineton',
   'country': 'Wales',
   'postcode': 'AL2R 0BL'}},
 {'first_name': 'Kerry',
  'last_name': 'Brown',
  'dob': '1979-09-12',
  'contact': {'email_address': 'angelaturner@hotmail.co.uk'},
  'address': {'street': 'Flat 81\nSimmons courts',
   'city': 'Marcton',
   'country': 'England',
   'postcode': 'IG6 9NH'}}]
```