Python to JSON and Back
Tags: python, json, api, rapidjson
Published: 2020-04-24 00:33:36

<cite>Photo by [David Clode](https://unsplash.com/@davidclode?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/python?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)</cite>

As JavaScript Object Notation ([JSON](https://en.wikipedia.org/wiki/JSON)) has become the ubiquitous language of communication between APIs on the web, being able to take [Python](https://www.python.org/) objects and serialize them to JSON strings is an important part of communicating with other services, clients, APIs, what have you.

The default Python [`json`](https://docs.python.org/3/library/json.html) module supports both [`dump`](https://docs.python.org/3/library/json.html#json.dump) and [`dumps`](https://docs.python.org/3/library/json.html#json.dumps) methods - the more common one used is `dumps` - dumping the object to a JSON String - all based on this [Conversion Table](https://docs.python.org/3/library/json.html#py-to-json-table) - which pointedly omits the `datetime` objects - most importantly because the [JSON Specification](https://www.json.org/) has no reference to any specific date/time format.
This is left up to the implementer, based on what format they need - leaving coders everywhere figuring out how to Do The Right Thing - since any date/time mismatches have potential to cause all manner of havoc if incorrectly parsed. If you want to read more on time, check out [this collection of articles](https://github.com/kdeldycke/awesome-falsehood#dates-and-time).

A simple demonstration of the failure condition looks like this:
```python
>>> import datetime
>>> import json
>>> json.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File ".../lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File ".../lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File ".../3.6.8/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable
```

And if you've hit this, and search the internet, you'll find all manner of approaches to resolve - a common approach seen is [implementing a function](https://stackoverflow.com/a/3235787) that iterates through an object, and if it finds a `datetime` value, convert it to the [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, with [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat). Example of the conversion step:

```python
>>> json.dumps(datetime.datetime.utcnow().isoformat())
'"2020-04-23T23:36:36.620619"'
```

However this then becomes custom code to iterate through objects, and sometimes we may need to iterate through deeply nested objects - for example a blog page, its posts and comments in a single JSON payload may all have timestamps that would need to be iterated through, detected, converted, and then `json.dumps()` the whole thing.

Some popular frameworks may handle this for you automatically - notable callouts are the Django REST Framework [`DateTimeField`](https://www.django-rest-framework.org/api-guide/fields/#datetimefield) will default to ISO8601, as will the excellent [`pydantic`](https://pydantic-docs.helpmanual.io/) library (here's the [implementation under the hood](https://github.com/samuelcolvin/pydantic/blob/cab6313798765d634e97e3a923c48dfe64367775/pydantic/json.py#L16-L17)).

However, oftentimes we may not want to take on importing a framework or a full-fledged library, but still want to avoid having to create custom solutions like this - and that's where we can lean on well-trodden paths from library maintainers, like the [`rapidjson`](https://rapidjson.org/) library - and more importantly, the [Python wrapper to this fast C library](https://python-rapidjson.readthedocs.io/).

## `rapidjson`

`rapidjson` aims to be a swap-in replacement for the native Python `json` module, and allows the coder to extend the behavior to support more behaviors with less effort. **Note:** This package must be installed via the [`python-rapidjson`](https://pypi.org/project/python-rapidjson/) package name.

*Keep in mind:* there are a [few incompatibilities](https://python-rapidjson.readthedocs.io/en/latest/quickstart.html#incompatibilities) you should be aware of when using `rapidjson` - but most implementations may not be using these, in fact the coercion of `"true"` strings to Python `True` values is actually an improvement!

So if we were to run our example from above, replacing `json` with `rapidjson`, we get the same result, albeit the traceback is a little clearer now:

```python
>>> import datetime
>>> import rapidjson
>>> rapidjson.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: datetime.datetime(2020, 4, 24, 0, 11, 16, 105719) is not JSON serializable
```

However now we can pass an argument to `rapidjson.dumps()` to inform it how we want to handle `datetime` instances, like so:
```python
>>> rapidjson.dumps(datetime.datetime.utcnow(), datetime_mode=rapidjson.DM_ISO8601)
'"2020-04-24T00:14:35.531078"'
```

Voilá, we've got a valid ISO8601-formatted string of a `datetime` with little added!

I've put together a more complete example of full object deserialization and loading below - feel free to run, fork, extend, experiment with the code, and read more about the [other `datetime_mode` options in `rapidjson`](https://python-rapidjson.readthedocs.io/en/latest/dumps.html#dumps-datetime-mode).

Enjoy!

{% replit @miketheman/TartUsablePipelining %}