Data-Oriented Programming is dope
Tags: python, programming, books, dataorientedprogramming
Published: 2022-01-03 06:07:38

<small>Photo by <a href="https://unsplash.com/@ravi_roshan_inc?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Ravi Roshan</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>.</small>
<small> This post was published first in https://techblog.deepki.com/data-oriented-programming/. This is an English translation by the same author.</small>

Data-oriented programming (DOP) is not a new concept. It's a paradigm that can be used by developers in any programming language, it's purpose is to reduce complexity of information system that they are designing.

[Yehonathan Sharvit](https://blog.klipse.tech/) explains that in his book [Data-oriented programming](https://www.manning.com/books/data-oriented-programming).
The book explores tenets of this paradigm, as a dialog between two people.

The narrator is a junior javascript developer, he develops a Library Management System of a client. Initial features are easy, the software is coded in object oriented. But when the client asks for some new features at the very last moment, everything becomes complicated. He seeks support from a veteran developper, Joe.

Throughout these pages Joe demonstrates difficulties he encounters, shows him how to confront them. _In fine_, he teaches him a new way to organize his source code that is more easy to decipher and to evolve.

---

Examples in the book are in Javascript, I wanted to introduce my interpretation in Python on a small part of these rules: code and data separation.

In the book, the heroes talk about user management. The narrator had to design two types of users, the librarian and the member:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/85f8ztl3lond67l8wprv.png)

Once this logic implemented, the client requested him to add super members, then VIP members.
It achieves to be this UML class diagram:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0rqxjbq1cpmwzw93tgu5.png)

It's really hard to manage for the young narrator. Although everything is perfectly logic, the classes hierarchy is hard to work with, mixing inheritance with dependencies.

Joe understands affirms that is "feelings" are due to « Data encapsulation has its merits and drawbacks: Think about the way you designed the Library Management System. According to DOP, the main cause of the complexity of systems and their lack of flexibility is because code and data are mixed together in objects »

That's what Yehonathan Sharvit fights all along the book: he depicts the difficulty to just understand something and be able to upgrade it without difficulties.

Complexity is a thing that have been accumulated insidiously. When it's not kept under control, implementing new features can take weeks instead of days. But DO comes with a radical approch to fight this complexity. To achieve this, data and code must be separated:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eq2dno3fk7m5117h0w6p.png)

In order to explain this separation, here is my implementation in python.

I've followed technics that are described in the book. I started from the client specifications, I've made a list of names that seem to represent Entities of the system, and another list of everything that look like a feature. Then I've organized what I've found:

* Two kind of users: **members** and **librarians**
* **Users** _log in_ to the system via email and password
* **Members** can _borrow **books**_
* **Members** and **librarians** can _search **books**_ by **title** or by **author**
* **Librarians** can _block_ and _unblock **members**_
* **Librarians** can _list the **books currently lent** by a **member**_
* There could be **several copies of a book**

Entities classified by groups:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pfd0qerc7c5aup9rboud.png)

Features put in several code modules:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vizx1mythj76ivmmhedz.png)

On this basis, I will implement book lending.

The **catalog**'s data part:

```yaml
$schema: "https://json-schema.org/draft/2020-12/schema"
properties:
  lendings:
    additionalProperties:
      type: object
      properties:
        id: { type: string }
        user_id: { type: string, format: uuid }
        book_item_id: { type: string }
      required: [id, user_email, book_item_id]
  propertyNames: { type: string, format: uuid }
required: [lendings]
```

The **user_management**'s data part:

```yaml
$schema: "https://json-schema.org/draft/2020-12/schema"
properties:
  members_by_id:
    type: object
    additionalProperties:
      type: object
      properties:
        is_blocked: { type: boolean }
      required: [is_blocked]
    propertyNames: { type: string, format: uuid }
required: [members_by_id]
```

Here I've used JSON Schema, because data does not have to be contained in rigid structures. Only the keys are relevant and need to be specified. In DO, data requires to obey three other rules:

* all types are generic
* all types are immutable
* shape of data and data schema are separated

Here is a mock that validates these two schemas:

```python
library_data = {
    "catalog": {
        "books_by_isbn": {
            "9781234567897": {
                "title": "Data Oriented Programming",
                "author": "Yehonathan Sharvit",
            }
        },
        "book_items_by_id": {
            "book-item-1": {
                "isbn": "9781617298578",
            },
            "book-item-2": {
                "isbn": "9781617298578",
            }
        },
        "lendings": [
            {
                "id": "...",
                "user_id": "member-1",
                "book_item_id": "book-item-1",
            }
        ],
    },
    "user_management": {
        "members_by_id": {
            "member-1": {
                "id": "member-1",
                "name": "Xavier B.",
                "email": "xavier@deepki.com",
                "password": "aG93IGRhcmUgeW91IQ==",
                "is_blocked": False,
            }
        }
    },
}
```

By convention, `dict` are used like some `Mapping`, and I forbid myself to update them.

Please note that examples will use the `classes+static` method form in order to make this article readable. In a production code, the `modules+functions` form is the way to go.

And now the code part:

```python
from __future__ import annotations

from typing import Tuple, TypeVar
from uuid import uuid4

T = TypeVar("T")


class Library:
    @staticmethod
    def checkout(library_data: T, user_id, book_item_id) -> tuple[T, dict]:
        user_management_data = library_data["user_management"]
        if not UserManagement.is_member(user_management_data, user_id):
            raise Exception("Only members can borrow books")
        if UserManagement.is_blocked(user_management_data, user_id):
            raise Exception("Member cannot borrow book because he is bloqued")
        catalog_data = library_data["catalog"]
        if not Catalog.is_available(catalog_data, book_item_id):
            raise Exception("Book is already borrowed")
        catalog_data, lending = Catalog.checkout(catalog_data, book_item_id, user_id)
        return (
            library_data | {
                "catalog": catalog_data,
            },
            lending,
        )


class UserManagement:
    @staticmethod
    def is_member(user_management_data: T, user_id) -> bool:
        return user_id in user_management_data["members_by_id"]

    @staticmethod
    def is_blocked(user_management_data: T, user_id) -> bool:
        return user_management_data["members_by_id"][user_id]["is_blocked"] is True


class Catalog:
    @staticmethod
    def is_available(catalog_data: T, book_item_id) -> bool:
        lendings = catalog_data["lendings"]
        return all(lending["book_item_id"] != book_item_id for lending in lendings)

    @staticmethod
    def checkout(catalog_data: T, book_item_id, user_id) -> Tuple[T, dict]:
        lending_id = uuid4().__str__()
        lending = {"id": lending_id, "user_id": user_id, "book_item_id": book_item_id}
        lendings = catalog_data["lendings"]
        return (
            catalog_data | {
                "lendings": lendings + [lending]
            },
            lending
        )
```

As we can see, code is a series of pure functions.
Functions that modify a state return a new state object rather than upgrading the previous state.

In each module, functions are made simple and easy to test. They can be reused in any context, like the main module. Globally, they are composed with other existing function. It becomes very easy to adapt them for the client's needs.

And now, which path will data lead if my alter-ego borrows another copy of the book?

```python
library_data, lending = Library.checkout(
    library_data,
    user_id="member-1",
    book_item_id="book-item-2",
)
```

Two things occur:

1. Data is systematically transmitted to every function calls. This object is quite opaque, each level use only a fragment that he knows without worrying about the remaining:

    ```python
    # 1. injects data into Library.checkout module
    library_data, lending = Library.checkout(library_data, ...)

    # 2. extracts data from user_management
    user_management_data = library_data["user_management"]

    # 3. uses this data fragment into UserManagement module
    if not UserManagement.is_member(user_management_data, ...):
        ...
    if UserManagement.is_blocked(user_management_data, ...):
        ...

    # 4. picks catalog data
    catalog_data = library_data["catalog"]

    # 5. uses this data fragment into Catalog module
    if not Catalog.is_available(catalog_data, ...):
        ...
    ... = Catalog.checkout(catalog_data, ...)
    ```

2. When a function is about to change a state, it returns a new version of data. Every level of the call stack must returns a new version of data:

    ```python
    # 1. handles the request in Catalog.checkout
    lending = ...
    lendings = catalog_data["lendings"]
    # 2. creates a new version of catalog_data
    catalog_data = catalog_data | {
        "lendings": lendings + [lending]
    }
    # 3. interception of the new catalog_data by Library.checkout
    catalog_data, ... = Catalog.checkout(...)
    # 4. creation of a new version of library_data
    library_data = library_data | {
        "catalog": catalog_data,
    }
    ```

    Then, this new version of data can be exposed to whole system.

In my example, I don't talk about data consolidation. I suggest you to read the book which gives informations concerning this subject.

## Is it pythonic?

Broadly speaking, this paradigm fits well in Python if we shelve the object oriented capabilities of the language.
The notion of modules in data-oriented are naturally superimposed on modules in Python, which facilitates adherence.

Borrowings of functional languages such as `map()`, `filter()` as well as functions from standard ``operator`` module also contribute to make this paradigm quite natural in Python.

In our example, [standard typing](https://techblog.deepki.com/typage-en-python/) will not work. However, it is quite easy to do custom typing, such as:

```python
from __future__ import annotations

from typing import Any, Mapping


class M(Mapping[str, Any]):
    def __or__(self, other: dict) -> M:
        return {**self, **other}  # type: ignore

    def __hash__(self) -> int:
        ...

# which can be used as well in the source code

def my_func(members_data: M[str, M], member_id: str):
    member = members_data[member_id]
```

## Code purity

In my implementation my functions are not really pure because I have used exceptions. However, this disgression is acceptable if it is applied under a certain condition. Indeed, exceptions are only used to express illegal operations, in a _throw early, catch late_ way. Using them this way contributes to the readability of the code. The higher layers of the system will know how to deal with them.

For example in a [Flask application](https://flask.palletsprojects.com/en/2.0.x/errorhandling/#error-handlers):

```python
@app.post("/checkout")
def checkout_view():
    ...
    try:
        ..., lending = Library.checkout(library_data, user_id, book_item_id)
        # la fonction 'checkout' peut lever des exceptions
        ...
        return jsonify(lending), 201
    except Exception as error:
        result = {"error": str(error)}
        return jsonify(result), 400
```

The author is also clear that embracing DO comes at a cost. For example, the fact that DO is relatively agnostic of any programming language undermines the guarantees offered by object modeling (or other tools such as code analysis that some IDEs allow). However, he sometimes offers alternatives for this, such as JSON Schema used here.

What I presented to you was just a preview of DOP using pure Python. The author gives a lot of details about unit tests, data structures, state management, structural sharing, atomicity, transformation pipeline, etc.

I highly recommend you to read [Data-oriented programming](https://www.manning.com/books/data-oriented-programming?utm_source=viebel&utm_medium=affiliate&utm_campaign=book_sharvit2_data_1_29_21&a_aid=viebel&a_bid=d5b546b7) by Yehonathan Sharvit, and please follow his approach [in his blog](https://blog.klipse.tech/visualization/2021/02/16/data-related-paradigms.html).

## Finally

The author is a multilingual developer and without citing it, many concepts come from the Clojure language. According to its defenders, [Clojure](https://clojure.org/) is the easiest programming language in the world because it has [almost no syntax or grammar](https://blog.cleancoder.com/uncle-bob/2019/08/22/WhyClojure.html) and it was designed by Rich Hickey in such a way as to [facilitate code changes](https://www.youtube.com/watch?v=2V1FtfBDsLU).

This language can be inspiring for other languages. To convince yourself, you can consult this other talk [Design, Composition, and Performance Short](https://www.youtube.com/watch?v=yoUPB62slns) by Rich Hickey.

It makes me happy to see some of these principles reused in other languages. Indeed, languages must nurture on each other.

_Bref_, making functional code in Python _is dope_.