Automatic synchronization of Django services
Tags: python, django, microservices, opensource
Published: 2020-06-18 22:18:50

> Using microservices is like going to the bar: it’s great when the party starts, but you can only hope that you won’t have a headache in the morning.

Please, don’t get me wrong. Microservices are definetly a great pattern, but as always, it comes with the cost. Let me briefly describe one of the main technical problems that you can face dealing with microservices.

## Problem 🧐

Each business service operates in a certain subdomain/[bounded context](https://martinfowler.com/bliki/BoundedContext.html), which may be stored in a dedicated database. As services are distributed over the network, data is also distributed. And as business domains are pretty complex, one domain object may be shared between several contexts. In other words, we need the same pieces of data in different loosely related isolated environments. That sounds like a problem! Let’s find a solution.

## Theoretical Solution 📚

[CQRS](https://martinfowler.com/bliki/CQRS.html) is rushing to help us! __C__ommand __Q__uery __R__esponsibility __S__egregation is the pattern that separates read/query and update/command operations for a data store. This pattern can be applied within one service (or even one DB instance), as well as adopted to be used in different isolated services with separate stores.

![CQRS usage examples](https://dev-to-uploads.s3.amazonaws.com/i/kao2j1u0aug5d0cmdv1j.PNG)

Examples:
* __A__: simple views to expand ORM bounds or materialized views to avoid complex joins and optimize queries
* __B__: DB shards with one write instance and several read instances; different DB types, where write type is SQL and read one is a NoSQL search engine, like [ElasticSearch](https://www.elastic.co/elasticsearch/)
* __C__: Shared domain data between several services with one producing service and several consumers for solving performance issues like API JOINs (our case!)

If you are already thinking about integrating it into your system, please, think twice. CQRS is a great tool for the right job, but there are several issues and considerations, which you need to be aware of:

* CQRS is simple in theory, but can be pretty complex in practice (especially when [Event Sourcing](https://microservices.io/patterns/data/event-sourcing.html) is added to the mix)
* Good messaging system will be needed for data synchronization between services (simple HTTP is rarely enough due to many reasons, like message failures)
* Data will only be __eventually__ (!) consistent (that may not fit for business critical transactions: there are other patterns for such cases)

Let’s now look at the real world example, where CQRS pattern could be exceptionally useful:

![CQRS Domain Example](https://dev-to-uploads.s3.amazonaws.com/i/gj5ljmhw5wko2fii3m3t.PNG)

Any supply-chain automation platform must be able to register incoming orders from customers. There may be millions of orders monthly, so the load is pretty high. The platform may have several certified integrations, which need to receive webhooks on different order actions (e.g. on order creation). And our UI portal must be able to show some aggregated daily/monthly stats for processed orders.

Technically that means that we have:

* Orders microservice with CRUD `/orders` REST API and own SQL DB — source/master service for *Order* domain objects
* *Webhooks* microservice, that sends outbound HTTP messages to our integrations (this service may have other [SLAs](https://en.wikipedia.org/wiki/Service-level_agreement) than *Orders*). This service has DB to store replicated order data for schedules, manual triggers or execution history.
* *Stats* service with read only `/stats/orders` Public API, that provides aggregated orders data within some time range. Incoming orders data (replica) are stored to avoid data duplication (and maybe some NoSQL DB like [Mongo](https://www.mongodb.com/) is used). There are several CQRS consumers, which simultaneously pull messages from the queue.
* Reliable fast message transport like [RabbitMQ](https://www.rabbitmq.com/) or [Apache Kafka](https://kafka.apache.org/) with support for persistence and ability to send messages to several consumers like *Webhooks* or *Stats* services.

I will try to dispel some of you doubts immediately. We could technically merge all three services into one monolithic *Super-Orders* service to avoid orders data duplication. But the cost of development and operating of such service would be extra high. *Webhooks* may run on the asynchronous engine, while *Orders* may be a simple synchronous service. Scaling strategies and SLAs differ for these services, so it’s not reasonable to merge them together. *Stats* could be the part of *Orders* until there is a common algorithm or functionality for collecting and aggregating stats from different domain objects. Stats could be aggregated by some background task or cron job that would add additional load to a mission-critical *Orders* DB. In a long run separation of *Stats* is the most reasonable solution.

I hope that CQRS theory is pretty clear at this moment. If you need deeper integration, here are some helpful links for you:
* [CQRS article from Microsoft](https://docs.microsoft.com/en-gb/azure/architecture/patterns/cqrs)
* [Microservices Patterns - CQRS](https://microservices.io/patterns/data/cqrs.html)

Let’s jump to practice already!

## Practical Solution 💡

{% github cloudblue/django-cqrs no-readme %}

Meet __Django-CQRS__ (!)— the open source Django Application for automatic data synchronization between several Django services. This library adds the power of CQRS to your applications with 10 lines of configuration and code! *Made with 💙 by CloudBlue Connect*.

```python
pip install django-cqrs    # You can try yourself!
```

> Django-CQRS is a new, but stable library. We have been using it in our production for about a year already. There has never been a single major issue and we use it more and more in our services. Library has 95+% coverage due to a big pack of unit tests and is also shipped with a set of integration tests. This library is designed to be transport agnostic (RabbitMQ is supported by default).

Let me share the motivation behind creation of this library and afterwards I will show an example of usage.

### Motivation

1. [Django](https://www.djangoproject.com/) is the main development framework in CloudBlue Connect and it is used in the majority of our services. It may not be the most trendy framework, but “stability and quality are our #1 priority”. Plus Django has a great community and a long proven track record of being used in the best Python projects. In other words, we needed something to perfectly fit this framework: something that will benefit from ORM, that is easy to configure and that will work behind the scenes in most of the times.
2. There are some open source python libraries like [EventSourcing](https://github.com/johnbywater/eventsourcing), but all of them try to implement CQRS pattern together with Event Sourcing. This may be an overhead for 90% of the real life cases and we needed something simple.

### Example

We will start with creating a simple model in Orders service.

{% gist https://gist.github.com/maxipavlovic/dd417b13889b9d7fc3f498b069130580 %}

> Each domain model needs a unique system-wide `CQRS_ID`. This ID is used everywhere: from model registry and utility functions to transport queues. There can be only one source model with such ID and several replicas (without any framework limitation).

And the default replica model in any other service:

{% gist https://gist.github.com/maxipavlovic/a04bf6c809260bdda7ba39d79b465305 %}

We need to set CQRS Django settings in both services, make migrations:

{% gist https://gist.github.com/maxipavlovic/4e5d52ee0ed83cf3ef1f562606830871 %}

And run replica with the shipped management command: 

```commandLine
python manage.py cqrs_consume -w 2
```

__And that’s it!__🥳 If the transport (RabbitMQ) is ready and services are up and running, CQRS is already working. When a new order is created, updated or deleted, it is automatically replicated to the replica service!
It seems like real magic, but the code and library architecture are actually pretty simple. Let’s look at the details of the architecture through the synchronization message data flow.

### Architecture

![Architecture](https://dev-to-uploads.s3.amazonaws.com/i/lt1kfjwxbjpedikrbdwk.PNG)

Each *Master/Source* model is extended with the *MasterMixin*. This mixin validates CQRS configuration and registers the model in the *Registry* and the [Signals Framework](https://docs.djangoproject.com/en/3.0/topics/signals/). It also extends the model class with CQRS Manager (for the support of bulk operations) and two very important fields: `cqrs_revision` and `cqrs_updated`.

> Each synchronized domain object must have a timestamp and a special revision number to solve synchronization problems like data duplication or message loss. CQRS does that out of the box on replica side.

After source signal is emitted, CQRS collects an object payload data together with synchronization primitives and calls the `producer`. `Producer` calls the transport layer where the message is sent to the actual transport, like RabbitMQ.

> Framework supports complex object payload serialization, where even other related objects may be included automatically with proper SQL optimizations. There is a special option `CQRS_SERIALIZER = ‘api.OrderSerializer’`, which allows to specify the serialization class for payload building. Obviously, for such cases some deserialization replica code needs to be written manually.

On the replica side each *Replica* model is extended with the *ReplicaMixin*. This mixin extends model with synchronization fields, validates CQRS configuration and registers the model in the *Registry*. When some message comes from transport queue, *Consumer* routes it to the target replica model and its related manager. Manager makes synchronization checks and saves the data to DB.

### Utilities

As distributed systems tend to fail, Django-CQRS is shipped with a great set of utilities for migration and operations purposes, which are implemented as Django management commands.

* __Bulk synchronizer__ without transport (use case: initial configuration). May be used at planned downtime.

```commandLine
# ON MASTER
python manage.py cqrs_bulk_dump --cqrs-id=author -> author.dump

# ON REPLICA
python manage.py cqrs_bulk_load -i=author.dump
```

* __Filter synchronizer__ over transport (use case: solve production synchronization problems in real-time).

```commandLine
# TO SYNC ALL REPLICAS
python manage.py cqrs_sync --cqrs-id=author -f={"id__in": [1, 2]}

# TO SYNC CHOSEN REPLICA
python manage.py cqrs_sync --cqrs-id=author -f={} -q=replica
```

* __Diff sync tools__ over pipes and transport (use case: find synchronization problems and solve them automatically, for example by cron job).

```commandLine
# k8s pipe example

kubectl exec -i MASTER_CONTAINER -- python manage.py cqrs_diff_master --cqrs-id=author 
| 
kubectl exec -i REPLICA_CONTAINER -- python manage.py cqrs_diff_replica 
|
kubectl exec -i MASTER_CONTAINER -- python manage.py cqrs_diff_sync
```

---

This is a pretty long article already and I am very happy if you have read up to this point. You are cool, thank you very much! 🙏

If you have any questions on any of these topics, you can 🤙 in [LinkedIn](https://www.linkedin.com/in/mxms/).

Stay healthy and inspired! See you.