OverComing Weak Consistency using Cache
Tags: redis, database, node
Published: 2022-11-25 10:42:54

Recently I had faced off with weak consistency for the first time in my professional career(< 2 yrs). I didn’t know that the DB which we were using in our PROD env was weakly consistent when reads are done using secondary replica. (how silly of me 😓)

> **Weak/Eventual Consistency** is a guarantee that when an update is made in a distributed database, that update will eventually be reflected in all nodes that store the data, resulting in the same response every time the data is queried.
> 

**What was I doing?**

Consider two tables

1. **Users** table, having the below columns
    
    
    | id | name | age |
    | --- | --- | --- |
2. **Relationship** table, which has the below columns
    
    
    | contactId | id |
    | --- | --- |
    
    > id is the primary key of the **users** table
    > 

Usually the relationship is created whenever some info is to be persisted for a particular user in the users table. When there is no relationship existing, a new DB row is created. And a new entry is saved in the relationship table mapping to the new user entry

When, new details are to be saved, the **users `id`** is fetched from the relationship table using the `contactId` and the details are persisted.

**What was problem?**

Consider this scenario, when someone lands on a webpage, we will have some details like `ip, timezone, email` which can be auto collected. 

When trying to save the first field, `email` we check the **Relationship** table, if there is no data present, a new row is created in the users table and add a new `relationship` entry is also created.

When the next update of `ip` is to be done we check the **Relationship** table, and get the `id` and do the updates.

```jsx
function getRelationship(contactId) {}
function createRelationship(contactId, user) {}
function createUser() {}
function updateUser(relationship, dataToUpdate) {}

function forEachUpdate(contactId, dataToUpdate) {
	relationship = getRelationship(contactId);
	if(!relationship) {
		newUser = createUser()
		relationship = createRelationship(contactId, user);
	}
	updateUser(relationship, dataToUpdate);
}

// main
forEachUpdate('sasi@hotmail.com', {email: 'sasi@hotmail.com'})
forEachUpdate('sasi@hotmail.com', {ip: '127.0.0.1'})
```

The problem happens when we don’t find a `relationship` when checking after a few milliseconds of delay. *But how? A new entry was added previously? What happened was,* The writes are done in a primary replica, the reads goes to secondary replica. Which meant that the reads and writes were going to different servers which will not have the exact same data at any given time. This is called *replication lag*. Because of this duplicate user entries were created in the users table.  

**What is the intermediate fix?**

Pointing the read queries to primary solved the issue. But, there is a reason why we have read replicas so that the primary will not be overloaded with all the read queries and respond quickly for writes. So this was not a permanent fix. For a permanent fix I had to switch to a different database, for which I didn’t have the time.

**Solution:**

Every time, I have to update some data it’s done sequentially. So, whenever a `relationship`  is created, a copy of it is stored in Redis(an in-memory cache) too.  

The sudo-code will look something like the below.

```jsx
function saveInRedis(contactId, user) {}
function saveInDB(contactId, user) {}
function createRelationship(contactId, user) {
	saveInDB(contactId, user);
	saveInRedis(contactId, user);
}
```

Therefore, before every update Redis is checked first to see if there is any relationship, if there is none, then the DB is checked

The function to getRelationship has been transformed like the below

```jsx
function getDataFromRedis(contactId) {}
function getRelationship(contactId) {
	relationShip = getDataFromRedis(contactId);
	if(!relationShip) {
		// check in DB
	}
	return relationShip
}
```