Cache Master data in Redis using Python
Tags: architecture, caching, python, performance
Published: 2022-01-12 12:42:45

## Introduction

According to its [GitHub repository](https://github.com/redis/redis), Redis (stands for **RE**mote **DI**ctionary **S**erver) is an in-memory data structure store. It is a disk-persistent key-value database with support for multiple data structures such as strings, hashes, lists, sets, bitmaps, etc.

Data structures implemented into Redis have a few special properties:

- Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that it is also non-volatile.
- The implementation of data structures emphasizes memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modelled using a high-level programming language.
- Redis offers a number of features that are natural to find in a database, like replication, durability, clustering, and high availability.


## Use Cases

When speaking of use cases of using Redis, the most generic use case that comes to many is for caching data and for session store (i.e. web session). Redis has a lot of use cases that can be applied and useful for any situations especially that involves speed and scalability wise, it's easy to manage either scaling up or scaling down.

Few use cases are:

1. Caching
2. Session store
3. Chat and messaging application
4. Gaming leader board applications

## Caching Data in Redis
The cache is temporary storage where data is stored so that in the future data can be accessed faster. So, caching is the process of storing data in Cache. One of the caching use cases is caching the master data that can be database query results, pandas data frame, etc. 

**Prerequisite**
Install python and Redis. Following versions are used in this blog:

python==3.9.0

redis==4.1.0

[redis-py](https://pypi.org/project/redis/) requires a running Redis server, and Python 3.6+. See the [Redis quickstart](https://redis.io/topics/quickstart) for Redis installation instructions.

redis-py can be installed using pip via “pip install redis”.

To test that the Redis connection to the server is successful, we can open a new terminal window and run:

`% redis-cli ping`

The output should be

`PONG`

**Create instance of Redis Class**

We need to create instance of Redis class that expects mainly Redis broker URL, port, and database, and it can be further used to do all CRUD operations on Redis.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xsb8ttu8jrpasbxajocc.png)

**Key-Value storage**
Cache is maintained in key-value pair, which means we can store any data with respect to a unique key like dictionary. Here key can be any string with max size of 512 MB. Values, on the other hand, can vary in size depending on their type. For aggregate data types (i.e. hash, list, set, and sorted set), the maximum value size is 512 MB for each element, although the data structure itself can have up to 2^32 - 1 elements.

 

Data structures supported in values by Redis:

- Binary-safe strings
- Lists
- Sets
- Sorted sets
- Hashes
- Bit arrays (or simply bitmaps)
 

**Note:** Redis can handle up to 232 keys and was tested in practice to handle at least 250 million keys per instance.

Every hash, list, set, and sorted set, can hold 232 elements.

In other words, your limit is likely the available memory in your system.

**Check if key exist in Redis**
Since data is saved in Redis as key-value pair, we can check if key is already there in cache:

```
app_redis_handler.exists(key)
```

For example:

```
if not app_redis_handler.exists("balance"):
    return None
```

**Create cache using key**
We can create a cache based on provided key and value:

```
app_redis_handler.set(key, data)
```

For example:

```
app_redis_handler.set("balance",1777)
```

**Save json dictionary as value**
We can also save json as data in Redis cache:

```
app_redis_handler.set(key, json.dumps(data))
```

For example:

```
app_redis_handler.set("balance", json.dumps({"due":577,\
 "credit":680}))
```

**Set expiry on Cache key**
We can provide expiry of cache key in seconds. Keys expiring information is stored as absolute Unix timestamps (in milliseconds in case of Redis version 2.6 or greater).

```
app_redis_handler.expire(key, expiry)
```

For example:

```
app_redis_handler.expire("balance", 10)
```

**Create cache using key with expiry**
We can create a cache based on provided key and value along with expiry:

```
app_redis_handler.setex(key, expiry, data)
```

For example:

```
app_redis_handler.setex("balance",60, 1777)
```

**Get data based on key**
Value can be fetched from cache based on key:

```
data = app_redis_handler.get(key)
```

For example:

```
data = app_redis_handler.get("balance")
```

**Delete/invalidate data based on key:**
Delete/invalidate cache key once not required:

```
app_redis_handler.delete(key)
```

For example:

```
app_redis_handler.delete("balance")
```

**Scan all keys prefixed with "prefix_text":**
Cache key is searchable based on prefix, and it fetches all keys prefixed with the provided text:

```
redis_handler.scan_iter(f'{prefix_text}*')
```

For example:

```
[key for key in app_redis_handler.scan_iter(f'{bal}*')]
```

**Save master data in cache**
We can also save the master data that can be database query results, pandas data frame data, etc. in cache:

```
app_redis_handler.set("employee", [(1, 'John', 'Finance'), \
(2, 'Richard', 'Energy'), (3, 'Carter', 'HR'),……])
```

Here we can save the master data in cache, and fetch it whenever required and load it in data frame like structure to apply filters like "department=HR". It can be useful if we want to show top employees of all departments from cache, so it is better to save it in one cache and apply filter on cache data to save the extra transactions to database and managing the extra keys in cache.

 

## Summary
Redis offers highly performant and efficient read and write along with few operations mentioned in this article. There are lots more. For more information, you can visit the [documentation](https://redis.io/).

 

## References
https://redis.io

https://github.com/redis/redis

https://redis-py.readthedocs.io/en/stable/

https://www.kite.com/python/docs/redis.StrictRedis

 

**Disclaimer:** This is a personal blog/post. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.