Use Redis Stack to Replace Postgres in a RESTful API - Learn and Output
Tags: redis, postgres, restful, webdev
Published: 2022-08-25 05:45:00

![architecture change](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f661ebfetdrhnsd9ej6a.png)
 


## Steps

1. Refactoring existing code base, to split db logic and business logic
2. [Integration test](https://www.smashingmagazine.com/2020/09/automate-api-testing-postman/#test-manually)
3. Implementation
4. Load test and Performance Result

## Load test

[Load test](https://github.com/tim-hub/sanic-currency-exchange-rates-api/tests/load-test.js) is used here to measure the performance.
- same Machine (my personal macbook laptop)
- same spec for DB 
  - 10GB Volume
  - Same ARM64 Instance


## Load Test result and Performance Compare

- Original Postgres Result
![postgres performance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tee5tld95fsfs983ja7t.png)

 
- Switch to use Redis through Redis-Py
![redis-py](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cbavn3fsbabi2p41wlgh.png)
 

- Switch to use Redis through Redis OM
![redis om](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eql47kq18uju7gmfwnzp.png)


The 2 key results here are `iterations` (higher is better) and `iteration_duration(avg)`(lower is better)

| Fact | Postgres | Redis-Py | Redis OM |
| :---: |:--------:|:-----------:|:--:|
| Iterations | 25.26/s  |   5.52/s    | 2.30/s |
| Iteration Duration | 560.15ms |    5.53s    | 8.07s|

This performance benchmark is surprising to myself as well, Redis as a well performanced DB, 
in most of time, it give a very good result of performance.

## Why Performance get worse after switching, 3 Potential Reasons
![database](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/akqbh723f08c7mvpqdiy.png)
 

This performance compare may not be objective, the reasons behind it requires more researching. These are some potential reasons I assume,

### 1. Min Specs for different DB are difference
Even the spec for Postgres and Redis Stack here are the same, but their min spec requirements may be different.
   - [Postgres](https://www.enterprisedb.com/docs/supported-open-source/postgresql/installer/01_requirements_overview/): 1 GHz processor. 2 GB of RAM. 512 MB of HDD.
   - [Redis](https://docs.redis.com/latest/rs/administering/designing-production/hardware-requirements/): 2.6GHz(2 CPUs), 4 GB of Ram, 10 GD of HDD  (Redis Stack supposed to be higher) 

### 2. Data Type Convert May Slow the API
Decimal in Python as a class could not be saved directly as a JSON attribute in RedisJSOM, 
so it was stores as string, and it will be converted to Decimal when read. This may slow the APP down.

### 3. Query was not used (RedisSearch) was not used, 
So when query history data, a loop of `GET` are sent to DB, which is not efficient


### Conclusion 1

This comparing result doesn't tell which is better between Postgres and RedisJSON, it only says that at current stage, at current DB Instance spec, for this API,
postgres gives better performance, and the way of `Decimal Convert` and `Loop of Get` in Application level slows the app too. 
(If anyone knows better about Redis to solve these problems, please let me know). So this comparing seems not very fair.


## 2022 October 25, Get Rid of Assuming Data Conversion in Codes

![thinking in front of desk](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u5q7srysmc6d57yfqnn9.png)

> The result is in the 3rd Table Column Above.
 

> Try solve the issue No 2, stop using Python data convert in application level, and No 3, tried to use `find` from Redis Search

Tried use Redis-OM for the querying, in [this branch](https://github.com/tim-hub/sanic-currency-exchange-rates-api/pull/26),
The result is still not good enough, (actually even worse), issues found:

1. Redis-OM-Python function was limited, `sort by` function is not supported, 

2. `find` function does not support 2 + conditions (not sure way, this is different with official document)


### Conclusion 2

![dashboard for result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2gs2kzzv9zba99orbxa.png)
 

- Redis-OM-Py is very new, it requires more community contribution
- The root cause of the performance issue after switch is not from application level, it is still something either from db level, or db SDK/driver performance

## Repos

{% embed https://github.com/tim-hub/sanic-currency-exchange-rates-api %}