Moving data between Elasticache Databases
Tags: elasticache, redis, python
Published: 2022-09-26 13:14:01

## The problem
Some time ago, me and my team were asked to migrate several Elasticache databases (Redis) from EC2-Classic (yep, I know 🤷‍♂️) to VPC. All these Databases were being used in production services so, taking an snapshot and restoring it on a VPC was not a good idea due to the time that takes doing the whole process.
Using [AWS Online Migration](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/OnlineMigration.html) was not an option too, since Online migration is designed for data migration from hosted Redis on Amazon EC2 or on-premise self-hosted Redis to ElastiCache for Redis and not between ElastiCache for Redis clusters. 
So we kept looking for alternatives until we found a nice [gist](https://gist.github.com/kitwalker12/517d99c3835975ad4d1718d28a63553e) with a script that almost fit our needs. The only fail there was that it wasn't work ok on encryption in transit clusters.

## The solution
So we did some changes on the script to fit our use-case and et voila! We finaly got a [working version](https://gist.github.com/joseapeinado/fb0deb31793fe32b37ee6b5d38cc3761) to move the data.

# Demo 

From a terminal with access to both Elasticache clusters (very important!): 
1. Clone the repo:

```
git clone git@github.com:joseapeinado/ElasticacheMigration.git
```

2. Prepare the virtual env and install dependencies

```bash
cd ElasticacheMigration
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
```

3. Profit!!!
```bash
SRC_CLUSTER=src_cluster.spznk9.ng.0001.use1.cache.amazonaws.com
DST_CLUSTER=dst_cluster.spznk9.ng.0001.use1.cache.amazonaws.com

python3 migrate-redis.py $SRC_CLUSTER --src_port=6379 $DST_CLUSTER --dst_port 6379 --src_db 2 --dst_db 2
Connecting to Redis instances...
Counting keys to migrate...
19222 keys: 100% |###############################################################################################################################################################################################################################################| Time: 0:00:05
('Keys disappeared on source during scan:', 0)
('Keys already existing on destination:', 19195)
```

## Use cases
### Copying just one Database to other existing cluster
The script copies data between clusters in a per-database basis, so it is ready to perform this migration out-of-the-box.

### Copy all databases from one cluster to other
The only caveat here is the need of run the migration script once per database.

### Copy database data from one DB to other DB within the same cluster.

## Advantages doing this migration
The more obvious advantage I think is the time that could take using the script versus:
1. Stop writes on Redis cluster
2. Take a snapshot
3. Launch a new cluster from the snapshot
4. Resume writes on the new cluster

This process can take at least 15 minutes or so. 

## Contribute
Any improvements will be more than welcomed 😃
https://github.com/joseapeinado/ElasticacheMigration/

This is my very first tech post, so any advice would be appreciated. 
And I hope the migration script help you like it helped me 🙂