CD-Stream:CDC Replicator Tool
Tags: postgres, python, cdc, etl
Published: 2020-03-18 17:41:39

---
title: CD-Stream:CDC Replicator Tool
published: true
date: 2018-10-30 21:12:00 UTC
tags: postgres,python,cdc,etl
canonical_url: 
---

#### ++ Cons on ETL pipelines

Just another day at the workplace;

**_5 minutes post the boot:_**

You hear everyone complain that the production database is slow. You quickly start to investigate; exploring all possible outcomes on the dashboards...

> **Could it have been the long-running slow query which you had raised a ticket for the production support to fix? Or Is it one of the queries run based on a non-indexed column?**

![](https://cdn-images-1.medium.com/max/500/0*5b7aE4SKkbTm-jb8.gif)

**_6th Minute and 15 minutes later:_**

Next, you hear the fellow data-analysts lament over their failed reports.

You now realize that your CPU had taken a humongous amount of query load and you understand that your relational database system has gone for a toss into an eternal slumber.

### And all of this due to a slow running query of your ETL pipeline..!! Ding. Ding... Ding…!! We have a winner!!!

Alright, let’s back it up a little bit.

Probably you did/used one of the following:

- SELECT \* from production\_database.table where updated\_at between x and y;

- Bulk exports and Dumps once in every few minutes

- Long-running and forgotten Zombie Crontabs

![](https://cdn-images-1.medium.com/max/311/0*NWQ-pURqbOWtGPdr.jpg)

Let’s put it this way... DB Size < 500 GB; it’s OK to do selects but if it exceeds >500GB, unless you have cuts in the budget, do not ever do a bulk select and transfer it over the wire to the destination database in the form of a pipeline.

#### Enter CDC:

CDC A.K.A Change Data Capture is there to assist this data wrangling exercise and if your data is rapidly growing and if your BI/BA needs access to that sweet-sweet DWH (data warehouse); THIS right here, is the way to go.

#### You meant the replication? — I don’t wanna get stuck with my OLTP DB Engine:

Well, you are not alone. For many reasons, data-wranglers generally don’t prefer a similar database engine as that of the OLTP. The main reasons include query performance, need for triggers and ease of re-running transformation jobs. But, doing so, requires setting up airflow clusters and setting up connectors to transform and load.

![](https://cdn-images-1.medium.com/max/500/0*S7iFWzeO5Ts9cYJ2.jpg)

Wouldn’t it be great if you could do all this without doing a bulk selection from the production database? Of course, you can...

**CD-Stream** is a cross-database CDC driven replicator tool that currently supports replication between **_MySQL and Postgres_**.

The tool runs queues to process the information occurring in the binary logs of the source database and replicates it across to a destination database of an entirely different engine.

![](https://cdn-images-1.medium.com/max/600/0*zO-ZxOZB5LpJtGsY.png)

Post the setup, as given in the project page: [CD-Stream](https://github.com/datawrangl3r/cd-stream); there’s a directory called ‘sample’ in the project which contains some of the intensive DDL and Data Insertion scripts, for you to evaluate and exercise.

And that, _People_; is how you wrangle production data!!!

_Originally published at_ [_https://www.datawrangler.in_](https://www.datawrangler.in/2018/10/cd-streamcdc-replicator-tool-cons-on.html) _on October 30, 2018._

* * *