Debezium: Open-Source Technology for Change Data Capture
Tags: opensource, docker, kubernetes
Published: 2020-07-23 22:19:24

<!-- wp:heading -->
<h2>Introducing Debezium</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><a aria-label="undefined (opens in a new tab)" href="https://debezium.io/" target="_blank" rel="noreferrer noopener">Debezium</a> is an open-source tool for Change Data Capture (CDC). It does this as a distributed platform that turns data stores into event streams. This allows you to focus only on data that has changed instead of all data. For example, it enables you to keep disparate data systems in continuous sync and to respond quickly to new information. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Built on top of <a href="https://kafka.apache.org/" target="_blank" aria-label="undefined (opens in a new tab)" rel="noreferrer noopener">Apache Kafka</a> &amp; Kafka Connect, a traditional "Debezium-centric" architecture comprises source and sink data stores with a selection of Kafka Connect compatible connectors able to communicate changes taking place in the source data store to the sink data store.  </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you're comfortable with <a aria-label="undefined (opens in a new tab)" href="https://www.docker.com/" target="_blank" rel="noreferrer noopener">Docker</a> and <a aria-label="undefined (opens in a new tab)" href="https://kubernetes.io/" target="_blank" rel="noreferrer noopener">Kubernetes</a>, a basic Debezium setup should only be moderately difficult. The main ingredients are Apache Kafka and Kafka Connect. Fortunately, in Debezium's <a aria-label="undefined (opens in a new tab)" href="https://hub.docker.com/u/debezium" target="_blank" rel="noreferrer noopener">public Docker repo</a>, there are Zookeeper, Kafka, and Kafka Connect images ready to go.  If you can get Apache Kafka with Kafka Connect up &amp; running, you have several Debezium deployment options to choose from. Let's go over a few.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Some Kafka Connect Deployment Options</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Since Kafka Connect is Debezium's foundation, for the time-being, we'll focus mainly on Kafka Connect deployment options.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a aria-label="undefined (opens in a new tab)" href="https://camel.apache.org/" target="_blank" rel="noreferrer noopener">Apache Camel</a> and Debezium are a natural fit since Camel is known for its ability to integrate data consumer and producer systems. We also can't lose sight of the fact both technologies are strongly supported by <a aria-label="undefined (opens in a new tab)" href="https://www.redhat.com/en" target="_blank" rel="noreferrer noopener">Red Hat</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can also use Debezium in <a aria-label="undefined (opens in a new tab)" href="https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/cdc-parent" target="_blank" rel="noreferrer noopener">Google Cloud</a> with technologies like <a aria-label="undefined (opens in a new tab)" href="https://cloud.google.com/pubsub" target="_blank" rel="noreferrer noopener">PubSub</a> and <a aria-label="undefined (opens in a new tab)" href="https://cloud.google.com/dataflow" target="_blank" rel="noreferrer noopener">DataFlow</a>. In this architecture, PubSub plays a similar role as Apache Kafka in a traditional setup, but you'll still need some knowledge of the Kafka Connect libraries. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In a traditional 'Debezium-centric' architecture, a relational database is used for the sink data store, but there are use cases such as database caching where an in-memory computing platform like GridGain can be used.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Before I forget, I have to thank Gunnar Morling at Red Hat and his team for all the work they are putting into Debezium.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2><strong>References:</strong></h2>
<!-- /wp:heading -->

<!-- wp:list {"ordered":true} -->
<ol><li>Extending Kafka connectivity with Apache Camel Kafka connectors. <a aria-label="undefined (opens in a new tab)" href="https://developers.redhat.com/blog/2020/05/19/extending-kafka-connectivity-with-apache-camel-kafka-connectors/" target="_blank" rel="noreferrer noopener"> https://developers.redhat.com/blog/2020/05/19/extending-kafka-connectivity-with-apache-camel-kafka-connectors/</a>. Last accessed: 7/16/2020.</li><li>How do I move data from MySQL to BigQuery? <a aria-label="undefined (opens in a new tab)" href="https://cloud.google.com/blog/products/data-analytics/how-to-move-data-from-mysql-to-bigquery" target="_blank" rel="noreferrer noopener">https://cloud.google.com/blog/products/data-analytics/how-to-move-data-from-mysql-to-bigquery</a>. Last: 7/16/2020.</li><li>Change Data Capture Between MySQL and GridGain With Debezium. <a aria-label="undefined (opens in a new tab)" href="https://www.gridgain.com/resources/blog/change-data-capture-between-mysql-and-gridgain-debezium" target="_blank" rel="noreferrer noopener">https://www.gridgain.com/resources/blog/change-data-capture-between-mysql-and-gridgain-debezium</a>. Last accessed: 7/21/2020.</li><li><a href="https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/change-data-capture-in-postgres-how-to-use-logical-decoding-and/ba-p/1396421">Change data capture in Postgres: How to use logical decoding and wal2json</a>. <a href="https://developers.redhat.com/blog/2020/05/19/extending-kafka-connectivity-with-apache-camel-kafka-connectors/" target="_blank" aria-label="undefined (opens in a new tab)" rel="noreferrer noopener">https://developers.redhat.com/blog/2020/05/19/extending-kafka-connectivity-with-apache-camel-kafka-connectors/</a>. Last accessed: 7/21/2020.</li></ol>
<!-- /wp:list -->