GOCDC and Postgres
Tags: showdev, tutorial, opensource, go
Published: 2020-02-18 09:25:30

## Change Data Capture (CDC) and Postgres

Recently, I've written [this](https://dev.to/thiagosilvaf/how-to-use-change-database-capture-cdc-in-postgres-37b8) post here, where I show you guys how to setup __Change Data Capture (CDC)__ in Postgres. So, if you don't know how to setup CDC in Postgres, I totally recommend you guys to take a quick look at that tutorial first. 

## GOCDC

[GoCdc](https://github.com/ThiagoSilvaF/gocdc) it's an Opensource API for data streaming, developed (I mean, in development) in Golang that I would like to share with you guys. In short, the concept behind it is similar to [Debezium](https://github.com/debezium), but in Golang, which in my opinion, it's easier for anyone to hack and adapt the application according to your needs. The focus of GoCdc is on simplicity. Simplicity to setup, to modify, etc. 


### Hands-on 🛠️

__Required:__
Docker -> https://www.docker.com/get-started

### 1 - Creating the Project

First of all, create a _docker-compose.yml_ file in the directory of your preference.
```yaml
version: "3"
services:
  gocdc:
    image: "133thiago/gocdc:latest"
    ports:
      - "8000:8000"
      
  db:
    image: "postgres:11"
    container_name: "my_postgres"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=example_db
    ports:
      - "5432:5432"
    command:
      - "postgres"
      - "-c"
      - "wal_level=logical"
    volumes:
      - my_dbdata:/var/lib/postgresql/data
  
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
      
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  my_dbdata:
```

### 2 - Running in Docker

Now you're going to run the _docker-compose up -d_, and then run __docker ps__. Then you should see the containers created.

### 3 - Postgres Setup

As I said at the beginning of this post, this step is going to be this tutorial in here -> (How to use Change Data Capture (CDC) with Postgres)[https://dev.to/thiagosilvaf/how-to-use-change-database-capture-cdc-in-postgres-37b8].

__IMPORTANT__: Just make sure you create either create your database as __example_db__, as it is in the value of __POSTGRES_DB__ in our docker-compose.yml, or you change the value in our docker-compose.yml to the database that you created. It is up to you.

### 4 - Kafka config:

Now, it's time to create our Kafka Topic, which is going to be the topic that our connector is going to send the database changes. 
First, run __docker ps__ again and get the _CONTAINER ID_ of your Kafka container, it will be something like this _4bed6164a8e9_
```bash
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS
         
4bed6164a8e9        wurstmeister/kafka             "start-kafka.sh"         5 hours ago         Up About an hour    0.0.0.0:9092->9092/tcp
```

Then, run the following command:
```bash
docker exec -it 4bed6164a8e9 "bash" # Remember to replace the Container ID with yours
```

Finally, let's create the topic, I'm going to call it _test_, but it is up to you. just bear in mind, you will need the topic later!
```bash
bash-4.4# Kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test
```

### 5 - Tinder match: Postgres ❤️ Kafka 

GOCDC provides a REST Interface for creating the connection between the Database and Kafka. If you look again to our docker-compose.yml file, GOCDC is using the Port 8000. So let's do a POST to our localhost:8000/connectors/Postgres
```bash
curl --location --request POST 'http://localhost:8000/connectors/postgres' \
--header 'Content-Type: application/JSON' \
--data-raw '{
    "connector_name":"Conn PG Test",
    "db_host": "localhost",
    "db_port": 5432,
    "db_user": "postgres",
    "db_pass": "postgres",
    "db_name": "example_db",
    "db_slot": "slot",
    "kafka_brokers": [ "localhost:9092" ],
    "kafka_topic": "test"
        "lookup_interval": 5000
}
'
```

Let's dive into the JSON object sent in this request:
_connector_name_: No big deal here, just an Identifier. It is unique though, so you can use it to Edit via __PUT__ request. 

_db_host_: The IP where your Postgres is running. In our example, it is _localhost_.

_db_port_: The Port to access our Postgres database.

_db_user_ AND _db_pass_: User and Password of our Postgres database.

_db_name_: The name of the database

_db_slot_: The name of the _Replication Slot_.

_kafka_brokers_: An array with our Kafka brokers. Well, we're running only one in our example, but you can set multiple.

_kafka_topic_: Here is the __Topic__ that we created in Step 4.

_lookup_interval_: Here you tell the connector how often it should execute a CDC lookup. In our example, every 5 seconds (yes, the parameter value is in _milliseconds_).

### 6 - Kafka Consumer:

So, we now have our Postgres database up and running, our Kafka "cluster" also up and running and the Connector created! 
Assuming that you have the Database and a Table created in your Postgres (Step 3), Connect to your _Kafka_ once again (__Step 4__) and run the following command:

```bash
bash-4.4# Kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
```
And now, you will be able to see every _Insert, Update and Delete_ from your Database, being sent to your Kafka consumer.

I hope you enjoyed the Tutorial, if you've got stuck at some step, please leave a comment and do my best to help! 🖖