Supabase CDC Webhooks to Memphis REST Gateway
Tags: supabase, rest, cdc, webhooks
Published: 2023-01-29 12:43:37

## Introduction
[Supabase](https://supabase.com/) is an open-source Firebase alternative.
In databases, change data capture (or CDC) is a set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data or because of it.

In simple words: Each time a new event occurs on a specific table, such as INSERT / UPDATE / DELETE, an event will be created and, if configured – published to a destination so some service will be notified for it.

## Flow
1. Install Memphis over Kubernetes.
2. Expose Memphis REST Gateway.
3. Create a new Supabase table.
4. Configure a webhook for the newly created table.
5. Check the incoming CDC events in Memphis.

![usecases](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fdia2m0w0t2p2estzv3g.jpeg)

---

## Step 1: Install [Memphis](https://docs.memphis.dev/memphis/getting-started/readme) over Kubernetes 
Memphis can be installed on any Kubernetes cluster above v1.20.

For this tutorial, I used Memphis [Terraform](https://docs.memphis.dev/memphis/deployment/cloud-deployment/deploy-on-aws) to deploy a 3-node AWS EKS cluster with a Memphis cluster on top.
**Output of AWS EKS creation -**

```
Apply complete! Resources: 67 added, 0 changed, 0 destroyed.

Outputs:

cluster_endpoint = "https://CA2D8E9674B2025*******.gr7.eu-west-1.eks.amazonaws.com"
cluster_id = "memphis-cluster-ed****q7"
cluster_name = "memphis-cluster-ed****q7"
cluster_security_group_id = "sg-09a8f95********02"
region = "eu-west-1"
```
**Output of Memphis cluster deployment -**

```
To access Memphis using UI/CLI/SDK with service EXTERNAL-IP, use the following URLs:
Dashboard/CLI : http://a34d36344a12b48a7a9f1ef09z576023-1416400140.eu-west-1.elb.amazonaws.com:9000
Memphis broker: http://a34d36344a12b48a7a9f1ef09z576023-1416400140.eu-west-1.elb.amazonaws.com:6666 (Client Connections)
```
**Bonus**:Create a DNS A record for the address above to work with a real hostname easily.

Reach the Memphis dashboard and initiate the cluster.
The address is the one with port 9000.

![UI welcome](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g4uierhdmhqmw1n8sxep.png)

---

## Step 2: Expose Memphis REST Gateway
By default, Memphis terraform does not deploy an LB to the “HTTP proxy” pod.

To do it, we need to transform the “memphis-http-proxy” service from “ClusterIP” to “LoadBalancer” by running the following –

```
kubectl patch svc memphis-http-proxy -n memphis --type='json' -p '[{"op":"replace","path":"/spec/type","value":"LoadBalancer"}]'
```

```
kubectl get svc

NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                        AGE
memphis-cluster            ClusterIP      None             <none>                                                                    9000/TCP,7770/TCP,6666/TCP,8222/TCP            104m
memphis-cluster-external   LoadBalancer   172.20.251.246   a34d36224a00b48a7a9f1ac09c576023-1416400140.eu-west-1.elb.amazonaws.com   9000:32173/TCP,6666:32013/TCP,7770:31214/TCP   102m
memphis-http-proxy         LoadBalancer   172.20.131.157   a5ce9a9c1006a2d7a64faaa6949dcfbb-1553258514.eu-west-1.elb.amazonaws.com   4444:30686/TCP                                 104m
mongo                      ClusterIP      None             <none>                                                                    27017/TCP                                      104m
```
Let’s save the newly created LB external IP of the HTTP proxy –
a5ce9a9c1006a2d7a64faaa6949dcfbb-1553258514.eu-west-1.elb.amazonaws.com

---

## Step 3: Create a new Supabase table

![Supabase table editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aj3gvbozpzu1y4iuc014.png)

![Supabase UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ofh382f96thtg5w0kj7a.png)

---

## Step 4: Configure a webhook for the newly created table
First, we need to generate an "Application" type user on Memphis -

![Memphis users](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/46tw0lvxvrnoay7h5srw.png)
Please save both the username and connection token -
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tafy8j5qy2yfbifzf2gn.png)
To generate a JWT token for producing events via HTTP, the following command needed to be run -
(Will be replaced with API tokens soon)

```
curl --location --request POST 'a5ce9a9c9506a4c7a99faaa6949dcfbb-1553258514.eu-west-1.elb.amazonaws.com:4444/auth/authenticate' --header 'Content-Type: application/json' --data-raw '{
    "username": "supabase",
    "connection_token": "******************",
    "token_expiry_in_minutes": 600000,
    "refresh_token_expiry_in_minutes": 10000092
}'
```
The result will be the JWT token for the Supabase webhook. Save it.

```
{"expires_in":36000000000,"jwt":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MDk4MTIzOTJ9.Xlxq56c0wsfOwMsFheUNPKrnECX4JX_vrz45su5JmVU"}
```
Back to Supabase. Go to Database -> Webhooks -> Create a webhook

![Webhooks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m2ce7cg4o0g6g9nzy2wh.png)
Configure it as follows –

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/di99x1p13duey4x1c9h8.png)

![AWS supabase Lambda](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pv0cw72ygex0l4f3jr6x.png)

- Method: POST
- URL: http://<HTTP Proxy>:4444/stations/<some station name>/produce/single
- Headers:

Content-type: application/json
Authorization: Bearer <JWT Token>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/os606gdmtbw6y8abad03.png)

---

## Step 5: Check for incoming CDC events
Let’s insert a new record to our table and delete it right after
so we would get two events –

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0uznl0aw1dp3jzdrp09y.png)
Going back to the Memphis dashboard and into the “supabase” station,
events can be seen in the center panel, and the parsed payload on the right-bottom.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hplx3gy1ohassmtxvjmt.png)
Have a great day!

---
Install Memphis [here](https://docs.memphis.dev/memphis/getting-started/readme).

---

[Join 4500+ others and sign up for our data engineering newsletter.](https://mailchi.mp/memphis.dev/newslettersub)

---

Originally published at [Memphis.dev](https://memphis.dev/) By [Yaniv Ben Hemo](https://www.linkedin.com/in/ybenhemo/),Co-Founder & CEO at @Memphis.dev.

---

Follow us to get the latest updates!
[Github](https://github.com/memphisdev/memphis-broker) • [Docs](https://docs.memphis.dev/memphis/getting-started/readme) • [Discord](https://discord.com/invite/DfWFT7fzUu)