Installing PubNub Kafka Sink Connector
Published: 2024-02-07 05:17:30

If you have an existing Kafka cluster and you're looking to deliver data records from your Kafka topics to devices such as web browsers and mobile phones in real-time, PubNub Kafka Sink Connector is the solution. This guide will walk you through the process of setting up the PubNub Kafka Sink Connector.

Setting up a Local Test Kafka Cluster (optional)
------------------------------------------------

The PubNub Kafka Sink Connector leverages Kafka Connect, a framework developed specifically for delivering data from Kafka topics to secondary systems such as Hadoop, Elastic, and PubNub. For demonstration purposes, we've included an entire Kafka cluster in a single Docker composed file. It includes everything you need to set up your Docker, the composed YAML files, Kafka broker, and the cluster itself. Additionally, we've included a producer for testing purposes that automatically produces data into a topic on your locally running Kafka cluster.

Clone the repository [https://github.com/pubnub/pubnub-kafka-sink-connector](https://github.com/pubnub/pubnub-kafka-sink-connector) and run the command `docker compose up` to boot these images on your local system. This might take a while, as there's a lot to Kafka itself.

The Docker Compose will also build the PubNub Kafka Sink Connector package. We've included an image that will build the Maven package for us for simplicity purposes. This makes it quick and easy to import into your Kafka Connect using the Kafka Connect framework. Once the builder image has finished building the connector package, it's ready for deployment. You'll know it's ready when the consumer notification messages appear at regular intervals showing: `consumer | {"timestamp":1705690559}` in a series of records. This represents that our test data container is operating successfully and we are ready to install the Sink Connector.

Deploying the PubNub Sink Connector
-----------------------------------

The Sink Connector is the bridge running on the Kafka Connect framework between the Kafka topics and the records in those topics over to your devices. All you need is the connector package information, which is in the PubNub Sink Connector JSON file. Post this file to Kafka Connect where Kafka Connect is running, and it will begin copying data from the configured topics directly over to your end target devices.

You need to configure the [JSON file examples/pubnub-sink-connector.json](https://github.com/pubnub/pubnub-kafka-sink-connector/blob/main/examples/pubnub-sink-connector.json) with your PubNub API key information. You can find your API keys at [https://admin.pubnub.com/](https://admin.pubnub.com/) This is what's going to be used to deliver the end results to your target devices. You can set your topics and your patterns, and then run the command to deploy the Sink Connector.

Run the following command to deploy the sink connector:

```js
curl -X POST \
    -d @examples/pubnub-sink-connector.json \
    -H "Content-Type:application/json" \
    http://localhost:8083/connectors
```

Your connector is now running and all data sent to the configured Kafka topic will be instantly copied to any device subscribed with the PubNub SDK.

Validating the Sink Connector
-----------------------------

You can validate that the connector is running by checking the messages being delivered to a web browser device. As soon as messages are put onto the Kafka topic, they are immediately copied over to the PubNub infrastructure, which is responsible for reliably delivering those messages to your end target devices. The following screenshot shows data streaming into a web browser application instantly the moment an event record is received in the Kafka topic.

![Validation of data being received on a web browser app from event records originating in a Kafka topic.](https://pubnub.com/cdn/3prze68gbwl1/9SwRTW3wCqjRs65X5vwyL/745d0c0fe06f48a5bfb8f4f639713fc6/Screenshot_2024-01-19_at_1.00.04_PM.png)

Building PubNub Kafka Sink Connector
------------------------------------

When you're ready for production, you can build PubNub Kafka Sink Connect using the Maven build command. Ensure you have all the dependencies installed such as Java 11 or greater and a new version of Maven. Run `mvn clean package` to run all the tests associated with the package and compile a couple of packages including a jar file and a zip file. This is the package that will be used in the Kafka Connect Framework.

Video Walkthrough for Installing PubNub Kafka Sink Connector
------------------------------------------------------------

In the video a guided installation of the PubNub Kafka Sink Connector is demonstrated.

[Click here to view the Video](https://videos.ctfassets.net/3prze68gbwl1/4S1JBjNrtGzLuUuJqmUc03/e2814500520648f92768d449a7c6c085/PubNub-Kafka-Connector.edited.mp4)

Setting up the PubNub Kafka Sink Connector is straightforward, and it's important to follow the steps carefully. If you encounter challenges along the way, reach out to us at [support.pubnub.com](https://support.pubnub.com), and we'll be happy to assist.

For more details, head to the [official PubNub documentation](https://www.pubnub.com/docs/general/integrations/kafka/pubnub-kafka-sink-connector).

How can PubNub help you?
========================

This article was originally published on [PubNub.com](https://www.pubnub.com/blog/installing-pubnub-kafka-sink-connector/)

Our platform helps developers build, deliver, and manage real-time interactivity for web apps, mobile apps, and IoT devices.

The foundation of our platform is the industry's largest and most scalable real-time edge messaging network. With over 15 points-of-presence worldwide supporting 800 million monthly active users, and 99.999% reliability, you'll never have to worry about outages, concurrency limits, or any latency issues caused by traffic spikes.

Experience PubNub
-----------------

Check out [Live Tour](https://www.pubnub.com/tour/introduction/) to understand the essential concepts behind every PubNub-powered app in less than 5 minutes

Get Setup
---------

Sign up for a [PubNub account](https://admin.pubnub.com/signup/) for immediate access to PubNub keys for free

Get Started
-----------

The [PubNub docs](https://www.pubnub.com/docs) will get you up and running, regardless of your use case or [SDK](https://www.pubnub.com/docs)