A beginners guide to data streaming with Kafka, NodeJs and Socket.io
Tags: javascript, node
Published: 2021-02-17 18:34:42

Through this small guide, my intention is to enable anyone to understand and begin their journey of working with Kafka in a few minutes.
After following this guide and using the tools mentioned, which is not the only way to achieve the output, you should have a working example on your hands and the understanding to use more advanced concepts.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wg8fojmzgl0i2u28xpgq.gif)

We shall start by building the individual modules, step-by-step and integrate all of them in the end. So, let's get started!
#### 1. Setting up Kafka
This is where we set up Kafka in our system. So, head down to the official [Apache Kafka quick-start guide](https://kafka.apache.org/quickstart). Follow the steps till **Step 3** and create a topic with any desired name. In my examples I have used the name **_topic_stream_**.

#### 2. Creating a Kafka Consumer
Glad you are back! Let's now use the [Kafka-node consumer](https://www.npmjs.com/package/kafka-node#consumer) to see how a consumer functions. A consumer simply reads messages from the Kafka topic.
Below, we have created a Consumer that subscribes to `topic_stream` and prints the data that is produced.
I have setup the Kafka client on `port 9092` in step 1. You should change the configuration according to your settings.

```javascript
const kafka = require('kafka-node');

const Consumer = kafka.Consumer,
  client = new kafka.KafkaClient('localhost:9092'),
  consumer = new Consumer(
  client, [ { topic: 'topic_stream', partition: 0 } ], { autoCommit: false });

  consumer.on('message', function (message) {
    console.log(message);
  });
```
Pretty straightforward, right?

#### 3. Creating a Socket API
Let's now setup a WebSocket API in our NodeJs service so we start streaming data to a webpage. We use [Socket.io](https://www.npmjs.com/package/socket.io) in conjunction with [express](https://www.npmjs.com/package/express). In a file called `server_consumer.js` create the following:
```javascript
const express = require('express');
const port = 3000;
const app = express();

const server = app.listen(port, () => {
  console.log(`Listening on port ${server.address().port}`);
});
const io = require('socket.io')(server, {
  cors: {
    origin: '*',
  }
});

io.on('connection', client => {
  console.log('Connected', client);

  client.on('event', data => { 
    console.log('Event triggered by client')
   });

  client.on('disconnect', () => { 
    console.log('Client disconnected');
   });
});
```
#### 4. Creating the Consumer, Socket API
We now know what a consumer is and how we intend to use the Socket.io API. Let's bring it all together now. Edit the `server_consumer.js` such that it now contains the Consumer as well:
```javascript
const kafka = require('kafka-node');
const express = require('express');
const port = 3000;
const app = express();

const Consumer = kafka.Consumer,
 client = new kafka.KafkaClient('localhost:9092'),
 consumer = new Consumer(
 client, [ { topic: 'topic_stream', partition: 0 } ], { autoCommit: false });

const server = app.listen(port, () => {
  console.log(`Listening on port ${server.address().port}`);
});
const io = require('socket.io')(server, {
  cors: {
    origin: '*',
  }
});

io.on('connection', client => {
  console.log('Connected', client);
  
consumer.on('message', function (message) {
    client.emit('request', message.value);
  });
  
client.on('disconnect', () => { 
    console.log('Client disconnected');
 });
});
```
#### 5. Creating the frontend using Chart JS
On our web browser, we will create a webpage that invokes the socket API and displays the data using Chart JS. You may use `D3.js` or any other libraries that you are comfortable with to display the data.
Let's create a simple `index.html`:
```
<html>
<head>
 <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
 <script src="https://cdn.jsdelivr.net/npm/chart.js@2.8.0"></script>
 <script src="https://cdn.socket.io/socket.io-3.0.1.min.js"></script>
 <script type="text/javascript" src="index.js"></script>
</head>
<body>
  <canvas id="myChart" width='300' height='100'></canvas>
</body>
</html>
```
In our `index.js`, we will make a call to our socket API and initiate listeners for data:
```
$(document).ready(function(){
  const count = 10;
  const data = {
    labels : [...Array(count).keys()].map(i => i.toString()),
    datasets : [
      {
        label: '# - Streamed number',
        backgroundColor: "rgba(50,220,220,0.5)",
        data : Array(count).fill(0),
      }
    ]
  }
  const updateData = function(newVal){
    const labels = data["labels"];
    const dataSetInitial = data["datasets"][0]["data"];
    count++;
    labels.push(count.toString());
    labels.shift();
    const newData = Math.floor(newVal);
    dataSetInitial.push(newData);
    dataSetInitial.shift();
  };
  const ctx = document.getElementById("myChart").getContext("2d");
  const chart = new Chart(ctx, {
    type: 'line',
    data,
    options: {animation : false}
  });
  
  function webSocketInvoke() {
    var socket = io('http://localhost:3000');
    socket.on('event', (value) => {
      updateData(value);
      chart.update();
    });
  }
  webSocketInvoke();
 });
```
We will now open the `index.html` and wait for the data to stream through.

#### 6: Producing data
Let's get to the final step where we produce data that can then be streamed to our setup. I will mention with 2 options to do this:
* Follow the instruction given in [Step 4](https://kafka.apache.org/quickstart#quickstart_send) of the Kafka quick-start guide. That's it.
* If you would like to explore the kafka-node library further, you may experiment and create a `producer.js` file and run it using your node environment; example file given [here](https://github.com/yuvraj2112/kafka_socket_stream/blob/master/producer.js).

***

That's it! You should now have a fully functional Apache Kafka stream at your hands and the understanding on how the end-to-end flow works.

Github link for the project: https://github.com/yuvraj2112/kafka_socket_stream