How to integrate kafka with nodejs ?
Tags: kafka, node, javascript, microservices
Published: 2022-12-18 16:19:38

Kafka is a powerful, open-source stream processing platform that has become increasingly popular in recent years for its ability to handle large amounts of data and real-time streaming scenarios. Node.js, on the other hand, is a JavaScript runtime built on Chrome's V8 JavaScript engine, known for its ability to create scalable, high-performance network applications. In this blog post, we'll explore the ways in which Kafka and Node.js can be used together to build efficient, real-time data processing systems that can handle high volumes of data and traffic.

👋 You can also check my weekly posts in [Linkedin](https://www.linkedin.com/posts/tarek-chafroud-6725711bb_kafka-activity-7005817676182740993-N2ls?utm_source=share&utm_medium=member_desktop)

## Requirements: 

- Basic knowledge of nodejs
- Basic Knowledge of docker
- Basic knowledge of javascript

## What we will do: 

1. Set up kafka.
2. Set up two Node.js apps: one for the producer and one for the consumer.
3. sent message from producer and consume it 
4. **scenario:** A user will be automatically created in the consumer when a producer is created.


## Discuss some crucial terms:

- **producer**: a producer is a program that writes data to a Kafka cluster. The producer sends messages to one or more topics.
- **consumer**: a consumer is a program that reads data from a Kafka cluster.
- **borker**: A broker is a Kafka server that stores and serves data.
- **topic**: topics are the channels through which producers and consumers communicate
- **partition**: a partition is a unit of data storage in a topic. Each topic is divided into one or more partitions




## Let's play the game:


**To begin, we will install Kafka**

   

- Install Docker: Kafka is typically run in a Docker container, so you will need to install Docker on your system if it is not already installed. Follow the instructions for your operating system to install Docker.

-   we will need two Docker images:
           

    - wurstmeister/zookeeper
           

    - wurstmeister/kafka
   

 1️⃣ Create a docker-compose.yml file and include the following in it.

`version: "3"
services:
  zookeeper:
    image: 'wurstmeister/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'wurstmeister/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper`  

 2️⃣ Run the following command to start the containers (-d to start the containers in the foreground)

```
docker-compose up -d
```

 3️⃣ Start kafka shell:(change kafka_kafka_1 by your container name)

```
docker exec -it <u>kafka_kafka_1</u> /bin/sh
```

 4️⃣ Create Topic:

```
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic your-topic-name-here
```
**Additional CLIs:**
       ▪️ kafka-topics.sh --list --zookeeper zookeeper:2181: list all topics.
       ▪️ docker ps: to show running containers.

and that's all.

**Next step: Set up two Node.js apps**

1️⃣ create a package.json:

```
npm init

```

2️⃣ install packages:

```
npm i express nodemon dotenv mongoose kafkajs
```
3️⃣ Now we will configure our producer:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tfhktqy4hr7yxaj7ypsj.png)
4️⃣ Add the following two lines: (to your index)

![our producer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vr7hrpacalr91gqwou7l.png)

**the result :** 👇

![producerresult](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dezoqtt52k3jkhfpkks2.png)

5️⃣ configure our consumer by adding this following in it:

![our consumer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v3vg2pt8fe2vzyjkb5vx.png)

6️⃣ now we will start our scenario: 

First of all, create your model, controller, and route. Inside the controller, you must add the following two lines:
`import {run} from "../utils/producer.js";
 run(response);
`
like this example below: 👇

![User controller](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/77edqn0h72e5em30bfbr.png)

7️⃣ update our producer:

![producer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8bk91r1hdzf0pqsdzx9p.png)

8️⃣ update our consumer:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b4simxf5869n3n1tzmvp.png)

**THE RESULT** 👏

from producer:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w3zf3518gh40h2xzvb0j.png)

from consumer:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yor7o7ks53by7ofwtznh.png)

---

🪄[Github-repo
](https://github.com/chafroudtarek/kafka-and-nodejs-blog)

In conclusion, Apache Kafka is a powerful distributed streaming platform that enables real-time data processing and stream processing applications. It can be integrated with Node.js, a popular JavaScript runtime, through the use of Kafka client libraries, allowing developers to build event-driven and data-driven applications using the two technologies together. Kafka is known for its scalability, high performance, and ability to handle high volumes of data, making it a valuable tool for building robust and scalable applications. However, it can also be complex to set up and maintain, and requires a good understanding of its architecture and configuration options in order to use it effectively. Overall, Kafka and Node.js can be a powerful combination for building efficient and scalable applications, and are worth considering for any project that requires real-time data processing or stream processing capabilities.