Kafka Microservice using Nest Js
Tags: node, nestjs, kafka, microservices
Published: 2020-09-19 15:21:24

Hello All,

Hope you are doing well. 
In this article, we are going to learn about how to set up a microservice using NestJs and Kafka.

Why did I start writing this? 
I recently wanted to set up one and I was struggling to find a good example. After hours of googling, I was able to. So to make life easier for my fellow developers here I am.

*Note: In this, I will not be explaining about Kafka or NestJs. So some basic knowledge on those would be helpful while implementing it.*

Let's Get Started!!!

**To install on windows/ubuntu please refer:**
[windows](https://dzone.com/articles/how-to-configure-an-apache-kafka-cluster-on-ubuntu)
[ubuntu](https://dzone.com/articles/how-to-configure-an-apache-kafka-cluster-on-ubuntu)

**To install Kafka on Mac using Homebrew:**

```
$ brew cask install java
$ brew install Kafka
```

**To start Zookeeper:**
```
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
```

Note: Zookeeper should be always started before starting the Kafka server.

**To start Kafka:**
```
kafka-server-start /usr/local/etc/kafka/server.properties
```

#### Lets check whether Kafka is working properly

**To create a topic:**
```
kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-first-topic
```

**To create a Producer console:**
```
kafka-console-producer --broker-list localhost:9092 --topic my-first-topic
```

**To create a Consumer console:**
```
kafka-console-consumer --bootstrap-server localhost:9092 --topic my-first-topic --from-beginning
```

Now Producer console will be able to accept user inputs. Whatever we type on the Producer console should be visible in the Consumer console.

####Let's write some code now

Let's create a NestJs application

```
nest new kafka-consumer
```

Once the application is created we need the microservices module and Kafka client library.

```
npm i --save @nestjs/microservices
npm i --save kafkajs
```

Now we need to update our main.ts to accept Kafka Transport and provide Kafka configurations

```
// main.ts

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';

import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
    transport: Transport.KAFKA,
    options: {
      client: {
        brokers: ['localhost:9092'],
      },
      consumer: {
          groupId: 'my-kafka-consumer',
      }
    }
  });

  app.listen(() => console.log('Kafka consumer service is listening!'))
}
bootstrap();

```

In app.controller.ts we are listening to our topic.

```
import { Controller } from '@nestjs/common';
import { MessagePattern, Payload } from "@nestjs/microservices";
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) { }

  @MessagePattern('my-first-topic') // Our topic name
  getHello(@Payload() message) {
    console.log(message.value);
    return 'Hello World';
  }
}
```

Start our server
```
npm start
```

Now let's go back to the producer console and send a message now it should be logged in our application console.

Voila!. Now we have set up our Kafka microservice successfully.

In case you want to set up producer inside a nest js application Please follow along.

Create an application and install the necessary dependencies

```
nest new kafka-producer
npm i --save @nestjs/microservices
npm i --save kafkajs
```

In app.controller.ts will set up our producer:
```
import { Controller, Get } from '@nestjs/common';
import { Client, ClientKafka, Transport } from "@nestjs/microservices";
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) { }

  @Client({
    transport: Transport.KAFKA,
    options: {
      client: {
        clientId: 'kafkaSample',
        brokers: ['localhost:9092'],
      },
      consumer: {
        groupId: 'my-kafka-consumer' // Should be the same thing we give in consumer
      }
    }
  })
  client: ClientKafka;

  async onModuleInit() {
    // Need to subscribe to topic 
    // so that we can get the response from kafka microservice
    this.client.subscribeToResponseOf('my-first-topic');
    await this.client.connect();
  }

  @Get()
  getHello() {
    return this.client.send('my-first-topic', 'Hello Kafka'); // args - topic, message
  }
}

```

Quit the producer console and consumer console that we started earlier if not already.

Start our server
```
npm start
```

Now open any Rest API client and hit
GET: http://localhost:3000/

Now we can see the ` Hello World ` as our response sent from Kafka microservice.

You can find the source code here:
[Consumer Sample](https://github.com/Kannndev/kafka-consumer)
[Producer Sample](https://github.com/Kannndev/kafka-producer)

*P.S This is my first blog post. Please feel free to give feedbacks.*