NestJS with RabbitMQ in a Monorepo: Building a Scalable Credit Card Payment System with Decoupled API and Consumers
Published: 2024-01-07 18:17:37


## Introduction
In this article, we will explore the creation of a credit card payment application using NestJS and RabbitMQ to handle billing generation with a Payment Service Provider (PSP). Additionally, we will incorporate Docker and Docker Compose to streamline container management.
Initially, we will create an endpoint simulating the billing creation process, performing a database insert and an HTTP request. We will observe that the response time for this endpoint is unacceptably high, around 1100 ms. Next, we will introduce a message broker service to asynchronously process the HTTP request for billing generation, resulting in a significant reduction in the endpoint's response time.
To conclude, we will address the separation of the consumer from the API in a monorepo, enabling both to scale independently. I want to highlight that I developed this project using the Clean Architecture, but you have the flexibility to choose the architecture that best suits your needs.

## Why should I use asynchronous processing and RabbitMQ?
 
- **Asynchronous Processing for Improved Performance**:
        The introduction reveals that the initial response time for the billing creation process is unacceptably high, around 1100 ms. Introducing a message broker allows you to shift to asynchronous processing, significantly reducing the endpoint's response time. RabbitMQ excels in handling asynchronous communication, enabling your application to scale more efficiently.

- **Enhanced Scalability and Responsiveness**:
        By leveraging RabbitMQ, you can decouple the billing generation process from the API endpoint. This decoupling facilitates improved scalability, as both the API and the billing generation service can scale independently. This flexibility ensures that your application remains responsive, even under increased load.

- **Reliable Message Delivery**:
        RabbitMQ provides reliable message delivery mechanisms, ensuring that messages are successfully delivered even in the event of system failures or network issues. This reliability is crucial in financial applications like credit card payment processing, where data integrity and consistency are paramount.

In summary, utilizing RabbitMQ in conjunction with NestJS for a credit card payment application brings tangible benefits such as improved performance, scalability, reliability, and flexibility, making it a well-rounded choice for handling asynchronous communication and optimizing your application's overall architecture.

##1. Start new nestjs app

`$ nest new nestjs-rabbitmq-example` 
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/197cdfda14b31a34b8a1b9d87e6da00a35d52e6e).

After installation, I removed app.controller.ts and app.service.ts 
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/616ff73936f7f2415a7529c91d62dac85ca9c015).

##2. Include entity, controller, and services to simulate the creation of a charge using a payment service provider such as Pagarme

`$ nest g module credit-card`

_You can see the classes created in this_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/6233d66380eb6153a12181ebf0f4adaf06101d8b).
![folder structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6mn3lqbmp9xenvpphg5z.png)

Now we have an endpoint that **simulates** the creation of a credit card charge, as if making an insertion in the database (100ms) and then an http request to _pagarme_ (1000ms).
![postman](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80ur3weg042oegvdxfxa.png)

In this scenario, the client is not required to linger on the HTTP request, awaiting a response from Pagarme; we can efficiently handle this process asynchronously. To achieve this, we will leverage **RabbitMQ** to queue the creation requests for charges.

##3. Add the necessary libs to implement rabbitmq
`$ yarn add @nestjs/microservices amqplib amqp-connection-manager` 
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/6ac0d9b411a9b2505f0fb4dfffcfb23cd5142855).

##4. Dockerize application
Dockerfile
```
FROM node:16

WORKDIR /usr/src/app
COPY package*.json ./

RUN yarn
COPY . .

RUN yarn build

EXPOSE 3000
CMD [ "yarn", "start:prod" ]
```
docker-compose.yml
```
version: '3.7'

services:
  credit-card-api:
    container_name: credit-card-api
    restart: on-failure
    build:
      context: .
    volumes:
      - .:/usr/src/app
    ports:
      - 3000:3000
    command: yarn start:dev
    depends_on:
      - rabbitmq
    
  rabbitmq:
    image: rabbitmq:3.9-management
    container_name: rabbitmq
    restart: always
    hostname: rabbitmq
    ports:
      - 5672:5672  
      - 15672:15672  
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq  

volumes:
  rabbitmq_data:

```
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/0d3b5bddf812e3cd36194585554cadbae8bb2ee2).

##4. Connect application with rmq

update file main.ts to use  _app.connectMicroservice()_ and _app.startAllMicroservices()_
I'm not going to delve into rabbitmq settings.
```
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { RmqOptions, Transport } from '@nestjs/microservices';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  app.connectMicroservice<RmqOptions>({
    transport: Transport.RMQ,
    options: {
      urls: [`amqp://rabbitmq:5672`],
      queue: 'create_charge_psp',
      prefetchCount: 1,
      persistent: true,
      noAck: false,
      queueOptions: {
        durable: true,
      },
      socketOptions: {
        heartbeatIntervalInSeconds: 60,
        reconnectTimeInSeconds: 5,
      },
    },
  });

  await app.startAllMicroservices();

  await app.listen(3000);
}
bootstrap();

```
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/6f0b6796042cbd34cdfdc3b1db6d087540246cf4).

##5. Start application
`$ docker-compose up --build`

![app running console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x5jkkc7fmzpwnz0lptbf.png)

Access rabbitmq panel in [http://localhost:15672](http://localhost:15672) with default crendentials login: **guest** and pass: **guest**

![rmq panel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mnnjnmn17ys6urzertl8.png)


##6. Create publisher and consumer
create-charge.publisher.ts

```
import { Inject } from '@nestjs/common';
import { ClientProxy } from '@nestjs/microservices';
import { catchError, firstValueFrom, throwError } from 'rxjs';
import { CreateChargeInputProps } from 'src/credit-card/domain/contracts/psp-service.interface';

export class CreateChargePublisher {
  constructor(
    @Inject('create_charge_publisher')
    private readonly clientProxy: ClientProxy,
  ) {}

  async publish(data: CreateChargeInputProps): Promise<void> {
    await firstValueFrom(
      this.clientProxy.emit('CREATE_CHARGE_PSP', data).pipe(
        catchError((exception: Error) => {
          return throwError(() => new Error(exception.message));
        }),
      ),
    );
  }
}
```
create-charge-on-psp.event-handler.ts
```
import { Controller, Inject } from '@nestjs/common';
import { Ctx, EventPattern, Payload, RmqContext } from '@nestjs/microservices';
import {
  CreateChargeInputProps,
  ICreateCharge,
} from 'src/credit-card/domain/contracts/psp-service.interface';
import { Pagarme } from 'src/credit-card/infra/psp/pagarme/pagarme.service';

@Controller()
export class CreateChargeOnPSPEventHandler {
  constructor(
    @Inject(Pagarme)
    private readonly pspService: ICreateCharge,
  ) {}
  @EventPattern('CREATE_CHARGE_PSP')
  async handle(
    @Payload() payload: CreateChargeInputProps,
    @Ctx() context: RmqContext,
  ): Promise<void> {
    console.log(payload);
    const channel = context.getChannelRef();
    const originalMsg = context.getMessage();
    try {
      await this.pspService.createCharge(payload);
    } catch (error) {
      console.log(error);
    }
    channel.ack(originalMsg);
  }
}
```
credit-card.module.ts
```
import { Module } from '@nestjs/common';
import { Pagarme } from './infra/psp/pagarme/pagarme.service';
import { CreateChargeUseCase } from './app/use-cases/create-charge.use-case';
import { CreateChargeController } from './presentation/controllers/create-charge.controller';
import { CreateChargeOnPSPEventHandler } from './presentation/event-handler/create-charge-on-psp.event-handler';
import { CreateChargePublisher } from './infra/rmq/publisher/create-charge.publisher';
import {
  ClientProxy,
  ClientProxyFactory,
  Transport,
} from '@nestjs/microservices';
import { ICreateCharge } from './domain/contracts/psp-service.interface';

@Module({
  providers: [
    Pagarme,
    {
      provide: CreateChargeUseCase,
      useFactory: (pspService: ICreateCharge) => {
        return new CreateChargeUseCase(pspService);
      },
      inject: [Pagarme],
    },
    CreateChargePublisher,
    {
      provide: 'create_charge_publisher',
      useFactory: (): ClientProxy => {
        return ClientProxyFactory.create({
          transport: Transport.RMQ,
          options: {
            urls: [`amqp://rabbitmq:5672`],
            queue: 'create_charge_psp',
            prefetchCount: 1,
            persistent: true,
            noAck: true,
            queueOptions: {
              durable: true,
            },
            socketOptions: {
              heartbeatIntervalInSeconds: 60,
              reconnectTimeInSeconds: 5,
            },
          },
        });
      },
    },
  ],
  controllers: [CreateChargeController, CreateChargeOnPSPEventHandler],
})
export class CreditCardModule {}

```
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/558df94b3b896f7728f71fbde440597b0e893b1b).

##7. Change the use case to send charge data to the rabbitmq queue instead of making the call to pagarme directly

create-charge.use-case.ts

```
import {
  CreateChargeInputProps,
  CreateChargeOutputProps,
} from 'src/credit-card/domain/contracts/psp-service.interface';
import { CreditCardChargeEntity } from 'src/credit-card/domain/entities/credit-card-charge.entity';
import { ICreateChargeUseCase } from 'src/credit-card/domain/use-cases/create-charge.use-case';
import { IPublisherCreateCharge } from 'src/credit-card/infra/rmq/publisher/create-charge.publisher';

export class CreateChargeUseCase implements ICreateChargeUseCase {
  constructor(private readonly publisher: IPublisherCreateCharge) {}

  async execute(
    props: CreateChargeInputProps,
  ): Promise<Omit<CreateChargeOutputProps, 'pspId' | 'value'>> {
    const entity = CreditCardChargeEntity.newCharge(props);
    console.log(entity);
    await new Promise((resolve) => setTimeout(resolve, 100)); //simulate database access
    await this.publisher.publish(props);
    return { ...props, status: 'PENDING' };
  }
}
```
_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/51b862d9d44dc0cace531567f75378e24a3c5531).

Now our endpoint for creating a credit card charge has an acceptable response time.

![postman](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9bnuqcz62vlj3mgo1u4q.png)

Several might halt their progress at this juncture; I've witnessed products in production following this approach. While it's not inherently incorrect, it hinders our ability to scale both the consumer and API horizontally and vertically independently. Additionally, the processing load on the consumer can adversely affect API response times. To address this, we aim to decouple startAllMicroservices() from listen(), instigating a shift towards a monorepo structure. Simultaneously, we'll develop a dedicated app for the consumer, fostering improved scalability and performance.

##8. Switch from standard mode to monorepo mode
Let's change the project structure to monorepo.
`$ nest generate app credit-card-consumer`

_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/142c3f2027d721d0d72bbf23c47427044ef6d46a).

It is imperative to modify the scripts responsible for building and launching the application. Additionally, we must craft a Dockerfile tailored for the consumer, configuring it to spawn three replicas, and subsequently, fine-tune the docker-compose.yml configuration accordingly.

_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/add7851d733339ccbc781efe35e7db1bfd5be813).

##9. Start application again

`$ docker-compose up --build`

Now, running the `$ docker ps` command we can see 3 instances of the consumer and 1 of the api running

![containers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2eluf984eazwqxsw8611.png)

##10. Decouple the consumer from the api
Currently, the consumer remains tightly coupled to the API, as we've only created the new credit-card-consumer app without implementing any modifications. The next step involves decoupling these components by transferring the responsibility of initiating the  consumer  to the credit-card-consumer application.

credit-card-consumer/src/main.ts
```
import { NestFactory } from '@nestjs/core';
import { CreditCardConsumerModule } from './credit-card-consumer.module';
import { RmqOptions, Transport } from '@nestjs/microservices';

async function bootstrap() {
  const app = await NestFactory.create(CreditCardConsumerModule);
  app.connectMicroservice<RmqOptions>({
    transport: Transport.RMQ,
    options: {
      urls: [`amqp://rabbitmq:5672`],
      queue: 'create_charge_psp',
      prefetchCount: 1,
      persistent: true,
      noAck: false,
      queueOptions: {
        durable: true,
      },
      socketOptions: {
        heartbeatIntervalInSeconds: 60,
        reconnectTimeInSeconds: 5,
      },
    },
  });

  await app.startAllMicroservices();
}
bootstrap();
```
See that here we do not need to invoke the app.listen() method.

nestjs-rabbitmq-example/src/main.ts
```
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();
```
See that here we only keep the app.listen() method.
Moreover, I have eliminated the services and controllers produced by the `$ nest generate app credit-card-consumer` command.

You can see that changes in this [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/bed1ea5cc07d77616b3b1f87889f6f094ed54560).

To finalize the decoupling process, it's essential to relocate the event handler to the credit-card-consumer, as it's responsible for both removing and processing messages from the queue. However, a challenge arises as the event handler relies on the Pagarme class within the nestjs-rabbitmq-example structure. To resolve this issue, let's transfer all shared components between the API and the consumer to the 'libs' folder, utilizing the 'nest generate library' resource. Importantly, we must refrain from importing any class from the API into the consumer under any circumstances.
I moved all the shared code to the credit-card lib, basically i kept only the _presentation_ layer for API and consumer.
Now our API is completely decoupled from the consumer, allowing us to scale each component independently.

_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/51d89b624a1c4c37959057db8e76e3166a95177b).

##11. Limit cpu and memory for containers
Establishing a controlled environment serves to restrict the RAM and CPU usage of containers.

_See these changes in the_ [commit](https://github.com/eduardoconti/nestjs-rabitmq-example/commit/0214f4c86066394f391f736e413dfc0c80ebfe3e).

##12. Conclusion
In conclusion, this article outlined the journey of creating a credit card payment application, emphasizing the efficient integration of technologies such as NestJS, RabbitMQ, Docker, Docker Compose, and the use of a monorepo. When addressing the initial challenge of inadequate response time in our endpoint, we implemented a message broker service to enable asynchronous processing of HTTP requests, resulting in significant improvements in system efficiency and responsiveness.

Additionally, we explored separating the consumer from the API within a monorepo, providing flexibility and individual scalability for both parts of the system. It is worth noting that, while I opted for the Clean Architecture, the choice of architecture and implementation within a monorepo remains at the developer's discretion.

This approach not only enhances application performance but also provides a solid foundation for future adaptations and customizations as needs evolve. By seamlessly integrating these technologies within a monorepo environment, we hope this article serves as a valuable guide for those seeking to optimize efficiency and scalability in their payment applications.

repo: [nestjs-rabitmq-example](https://github.com/eduardoconti/nestjs-rabitmq-example)

links:
- [nestjs monorepo docs](https://docs.nestjs.com/cli/monorepo)
- [nestjs rabbitmq docs](https://docs.nestjs.com/microservices/rabbitmq)
- [rabbitmq docs](https://www.rabbitmq.com/#getstarted)