message brokers, a brief walk-through
Tags: microservices, go, rabbitmq, python
Published: 2022-09-12 17:30:23

In an ever changing world of software development,utilizing microservices and distributed systems is the `de facto` standard for architecting software solutions, there is always a need to employ efficient means of message transfer between these various microservices. 
One of the fundamental aspect of developing these services infact, is to incorporate an `Inter-service` communication between them. And although there are numerous methods of approaching this; `Message Brokers` provide an efficient, scalable and cheaper means of data sharing betwixt these services. 
In this article we are briefly going to uncover the basics of message brokers, and build a distributed alert system that publishes alert messages to listed email addresses and a twitter account; using `fastapi` `golang/gin` and `rabbitmQ`.

## Message Brokers
A broker in layman's term is someone who negotiates/arranges a transaction between two entities (be it people, businesses e.t.c) similarly; Message brokers are an inter-service communication technology, that utilizes standard messaging protocols to promote rapid development, and data-sharing between distributed applications. Message brokers provide data-sharing, data-marshaling, and data persistences features to traditional middleware applications out of the box, and they as well guarantee the transfer of messages with low latency. They ensure delivery of messages even when a consumer(client) is inactive, this enhances reliability/uptime of  application's flow since each service is independent. A typical case for example is a service responsible for sending emails to recently signed up users, now assuming the service went down at the time a user is being registered, the downtime doesn't interupt the user's experience since the native backend only has to publish to the `sign-up queue`, which the consuming application can process from whenever it is restored.

## Publishers, Consumers and Queues
Message brokers primarily rely on three fundamental concepts for message transfer, they are `Publishers`, `Consumers` and `Queues` and although they might differ in name and behaviour pending on the message brokers you use, they can still be used interchangably(or not) with `Producers`, `Subscribers` and `Channels`; So lets have a brief walkthrough of each of these concepts.

`Publishers`: In messaging, publishers (or producers) represents an application that sends/publishes messages to a message broker instance. Depending on the protocol, the publisher sends the message to a broker which further routes the message to the approriate queue/channel. 

`Consumers`: consumers (or subscribers) represents application that readily listens on a queue or channel for new messages from the publisher. In order to consume messages, a consumer has to be registered to a queue.

`Queues`: queues as we know them are data structures that provides two kind of operations; `enqueue` whereby data is enterred into the queue through the rear or tail, and `dequeue` whereby data is removed from the queue through the front or head. In message brokers, queues are referenced via their names.

## Data Sharing
Since the essence of this article revolves around data tranfer, i'd quickly like to deviate your attention to commonly employed design practices when defining how applications should communicate with each other.
In the process of building a distributed system where message sharing is priority, you should be able to answer the following questions:-

- What data does your services need to share?
- What service(s) are the primary providers of the data, can they be identified?
- Does the nature of the data imply the introduction of a third-party software to share it?

Answering these questions are quintessential to the development of solid distributed ecosystems. When we start to write the codes for this article, we'd incoporate this services and see if our solution answer these questions.

## When and Why Should We Use Message Brokers?
Unlike the typical `client-server` paradigm, where a client (eg. web browser) sends a requests to a server, and blocks till it gets a response back from the server, message brokers allows a client to continue with other operations, whilst it processes the former requests. This is incredible because it saves time, and enhances the user's experience; Imagine a scenerio where you have to successfully send a user their registration email, before redirecting them to their profile or the login page, the time spent waiting on the email service would imply a very poor user experience.
Suffice to say it's best to use message brokers whenever you're  building something whose average processing time spans beyond the traditional http requests/responses duration. Keep in mind `time` isn't the only yardstick to consider when trying out message brokers, but it is one to always have in mind.

## Building an alert dispatch service
Up until this point we've seen ideal situations where we would want to build our applications using message brokers. This section aims at furthering your practical knowledge on the use of message brokers, and we are going to be building an `Alert Dispatch Service`, which is essentially an `API` that recieves a json post request:

![json sample](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9vldwzebir5ip1h0q9r6.png)
the API then serializes this json and publishes it to a message broker (`RabbitMq`) which in turn broadcasts it to any consumer listening on the queue, the consuming service serializes the recieved data and publishes the message to a twitter account and a list of email addresses. 
The Api is written in `python fastapi framework` and the consumer is written in `golang`, this example elaborates the use cases of message brokers as we can see how we've enabled communication between two independent  microservices, written in entirely different languages; whether this service has a use case in real world scenerios isn't considered as i only thought of a perfect example for using message brokers, which would be straight to beginners. 

## Fastapi Service (Publisher)
As i mentioned earlier, the `Api` and `publisher` service is written with python's modern web api framework [fastapi](https://fastapi.tiangolo.com/), it is quite easy to set up. First we need to set up our virtual environment, do this by running `python -m venv <name_of_your_virtual_env>` you can activate your virtual environment on windows by running 
 
`<name_of_venv>/bin/activate.bat` or `source <name_of_venv>/bin/activate` on linux/macOs. Next we need to install `fastapi` by running:
`pip install fastapi` 
this installs fastapi, alongside other dependencies it needs to function properly. Then we need to install `uvicorn` which will act as the ASGI (Asynchronous server gateway interface)  server for our application; to do this run `pip install uvicorn`. Once you've completed this setups, we can move on to writing our codes;

![Importing our dependencies and declaring the alert schema](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/42n10x6mwt8hyzkhnkxg.png)

From the code above,on line one and two we imported the `FastAPI` class from fastapi and also the `BaseModel` class from [pydantic](https://pydantic-docs.helpmanual.io/)
we then instantiate the fastapi class with a variable called app on line 4, we will see the use of this app variable shortly.On line 8 we create an Alert class which is the schema of the json data our api expects, and within the class we declare two fields, `topic` which is the topic of the alert and `message` which is the alert's content. Notice how our alert class inherits the `pydantic.BaseModel` class.

Next we write our api function on line 14, the `app` variable defines the http method required by the function and also declares it's route, then the function basically checks if an http request body contain's an alert type, _notice how all the `Serialization` is handled neatly by Fastapi behind the scenes_; if the alert is valid we calls our rabbitmQ publisher function else, we throw an error at the client. The Publisher function:

![Publisher function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58iwy284rp8uqwjiwxvn.png)
For the publisher, we import two packages `Pika` which is a python library for communicating with rabbitmQ and `json` to enable us serialize the alert instance. Then we create a connection to our rabbitmQ instance, and declare a `Queue` named `alert_queue`, when you declare a queue, rabbitmQ checks to see if the queue already exists, if it doesn't it creates the queue else it just goes ahead to publish the message. Next we declare a dictionary, to map our Alert's value, this is to enable us send messages in json format to our message broker, which can be delivered and serialized by our consumer service. Finally we publish the message to the `alert_queue` and return from the function body. 


## Golang Service (Consumer)
The consumer service is written in go, to elaborate the essence of this article i.e _using message brokers to transfer messages between different, independent microservices_. 
Unfortunately i won't be able to detail the code bit by bit since this article is becoming too lengthy, but it's quite straight-forward if you write Go regularly;

![connecting to rabbitmQ with go](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iyhqq2x8hculbtby3u3n.png)
Basically this code, connects to our rabbitmQ using the [streadway/amqp](https://github.com/streadway/amqp) package which is the golang standard library for communicating with rabbitmQ, next we declare a unique channel to enable us pass across message to a specific queue. Then we call the `Consume` function which takes in a parameter of `*amqp.Channel` which is the channel we created previously. 

![consume message from the queue](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e8yc8f3c4rcyl2zqda2d.png)
So the `Consume` function basically reads all the messages currently available in the queue and serializes them to an `alert` variable, then it sends a tweet to a specified twitter account with the alert's message as the tweet's content, it also sends an email with the alert's topic as subject and the alert's message as the email's content to my email address. I'm obviously not gonna go into the details of how to programmatically create a tweet or send an email. But if you're interested in learning how to send **Emails** with golang, you should definitely take look at my article [Sending E-mails with Go](https://dev.to/salemzii/sending-e-mails-with-go-ff0). All the sample codes written here are publicly available on this [github](https://github.com/salemzii/MQ-dispatcher) repository with instructions to run them.

## Conclusion
Being able to effectively exchange messages across multiple decoupled software components is vital for every distributed ecosystem; message brokers provides an easy, scalable and reliable means of achieving this.
In this article we've briefly covered the basics of publishing and consuming messages with rabbitmq, we went further to build an alert dispatch service that distributes alerts to listed email addresses and a twitter account, using **python** and **golang**. 

Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!