📈 Working with RabbitMQ in Golang by examples
Tags: go, tutorial, beginners, rabbitmq
Published: 2021-03-31 20:59:59

## Introduction

Hey, DEV people! 😉 Today, I'll cover the topic of working with a Message broker called RabbitMQ in your Go projects. There will be both a theoretical part and practice.

Of course, the article is more aimed at those who _just want_ to understand this topic. But I call on more **experienced** people to help cover this topic even better in the comments on this article. Together we can do a lot!

<a name="toc"></a>
### 📝 Table of contents

- [What is a message broker?](#what-is-a-message-broker)
- [What problems will be able to solve?](#what-problems-will-be-able-to-solve)
- [Full project code](#full-project-code)
- [Setting up RabbitMQ](#setting-up-rabbitmq)
- [Setting up Fiber as a sender](#setting-up-fiber-as-a-sender)
- [Setting up a message consumer](#setting-up-a-message-consumer)
- [Running the project](#running-the-project)

<a name="#what-is-a-message-broker"></a>
## What is a message broker?

This is an architectural pattern in distributed systems, where a message broker is an application that converts a single protocol message from the source application to a protocol message from the destination application, thereby acting as an intermediary between them.

Also, the tasks of the message broker include:

- Checking the message for errors;
- Routing to specific receiver(s);
- Splitting the message into several smaller ones, and then aggregating the receivers' responses and sending the result to the source;
- Saving the messages to a database;
- Calling web services;
- Distributing messages to subscribers;

> 🤔 But what is it anyway? Well, let's translate it into our language.

If you simplify this huge description, you can portray the message broker as a post office in real life (_which you have encountered many times_):

1. A sender (_user of your product_) brings a parcel (_any data_) to the post office and specifies the addressee for receipt (_another service_).
2. A post office employee accepts the parcel and places it in the storage area (_puts it in the queue to be sent_) and issues a receipt that the parcel has been successfully accepted from the sender.
3. After some time, the parcel is delivered to the addressee (_another service_), and he doesn't have to be at home to accept the parcel. In this case, his parcel will wait in a mailbox until he receives it.

![message broker schema](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vmumd57vnbnq7h1z25xw.jpg)

[↑ Table of contents](#toc)

<a name="#what-problems-will-be-able-to-solve"></a>
## What problems will be able to solve?

One of the most important problems that can be solved by using this architectural pattern is to parallelize tasks with a guaranteed result, even if the receiving service is unavailable at the time of sending the data.

With the total dominance of microservice architecture in most modern projects, this approach can maximize the performance and resilience of the entire system.

> 👌 It sounds a bit confusing... but let's use the post office analogy again! 

Once the sender gives his parcel to the post office employee, he no longer cares how his parcel will be delivered, but he does know that it will be delivered anyway!

[↑ Table of contents](#toc)

<a name="#full-project-code"></a>
## Full project code

For those who want to see the project in action:

{% github koddr/tutorial-go-fiber-rabbitmq no-readme %}

[↑ Table of contents](#toc)

<a name="#setting-up-rabbitmq"></a>
## Setting up RabbitMQ

As usual, let's create a new Docker Compose file:

```yaml
# ./docker-compose.yml

version: "3.9"

services:
  
  # Create service with RabbitMQ.
  message-broker:
    image: rabbitmq:3-management-alpine
    container_name: message-broker
    ports:
      - 5672:5672   # for sender and consumer connections
      - 15672:15672 # for serve RabbitMQ GUI
    volumes:
      - ${HOME}/dev-rabbitmq/data/:/var/lib/rabbitmq
      - ${HOME}/dev-rabbitmq/log/:/var/log/rabbitmq
    restart: always
    networks:
      - dev-network

networks:
  # Create a new Docker network.
  dev-network:
    driver: bridge
```

> ☝️ Please note! For the initial introduction to RabbitMQ we will not create a cluster and use a load balancer. If you want to know about it, write a comment below.

[↑ Table of contents](#toc)

<a name="#setting-up-fiber-as-a-sender"></a>
## Setting up Fiber as a sender

To connect to the message broker, we will use the [Advanced Message Queuing Protocol](https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) or `AMQP` for short. The standard port for RabbitMQ is `5672`.

Okay, let's write a simple data sender using [Fiber](https://github.com/gofiber/fiber) web framework:

```go
// ./sender/main.go

package main

import (
	"log"
	"os"

	"github.com/gofiber/fiber/v2"
	"github.com/gofiber/fiber/v2/middleware/logger"
	"github.com/streadway/amqp"
)

func main() {
	// Define RabbitMQ server URL.
	amqpServerURL := os.Getenv("AMQP_SERVER_URL")

	// Create a new RabbitMQ connection.
	connectRabbitMQ, err := amqp.Dial(amqpServerURL)
	if err != nil {
		panic(err)
	}
	defer connectRabbitMQ.Close()

	// Let's start by opening a channel to our RabbitMQ
	// instance over the connection we have already
	// established.
	channelRabbitMQ, err := connectRabbitMQ.Channel()
	if err != nil {
		panic(err)
	}
	defer channelRabbitMQ.Close()

	// With the instance and declare Queues that we can
	// publish and subscribe to.
	_, err = channelRabbitMQ.QueueDeclare(
		"QueueService1", // queue name
		true,            // durable
		false,           // auto delete
		false,           // exclusive
		false,           // no wait
		nil,             // arguments
	)
	if err != nil {
		panic(err)
	}

	// Create a new Fiber instance.
	app := fiber.New()

	// Add middleware.
	app.Use(
		logger.New(), // add simple logger
	)

	// Add route.
	app.Get("/send", func(c *fiber.Ctx) error {
		// Create a message to publish.
		message := amqp.Publishing{
			ContentType: "text/plain",
			Body:        []byte(c.Query("msg")),
		}

		// Attempt to publish a message to the queue.
		if err := channelRabbitMQ.Publish(
			"",              // exchange
			"QueueService1", // queue name
			false,           // mandatory
			false,           // immediate
			message,         // message to publish
		); err != nil {
			return err
		}

		return nil
	})

	// Start Fiber API server.
	log.Fatal(app.Listen(":3000"))
}
```

As you can see, at the beginning we create a new connection to RabbitMQ and a channel to send data to the queue, called `QueueService1`. With a GET request to [localhost:3000/send](http://localhost:3000/send), we can pass a needed text in a `msg` query parameter, which will be sent to the queue and next to the consumer.

Now create a new Dockerfile called `Dockerfile-sender` in the root of the project in which we describe the process of creating the container for sender:

```dockerfile
# ./Dockerfile-sender

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./sender/main.go .

# Set necessary environment variables needed 
# for our image and build the sender.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o sender .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/sender", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/sender"]
```

All that remains is to update the Docker Compose file so that it takes this Dockerfile into account when creating a container with RabbitMQ: 

```yaml
# ./docker-compose.yml

version: "3.9"

services:
  
  # ...
  
  # Create service with Fiber sender.
  sender:
    container_name: sender
    ports:
      - 3000:3000
    build:
      context: .
      dockerfile: Dockerfile-sender
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - message-broker
  
  # ...

# ...
```

<a name="#setting-up-a-consumer"></a>
## Setting up a message consumer

The message consumer should be able to accept messages from the broker's queue and output in the logs the message sent from the sender.

Let's implement such a consumer:

```go
// ./consumer/main.go

package main

import (
	"log"
	"os"

	"github.com/streadway/amqp"
)

func main() {
	// Define RabbitMQ server URL.
	amqpServerURL := os.Getenv("AMQP_SERVER_URL")

	// Create a new RabbitMQ connection.
	connectRabbitMQ, err := amqp.Dial(amqpServerURL)
	if err != nil {
		panic(err)
	}
	defer connectRabbitMQ.Close()

	// Opening a channel to our RabbitMQ instance over
	// the connection we have already established.
	channelRabbitMQ, err := connectRabbitMQ.Channel()
	if err != nil {
		panic(err)
	}
	defer channelRabbitMQ.Close()

	// Subscribing to QueueService1 for getting messages.
	messages, err := channelRabbitMQ.Consume(
		"QueueService1", // queue name
		"",              // consumer
		true,            // auto-ack
		false,           // exclusive
		false,           // no local
		false,           // no wait
		nil,             // arguments
	)
	if err != nil {
		log.Println(err)
	}

	// Build a welcome message.
	log.Println("Successfully connected to RabbitMQ")
	log.Println("Waiting for messages")

	// Make a channel to receive messages into infinite loop.
	forever := make(chan bool)

	go func() {
		for message := range messages {
			// For example, show received message in a console.
			log.Printf(" > Received message: %s\n", message.Body)
		}
	}()

	<-forever
}
```

Okay, in the same way with sender, let's create a new Dockerfile called `Dockerfile-consumer` to describe the process of creating a container for the message consumer:

```dockerfile
# ./Dockerfile-consumer

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./consumer/main.go .

# Set necessary environment variables needed 
# for our image and build the consumer.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o consumer .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/consumer", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/consumer"]
```

Yes, you guessed it! 😅 Once again, we will have to put a description in the Docker Compose file to create the container for the consumer:

```yaml
# ./docker-compose.yml

version: "3.9"

services:
  
  # ...
  
  # ...
  
  # Create service with message consumer.
  consumer:
    container_name: consumer
    build:
      context: .
      dockerfile: Dockerfile-consumer
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - sender
      - message-broker

# ...
```

Great! Finally, we're ready to put everything together and run the project.

[↑ Table of contents](#toc)

![running the project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5lta4yywl1szpwdhkmgs.jpg)

<a name="#running-the-project"></a>
## Running the project

Just run this Docker Compose command:

```bash
docker-compose up --build
```

Wait about 1-2 minutes and make a few HTTP request to the API endpoint with different text in `msg` query:

```bash
curl \
    --request GET \
    --url 'http://localhost:3000/send?msg=test'
```

Next, go to this address: `http://localhost:15672`. Enter `guest` both as login and password. You should see the RabbitMQ user interface, like this:

![rabbitmq gui](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjutaiugkbb7a1yh1ds6.png)

To simply view the logs and metrics inside the Docker containers, I recommend to use a command line utility, called [ctop](https://github.com/bcicen/ctop):

![ctop](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uxgimwizcomzaqpldno7.png)

If you turn off the consumer container, but continue to make HTTP requests, you will see that the queue starts to accumulate messages. But as soon as the consumer is started up again, the queue will clear out because the consumer will get all the messages sent.

🎊 Congratulations, you have fully configured the message broker, sender, consumer and wrapped it all in isolated Docker containers!

[↑ Table of contents](#toc)

<a name=""></a>
## Photos and videos by

- Vic Shóstak https://shostak.dev
- Fatos Bytyqi https://unsplash.com/photos/Agx5_TLsIf4

<a name=""></a>
## P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My projects that need your help (and stars) 👇

- 🔥 [gowebly][gowebly_url]: A next-generation CLI tool for easily build amazing web applications with Go on the backend, using htmx & hyperscript and the most popular atomic/utility-first CSS frameworks on the frontend.
- ✨ [create-go-app][cgapp_url]: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.
- 🏃 [yatr][yatr_url]: Yet Another Task Runner allows you to organize and automate your routine operations that you normally do in Makefile (or else) for each project.
- 📚 [gosl][gosl_url]: The Go Snippet Library provides snippets collection for working with routine operations in your Go programs with a super user-friendly API and the most efficient performance.
- 🏄‍♂️ [csv2api][csv2api_url]: The parser reads the CSV file with the raw data, filters the records, identifies fields to be changed, and sends a request to update the data to the specified endpoint of your REST API.
- 🚴 [json2csv][json2csv_url]: The parser can read given folder with JSON files, filtering and qualifying input data with intent & stop words dictionaries and save results to CSV files by given chunk size.

<!-- Footer links -->

[gowebly_url]: https://github.com/gowebly
[cgapp_url]: https://github.com/create-go-app
[yatr_url]: https://github.com/koddr/yatr
[gosl_url]: https://github.com/koddr/gosl
[csv2api_url]: https://github.com/koddr/csv2api
[json2csv_url]: https://github.com/koddr/json2csv