RabbitMQ on Kubernetes
Tags: devops, rabbitmq, kubernetes, opensource
Published: 2024-05-14 09:42:12

[RabbitMQ](https://www.rabbitmq.com/) is an open-source message broker software that implements a handful of messaging protocols, originally the [AMQP](https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) (Advanced Message Queuing Protocol), and also includes web-based ones such as [STOMP](https://stomp.github.io/) (Simple Text Orientated Messaging Protocol), [MQTT](https://mqtt.org/) (Message Queuing Telemetry Transport), and [WebSockets](https://en.wikipedia.org/wiki/WebSocket) to decouple applications that share asynchronous data. RabbitMQ not only serves as an attractive messaging system choice due to its robustness and well-maintained open-source nature but also stands out for its ease of use and configuration. Before creating our first RabbitMQ instance and cluster, let's explore some fundamental concepts around messaging and check out some common use cases.

> ℹ️ As of writing the latest RabbitMQ version is `3.13.2`. See change log [here](https://www.rabbitmq.com/release-information).

## Fundamentals

### Sync vs Async messaging 📟

In synchronous messaging, the sender waits for a response before proceeding, which can lead to **bottlenecks** or **inefficiencies**. Asynchronous messaging, on the other hand, frees up the sender by allowing messages to be sent without an immediate response, breaking the synchronous relationship between apps or microservices is known as `decoupling`.

Messaging systems excel in scenarios requiring asynchronous communication. Decoupled applications that exchange high message volumes and depend on reliable delivery are prime candidates to use it. From `distributed systems` to `microservices architectures` and `event-driven setups`, messaging brokers like RabbitMQ facilitate seamless communication across multipe diverse technological architectures.

### Producers and consumers 📲

In its simplest form, the messaging graph looks like this: `Producer - Queue - Consumer`

![simple-graph](https://github.com/glasskube/glasskube/assets/38757612/84a407f5-f44e-4026-9058-ef95f0feffa2)

### Messaging Brokers 🎛️

The Messaging broker is the core component of the messaging system, orchestrating the management of various messaging core entities, such as [connections](https://www.rabbitmq.com/docs/connections), [channels](https://www.rabbitmq.com/docs/channels), [queues](https://www.rabbitmq.com/docs/queues), and [exchanges](https://www.rabbitmq.com/docs/e2e) among others.

As the central hub, the broker encompasses all RabbitMQ features, ranging from clustering and mirroring to GUI and plugin management. It facilitates communication between clients and the broker itself, adhering to the open standard AMQP protocol for message transportation.

### Other key components 🧩

A producer application establishes a `connection` with the messaging broker via a `TCP connection`. Within this connection, `multiple channels` can be created, serving to logically separate data flows. Each channel facilitates the transmission of data to a `queue`. These queues act as intermediary storage, from which `consumer apps` connect to consume messages from. If the `Pub/Sub messaging pattern` is used, an `exchange` might also be present.

RabbitMQ also introduces the concept of virtual hosts `(vhosts)` to partition and isolate messaging resources within the broker, enabling distinct environments or applications to operate independently within the same RabbitMQ instance, and enhancing security and manageability.

In scenarios demanding heightened fault tolerance and message reliability, messaging `replicas` come into play. These [replicas](https://www.rabbitmq.com/docs/quorum-queues#replication), used in [quorum](https://www.rabbitmq.com/docs/quorum-queues) queues, ensure data redundancy and resilience by maintaining synchronized copies of messages across multiple nodes or clusters.

### Messaging patterns 🌐

#### Publish / Subscribe

The **Pub/Sub** pattern is used to broadcast messages to multiple subscribers who are interested in receiving notification updates. Here, instead of publishing a message to a single queue, messages are processed by an exchange which then routes the messages to multiple queues based on predetermined routing [rules](https://www.rabbitmq.com/tutorials/tutorial-four-go) and [bindings](https://www.rabbitmq.com/tutorials/tutorial-four-go#bindings). Each queue is bound to the exchange with a routing key.

![pub-sub](https://github.com/glasskube/glasskube/assets/38757612/0660831b-4b3b-46dd-9213-aa2df179a34c)

#### Worker queues

Worker queues are a way to distribute and parallelize the processing of tasks or messages across multiple workers or consumers. It can be simpler than the Pub/Sub pattern since it doesn’t have the added abstraction of an exchange agent. In this pattern, tasks or messages are published to a single queue. Multiple workers consume messages from this queue, with each worker processing tasks independently.

![worker-queue](https://github.com/glasskube/glasskube/assets/38757612/866e0eb5-7fd2-44cf-814b-103a04fa4b95)

## Oh by the way

If you are a fan of supporting Open Source projects working to make Kubernetes package management better for everyone then please consider supporting Glasskube, by giving us a Star on [GitHub](https://github.com/glasskube/glasskube) 🙏

[![Even kitty cats like giving stars](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1h9v5nqc7pfp6u3h1kqd.jpeg)](https://github.com/glasskube/glasskube)

## Benefits of messaging systems 👍

### Scaling

Queues can be added to `expand capacity` if needed. Even multiple broker nodes can be coupled to form messaging `clusters`.

### Batching

Batching involves grouping multiple messages into a single batch or bundle before transmitting them over the network. This can positively impact **performance**, **efficiency**, and **resource utilization**.

### Architecture decoupling

Applications can be developed and scaled **independently**, optimizing **resource usage** and **interoperability**.

### Reliability and Persistence

By `clustering`, `replicating` and `node federation`, no node or storage volume is fully responsible for data persistence. No node need be a single point of failure.

## Use cases

So which architectures or apps might benefit the most from messaging systems? Think of those that require asynchronous communication, scalability, reliability, and decoupling between components. Solutions like social media apps, financial services, IoT products, telecommunication apps, and supply chain management tools are just a few.

**Messaging app example:** **💬**  

For instance, every time you receive a message on WhatsApp or via text, the message payload is processed asynchronously using queues.

**Financial app example:** **📈**  

The same goes for when you use a financial app to put a buy order on your favorite stock. The buy details will be delivered from the front end of the producer app to the queue so that a backend consumer app can pick it up and continue processing your buy order.

```
       +-------------+
       |  Financial  |
       |    App      |
       +------+------+               +-----------------+
              |                      |                 |
              v                      |    RabbitMQ     |
  +-----------+-----------+          |    Broker       |
  |                       |          |                 |
  |  User Initiates Buy   +--------->+   Queue         |
  |      Order            |          |                 |
  |                       |          +-----------------+
  +-----------+-----------+                    |
              |                                |
              v                                |
       +-------------+                         |
       |  Consumer   |                         |
       |    App      |<------------------------+
       +-------------+

```

## Features of RabbitMQ 🧰

### Ease of use
    
Compared to other messaging systems, RabbitMQ’s simplicity of configuration and use stands out among the rest.
        
### Delivery Acknowledgement and consumer confirms
    
Typical of synchronous communication but less present in async is a received acknowledgment when a message was successfully consumed. RabbitMQ has a [feature](https://www.rabbitmq.com/docs/confirms) that can be enabled to achieve this.
        
### Distributed network

RabbitMQ as a messaging broker fits into distributed architectures, using producer and consumer apps connecting from remote hosts, this makes it desirable for the RabbitMQ broker to also be distributed there are three ways to accomplish this: [clustering, federations, or the shovel plugin](https://www.rabbitmq.com/docs/distributed).
        
### Tools + plugins
    
RabbitMQ has a series of tools and [plugins](https://www.rabbitmq.com/docs/plugins) to make managment and configuration much easier. Just as the management plugin which delivers a useful GUI from which mange management and monitoring tasks can be done as well as deployment plugins. Some useful plugins
        
  -   **rabbitmq\_management:** for management console access
            
  -   **rabbitmq\_prometheus:** exports metrics in Prometheus-compatible format, facilitating monitoring and observability.
            
  -   **rabbitmq\_federation:** enables cluster and exchange formation and data replication
            
  -   **rabbitmq\_peer\_discovery\_k8s:** used for automatic node discovery (cluster formation) in k8s environments
            
  -   RabbitMQ comes with some useful CLI tools `rabbitmqcli` ,`rabbitmq-plugins`,`rabbitmqadmin` and `rabbitmq-queues`. To be executed inside the RabbitMQ nodes they enable cluster management, plugin configurations, and an array of customizations.
        

## RabbitMQ Installation methods 

### Docker 🐳

```
# latest RabbitMQ 3.13
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management
```
> ℹ️ Check out a great in-depth installation [guide](https://www.youtube.com/watch?v=hfUIWe1tK8E) from Marcel Dempers using Docker

### Open Source RabbitMQ Server on multiple operating systems

-   Linux: ([Debian](https://www.rabbitmq.com/docs/install-debian), [RHEL, CentOS Stream, Fedora, Amazon Linux 2023](https://www.rabbitmq.com/docs/install-rpm))
    
-   [Windows](https://www.rabbitmq.com/docs/install-windows)
    
-   [MacOS](https://www.rabbitmq.com/docs/install-homebrew)
    

### RabbitMQ Cluster Kubernetes Operator ☸️

We will focus on this installation method since it’s the recommended method for working with RabbitMQ in Kubernetes environments.

## Install the RabbitMQ Cluster Kubernetes Operator

The RabbitMQ Cluster Kubernetes Operator automates the provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.

## Installation

Installing the RabbitMQ cluster operator can easily be achieved with the Glasskube package manager.

### Install Glasskube

If you already installed `glasskube` you can skip this step.
If not, `glasskube` can easily be installed the way you usually install packages for your operating system.

#### MacOs:
```
brew install glasskube/tap/glasskube
```
#### Linux (Ubuntu/Debian)
```
curl -LO https://releases.dl.glasskube.dev/glasskube_v0.4.0_amd64.deb
sudo dpkg -i glasskube_v0.4.0_amd64.deb
```
> More installation guides [here](http://localhost:3000/docs/getting-started/install/)


### Install the RabbitMQ cluster operator
Start the UI via the command line:
```
glasskube serve
```
Install RabbitMQ cluster operator via the Glasskube UI.
A package can be installed with a simple command.

```
glasskube install rabbitmq-operator
```

The process will wait until the package got successfully installed.

You will need access to a Kubernetes cluster running `Kubernetes 1.19` or later (You can easily create a local cluster by using [Minikube](https://minikube.sigs.k8s.io/docs/)) `kubectl` isn’t strictly speaking a dependency for installing packages via Glasskube, but it is the recommended way to interact with the cluster. Therefore, it is highly recommended. Installation instructions are available for macOS, Linux, and Windows.

> ℹ️ Glasskube will automatically install the cluster operator in a newly created namespace called `rabbitmq-system`

## Confirm Service Availability[](https://www.rabbitmq.com/kubernetes/operator/using-operator#service-availability "Direct link to Confirm Service Availability")

Before configuring your app to use RabbitMQ Cluster Kubernetes Operator, ensure that `RabbitmqCluster` Custom Resource is deployed to your Kubernetes cluster and is available.

To confirm this availability, run

```
kubectl get customresourcedefinitions.apiextensions.k8s.io
```

## Create a RabbitMQ Instance[](https://www.rabbitmq.com/kubernetes/operator/using-operator#create "Direct link to Create a RabbitMQ Instance")

The RabbitMQ Cluster Kubernetes Operator creates the necessary resources, such as `Services`, `StatefulSets`, `Secrets` and `ConfigMaps` in the same namespace in which the `RabbitmqCluster` was defined.

First, create a YAML file to define a `RabbitmqCluster` resource named definition.yaml.

> ℹ️ **Note:** The YAML file can have any name, but the steps that follow assume it is called `definition.yaml`.

Then copy and paste the below snippet into the file and save it:

```
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: definition
```

Next, apply the definition by running:

```
kubectl apply -f definition.yaml
```

Then verify that the process was successful by running:

``` 
kubectl get all -l app.kubernetes.io/name=definition
```

## Configure a RabbitMQ Instance[](https://www.rabbitmq.com/kubernetes/operator/using-operator#configure "Direct link to Configure a RabbitMQ Instance")

To configure a RabbitMQ instance in other words the cluster itself, open definition.yaml or edit the configuration in place by running:

```
kubectl edit rabbitmqcluster definition
```

By default all of the required manifests and objects to support your RabbitMQ have been created, to edit them you simply have to edit the Kubernetes manifest files.

## Access the RabbitMQ dashboard on Kubernetes

You can access the RabbitMQ management dashboard in multiple ways but editing the service object and using a load balancer. The quickest way that does not require you to change any default configuration is by port-forwarding the `definition-server-0` pod.

```
kubectl port-forward -n rabbitmq-system definition-server-0 8080:15672
```

Then you can access the dashboard login page to access the console. To find the credentials to access your deployment you will have to decode the username and password in the provisioned Kubernetes secret.

```
kubectl get secret definition-default-user -n rabbitmq-system -o yaml
```

The output will be something like this:

```
apiVersion: v1
data:
  default_user.conf: ZGVmYXVsdF91c2VYXVsdF91c2VyXzhMZklqOGFFQmROZmcKZGVmYXVsdF9wYXNzID0gd3IyWTl3MVJLMUtOdWN2S2ptS0FaX2R3dlRDSEhiYWEK
  host: ZGVmaW5pdGlvbi5yYWJiaXRtcS1zeXN0ZW0uc3Zj
  password: d3IyWTl3MVJLMUtOdWN2S2ptR3dlRDSEhiYWE=
  port: NTYMg==
  provider: cmFiml0bXE=
  type: cmFiYmlbXE=
  username: ZGVmYXVsdF91c2VMZklqFFQmRsbHd4b1BOZmc=
```

Decode the base64 `password` and `username` values and pass output into the login page

```
# Password
➜  ~ echo -n d3IyWTl3MVJLMUtOdW2S2ptS0FaX2R3dlRDSEhiYWE=  | base64 --decode
wr2Y9w1RK1KNucvKjmKAZ_dwvTCHHba%
# Username
➜  ~ echo -n ZGVmYXVsdF91c2VyXhMZklqOGFFQmRsbHd4b1BOZmc=  | base64 --decode
default_user_8LfIj8aEBdllwxoPNf%
```

![Screenshot 2024-05-09 at 15 33 08](https://github.com/glasskube/glasskube/assets/38757612/f90e367d-fa87-47f4-93b2-017c96757f68)

> ℹ️ To monitor your RabbitMQ instance check out this [documentation](https://www.rabbitmq.com/kubernetes/operator/operator-monitoring).

## Create a queue 🚃

There are multiple ways to declare queues in RabbitMQ, depending on your workflow and preferences.

### Option 1: Queue Declaration in a Producer App

In most cases, you'll declare a queue within the producer app written in your preferred programming language. When the app publishes its first message, the queue declaration will automatically create the queue if it doesn't already exist. This approach allows for dynamic queue creation as needed.

### Option 2: RabbitMQ Configuration Files

Alternatively, you can use RabbitMQ [configuration files](https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/import-definitions/definitions.json) to declare multiple types of RabbitMQ objects, including [vhosts](https://www.rabbitmq.com/docs/vhosts), [channels](https://www.rabbitmq.com/docs/channels), [exchanges](https://www.rabbitmq.com/tutorials/amqp-concepts), and [queues](https://www.rabbitmq.com/docs/queues). This method provides a more structured approach to managing RabbitMQ resources and configurations.

### Option 3: Management UI

For manual management, you can create queues directly from the RabbitMQ Management UI. It provides a user-friendly interface for interacting with RabbitMQ, allowing you to perform administrative tasks without using the command line.

#### **User and Virtual Host Creation (Optional)**

Before creating queues, it's recommended to set up users and virtual hosts to organize messaging workflows effectively. For example:

```
rabbitmqctl add_user jake jakepassword
rabbitmqctl set_user_tags jake administrator
rabbitmqctl set_permissions -p / jake ".*" ".*" ".*"

rabbitmqctl add_vhost vhost-1
rabbitmqctl set_permissions -p vhost-1 jake ".*" ".*" ".*"
```

#### Queue Creation via Management UI

1.  **Access RabbitMQ Management Console**: Open the RabbitMQ Management UI in your web browser.
    
2.  **Login with Credentials**: Use the credentials of the user you created (e.g., username: **jake**, password: **jakepassword**) to log in to the Management UI.
    
3.  **Navigate to Queues Section**: Click on the "Queues" tab in the Management UI to view existing queues.
    
4.  **Add Queue**: Scroll to the bottom of the page and click on the "Add a new queue" button to create a new queue.
    
- Specify the queue name, type, and other desired properties like node preference.
        
- Optionally, configure queue bindings, policies, and other advanced settings based on your requirements.
        
5.  **Save Changes**: Click on the "Add queue" or "Save" button to confirm and create the new queue.
    

By following these steps, you can easily create and manage queues in RabbitMQ according to your specific needs.

![Screenshot 2024-05-12 at 18 18 25](https://github.com/glasskube/glasskube/assets/38757612/b64a79c8-3b9c-4bfc-bef0-41344dc4eae5)

The queue is now ready:

![Screenshot 2024-05-12 at 18 19 48](https://github.com/glasskube/glasskube/assets/38757612/bae29873-901a-47d2-955b-8838f9023a55)  

## Build a RabbitMQ cluster 🐇

When building a production application with a messaging component to its architecture, considerations around high availability must be made. RabbitMQ offers a clustering feature that consists of joining multiple RabbitMQ brokers and replicating the configuration and the data contained inside the message queues.

![kubectl-graph-cluster](https://github.com/glasskube/glasskube/assets/38757612/69b9d748-f616-4944-8a4f-b0d9f0aed85f)

> ℹ️ To understand how clusters are formed we will need to understand a few concepts.  
By using the operator, much of the concepts we explore below will be configured for you by default. Nonetheless, it’s still valuable to understand what is happening under the hood.

### Authentication

For a RabbitMQ node to join forces and form a cluster with another node that need to share an Erlang cookie environment variable. This will ensure that nodes share the configuration data needed to consider themselves a cluster. This doesn’t ensure queue data replication though. This is where the concept of mirroring comes in.

### Quorum queues

The RabbitMQ quorum queue is a modern queue type, which implements a durable, replicated FIFO queue based on the [Raft consensus algorithm](https://raft.github.io/). Previously Classic Mirrored Queues were recommended but they have since been deprecated. Quorum queues are implemented much the same way yet they offer high availability via replication and focus on data safety. Policies can be applied to queues and it’s through these policies that we can configure the mirroring specification.

**Queue declaration**

In the past, replication of queues was specified by using policies in conjunction with Classic Queues. Quorum queues are created differently but should be compatible with all client applications which allow you to provide arguments when declaring a queue. The x-queue-type argument needs to be provided with the value quorum when creating the queue. For example, using the Elixir AMQP client1, declaring a Quorum Queue is as follows:

```
Queue.declare(publisher_chan, "my-quorum-queue", durable: true, arguments: [ "x-queue-type": "quorum" ])
```

> ℹ️ Manage quorum queues easily by using the `rabbitmq-queues` CLI. Here are some useful CLI [commands](https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/docs/rabbitmq-queues.8).

### Automatic Failover

Each quorum queue is made up by a primary replica, known as the `leader` in [Raft](https://raft.github.io/) terminology, along with potentially multiple secondary replicas, referred to as `followers`.

Initially, a leader is elected when the cluster is formed, and subsequently, if the current leader becomes unavailable.

Suppose a node hosting the leader of a quorum queue fails or stops for any reason. In that case, another node hosting one of the followers for that quorum queue will assume leadership and resume operations.

When failed or rejoining followers come back online, they undergo a process of resynchronization, commonly known as "catching up," with the leader. Unlike traditional mirrored queues, where a temporary replica failure requires a full resynchronization from the current leader, only the differential changes are transferred if a rejoining replica lags behind the leader.

### Fault tolerance

For optimal performance in RabbitMQ clusters, it's beneficial to constrain the quorum queue size to a smaller, `odd number` of nodes.

It's also worth mentioning that performance tends to degrade noticeably with quorum queue node sizes that exceed 5. Therefore, it’s strongly advised against deploying quorum queues on more than 7 RabbitMQ nodes.

## Add instances to the cluster 🪴

> ℹ️ Upon deployment, the cluster operator automatically generates a set of Kubernetes manifest files defining the default configurations for the RabbitMQ instance. You can inspect these configurations using the following command: Run `k get all -n rabbitmq-admin` to see all created objects. These default configurations can be customized and updated to better align with your specific requirements and preferences.

The RabbitMQ Kubernetes operator simplifies the setup process by handling essential tasks such as Mirroring, Failover, and Node federation automatically. This means that creating a RabbitMQ cluster is as straightforward as adjusting the number of replicas in the `RabbitmqCluster` definition file. With these capabilities built-in, managing RabbitMQ clusters becomes much more seamless and hassle-free.

Here I updated the `RabbitmqCluster` manifest to scale to `3` replicas:

> ℹ️ When specifying the number of replicas for the `RabbitmqCluster` object [an even number of replicas is highly discouraged](https://www.rabbitmq.com/docs/clustering#node-count). Odd numbers (1, 3, 5, 7, and so on) must be used.

```
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rabbitmq.com/v1beta1","kind":"RabbitmqCluster","metadata":{"annotations":{},"name":"definition","namespace":"rabbitmq-system"}}
  creationTimestamp: "2024-05-07T17:04:53Z"
  finalizers:
  - deletion.finalizers.rabbitmqclusters.rabbitmq.com
  generation: 6
  name: definition
  namespace: rabbitmq-system
  resourceVersion: "2729957"
  uid: f9b4cad8-68b4-489e-b7a1-baa3b91882d5
spec:
  delayStartSeconds: 30
  image: rabbitmq:3.13.1-management
  override: {}
  persistence:
    storage: 10Gi
  rabbitmq:
    additionalConfig: |
      load_definitions = /etc/rabbitmq/definitions.json
  replicas: 3 # added three replicas which will for the RabbitMQ cluster
  resources: 
    limits:
      cpu: "2"
      memory: 2Gi
    requests:
      cpu: "1"
      memory: 2Gi
  secretBackend:
    externalSecret: {}
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800
...
```

![Screenshot 2024-05-12 at 18 27 08](https://github.com/glasskube/glasskube/assets/38757612/231868a4-2c14-490d-b164-a9417f5c6344)

Not only have the nodes been added to the cluster but the queue has been automatically replicated to all cluster nodes too

![Screenshot 2024-05-12 at 18 28 56](https://github.com/glasskube/glasskube/assets/38757612/5d1520ec-3e70-409d-8c66-a7873485e86a)

Messaging brokers such as RabbitMQ play a central role in [event-driven architectures](https://en.wikipedia.org/wiki/Event-driven_architecture), offering `flexibility` for `task processing`, `workflow orchestration`, and `service integrations` while offering a selection of `messaging patterns` to choose from to best suit your architecture. An obvious choice when prioritizing `delivery reliability` and `fault tolerance`, due to its native acknowledgment and message persistence features.

Getting started with RabbitMQ is easier than ever especially now that [Glasskube](https://github.com/glasskube/glasskube) integrates the cluster controller for easy installation and management. This guide aims to offer the necessary insights for setting up a functional instance or RabbitMQ cluster within a Kubernetes environment. While there's so much more to learn regarding RabbitMQ such as this and that, I hope this primer provides ample groundwork to kickstart your RabbitMQ journey.


If you like this sort of content and would like to see more of it, please consider supporting us by giving us a Star on GitHub 🙏

[![Even kitty cats like giving stars](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1h9v5nqc7pfp6u3h1kqd.jpeg)](https://github.com/glasskube/glasskube)