MQTT over TCP
Tags: mqtt, emqx, python, wireshark
Published: 2021-05-14 01:00:51

# MQTT over TCP

## Introduction

In this article I will be trying to understand how is the flow in a single MQTT connection. For those who don't know what is **MQTT** here is their [main page](https://mqtt.org/), where you can find reliable information about this protocol, along with several use cases. I personally recommend you to check this protocol.

## Prerequisites

1. Docker
2. emqx/emqx docker image
3. Python
4. paho-mqtt python library
5. Wireshark

### Setup

Having **Docker** and **Python 3** installed, run the following command:

* Downloading **emqx/emqx** Docker image

  ```bash
  docker pull emqx/emqx
  ```

* Running this image as a container

  ```bash
  docker run -d --name emqx -p 18083:18083 -p 1883:1883 emqx/emqx:latest
  ```

* Making sure container is up without problem

  ```bash
  docker ps -a
  ```

Check the status of the container, it should be **Up**.

* Install **paho-mqtt** python library

  ```bash
  pip install paho-mqtt
  ```


* Run Wireshark

  ```bash
  sudo wireshark
  ```

After all these we should be good to go.

## Preparing python scripts

The python scripts are the same that you can find in [MQTT in Python(Paho)](https://www.emqx.io/blog/how-to-use-mqtt-in-python), with a modification. We don't want to send that amount of message, instead I will send only one message. All the analysis will be with the publisher script, but you can perform the same task with the subscriber script. 

```python
# pub.py modified from original

import random
import time
from paho.mqtt import client as mqtt_client

# Configuration
broker = '0.0.0.0'
port = 1883
topic = "/python/mqtt"
client_id = f"python-mqtt-{random.randint(0, 1000)}"

def connect_mqtt():
    """
    connect_mqtt: handle the connection.
    """
    def on_connect(client, userdata, flags, rc):
        if rc == 0:
            print("Connected to MQTT Broker!")
        else:
            print("failed to connect returned code %d\n", rc)

    client = mqtt_client.Client(client_id)    
    client.on_connect = on_connect
    client.connect(broker, port)
    return client

def publish(client):
    """
    publish: publish data to the broker.
    """
    msg_count = 0    
    time.sleep(1)
    msg = f"messages: {msg_count}"
    result = client.publish(topic, msg)
    # result: [0, 1]
    status = result[0]
    if status == 0:
        print(f"Send `{msg}` to topic: `{topic}`")
    else:
        print(f"Failed to send message to topic `{topic}`")
    msg_count += 1

def run():
    client = connect_mqtt()
    client.loop_start()
    publish(client)

if __name__ == '__main__':
    run()
```

## General overview of this setup

If you are new to MQTT all this setup it may be pretty strange for you. I'll do my best to explain to you why all this is needed, a pretty sketch with [excalidraw](https://github.com/excalidraw/excalidraw) will help you to understand.

![Pub, EMQX, Sub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/skezbu6rptw6q9n157rc.png)

According to this sketch, both the publisher and the subscriber are connected to emqx/emqx. In this case I'll be looking only one part of this communication, the interaction between the publisher and the broker.

## Looking inside the packets

It's time to see the packets on this communication, with the idea that at least get an idea of how this works. If you followed all the steps above, you should have emqx running and waiting for connection in **1883** port. Now make sure also to select the appropriate network interface with Wireshark, as I explained on the tutorial related to [TCP](https://dev.to/gealber/tcp-handshake-3ac). The filter in this case will be `tcp.port == 1883 | mqtt`. With all these, it is time to run the `pub.py` script, let's see what happens.

```bash
python pub.py
```

**Output:**

```bash
Connected to MQTT Broker!
Send `messages: 0` to topic: `/python/mqtt`
```

If you read the output, it seems that the publisher just connected and send the message to the topic `/python/mqtt`. Mmm... :thinking: but that's not all, there's a lot of back and forward between the publisher and emqx, let's explore the packets that we captured with Wireshark.

After running the script you should be seeing something like this.

![Traffic mqtt and tcp.port 1883](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/72jz3vuaz2i4mo9rqyz6.png)

As I commented to you, is not just sent.

#### TCP handshake

The first three packets that you can see on Wireshark are related to the **TCP handshake** between the publisher and emqx. 

![TCP handshake](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u88dkokttcvd0wkw8vtw.png)

#### CONNECT command

![connect pub emqx](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xi0ea7261x4d6mlf50nx.png)

After the handshake is established the publisher send to emqx a CONNECT COMMAND, this is something new. What is this **CONNECT** command? This **CONNECT** command is part of **MQTT** protocol, let's see inside the packet

![CONNECT command mqtt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d9fafopsvjxqybeljpq1.png)

Mmm...:thinking: is not so easy to take an idea just with this picture, right? As a personal recommendation, you should take this picture and the [MQTT specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html), in case you are too lazy to read part of a specification I understand, so stay with me and question everything I say, that will help you to think more critical about this subject. Any specification of a protocol, is a document very dense and extremely technical, so don't worry if you don't understand everything. In our simple case you should be looking at section [2 MQTT Control Packet format](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901019) and also skimming section [1 Introduction](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901000). 

According to the specification an MQTT Control Packet is structured in the following way:

* Fixed Header, present in all MQTT Control Packets
* Variable Header, present in some MQTT Control Packets
* Payload, present in some MQTT Control Packets

Now back to the picture. Could we identify some of these parts? At least I will make my best with the Fixed Header. Let's see, according to the photo, which represent the traffic intercepted by Wireshark, the **Fixed Header** would be composed by:

* Header Flags, which specify the Message Type, in our case is a **Connect command**.
* Connect Flags,  which contains flags specific to our Message Type. One really important here, is the **QoS**, which stands for **Quality of Service**. **QoS** can take three values: 0, 1 or 2. The section [4.3 Payload, present in some MQTT Control Packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901234) is dedicated to it, check it out to get an idea of its importance.

There's also more information that Wireshark show you, like **Protocol name**, **Version**, **Msg length**, and **Client ID**. The first two are used to identify the request with the protocol, is like saying "hey I'm speaking MQTT v3.1.1", in this way the other part can return an appropriate response. While the **Client ID**, like the name suggests, is a way to identify the other part connected.

#### EMQX TCP ACK

After the **CONNECT** command, **emqx** sends an **ACK** packet over **TCP** with the common data related to this packet type.

![emqx ack tcp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lzcfafxfarcxlqhbjm7a.png)

#### EMQX CONNECT ACK command

![emqx conn ack mqtt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pgkqu64928lttp13d5ul.png)

Now, using the **MQTT** protocol, is sent to the publisher the **CONNECT ACK** command. This is the equivalent of saying: "Ok, I acknowledge your attempt of connection", all in the sense of **MQTT** protocol, because as we saw before they are already connected by **TCP**. They are just agreeing in "talking MQTT" for now on, here's a screenshot of how Wireshark sees this case.

![Conn ack command emqx, packet](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dgubebx44okxeo8ph41x.png)

Note the **Return Code**, as I mention before it tell us **"Connection accepted"**, thanks to Wireshark we can see this in human-readable format.

#### Publisher send ACK by TCP and  emqx response with an ACK by TCP

![publisher ack tcp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5cps7bg966db4p2wvfpf.png)

![emqx ack tcp again](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iy817or7rarv0b39dhxo.png)

**TCP** chatting between both points before sending the actual message.

#### Publisher send Publish message command

![publisher pub msg](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yebzw74oa1plsabv3e0b.png)

Now actually sending the data from the publisher to the emqx

![pub msg packet](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ovey2rtattj00xviicqg.png)

Here's where all the important information goes, take a look at the picture. We got **Msg Length**, **Topic Length**, **Topic** and the actual **Message**. It is in this last packet that the publisher really send all the data, he specified the **Topic** and the message, and many other things too. In this way a subscriber that is connected to emqx, and is "subscribed" to this specific topic, could read the message. 

## Conclusion

We saw how chatty are protocols, specially this one. In this case we analyze MQTT over TCP, but there's another case that deserve our attention, and is the case of MQTT over Websocket. We didn't covered the part of the subscriber here, but you should try it yourself.

That's all :wave:

## Bibliography

1. [EMQX Docker image](https://hub.docker.com/r/emqx/emqx)
2. [MQTT specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
3. [HiveQT](https://www.hivemq.com/)
4. [How to use MQTT in Python](https://www.emqx.io/blog/how-to-use-mqtt-in-python)