Dynamic Mapper - Map (MQTT) device data in a zero-code approach!
Tags: iot, mqtt, cumulocity, dynamic
Published: 2024-06-04 07:28:25

---
title: Dynamic Mapper - Map (MQTT) device data in a zero-code approach!
published: true
date: 2024-04-30 12:37:37 UTC
tags: iot, mqtt, cumulocity, dynamic
canonical_url: https://tech.forums.softwareag.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/265444/1
---

 

## Introduction

Since years Cumulocity IoT is providing a MQTT endpoint dedicated to the Cumulocity Domain Model to connect MQTT devices. But what about devices using a pre-defined topic structure and any custom payload which cannot be modified because they are so called “closed”? Since the latest release of Cumulocity there is a new Service in private-preview - the **MQTT Service** - which is a full-fledged MQTT Broker without the limitation of pre-defined topic structure and payload. Still, it does not solve the problem to map device data to the Cumulocity Domain Model. Therefor we [@Christof\_Strack](https://tech.forums.softwareag.com/u/christof_strack) and myself [@Stefan_Witschel](https://tech.forums.softwareag.com/u/Stefan_Witschel) implemented an open-source tool called **Dynamic Mapper** which allows you to map any data bi-directional in a zero-code approach.

In the first part of this article I will explain some patterns and concepts behind Device Connectivity and MQTT devices. In the second part I will provide a step-by-step guide how to connect a sample MQTT device and to send telemetry data to Cumulocity IoT.

**Let’s go!**

## MQTT Endpoint

Cumulocity provides an MQTT endpoint that let you connect devices via MQTT but they must support a pre-defined topic structure and the SmartREST data model or Cumulocity domain model via JSON.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/a/e/ae56e811df292a7049821dde8bbb466a814771eb_2_690x350.png)](https://global.discourse-cdn.com/techcommunity/original/3X/a/e/ae56e811df292a7049821dde8bbb466a814771eb.png "image")

This MQTT endpoint concept therefore only works for devices that can be customized or allow an installation of an agent like [thin-edge.io](http://thin-edge.io) but not for devices that are _closed_ and cannot be modified with an additional agent. If you wanted to connect devices with a pre-defined topic structure and payload to Cumulocity you had to use an externally hosted MQTT Broker and add some logic to map it to Cumulocity IoT.

## MQTT Service

Since the latest release in Q2 2024, the **MQTT Service** feature will be available as a private preview, which covers this gap as part of the platform.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/7/5/751d802f321ac6409ee7f9e933b1538ecabfc727_2_690x201.png)](https://global.discourse-cdn.com/techcommunity/original/3X/7/5/751d802f321ac6409ee7f9e933b1538ecabfc727.png "image")

Still, it will not cover the **map** and the **transport** part of an agent. So this must be covered by a custom component (microservice) which I want to highlight as a potential solution in this article:

> **[The Dynamic Mapper](https://github.com/SoftwareAG/cumulocity-dynamic-mapper)**

## Dynamic Mapper

As stated in the chapter before the **Dynamic Mapper** tries to address the need to dynamically map **any** topic structure and data payload to the Cumulocity IoT domain model. We implemented it in a modular approach so it can support multiple message brokers and any data format like JSON, Protobuf, Binary, CSV, and many more.

### Architecture

To use the dynamic mapper you need to use a message broker either hosted externally or using the MQTT Service hosted by Cumulocity.

The Dynamic MQTT Mapper itself provides two artifacts:

1. A **microservice** which can be uploaded to Cumulocity IoT and
2. A **frontend** as a plugin which can extend existing Applications like the Administration App.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/7/f/7f90c6d38cbc2575a8cde6fbc08e0a991bc8b817_2_690x358.png)](https://global.discourse-cdn.com/techcommunity/original/3X/7/f/7f90c6d38cbc2575a8cde6fbc08e0a991bc8b817.png "image")

### Components

The Dynamic Mapper has multiple components:

- **Connector clients** - the mapper has a modular connector concept which allows to add multiple connectors for each broker with required properties. Out of the box it supports **MQTT** , **MQTT Service** and **Kafka**. They allow to connect, publish and subscribe to topics.
- **Data Mapper** - handling of inbound & outbound messages from/to Broker in regards to the Cumulocity IoT Domain Model… Also includes an expression runtime [JSONata](https://jsonata.org/) to execute expressions. Data Mappers again can be separated into Processors. For example, there is a _JSONProcessor_ handling any JSON payload and _FlatFileProcessor_ for processing flat-file payload (e.g. 10,Temperatur,20,°C) and it can be also easily extended with processor extensions.
- **C8Y Client** - implements part of the Cumulocity IoT REST API to integrate data. Also subscribes on the Notification 2.0 API to receive messages from Cumulocity IoT and forwards it to the data mapper.
- **REST Endpoints** - custom endpoints that are used by the MQTT frontend plugin or can be used to add mappings programmatically
- **Mapper Frontend** - A plugin for Cumulocity IoT to provide a UI for Broker Configuration, Data Mapping, Testing, Monitoring, and much more.

Also, the mapper microservice is fully **multi-tenant enabled**. You can deploy it once in your enterprise tenant and subscribe to subtenants to manage connections and mapping per tenant without using additional resources.

Let’s summarize the key features of the dynamic mapper:

- No-code data mapping capabilities
- Supports multiple types of message brokers like MQTT, MQTT Service or Kafka and can be extended easily.
- Supports multiple broker connections in parallel
- Bi-directional messaging: Broker ↔ Mapper ↔ C8Y
- Multi-data payload support like JSON, Protobuf, CSV, binary, and others
- Fully multi-tenant enabled
- Open-source developed and maintained
- Easy to extend (Broker connectors, payload processors, etc.)

No let’s check how to use it in practice. In the following I will guide you step-by-step to make use of that Dynamic Mapper.

## Step-by-step guide

### 1. Prerequisites

To follow this guide you need the following:

- A [Cumulocity IoT Tenant](https://signup.softwareag.cloud/#/basic-b?product=cumulocity&utm_referrer=TechCommunity) with microservice feature enabled. In trial tenants this feature is disabled. Please get in contact with your sales rep or support to request that feature for your tenant.
- A MQTT Client Tool like [MQTTx](https://mqttx.app)
- A publicly available MQTT Broker or MQTT Service. You can set up your own one or use existing sandboxes like [test.mosquitto.org](https://test.mosquitto.org/) or [others](https://mntolia.com/10-free-public-private-mqtt-brokers-for-testing-prototyping/). For this guide, I will use `test.mosquitto.org`.

### 2. Setup & Deployment

#### 1. Download of artifacts

As a next step we gonna download the artifacts of the [Dynamic Mapper](https://github.com/SoftwareAG/cumulocity-dynamic-mapper): [Releases · SoftwareAG/cumulocity-dynamic-mapper · GitHub](https://github.com/SoftwareAG/cumulocity-dynamic-mapper/releases)  
Make sure to select the latest release. In the assets part of the release you’ll find two zip files:

1. `dynamic-mapping-service.zip` - This is the microservice of the dynamic mapper.
2. `dynamic-mapping-ui.zip` - This is the frontend plugin of the dynamic mapper.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/2/4/24d3eade33db9d17715d13a555686426f4ae812e_2_690x184.png)](https://global.discourse-cdn.com/techcommunity/original/3X/2/4/24d3eade33db9d17715d13a555686426f4ae812e.png "image")

Download them and store them somewhere on your disk.

#### 2. Upload of artifacts

Now we go the our Cumulocity IoT Tenant and upload both assets.

##### Microservice

We start with the microservice `dynamic-mapping-service.zip`.

In the Application Switcher we go to **Adminstration**. Afterwards we select **Ecosystem** → **Microservices**. Now we should see the “Add microservice” button on top right. If this is not the case please check the [Prerequisites](https://tech.forums.softwareag.com#prerequisites) that the microservice-hosting-feature is assigned to your tenant.

[![image](https://global.discourse-cdn.com/techcommunity/original/3X/2/8/28b6562dfa53dfc7e0521f0fb6958ab1c76bdbb9.png)](https://global.discourse-cdn.com/techcommunity/original/3X/2/8/28b6562dfa53dfc7e0521f0fb6958ab1c76bdbb9.png "image")

Select the `dynamic-mapping-service.zip` or drag & drop it in the area of the popup. When asked to subscribe please click the “Subscribe” button.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/4/0/409aa1b20675f271956e46e65012ef760edf2ffe.png)

After a couple of seconds, the microservice should be uploaded and activated. We can check that by selecting Dynamic-mapping-service in the list of microservice and checking the status.

[![image](https://global.discourse-cdn.com/techcommunity/original/3X/d/b/dbb1f6ec243340a3a046fbea23a6d6d328e66e90.png)](https://global.discourse-cdn.com/techcommunity/original/3X/d/b/dbb1f6ec243340a3a046fbea23a6d6d328e66e90.png "image")

##### Frontend Plugin

Let’s continue with the frontend plugin (dynamic-mapping-ui.zip).

Now we can navigate to **Ecosystem** → **Applications** → **Extensions**.

Select the `Dynamic-mapping-ui` plugins or click on **Add extension package** and upload the `dynamic-mapping-ui.zip` manually.  
You should see the Dynamic-mapping packages in the packages list afterward.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/0/c/0cb49b54cc1b3b2b7801e1f4eaf48b9612ecd459_2_690x92.png)](https://global.discourse-cdn.com/techcommunity/original/3X/0/c/0cb49b54cc1b3b2b7801e1f4eaf48b9612ecd459.png "image")

Now we want to assign that plugin to the Administration App. Go to **Applications** and click on **Add application**. Click on **Duplicate existing application** and **Administration** afterward.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/a/0/a056247129adf9d8da91e026e2ae0b486f8be17e.png)

Click on `Duplicate`

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/3/4/34ed1dd636cf8b85948de52704a4fe5e1db1e758_2_338x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/3/4/34ed1dd636cf8b85948de52704a4fe5e1db1e758.png "image")

It will take a few seconds and you should see the new custom application **Administration** in the Application list. Select it by clicking on the name.

Go to the **Plugins** Tab. In the bottom right you see a button **Install plugin**. When clicked you see all available plugins in your tenant.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/c/e/ce5296bd4fec0c2ca21893fbfa4ac6532dd7497c.png)

Select **Dynamic Mapping Widget** and click **Install**.  
Now we need to refresh the page by pressing F5 or click the reload button in the browser.

In the left menu go to **Settings**. There you should see now **Dynamic Mapping** which is the plugin we have just installed.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/6/b/6b6187728c7f9930ad71c0dd3d59e2e16e1cc2cd_2_148x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/6/b/6b6187728c7f9930ad71c0dd3d59e2e16e1cc2cd.png "image")

**Congratulations you’ve successfully installed both components in your tenant!**

### 3.1 MQTT Service configuration

> **Please note:** MQTT Service is currently in private preview and is only available on request on your tenants. If you haven’t it enabled please proceed with chapter [3.2 MQTT Broker configuration](https://tech.forums.softwareag.com#h-32-mqtt-broker-configuration-16)

For adding MQTT Service you just need to add a MQTT Connector by clicking on **Connector** → **Add configuration**. Select **MQTT Service** and click **Save**

![image](https://global.discourse-cdn.com/techcommunity/original/3X/b/f/bf3ba07f6e1d2114d4d3b679a9c881093ade19e5.png)

Now activate that connector by clicking on the Action Button.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/2/e/2e4547130a1295791b41fa2aed49c1cf2b591435.png)

After a few seconds the status should change to ![image](https://global.discourse-cdn.com/techcommunity/original/3X/d/0/d0a4ea0c934724162ee86ab594c635ff32efb33d.png)

### 3.2 MQTT Broker configuration

To add an external MQTT Broker we use the following connection details of a public mosquitto MQTT broker:

```
Protocol: mqtts:/
MQTT Host: test.mosquitto.org
MQTT Port: 8886
ClientId: dynamic-mapping-client-{{anyNumber}}

```

Make sure to select **mqtts://** as protocol enter the **Mqtt Host** , **Mqtt Port** , **Client Id** and click on **Save**.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/e/e/ee9bf05ba58c5fd3a0de8464d05bfaa94343ed49_2_218x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/e/e/ee9bf05ba58c5fd3a0de8464d05bfaa94343ed49.png "image")

Now activate that connector by clicking on the Action Button.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/2/e/2e4547130a1295791b41fa2aed49c1cf2b591435.png)

After a few seconds the status should change to ![image](https://global.discourse-cdn.com/techcommunity/original/3X/d/0/d0a4ea0c934724162ee86ab594c635ff32efb33d.png)

### 4. Create a device mapping

Now let’s assume we have a device, a CCU installed on a bus, that sends some JSON data like this:

```
topic: device/express/berlin_01
{ 
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International" 
} 

```

We go to the **Mapping Inbound** tab and click on **Add mapping**. In the next pop-up we select **JSON**.

#### Define Topic

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/d/8/d8fd68cc8e15076f84c0ea095b277e2d7b3a23b0_2_682x499.png)](https://global.discourse-cdn.com/techcommunity/original/3X/d/8/d8fd68cc8e15076f84c0ea095b277e2d7b3a23b0.png "image")

- **Subscription Topic** : `device/#`
- **Mapping Topic** : `device/+/+`
- **Mapping Topic Sample** : `device/express/berlin_01`
- **Target API** : `Inventory`.
- **Map Device Identifier** & **Update Existing Device** : checked

Explanation:

- In **Subscription Topic** we enter `device/#` because we want to subscribe on all devices not only for express busses or the bus berlin\_01. This is the topic we subscribe to on the MQTT Broker.
- In **Mapping Topic** `device/+/+` is entered. The meaning is that this mapping will handle all messages for exactly 2 levels of sub-topics. As our topic is `device/express/berlin_01` we change that to the value `device/+/+`
- **Mapping Topic Sample** We define a concrete sample that is used for the testing. If we wouldn’t provide that testing is not possible.
- In Target API we select **Inventory** to create a new device in Cumulocity.
- Check **Map Device Identifier** & **Update Existing Device** - Map Device Identifier states that we want to correlate a unique ID to an external ID in Cumulocity and Update Existing Device states that it will also update the existing object in Cumulocity when it already exists.

Click on **Next**.

#### Define template

In the next view you see on the left the **source template** and on the right the **target template**. With the below we can manage the mappings.  
Let’s start by defining the source template.

Click on **Tree** in **Source** and change it to **Code**. We don’t touch the `_TOPIC_LEVEL_` but we add our example payload, copy & paste the following JSON message:

```
{ 
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International",
    "_TOPIC_LEVEL_": [
       "device",
       "express",
       "berlin_01"
  ]
} 

```

Now change back from **Code** to **Tree**.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/7/9/79a374912031bc4b98588b518cdbb181e38ae72c.png)

Let’s start with the important info the device identifier. We know that it is part of the topic _device/express/ **berlin\_01** _ so we select on the left side `berlin_01` and on the right side `id` and click on **+ Add substitution** to add the substitution.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/a/4/a4839f0341c99054900b93ab6d828a8364268290_2_530x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/a/4/a4839f0341c99054900b93ab6d828a8364268290.png "image")

In **Substitutions** list we see this afterward:  
`00. [* _TOPIC_LEVEL_[2] --> id ]`

The second mapping will be the `customType` to `type`.  
Select `customType` on the left and `type` on the right. Again click on **+ Add Substitution**.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/0/d/0d07e0e615ebebfc31f2d6bea4895da9a6e496c4_2_633x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/0/d/0d07e0e615ebebfc31f2d6bea4895da9a6e496c4.png "image")

For the `name` we use now two properties from the left `operator` and `line` to map it to one property of the right `name`.  
This can be done by using an expression language called [JSONata](https://jsonata.org/). We select `operator` on the left and `name` on the right. Next we go to the field **Evaluate Expression on Source** and add the following:

```
operator&"-"&line

```

In the field **Result Type** you see the preview how the expression result will look like. Click on **+Add substitution** to add this substitution.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/5/d/5d06d3644451c3478e8805d03ab19ce34929aa63_2_690x157.png)](https://global.discourse-cdn.com/techcommunity/original/3X/5/d/5d06d3644451c3478e8805d03ab19ce34929aa63.png "image")

You can click on **?** to get more examples for expressions or full spec of the expression language.

At last, we map the `capacity` from the left to a new property on the right.  
In the **Target Tree** go with the mouse to the line below “id” and click the arrow down to open the menu. Scroll down an click on **+Value**

![image](https://global.discourse-cdn.com/techcommunity/original/3X/6/0/60a3f14c8bd270448eb46f526caa767268873016.png)

Enter `capacity` as the field and `100` as the value. The 100 value will be overwritten by the value of the source mapping so it doesn’t matter what we enter here.

![image](https://global.discourse-cdn.com/techcommunity/original/3X/b/e/be9271f6a9fbf080c41a4c2f59c0b38e7c7f265e.png)

Now we have 4 substitutions and we are done with our mapping.  
Click on **Next**

#### Test mapping

In the next view we can evaluate how our mapping works. First click on **Transform Test Message** which will execute all substitutions based on provided source data.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/b/a/ba540c4ef82a631176f40e146eb564b9c39d42e1_2_549x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/b/a/ba540c4ef82a631176f40e146eb564b9c39d42e1.png "image")

With **Show Next Test Result** we can iterate through all messages if we have an array mapped containing multiple device information. In our case, we just have one message.

Click on **Send Test Message**. It will run the mapping and send it to Cumulocity IoT. After some time you will see the response of Cumulocity IoT in the **Cumulocity Response**.

Everything looks good so we **Confirm** our mapping.  
Now in the Mapping list, you should see your mapping. By clicking on the **Active** Toggle Button you can enable your mapping.  
Our mapping is finished and we can continue sending device data to the MQTT Broker.

#### Send device data

Now let’s simulate that we connect our device to the generic MQTT Broker we have configured earlier.  
Start MQTTx client on your desktop. On **Connections** click on the **+** to create a new connection.  
We provide the following information:

```
Name: mosquitto_sandbox
Client_ID: (don't touch)
Host: mqtts://test.mosquitto.org
Port: 8886

```

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/1/9/1931a6a0b2937c4c8bddc0e1b295c7a4d04273e0_2_684x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/1/9/1931a6a0b2937c4c8bddc0e1b295c7a4d04273e0.png "image")

Click on **Connect**.

> **Please note** : The mosquitto sandbox will terminate the connection after a few seconds with no activity. So you might have to re-connect.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/a/d/add005b8ef3cd032647212cf783454385b040c4e_2_599x500.png)](https://global.discourse-cdn.com/techcommunity/original/3X/a/d/add005b8ef3cd032647212cf783454385b040c4e.png "image")

In the bottom right corner change the **Payload** to JSON. Enter the **topic** `device/express/berlin_01` with the payload:

```
{
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International2" 
}

```

> Please recognize that we changed the customType to `type_International2`. This will result in an update of the existing Device we have created during Testing.

Click on the bottom right button to send the message.

#### Confirm Device creation

Let’s check what happened! Go back to the Cumulocity IoT Dynamic Mapping App. Click on **Monitoring**.  
There you should see now an entry:

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/1/3/13a5c83cebe8f8a31033e604aa0dde0fb7a32ba4_2_690x28.png)](https://global.discourse-cdn.com/techcommunity/original/3X/1/3/13a5c83cebe8f8a31033e604aa0dde0fb7a32ba4.png "image")

This means at least 1 message has been received without error.

Switch the application to **Device Management** using the Application Switcher in top right corner.

Click on **Devices** → **All devices** in the left navigation.  
Here you see a new device has been created of type `type_International2` we have just sent via MQTT Broker.

[![image](https://global.discourse-cdn.com/techcommunity/optimized/3X/7/7/77608bbb842aafba74944682a9ae7521fbc2b3e8_2_690x79.png)](https://global.discourse-cdn.com/techcommunity/original/3X/7/7/77608bbb842aafba74944682a9ae7521fbc2b3e8.png "image")

We can now repeat the last steps by sending the same message with different values on different topics e.g. `device/express/berlin_02` with

```
{
    "line": "Bus-Prag-Leipzig", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 128, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International" 
}

```

### 5. Sending Measurements Events Alarms (MEAs)

[Read full topic here](https://tech.forums.softwareag.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/265444/1) to learn how to send Measurements Events Alarms.

If you want to read the full documentation please have a look here:

> [GitHub - SoftwareAG/cumulocity-dynamic-mapper: The ultimate Mapper for building the bridge between any Message Broker and Cumulocity IoT in a zero-code approach!](https://github.com/SoftwareAG/cumulocity-dynamic-mapper)

Is there anything missing? Any connector you want to use or payload format which is currently not supported?

**We are very happy about feedback and contribution! So if you have an idea please approach us either here or using the GitHub project!**

* * *

> This article is part of the TECHniques newsletter blog - technical tips and tricks for the Software AG community. [Subscribe](https://info.softwareag.com/TechCommunity-Subscription-Page.html) to receive our quarterly updates or read the [latest issue](https://tech.forums.softwareag.com/techniques-latest).