Decoding USP protobuf data sent by USP agents, using the recent AWS IoT Rules Engine native SQL decode function
Tags: awsiot, tr369, protobuf, opensource
Published: 2023-01-03 10:41:13


## Contents

1. [Introduction](#Introduction)
2. [Context](#Context)
3. [Set-Up Steps](#SetUp)
4. [Exploring the payloads and testing with the AWS IoT Console MQTT Client](#Verify)
5. [Conclusion](#Conclusion)
6. [Author] (#Author)

## Introduction<a name="Introduction"></a>

This post is a follow-up to: {% link https://dev.to/iotbuilders/connect-the-open-broadband-usp-agent-to-aws-iot-core-using-mqtt5-and-validate-the-integration-with-aws-iot-core-device-advisor-2kb5 %}, which showed how to connect a device running obuspa to AWS IoT Core, over MQTT 5, ingest data and validate the MQTT 5 connection using AWS IoT Core Device Advisor. It is recommended you read the preceding post, in order to get a better understanding of integrating a USP agent with AWS IoT.  

In this post, you will see how to set up and use the [native AWS IoT Rules Engine protobuf decoding SQL function](https://aws.amazon.com/about-aws/whats-new/2022/12/aws-iot-core-rules-engine-google-protocol-buffer-messaging-format/), in order to decode USP records and messages sent by [USP agents](https://usp.technology/specification/02-index-architecture.html#sec:endpoints), on the fly, without the need for additional AWS Lambda functions with bespoke decoding logic.

## Context<a name="Context"></a>

[Protocol Buffers (protobuf)](https://developers.google.com/protocol-buffers) is an open-source data format used to serialize structured data in binary form. It is used for transmitting data over networks or storing it in files. Protobuf allows you to send data in small packet sizes and at a faster rate than other messaging formats. [USP] (https://dev.to/iotbuilders/connect-the-open-broadband-usp-agent-to-aws-iot-core-using-mqtt5-and-validate-the-integration-with-aws-iot-core-device-advisor-2kb5#Introduction) specifies Protocol Buffers Version 3 as a [mechanism to serialize data to be sent over a Message Transfer Protocol, such as MQTT](https://usp.technology/specification/05-index-encoding.html#sec:encoding).

The protobuf decoding support in the AWS IoT Rules Engine was recently introduced (https://aws.amazon.com/about-aws/whats-new/2022/12/aws-iot-core-rules-engine-google-protocol-buffer-messaging-format/) to enable developers to decode protobuf payloads to JSON format directly in the rule, and route them to downstream services, by means of an AWS IoT Rules SQL function: decode(value, decodingScheme) (https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html#iot-sql-decode-base64).

This feature is interesting because it removes the need to write and maintain custom protobuf decoding logic, which was, prior to this feature, usually done using AWS Lambda functions, invoked either as Rule Actions, or as part of IoT Rule SQL expressions (see the diagram below - before and after the native protobuf support in AWS IoT Rules Engine).

!(https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ssc6t3oihgvwqpp24ggp.png)

## Set-Up Steps<a name="SetUp"></a>

**Prepare the required protobuf files**

As part of the [AWS IoT Rule decoding pre-requisites](https://docs.aws.amazon.com/iot/latest/developerguide/binary-payloads.html#binary-payloads-protobuf-prerequisites), the protobuf descriptor file needs to be created and uploaded to an S3 bucket.

1. Install the [Protobuf Compiler (protoc)](https://github.com/protocolbuffers/protobuf/releases) on your system. On MacOS, you could use Homebrew, as described [here] (https://formulae.brew.sh/formula/protobuf).
2. Download the 2 proto files [record]( https://github.com/BroadbandForum/usp/blob/master/specification/usp-record-1-2.proto) and [message] (https://github.com/BroadbandForum/usp/blob/master/specification/usp-msg-1-2.proto) from the open source USP GitHub repository, corresponding to the USP version that you want to use in your project. This post will work with the latest version, USP 1.2. 
3. Create the descriptor file required by the AWS IoT Rules Engine, as described in the [docs](https://docs.aws.amazon.com/iot/latest/developerguide/binary-payloads.html#binary-payloads-protobuf)

You can use the following command:

```
protoc —descriptor_set_out=usp-1-2.desc —proto_path=<PATH TO YOUR PROTO FILES> —include_imports usp-record-1-2.proto usp-msg-1-2.proto
```

**Create the AWS Cloud resources**

You will need to create the AWS IoT Rule, correct Roles and Policies, and S3 Bucket where the descriptor file will be uploded. You can use the AWS CloudFormation template below for this set-up. To create the AWS resources using this template, you can use the [AWS CloudFormation CLI commands] (https://docs.aws.amazon.com/cli/latest/reference/cloudformation/index.html) or the [AWS SAM CLI commands] (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-command-reference.html). 

For the purpose of this example, the Rule will decode the protobuf record, then decode the protobuf message (both using the decode () SQL function), and then republish the resulting JSON object, on a new topic, so we can examine correctness.

``` yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  USPProtobufDecodingRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: USPProtobufDecodingRule
      TopicRulePayload:
        AwsIotSqlVersion: 2016-03-23
        RuleDisabled: False
        Sql: SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'
        Actions:
          - Republish:
              RoleArn: !GetAtt RepublishDecodedRole.Arn
              Topic: '/usp/controller/decoded'

  RepublishDecodedRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
              - iot.amazonaws.com
      Policies:
        - PolicyName: allowRepublish
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic//usp/controller/decoded" ] ]

  USPBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: usp-1-2-bucket

  USPBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref USPBucket
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
            - 's3:Get*'
            Effect: Allow
            Resource: !Join
              - ''
              - - 'arn:aws:s3:::'
                - !Ref USPBucket
                - /*
            Principal:
              Service:
                - iot.amazonaws.com

```

The AWS IoT Rule SQL expression for decoding could look like below:

```bash
SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'
```

The SQL rule behaves the following way: 

* First, the binary MQTT payload is transformed in a based64-encoded string, 
* Then, the base64 encoded string is protobuf-decoded at USP record level, as specified by the descriptor,
* Finally, the USP message inside the record, located in the payload object, is protobuf decoded.

Because USP messages are encoded and wrapped inside encoded USP records, you will need to call the decode function twice.

**Upload the descriptor file and start obuspa**

1. Upload the descriptor file created in step 3 into your S3 bucket. You can do this using the AWS CLI or AWS Console. 
2. You can now set up and start the obuspa, as described in the following sections: [Integrate obuspa with AWS IoT Core over MQTT5] (https://dev.to/iotbuilders/connect-the-open-broadband-usp-agent-to-aws-iot-core-using-mqtt5-and-validate-the-integration-with-aws-iot-core-device-advisor-2kb5#MQTT5config) and [Verify MQTT 5 Integration/Protobuf Data Publishing] (https://dev.to/iotbuilders/connect-the-open-broadband-usp-agent-to-aws-iot-core-using-mqtt5-and-validate-the-integration-with-aws-iot-core-device-advisor-2kb5#VerifyMQTT5Integration). 

Once the agent is sending data, the set-up is complete.

## Exploring the payloads and testing with the AWS IoT Console MQTT Client<a name="Verify"></a>

As the protobuf records arrive on the topic, the IoT Rule will pick them up, decode them, and republish the JSON objects on the `/usp/controller/decoded` topic. If you subscribe with the test MQTT Client in the AWS IoT Console, you will be able to see the protobuf records on the `/usp/controller` topic, and the JSON payloads on the `/usp/controller/decoded` topic, as shown in the images below. The first image shows the original protobuf encoded USP record, arriving on the topic, and the second image the decoded USP message, in JSON format.

![Original protobuf encoded USP record](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/paeprjy6qubcq6fmzi0b.png)

![Decoded USP record and message, in JSON format](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qqeul4e39nass6oibxhr.png)

## Conclusion<a name="Conclusion"></a>

This post shows how to to decode protobuf records and messages sent by a USP agent, directly in the AWS IoT Rules Engine, using the decode function, as part of the rule expression. This is an AWS IoT Rules Engine feature released [recently](https://aws.amazon.com/about-aws/whats-new/2022/12/aws-iot-core-rules-engine-google-protocol-buffer-messaging-format/), making it easier to use protobuf directly in your [IoT Rule] (https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html), without the need to write and maintain custom decoding code in AWS Lambda functions. 

## Author<a name="Author"></a>
{% user fay_ette %}