Fleet Provisioning for Embedded Linux Devices with AWS IoT Greengrass
Tags: greengrass, iot, yocto
Published: 2023-05-04 16:48:17

## Introduction

Managing a large fleet of embedded devices can be complex and challenging, particularly when it comes to creating a single image that can be flashed onto multiple devices. These devices must be able to self-provision, utilizing unique information such as their serial number, upon initial boot. In this blog post, we will discuss how AWS IoT Greengrass - Fleet Provisioning can streamline this process for embedded Linux devices, making it more efficient and reliable.

For embedded systems engineers experienced in Embedded Linux and Yocto, we will guide you through building a Raspberry Pi Yocto image with Greengrass with Fleet Provisioning Plugin. This ensures seamless device provisioning and management, as well as automatic registration and configuration. 

![Fleet provisioning by claim](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9rzwpi641sz8z8rhkar0.png)

>Please note here that the pre-provisioning lambda is optional but encouraged in order to additional layer of security. We will not be covering it in this post. You can learn more about it [here](https://docs.aws.amazon.com/iot/latest/developerguide/pre-provisioning-hook.html).

With the stage set, let's dive into the prerequisites for setting up AWS IoT Greengrass and Fleet Provisioning for your embedded Linux devices.

## Prerequisites

Before diving into the process of preparing the host and configuring the Yocto image build, it's essential to set up AWS IoT Core. This involves creating policies, obtaining claim certificates, and ensuring that the AWS CLI is installed and configured. General information on how to accomplish this can be found in the [AWS IoT Greengrass Developer Guide](https://docs.aws.amazon.com/greengrass/v2/developerguide/fleet-provisioning-setup.html).
In summary, we will need to:

1. A token exchange IAM role, which core devices use to authorize calls to AWS services and An AWS IoT role alias that points to the token exchange role.
2. An AWS IoT fleet provisioning template. The template must specify information needed for creating thing and policy which will be attached to greengrass core device created. You can either use existing IoT policy name or define the policy on the template.
3. An AWS IoT provisioning claim certificate and private key for the fleet provisioning template.

Devices can be manufactured with a provisioning claim certificate and private key embedded in them. When the device connects first time to AWS IoT, it uses the claim certificate to register the new device and exchange it to unique device certificate. Provisioning claim certificate needs to have AWS IoT policy attached which allows devices to register and use the fleet provisioning template.

To make this process more efficient, we can utilize a CloudFormation template that automates most of these steps:

```
AWSTemplateFormatVersion: "2010-09-09"

Parameters:
  ProvisioningTemplateName:
    Type: String
    Default: 'GreengrassFleetProvisioningTemplate' 
  GGTokenExchangeRoleName:
    Type: String
    Default: 'GGTokenExchangeRole'
  GGFleetProvisioningRoleName:
    Type: String
    Default: 'GGFleetProvisioningRole'
  GGDeviceDefaultPolicyName:
    Type: String
    Default: 'GGDeviceDefaultIoTPolicy'
  GGProvisioningClaimPolicyName:
    Type: String
    Default: 'GGProvisioningClaimPolicy'

Resources:
  
  GGTokenExchangeRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref GGTokenExchangeRoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - credentials.iot.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: '/'
      Policies:
        - PolicyName: !Sub ${GGTokenExchangeRoleName}Access
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - 'iot:DescribeCertificate'
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                  - 'logs:DescribeLogStreams'
                  - 's3:GetBucketLocation'
                Resource: '*'

  GGTokenExchangeRoleAlias:
    Type: AWS::IoT::RoleAlias
    Properties:
      RoleArn: !GetAtt GGTokenExchangeRole.Arn
      RoleAlias: !Sub ${GGTokenExchangeRoleName}Alias

  GGFleetProvisioningRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref GGFleetProvisioningRoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - iot.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: '/'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSIoTThingsRegistration'

  GGDeviceDefaultPolicy:
    Type: AWS::IoT::Policy
    Properties:
      PolicyName: !Ref GGDeviceDefaultPolicyName
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
            - 'iot:Connect'
            - 'iot:Publish'
            - 'iot:Subscribe'
            - 'iot:Receive'
            - 'iot:Connect'
            - 'greengrass:*'
          Resource: '*'
        - Effect: Allow
          Action:
            - 'iot:AssumeRoleWithCertificate'
          Resource: !GetAtt GGTokenExchangeRoleAlias.RoleAliasArn

  GGFleetProvisionTemplate:
    Type: AWS::IoT::ProvisioningTemplate
    Properties:
      TemplateName: !Ref ProvisioningTemplateName
      Description: 'Fleet Provisioning template for AWS IoT Greengrass.'
      Enabled: True
      ProvisioningRoleArn: !GetAtt GGFleetProvisioningRole.Arn
      TemplateBody: !Sub |+ 
        {
          "Parameters": {
            "ThingName": {
              "Type": "String"
            },
            "ThingGroupName": {
              "Type": "String"
            },
            "AWS::IoT::Certificate::Id": {
              "Type": "String"
            }
          },
          "Resources": {
            "GGThing": {
              "OverrideSettings": {
                "AttributePayload": "REPLACE",
                "ThingGroups": "REPLACE",
                "ThingTypeName": "REPLACE"
              },
              "Properties": {
                "AttributePayload": {},
                "ThingGroups": [
                  {
                    "Ref": "ThingGroupName"
                  }
                ],
                "ThingName": {
                  "Ref": "ThingName"
                }
              },
              "Type": "AWS::IoT::Thing"
            },
            "GGDefaultPolicy": {
              "Properties": {
                "PolicyName": "${GGDeviceDefaultPolicyName}"
              },
              "Type": "AWS::IoT::Policy"
            },
            "GGCertificate": {
              "Properties": {
                "CertificateId": {
                  "Ref": "AWS::IoT::Certificate::Id"
                },
                "Status": "Active"
              },
              "Type": "AWS::IoT::Certificate"
            }
          }
        }
  
  GGProvisioningClaimPolicy:
    Type: AWS::IoT::Policy
    Properties:
      PolicyName: !Ref GGProvisioningClaimPolicyName
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
            - 'iot:Connect'
          Resource: '*'
        - Effect: Allow
          Action:
            - 'iot:Publish'
            - 'iot:Receive'
          Resource: 
            - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/certificates/create/*'
            - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/provisioning-templates/${ProvisioningTemplateName}/provision/*'
        - Effect: Allow
          Action:
            - 'iot:Subscribe'
          Resource:
            - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/certificates/create/*'
            - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/provisioning-templates/${ProvisioningTemplateName}/provision/*'
  
Outputs:
  
  GGTokenExchangeRole:
    Description: Name of token exchange role.
    Value: !Ref GGTokenExchangeRole
  GGTokenExchangeRoleAlias:
    Description: Name of token exchange role alias.
    Value: !Ref GGTokenExchangeRoleAlias
  GGFleetProvisionTemplate:
    Description: Name of Fleet provisioning template.
    Value: !Ref GGFleetProvisionTemplate
  GGProvisioningClaimPolicy:
     Description: Name of claim certificate IoT policy.
     Value: !Ref GGProvisioningClaimPolicy
```

Save the file and create CloudFormation stack from template.yaml:

```
aws cloudformation create-stack --stack-name GGFleetProvisoning --template-body file://gg-fp.yaml --capabilities CAPABILITY_NAMED_IAM
```

Wait few minutes for resources being created. You can check status from CloudFormation console or with command:

```
aws cloudformation describe-stacks --stack-name GGFleetProvisoning
```

### Create claim certificate

These we will be embedded in our RPi SD Card Image and used to provision our devices.

```
mkdir claim-certs

export CERTIFICATE_ARN=$(aws iot create-keys-and-certificate \
    --certificate-pem-outfile "claim-certs/claim.cert.pem" \
    --public-key-outfile "claim-certs/claim.pubkey.pem" \
    --private-key-outfile "claim-certs/claim.pkey.pem" \
    --set-as-active \
    --query certificateArn)

curl -o "claim-certs/claim.root.pem" https://www.amazontrust.com/repository/AmazonRootCA1.pem

```

### Attach the AWS IoT policy to the provisioning claim certificate

As we created IoT policy named `GGProvisioningClaimPolicy` with CloudFormation we can just use the name to attach the policy:

```
aws iot attach-policy --policy-name GGProvisioningClaimPolicy --target ${CERTIFICATE_ARN//\"}
```

### Create a Thing Group

Once our devices get provisioned they will become part of this Thing Group allowing us later to target Thing Group Fleet Deployments.

```
aws iot create-thing-group --thing-group-name EmbeddedLinuxFleet
```

As of now we should be good to go and build our RPI image.

## Building RPi Image

Building a Yocto image for Raspberry Pi requires several steps, including setting up the build environment, cloning the necessary repositories, configuring the build, and finally, building the image itself. Here's a step-by-step guide to help you through the process:

Open a terminal window on your workstation which has all the prerequisits based on the [Yocto Project Build Doc](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html).


>NOTE For the sake of this tutorial, the variable `BASE` refers to the build environment parent directory. Here, this will be set to `$HOME`. If you are using another partition as the base directory, please set it accordingly.

```
export BASEDIR=$(pwd)
export DIST=poky-rpi4
export B=kirkstone
```

Clone the Poky base layer to include OpenEmbedded Core, Bitbake, and so forth to seed the Yocto build environment.

```
git clone -b $B git://git.yoctoproject.org/poky.git $BASEDIR/$DIST
```

Clone additional dependent repositories. Note that we are cloning only what is required for AWS IoT Greengrass.

```
git clone -b $B git://git.openembedded.org/meta-openembedded \
    $BASEDIR/$DIST/meta-openembedded
git clone -b $B git://git.yoctoproject.org/meta-raspberrypi \
    $BASEDIR/$DIST/meta-raspberrypi
git clone -b $B git://git.yoctoproject.org/meta-virtualization \
    $BASEDIR/$DIST/meta-virtualization
git clone -b $B https://github.com/aws4embeddedlinux/meta-aws \
    $BASEDIR/$DIST/meta-aws
```

Source the Yocto environment script. This seeds the `build/conf` directory.

```
cd $BASEDIR/$DIST
. ./oe-init-build-env
```

Add necessary layers to `bblayers.conf` using `bitbake-layer add-layer`:

```
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-openembedded/meta-python
bitbake-layers add-layer ../meta-openembedded/meta-filesystems
bitbake-layers add-layer ../meta-openembedded/meta-networking
bitbake-layers add-layer ../meta-virtualization
bitbake-layers add-layer ../meta-raspberrypi
bitbake-layers add-layer ../meta-aws
```

Configure the `local.conf:`

Here it is important to note that apart from standard raspberry pi configuration:

```
MACHINE ?= "raspberrypi4-64"

DISABLE_VC4GRAPHICS = "1"

# Parallelism Options
BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"

# Additional image features
USER_CLASSES ?= "buildstats"

# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"

# Disk Space Monitoring during the build
BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    HALT,${TMPDIR},100M,1K \
    HALT,${DL_DIR},100M,1K \
    HALT,${SSTATE_DIR},100M,1K"

CONF_VERSION = "2"

DISTRO_FEATURES += "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""

IMAGE_FSTYPES = "rpi-sdimg"
```

We should focus on our Greengrass FleetProvisioning configuration part, which should look like this:

```

IMAGE_INSTALL:append = " greengrass-bin "
GGV2_DATA_EP     = "xxx-ats.iot.<your aws region>.amazonaws.com"
GGV2_CRED_EP     = "xxx.iot.<your aws region>.amazonaws.com"
GGV2_REGION      = "<your aws region>"
GGV2_THING_NAME  = "ELThing"
GGV2_TES_RALIAS  = "GGTokenExchangeRoleAlias" # we got this from the cloudformation
GGV2_THING_GROUP = "EmbeddedLinuxFleet"

PACKAGECONFIG:pn-greengrass-bin = "fleetprovisioning"
```

Here it is important to note that we are adding `greengrass-bin` to our image and then providing additional configuration required by the `config.yaml` as well as adding `PACKAGECONFIG:pn-greengrass-bin = "fleetprovisioning"` in order to enable the functionality.

In order to get the AWS region and the IoT endpoints we can do the following:

```
echo "GGV2_REGION="$(aws configure get region)
echo "GGV2_DATA_EP="$(aws --output text iot describe-endpoint \
    --endpoint-type iot:Data-ATS \
    --query 'endpointAddress')
echo "GGV2_CRED_EP="$(aws --output text iot describe-endpoint \
    --endpoint-type iot:CredentialProvider \
    --query 'endpointAddress')
```

Please note that we will need a unique Thing Name to be generated for each device, so here the Thing Name is taken as a prefix and there is script inside of a `greengreass-bin` recipe that appends the unique device id to the Thing Name using the MAC address.

```
#!/bin/sh
file_path="$1"
default_iface=$(busybox route | grep default | awk '{print $8}')
mac_address=$(busybox ifconfig "$default_iface" | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}' | tr ':' '_')
sed -i "s/<unique>/$mac_address/g" "$file_path"
```

```
meta-aws
└── recipes-iot
    └── aws-iot-greengrass
        └──files
            └── replace_board_id.sh
```

Feel free to replace this file with any other way of obtaining the uniqueness such as serial number or similar 

Finally we should copy our claim credentials we generated at the beginning to be included in our build:

```
cp  "claim-certs/claim.cert.pem" \
    "claim-certs/claim.pkey.pem" \
    "claim-certs/claim.root.pem" \
    $BASEDIR/$DIST/meta-aws/recipes-iot/aws-iot-greengrass/files/
```

Please adjust the paths based on the location of generated certs and the recipe.

After all of this we should proceed with building our image.

```
bitbake core-image-minimal
```

⌛ Couple of hours later ⌛ the build should be complete, and we can find the resulting image in the following directory:

```
ls tmp/deploy/images/raspberrypi4-64/*sdimg
```

To flash the image onto an SD card, use a tool like 'dd'

```
sudo dd if=tmp/deploy/images/raspberrypi4-64/core-image-minimal-raspberrypi4-64.sdimg of=/dev/sdX bs=4M
```

Where we need to make sure to replace "/dev/sdX" with the appropriate device identifier for the SD card.

>⚠️ Please double check the SD card identifier as a mistake here can wipe your workstation system

## Powering the Device for the First Time

Once the SD card is reinserted into the Raspberry Pi with power and internet connected, the device should perform provisioning and appear in the list of Greengrass core devices.:

```
aws greengrassv2 list-core-devices

{
    "coreDevices": [
        {
            "coreDeviceThingName": "ELThing_11_22_33_44_55_60",
            "status": "HEALTHY",
            "lastStatusUpdateTimestamp": "2023-04-25T15:39:00.703000+00:00"
        },
       {
            "coreDeviceThingName": "ELThing_11_22_33_44_55_61",
            "status": "HEALTHY",
            "lastStatusUpdateTimestamp": "2023-03-31T03:11:17.911000+00:00"
        },
       {
            "coreDeviceThingName": "ELThing_11_22_33_44_55_62",
            "status": "HEALTHY",
            "lastStatusUpdateTimestamp": "2023-02-25T15:17:29.505000+00:00"
        },        
    ]
}
```

Success! 

## Conclusion

To sum it up, managing a large fleet of embedded Linux devices can be a complex and challenging task, especially when it comes to creating a single image that can be flashed onto multiple devices. However, AWS IoT Greengrass with Fleet Provisioning can streamline this process and make it more efficient and reliable. In this blog post, we have discussed the prerequisites for setting up AWS IoT Greengrass and Fleet Provisioning, including creating policies, obtaining claim certificates, and configuring the Yocto image build. We have also provided a step-by-step guide to building a Yocto image for Raspberry Pi with Greengrass Fleet Provisioning configuration. Using AWS IoT Greengrass - Fleet Provisioning, managing a large fleet of embedded devices can be made easier, more efficient, and secure.

If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on [Twitter](https://twitter.com/nenadilic84) or [LinkedIn](https://www.linkedin.com/in/nenadilic84/).

Feel free to checkout this video that goes over the mentioned setup: https://youtu.be/Eeo7GLVr0jw