How to add custom resources to AWS Amplify. Add an SQS queue that triggers lambda.
Tags: aws, serverless, programming, devops
Published: 2021-11-04 15:33:41

AWS Amplify is allowing you to build cloud-connected apps very easily. It has a broad range of categories it supports natively which you can use to build (see my  [post](https://sandro.volpee.de/what-can-you-build-with-aws-amplify-lets-see-all-categories-and-services)  here). But there are still some major components missing, such as SQS and RDS. In this post, I want to show you how to add AWS services that are not currently supported in Amplify. We start with the general steps you need to take for adding these resources. In the second part, I show you how to add an SQS queue that triggers a lambda. 

There are many ways of doing this but I think the easiest is to use CloudFormation. It is also possible to use frameworks like CDK or Terraform but CloudFormation is best integrated into Amplify. All deployments and updates will still happen from the Amplify CLI and no other tool is needed. All code is in this [repo](https://github.com/AlessandroVol23/amplify-custom-sqs).

## How to add custom resources in general

Amplify has a docs page [here](https://docs.amplify.aws/cli/usage/customcf/) on how to add custom AWS resources. We need to do basically two steps.

### **Add custom resource in backend config**

In the first step, you need to add a custom category name and custom resource name within your backend config `amplify/backend/backend-config.json`. 

```JSON
{
  "<custom-category-name>": {
    "<custom-resource-name>": {
      "service": <custom-aws-service-name>,
      "providerPlugin": "awscloudformation"
    }
  }
}
```

This could look like that. Remember that Amplify always uses the category name as a prefix. Think about that when using the names to not have the same category all over the resource names.

### **Add custom folders & CloudFormation**

After that, you need to add the folders you referenced your category and resource names to.

```json
amplify
  \backend
    \<custom-category-name>
      \<custom-resource-name>
        parameters_<custom-resource-name>.json
        template_<custom-resource-name>.json
```

`template_<custom-resource-name>.json`: CloudFormation Templat
`parameters_<custom-resource-name>.json`: Additional parameters for CloudFormation

You need both the parameters and the template files. Create them directly with the suffix `custom-resource-name` to avoid any name clashes in the future. 
Now you would need to add your CloudFormation code in the template file. We'll see that in a second.


### **Reference existing parameters**

The good thing here now is that you can reference output parameters of other CloudFormation categories. For example if you need the lambda ARN which was added by Amplify you can reference it in a `dependsOn` block in the `backend-config.json`.

```json
{
  "<custom-category-name>": {
    "<custom-resource-name>": {
      "service": <custom-aws-service-name>,
      "providerPlugin": "awscloudformation",
      "dependsOn": [
        {
          "category": "function",
          "resourceName": "lambda", // check `amplify status` to find resource name
          "attributes": [
              "arn" // Check Output Value of the resource specific cloudformation file to find available attributes
            ]
        }
      ]
    }
  }
}
```

Input parameters need to be defined in the created `template_<custom-resource-name>.json` file. The format is `<category><resource-name><attribute-name` .

```json
{
"Parameters": {
  "functionlambdaarn": { 
    "Type": "String"
  }
}
}
```

The last step for referencing is to get the value into CloudFormation as well. For that you need to add an `Fn:GetAtt` to your attribute name.

```json
{
  "functionlambdaarn": {
    "Fn::GetAtt": [
      "functionlambda",  
      "Outputs.ARN"
    ]
  }
}
```

Finally, you need to check out your environment again and push it to the cloud with `amplify env checkout <ENV>` and `amplify push`. 

So that is the theory and seems a lot first. Let's add an SQS queue and see how it works in reality. 

## Add a custom SQS queue

We want an SQS queue that has access to a lambda function. At this point we don't care about what triggers the queue and which messages it will send, we simply want to create it.

In this setup we have an active Amplify application and have created one lambda function named `service`.

### Backend Config

First of all, we add a custom SQS resource. I am in root of my project and show you the following steps you need to do to create the queue. We start by editing the `amplify/backend/backend-config.json` 

```JSON
{
  "function": {
    "service": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda"
    }
  },
  "queue": {
    "users": {
      "service": "SQS",
      "providerPlugin": "awscloudformation"
    }
  }
}
```

We added the `queue` parameter. Several things are defined here:

- `queue`: That is the new category name. In lambda, it is `function`.
- `standardQueue`: That is the resource name. That is how you name your queue. In lambda, this is `service`.
- `service`: This is the custom AWS service we want to use. In this case, it is `SQS`.
- `providerPlugin`: That is the provider we use for provisioning the resources. In our case and mostly with Amplify this is CloudFormation.

### CloudFormation

Now we need to add the folders we need:

```bash
mkdir -p amplify/backend/queue/users
touch amplify/backend/queue/users/parameters_users.json
touch amplify/backend/queue/users/template_users.json
```

Now let's first fill the `template_users.json`. We need to add the CloudFormation code here.

```JSON
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Queue resource stack creation using Amplify CLI",
    "Parameters": {
        "env": {
            "Type": "String"
        }
    },
    "Conditions": {
        "ShouldNotCreateEnvResources": {
            "Fn::Equals": [
                {
                    "Ref": "env"
                },
                "NONE"
            ]
        }
    },
    "Resources": {
        "SQS": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "QueueName": {
                    "Fn::If": [
                        "ShouldNotCreateEnvResources",
                        "users",
                        {
                            "Fn::Join": [
                                "",
                                [
                                    "users",
                                    "-",
                                    {
                                        "Ref": "env"
                                    }
                                ]
                            ]
                        }
                    ]
                }
            }
        }
    },
    "Outputs": {
        "Name": {
            "Value": {
                "Ref": "SQS"
            }
        },
        "Arn": {
            "Value": {
                "Fn::GetAtt": [
                    "SQS",
                    "Arn"
                ]
            }
        },
        "Region": {
            "Value": {
                "Ref": "AWS::Region"
            }
        }
    }
}
```

At this point, we can push the SQS queue to the cloud and create it.

```bash
amplify env checkout <ENV> # e.g. dev
amplify push
```

The output of `amplify push` should now be:

```bash
❯ amplify push
✔ Successfully pulled backend environment dev from the cloud.

    Current Environment: dev
    
┌──────────┬───────────────┬───────────┬───────────────────┐
│ Category │ Resource name │ Operation │ Provider plugin   │
├──────────┼───────────────┼───────────┼───────────────────┤
│ Queue    │ users         │ Create    │ awscloudformation │
├──────────┼───────────────┼───────────┼───────────────────┤
│ Function │ service       │ No Change │ awscloudformation │
└──────────┴───────────────┴───────────┴───────────────────┘
? Are you sure you want to continue? (Y/n)
```

Let's hit `Y` and deploy our SQS queue. 

Let's check the CloudFormation template. Go to your Amplify console → find your environment and click on **View in CloudFormation**. 


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403093511/NPA2qZQNQU.png)

If you look at one of the nested stacks you will see that your SQS queue was created:


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403111866/y5vyQLR3I.png)

Now open  [SQS](https://eu-central-1.console.aws.amazon.com/sqs/v2/)  and you see your queue being ready to process some messages. 


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403119996/ual2Nd06d.png)

### Lambda Consumer

Now we want to connect our SQS queue with our lambda function. To accomplish that we need to connect the lambda with our SQS queue and then define that queue as the trigger for the lambda. We already added the URL and ARN of our SQS queue in the `outputs` section of the CloudFormation template. That means that the CloudFormation template will output these two parameters after it is finished and other resources have access to these parameters. Let's now use this as an input for lambda. For that, we need to edit the `amplify/backend/backend-config.json` file again.

```json
"function": {
    "service": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda",
      "dependsOn": [
        {
          "category": "queue",
          "resourceName": "users",
          "attributes": ["Name", "Arn"]
        }
      ]
    }
  }
```

We change the function part of the file so, that we are depending on the category `queue` with our exact name `users` and the attributes we want to pass. 

Now we just need to update the CloudFormation template of lambda to receive and handle these params. 

Open `amplify/backend/function/service/service-cloudformation-template.json` and edit the following things: 

1. You need to add your queue name and queue ARN as an input parameter in  the section `Parameters`
2. You define the name of the SQS queue as an environment variable in the section `Resources/LambdaFunction/Environment/Variables`

```json
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Lambda Function resource stack creation using Amplify CLI",
  "Parameters": {
# THIS IS ADDED in the format <CATEGORY RESOURCE>Name (without space)
    "queueusersName": {
      "Type": "String",
      "Default": "queueusersName"
    },
# THIS IS ADDED in the format <CATEGORY RESOURCE>Arn (without space)
    "queueusersArn": {
      "Type": "String",
      "Default": "queueusersArn"
    }

  },
  ...
  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {
        "aws:asset:path": "./src",
        "aws:asset:property": "Code"
      },
        ...

        "Environment": {
          "Variables": {
# THIS IS ADDED: Same name as above
            "SQS_QUEUE": {
              "Ref": "queueusersName"
            },
            "ENV": {
              "Ref": "env"
            },
            "REGION": {
              "Ref": "AWS::Region"
            }
          }
        },
        "Role": {
          "Fn::GetAtt": [
            "LambdaExecutionRole",
            "Arn"
          ]
        },
        "Runtime": "nodejs14.x",
        "Layers": [],
        "Timeout": 25
      }
    },
   ...
  }
}
```

After checking out the environment and pushing again with `amplify env checkout dev && amplify push -y` you see that the URL was added as an environment variable in lambda.


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403173736/NmjH4hxGF.png)

### SQS as input trigger

Finally, let's call the lambda from the SQS queue. The connection of the environment variable should show how to pass arguments. Don't send a message from the SQS queue **and** call the lambda from the SQS queue because this ends up in a loop. 

To add the queue as an input trigger we need to edit the CloudFormation template again. We will add the following things:

1. Add `LambdaFunctionEventSourceMapping` and add the SQS URL here
2. Add an IAM policy statement to give it access to the resource

See the full file  [here](https://github.com/AlessandroVol23/amplify-custom-sqs/blob/main/amplify/backend/function/service/service-cloudformation-template.json) 

```JSON
{...
  "Resources": {
    "LambdaFunctionEventSourceMapping": {
      "DependsOn": [
        "lambdaexecutionpolicy"
      ],
      "Type": "AWS::Lambda::EventSourceMapping",
      "Properties": {
        "EventSourceArn": {
          "Ref": "queueusersArn"
        },
        "FunctionName": {
          "Ref": "LambdaFunction"
        }
      }
    },
# ADD THIS
    "lambdaexecutionpolicy": {
      "DependsOn": [
        "LambdaExecutionRole"
      ],
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "lambda-execution-policy",
        "Roles": [
          {
            "Ref": "LambdaExecutionRole"
          }
        ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
# ADD THIS
            {
              "Effect": "Allow",
              "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
              ],
              "Resource": {
                "Ref": "queueusersArn"
              }
            },
            ...
                ]
    }
...

```

Let's push again `amplify env checkout dev && amplify push -y` and see that our lambda function now has SQS as a trigger

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403249049/Fo-sjwVIJ.png)

Finally just change the resource in the IAM policy of the CloudFormation template to: 

```JSON
"Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "sqs:ReceiveMessage"
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
              ],
              "Resource": {
                "Ref": "queueusersArn"
              }
            },
```

With that, we follow the least privileged principle and don't give lambda access to all resources. This wasn't possible before because Amplify wouldn't recognize it. 

Now we see that the IAM policy is restricted on this one resource.


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635403262365/kht9wuNpU.png)

Now we can go to SQS, send a message and see that the lambda was executed successfully. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1635427139700/Uxam9vkUO.png)

The video shows it as well but can't be embedded right now 🤷🏽‍♂️

{% youtube pRs3M7m0JX8 %}

## Summary

That's it. It seems a bit much at the beginning but with CloudFormation and Amplify you are very flexible in how to add and connect different resources. 

Here is a short summary of the steps needed to do:

1. Custom resource in backend config
2. Folders & CloudFormation for custom resource
3. Pass parameter from SQS to lambda from backend config
4. Adjust CloudFormation template in lambda to use the parameter

If you want to know more about serverless, AWS, and building SaaS products on the cloud follow my [Twitter](https://twitter.com/sandro_vol) 🙂

References: [https://medium.com/@navvabian/how-to-add-an-sqs-queue-to-your-amplify-cli-bootstrapped-project-cb7781c636ed](https://medium.com/@navvabian/how-to-add-an-sqs-queue-to-your-amplify-cli-bootstrapped-project-cb7781c636ed)