SQS Queues as an EventBridge Rule Target (with CloudFormation)
Tags: sqs, eventbridge, aws, cloudformation
Published: 2021-12-02 06:40:28

AWS offers a [dizzying array of targets for EventBridge Rules](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html#eb-console-targets), but even if you set them up through the console, the documentation for each target type can be a bit light.

SQS, being an early AWS service, can come with some extra quirks that make integrations a bit trickier, especially when it comes authorization or configuration.

In this article, I show you how to set up SQS queues and SQS FIFO queues as a target of an Event Bridge Rule using CloudFormation syntax. 

*NOTE: I've assumed you're familiar with EventBridge events and SQS, and just want to get up and running.*

## Event Bridge to SQS Queue - the simple case

There is nothing special in the EventBridge rule - just the target ARN of the Queue:

```yaml
  MyQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      # Configure the event pattern to filter your events (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html)
      EventPattern:
        source: ["com.mycompany.events"]
        detailType: ["CustomerPurchase"]
      Targets:
        - Arn: !GetAtt MyQueue.Arn
          Id: QueueTarget # set as needed
```

You don't need a role ARN because the permissions are handled using queue policies

## Queue and Queue Policy

Define your queue - it requires no special properties:

```yaml
  MyQueue:
    Type: AWS::SQS::Queue
```

and define a queue policy, which will permit EventBridge to write events to your queue:

```yaml
  MyQueueEventBridgePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Principal: { Service: events.amazonaws.com }
            Action: SQS:SendMessage
            Resource: !GetAtt MyQueue.Arn
      Queues:
        - !Ref MyQueue
```

You can further restrict the policy with a condition key limiting access to the event rule that invoked the policy (you shouldn't consider this optional, otherwise you're granting access to any EventBridge rule):

```yaml
         Statement:
           -
            ...
            Condition:
              ArnEquals: { "aws:SourceArn": !GetAtt MyQueueTarget.Arn }
```

## Event Bridge to FIFO Queue - the complex case

Similar to the normal SQS queue case, you define your event bridge rule:

```yaml
  MyFifoQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      # Configure the event pattern to filter your events (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html)
      EventPattern:
        source: ["com.mycompany.events"]
        detailType: ["CustomerPurchase"]
      Targets:
        - Arn: !GetAtt MyFifoQueue.Arn
          Id: QueueTarget # set as needed
          SqsParameters:
            MessageGroupId: myMessageGroupId
```

Similar to SQS queues, you don't need a Role ARN, but you do need to specify a `MessageGroupId`. This value is a fixed string - there is currently no support to make this dependent on a value in the incoming event, which means all your messages will be in the same [message group](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html).

## Queue and Queue Policy

Your queue is similar, but will need to be a FIFO queue, and have content-based deduplication turned on (this seems to mandatory to avoid silent failures):

```yaml
  MyFifoQueue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: true
      ContentBasedDeduplication: true
```

The queue policy is the same (see the previous section for an example).

## What about encryption?

SQS [supports two types of encryption at rest](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html):

* **SSE-SQS** - SQS manages the encryption for you. This currently isn't available with CloudFormation, so I haven't tested its use with EventBridge
* **SSE-SQS** - KMS is used to perform encryption, either with a AWS-managed key or a Customer-Managed Key (CMK)

Transport encryption still relies on TLS, and IAM (via queue policies) is used to perform authorization and control access to the queue.

Using the AWS-managed key with SQS (by specifying `KMSMasterKeyId: alias/aws/sqs`) doesn't appear to work, and this is probably because specific access hasn't been granted to EventBridge to access the key, and probably can't be made to work because AWS managed keys do not let you edit their IAM key policy.

A customer managed key may be able to be made to work - [see this FAQ for an example](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-troubleshooting.html#eb-sqs-encrypted).

Hopefully once SSE-SQS is available in CloudFormation, it should be simple matter of enabling it so encryption can be enabled in SQS when used as an EventBridge target.

## Processing messages

The records that arrive in your SQS queue will have a body of the whole EventBridge event by default ([unless you configure your EventBridge rule to transform, or select part of the event, or pass a different JSON entirely using the InputTransformer/InputPath/Input properties](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-events-rule-target.html#aws-properties-events-rule-target-properties) - see [this article](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html) for details on using `InputTransformer`).

If you process your "EventBridge over SQS" events with AWS Lambda, your input event will look something like:

```json
{
    "Records": [
        {
            "messageId": "9c3aeeeef-6eee-4a18-9abc-89753969a233",
            "receiptHandle": "AQEBUjLMNIEkASgzMS9sSrfQdkZZa7tfH0sLH/KEXpr3P3c+Aen0WQ3x3gZoKXycvhkjU3F6vA+VwRky3JOCRxUN5o1d97y52Iqw0ID/HUflIXDoJWEUCRDUIRCONTEtbYW39EE5Fxxz+rc4YyBw5v8dlxCDtXdgdyEucUVIdQt4K+94YJdz3GTpQF2ASg0YdhRQvWQb4to+wfJLVW/C0f/cgY43zFTNHrRCSuRKBMNEUONSKntc0ubh7QHyBjNoIUC+2QXGq2ECdkBlBJ9BVDQZLXuObgjcoL1hi2XiLebTKCRT0Jo0rMEfs17giLHIFwy1ZrKeOqim",
            "body": "{\"version\":\"0\",\"id\":\"5b2fcd39-5fd6-b4ef-51a6-7d5624ced5da\",\"detail-type\":\"CustomerPurchase\",\"source\":\"com.mycompany.events\",\"account\":\"3829362938291\",\"time\":\"2021-12-02T06:01:09Z\",\"region\":\"us-east-1\",\"resources\":[],\"detail\":{\"type\":\"order\",\"orderNumber\":\"2928\"}}",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:3829362938291:my-queue.fifo",
            "awsRegion": "us-east-1"
        }
    ]
}
```

If you don't care about the EventBridge envelope, and would prefer to receive the raw `detail` component of the message, the simplest input transformation is to set `InputPath` to `$.detail`.


```yaml
  MyFifoQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      ....
      Targets:
        - 
          Id: QueueTarget # set as needed
          ...
          InputPath: $.detail # just send the event detail
```

This is most useful when porting existing integrations (e.g. SNS -> SQS with raw message delivery), when you're using your own events, and/or when your event rule selects the Event Source and Event Detail Type such that you know the schema of the detail component without needing the EventBridge metadata to distinguish it.