Don't Lose Your Events! Use the Transactional Outbox Pattern.
Tags: aws, serverless, architecture, tutorial
Published: 2023-10-23 07:22:01

## 1. Introduction

Events are the key part of an Event-driven Architecture (EDA). They are the mechanism that allows services to communicate with each other in a decoupled way.

For example, an Order Service emits an **OrderCreated** event when a new order is created. The Shipping Service then consumes this event that starts the shipping process.

But how can you ensure that you can reliably publish events? Take a look at the following code:

```javascript
exports.handler = async (event, context) => {
  const order = domain.NewOrder(...);

  await persistOrder(order);
  await publishDomainEvent(new OrderCreated(order));
}

```

This looks straightforward, right? The order service receives a request (or a command) to create a new order. It validates the order, persists it, and publishes a domain event.

When building distributed systems, [eight fallacies](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing) describe false assumptions that developers often make. One of which is **the network is reliable**.

Look at the above code again. What happens if the `publishDomainEvent` call fails? The order will be persisted, but no domain event will be published. If you don't handle this case, you will end up with an inconsistent state. The shipping service will not initiate the shipping process.

There are many solutions of varying complexity to this problem. In this article, you will see how to implement the **Transactional Outbox Pattern** to solve it in a Serverless environment.

## 2. What is the Transactional Outbox Pattern?

In the transactional outbox pattern, you persist updates to business entities and any domain events related to those updates in the same transaction. A separate process reads the persisted domain events and publishes them to a message broker. The transactional nature ensures that the entity and related events are either all persisted or none of them are.

![The transactional outbox pattern.](https://www.eliasbrange.dev/images/transactional-outbox-pattern-dynamodb/transactional-outbox.png)

In the diagram above, when the _Order Service_ receives a request, it:

1. Persists any updates to business entities in the _Order Table_ and any related domain events to the _Outbox Table_ in a single transaction.
2. The _Message relay_ process gets the domain events from the _Outbox Table_.
3. The _Message relay_ process publishes the domain events to a message broker.

## 3. Using DynamoDB Streams for Change Data Capture

[DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) is a powerful feature of DynamoDB that lets you capture all item-level changes made to a table. When you enable a stream on a table, you can configure a Lambda function to be invoked on every change to the table. Streams are fully managed by AWS and are a perfect fit to implement the Transactional Outbox Pattern.

Looking back at the code in the [Introduction](#1-introduction), we were worried about what happens if the `publishDomainEvent` call fails after the `persistOrder` call succeeds. DynamoDB streams can help us solve this.

Using streams, the Lambda function only needs to persist the order to the _OrdersTable_. This change will be sent through the stream to another Lambda function, which publishes the domain events. If there are any errors when publishing the events, the Lambda function will retry the operation until it succeeds or until the records expire.

## 4. Implementing the Pattern in Serverless Applications

### 4.1. Naive approach

Let's start with the naive approach. We will use the following architecture:

![A simple but naive transactional outbox pattern.](https://www.eliasbrange.dev/images/transactional-outbox-pattern-dynamodb/architecture-naive.png)

In this pattern, the _CreateOrder_ Lambda function persists the order to the _OrdersTable_. The table is configured with a DynamoDB stream that triggers the `PublishEvents` Lambda function on every change to the table.

The `PublishEvents` Lambda must parse the change and determine what kind of domain event to send. The code might look something like this:

```javascript
exports.handler = async (event, context) => {
  for (const record of event.Records) {
    const domainEvent = parseDynamoRecord(record);

    if (domainEvent) {
      await publishDomainEvent(event);
    }
  }
};

const parseDynamoRecord = (record) => {
  if (isOrderCreated(record)) {
    return new OrderCreated(record.dynamodb.NewImage);
  }
  return null;
};

const isOrderCreated = (record) => {
  // Check if the record is an INSERT event for an order.
  return (
    record.eventName === 'INSERT' &&
    record.dynamodb.NewImage &&
    record.dynamodb.NewImage.type === 'ORDER'
  );
};
```

This might look like a good and simple solution at first. But it has some drawbacks:

- **Coupling between the _PublishEvents_ Lambda and the _OrdersTable_**

    The _PublishEvents_ Lambda function is coupled to the schema of _OrdersTable_. If you change the schema, you must also change the _PublishEvents_ Lambda function.






- **Complex logic in _PublishEvents_ to handle different events**

    If you want to publish other events, such as _OrderCancelled_ and _OrderUpdated_, you must add complex logic to the _PublishEvents_ Lambda function. This means that you will inevitably have to duplicate the core domain logic since you must correctly determine which domain event to publish depending on what the change contains.

- **If you are using Single-Table Design in DynamoDB**

    If you use a [single-table design](https://www.alexdebrie.com/posts/dynamodb-single-table/) for your DynamoDB table, the complexity in _PublishEvents_ will be even greater. You might have multiple entities in the _OrdersTable_, such as _Orders_ and _OrderItems_. The _PublishEvents_ Lambda function must now determine what kind of entity has changed and what type of event to emit for that entity.


    > There are a lot of advocates for single-table design in DynamoDB. It is, however, not a silver bullet that solves all problems. [This article](https://www.gomomento.com/blog/single-table-design-for-dynamodb-the-reality) by **Pete Naylor** tells a different story and is well worth a read.

The above drawbacks are not ideal. They force you to move core domain logic to the outer bounds of your application while simultaneously coupling the logic to your database schema.

There's a better approach.

### 4.2. Use a Separate Outbox Table and DynamoDB Transactions

![A better approach to the transactional outbox pattern.](https://www.eliasbrange.dev/images/transactional-outbox-pattern-dynamodb/architecture.png)

This pattern introduces another table, the _OutboxTable_. It relies on [DynamoDB transactions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html) to atomically persist both the entity and domain event in a transaction. This means that either both the entity and domain events are persisted or none of them are.

The _PublishEvents_ Lambda function only relays events from the _OutboxTable_ to EventBridge. The code might look something like this:

```javascript
exports.handler = async (event, context) => {
  for (const record of event.Records) {
    if (record.eventName !== 'INSERT') {
      // We only want to publish events when they are inserted in the outbox table.
      continue;
    }
    await publishDomainEvent(record.dynamodb.newImage.event);
  }
};
```

The logic to determine what events to publish is kept within the core domain logic inside the _CreateOrder_ function. The outer bounds of the application are responsible for persisting the order and any domain events in an atomic transaction.

```javascript
exports.handler = async (event, context) => {
  const { order, events } = domain.NewOrder(...);

  await persistNewOrder(order, events);
}

const persistNewOrder = async (order, events) => {
  await dynamoClient.send(
    new TransactWriteItemsCommand({
      TransactItems: [
        {
          Put: {
            TableName: "OrdersTable",
            Item: order,
          },
        },
        ...events.map((event) => ({
          Put: {
            TableName: "OutboxTable",
            Item: {
              event,
            },
          },
        })),
      ],
    }),
  );
};

```

This approach has several benefits:

- **The _PublishEvents_ Lambda is decoupled**

    The _PublishEvents_ Lambda function is now decoupled from the schema of the _OrdersTable_. It doesn't need to know how entities are persisted.

- **The _PublishEvents_ Lambda has low complexity**

    The _PublishEvents_ Lambda function is now straightforward. It only needs to relay events from the _OutboxTable_ to EventBridge.

- **Core domain logic doesn't leak**

    The core domain logic is kept inside the domain layer. It doesn't leak to the outer bounds of the application.

- **The _OutboxTable_ can support different events and entities**

    The _OutboxTable_ can support different events and entities. You can add new events and entities without changing the _PublishEvents_ Lambda function.

## 5. Handling failures

The Transactional Outbox Pattern is all about reliably publishing events. Even though most of the services used are fully managed by AWS, errors can still occur. Let's look at what happens if the _PublishEvents_ Lambda function fails to publish events.

DynamoDB streams send batches of records to the _PublishEvents_ Lambda function. If an invocation of the _PublishEvents_ function fails, the entire batch will be retried until it succeeds or until the records expire.

For example, imagine that a batch of ten records is sent to the _PublishEvents_ Lambda function, and the function fails to publish the 10th event. The entire batch is retried, and this time it succeeds. Events 1-9 will now have been published twice.

There are a couple of different ways to handle this:

- **Making clients idempotent**

    Make sure that clients that consume the events implement idempotency into their operations. This means that they must be able to handle the same event being published multiple times. I'd recommend always doing this since many AWS services use at-least-once delivery.


    Even if you can guarantee that you only publish an event once, the client could still receive the same event more than once from EventBridge itself.



    For a deeper dive on idempotency, refer to [this excellent article](https://www.readysetcloud.io/blog/allen.helton/api-essentials-idempotency/) written by _Allen Helton_.

- **Using a batch size of 1**

    You can configure the Lambda function to only receive one record at a time. If the function fails, it will only retry that single record. This is the simplest option, but it could increase the number of invocations and the cost of the Lambda function. This could also increase latency if you write many events to the _OutboxTable_.

- **Bisecting batch failures**

    A more advanced concept in DynamoDB streams lets you [bisect a batch](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html#services-ddb-batchfailurereporting) when an invocation fails. When an invocation fails in the middle of a batch, a function can report the last successfully processed record. The batch is then bisected at this record so that only the unprocessed records are retried.

## 6. Limitations

Since this pattern relies on DynamoDB transactions, it inherits the same limitations. Transactions are limited to 100 items. This means that you cannot publish, say, 150 domain events in a single transaction.

I can't really see a reason for publishing that many events from a single operation, so this shouldn't be a problem in practice.

## 7. Conclusion

The Transactional Outbox Pattern is a great pattern to publish domain events reliably. DynamoDB streams let you easily implement the pattern in your Serverless applications. Combined with DynamoDB transactions, your domain logic can be kept nicely encapsulated in the domain layer, while the outer parts of the application can be kept as simple as possible.

## 8. Further reading

- [Change data capture for DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)
- [Using AWS Lambda with Amazon DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html)
- [Transactional Outbox Pattern](https://microservices.io/patterns/data/transactional-outbox.html)