Amazon SQS to AWS Lambda error handling
Tags: aws, serverless, sqs, lambda
Published: 2022-01-27 04:39:29

---
title: Amazon SQS to AWS Lambda error handling
published: true
description: How to gracefully process Amazon SQS messages using AWS Lambda.
tags: aws, serverless, sqs, lambda
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b4e1y5mos2hont4p9gyj.jpeg
---

As serverless developers, we often find ourselves stringing multiple services together. Be it for resiliency needs or to aid us in creating asynchronous communication channels. Thankfully, insightful people from AWS serverless community came up with various patterns one can tweak to achieve their architecture needs.

One of the most common integration patterns I see in the wild is the [_Amazon SQS_](https://aws.amazon.com/sqs/) to [_AWS Lambda_](https://aws.amazon.com/lambda/) pattern. There is a surprising amount of minutia that developers must keep in mind while using it in production – like error handling.

This blog post will learn about the _Amazon SQS_ to _AWS Lambda_ integration itself, how to handle errors while reading from _Amazon SQS_ queues, and what the latest pre-reinvent (2021) announcement has to do with it.

## The integration

The following is a diagram depicting _Amazon SQS_ to _AWS Lambda_ integration.

![The integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6e311h80g8r0roz5pvks.jpeg)

The diagram exposes a bit of the internals, at least the ones that I could infer from AWS documentation, of the integration.

The _event source mappings_ component plays a crucial role in this architecture – it is responsible for polling the events from the _Amazon SQS_ queue, batching them, and handling the _AWS Lambda_ responses. If you want to learn more about _event source mappings_, I have [written an article on this topic](https://dev.to/wojciechmatuszewski/the-unsung-hero-of-serverless-apps-event-source-mapping-1679).

Whether the _Data Queue_ backlog gets smaller (what you most likely want) depends on how our _AWS Lambda_ function behaves – whether or not it throws an error and, as os recently, if it returns a specific set of data.

Before showing you what I would do to ensure my _AWS Lambda_ plays nicely with the _event source mappings_, let us explore what, in my humble opinion, you should not be doing.

## How to get yourself into trouble

In my opinion, **in most scenarios**, the worst thing that could happen is for the _Reader AWS Lambda_ to throw an error.

![Lambda reporting runtime error](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b1gml021wdpgq21l7aly.jpeg)

This behavior in itself is **not inherently wrong**. The problem lies in the fact that many people **do not realize the consequences** of throwing a runtime error in the _Reader AWS Lambda_.

So what are the consequences, you might ask? According to the [_event source mappings_ AWS Documentation page](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html):

> By default, if your function returns an error, the entire batch is reprocessed until the function succeeds, or until the items in the batch expire. To ensure in-order processing, Lambda pauses processing for the affected shard until the error is resolved.

Now, imagine processing a batch of 100s of _Amazon SQS_ events. If one of them fails (might be the case of a [poison pill](https://github.com/jlhood/serverless-app-ideas/blob/master/design/poison-pill-defender.md)), the _event source mappings_ component will retry the whole batch! If your system is not [idempotent](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/), you might be in trouble.

What can you do to ensure the situation where the entire batch is retried never happens?

## How to handle errors gracefully

We have two options at our disposal. First uses the newly announced feature of [returning partial batch responses from the _Reader AWS Lambda_](https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-partial-batch-response-sqs-event-source/). The second is a method AWS serverless developers used before the feature mentioned above was available.

Let us talk about the "new way" signaling errors to the _Amazon SQS_ _event source mappings_ first.

### Partial batch responses

![Returning partial batch responses](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4hfqjnjen6pmxxq3psdj.jpeg)

The _Reader AWS Lambda_ code keeps track of failed messages. Those message identifiers are sent back to the _event source mappings_.

The following is an example TypeScript code snippet of how one might do just that.

```ts
import { SQSEvent, SQSRecord } from "aws-lambda";

type Response = { batchItemFailures: { itemIdentifier: string }[] };

export const handler = async (event: SQSEvent): Promise<Response> => {
  const records = event.Records;

  const response: Response = { batchItemFailures: [] };

  const promises = records.map(async record => {
    try {
      await doWork(record);
    } catch (e) {
      response.batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  });

  await Promise.all(promises);
  return response;
};

async function doWork(record: SQSRecord) {
  /** Your implementation*/
}
```

Since I'm using the `Promise.all` interface, the record processing happens in parallel. This way of handling errors is, in my opinion, much easier than the one I'm about to show you next – let us explore why.

### Before partial batch responses feature

Before AWS introduced the _partial batch response_ feature, the error handling part had to be more involved – we were required to **manually delete the successfully processed messages**. If we delete the message, even if we throw an error in the _AWS Lambda_ function, the message is gone.

The following is an example TypeScript code snippet of how one might write such logic.

```ts
import { SQSHandler, SQSRecord } from "aws-lambda";

export const handler: SQSHandler = async event => {
  const records = event.Records;

  const promises = records.map(async record => {
    await doWork(record);
    await deleteRecordViaSDK(record);
  });

  const response = await Promise.allSettled(promises);
  const hasFailedRecords = response.find(
    record => record.status === "rejected"
  );

  if (hasFailedRecords) {
    throw new Error("Failed to process one or more records");
  }
};

async function doWork(record: SQSRecord) {
  /** Your implementation*/
}

async function deleteRecordViaSDK(record: SQSRecord) {
  /**Use aws-sdk to delete the message */
}
```

Like in the previous example, the records are processed in parallel by using the `Promise.allSettled` ([difference between `Promise.allSettled` and `Promise.all`](https://stackoverflow.com/a/59784198)).

After skimming through the code, you might be asking yourself – "What if the `deleteRecordViaSDK` fails?". Great question!

Sadly, I do not have a one-size-fits-all solution for you. The message that we did not successfully delete will be re-queued and pushed to your _AWS Lambda_ function again by the _event source mappings_.

One piece of advice I could give here is to use the `messageId` attribute as a unique identifier used to determine if you have already processed the message or not. I encourage you to lean on existing resources as much as possible. For example, [this AWS blog post](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) is a good start.

## Closing words

With every service integration (not only AWS ones) comes various nuances developers have to account for while building features. The _Amazon SQS_ to _AWS Lambda_ is no different.

I believe that the best thing one can do in such situations is to **thoroughly read the documentation**. You will save yourself a lot of time in the long run.

For more AWS serverless related content, follow me on Twitter – [@wm_matuszewski](https://twitter.com/wm_matuszewski).

Thank you for your time.