Tracking failed SQS messages
Tags: aws, lambda, observability, serverless
Published: 2019-12-28 01:26:45

---
title: Tracking failed SQS messages
published: true
description: Find logs for failing Lambda, beginning with SQS message which caused the failure
tags: AWS,Lambda,observability,serverless
---

![Photo by Paweł Czerwiński on Unsplash](https://thepracticaldev.s3.amazonaws.com/i/e6gnb7jkw9el1t00t162.jpg)
<sub><sup>_Photo by Paweł Czerwiński on Unsplash_</sub></sup>


Tracking the message flow through the system is indispensable ability in large distributed applications. 

Amazon provides limited tools out of the box. There are better dedicated services for this purpose, but sometimes we are left with what we have in AWS.

In this article I will show you how you can track the logs of Lambda function. 

The function was invoked from CloudWatch scheduled event and had dead-letter queue (DLQ) assigned.
As a result the failing message landed in DLQ.



1) Get SQS DLQ message - navigate in AWS Cloud Console to SQS service, then find your DLQ and select "Queue Actions -> View/Delete Messages". 

You can then see all messages. Select one message and you should see something similar as below:

```
{
"version":"0",
"id":"29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2",
"detail-type":"Scheduled Event",
"source":"aws.events",
"account":"763369520800",
"time":"2019-09-19T23:30:15Z",
"region":"eu-west-1",
"resources":["arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function"],
"detail":{}
}

```


2) Get the id value (`29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2`) and lookup in [CloudWatch Log Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) for your Lambda function, e.g. use query:

fields @timestamp, @message
| sort @timestamp desc
| limit 20
| filter id='29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2'

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/us6mlrzxsbmeeor82vpd.png)

As a result you would get a bunch of messages like: 

```

@ingestionTime 1568936016992
@log 763369520800:/aws/lambda/my-lambda-function
@logStream 2019/09/19/[$LATEST]45f87b84a2fd4fb0a79df2c81c57cc06
@message 2019-09-19T23:33:30.528Z b276b9b4-bc43-4876-8052-0f94a38720a7 event: {"version":"0","id":"29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2","detail-type":"Scheduled Event","source":"aws.events","account":"763369520800","time":"2019-09-19T23:30:15Z","region":"eu-west-1","resources":["arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function"],"detail":{}}
@requestId b276b9b4-bc43-4876-8052-0f94a38720a7
@timestamp 1568936010528
account 763369520800
detail-type Scheduled Event
id 29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2
region eu-west-1
resources.0arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function
source aws.events
time 2019-09-19T23:30:15Z

```

Pay attention to `@logStream` value. Now you can lookup this stream in CloudWatch:

![Search CloudWatch stream](https://thepracticaldev.s3.amazonaws.com/i/sem00i42hshmaz8fw54u.png)

The stream can contain plenty of logs so navigating through all of them and searching interesting ones can be tedious work.

Instead of fetching Lambda logs in AWS Cloud Console there are better tools. 

You can check [saw](https://github.com/TylerBrock/saw) and the [other article](https://dev.to/piczmar_0/pleasant-parsing-of-aws-lambda-logs-fdp) which I dedicated to using this tool.