AWS Lambda recursive loop detection - but not (yet) for S3
Published: 2023-07-17 16:05:44

Recursive loops in AWS Lambda are the root of all evil because you create a large bill quickly. 


You might think the danger is over because of [new AWS recursive loop detection](https://aws.amazon.com/blogs/compute/detecting-and-stopping-recursive-loops-in-aws-lambda-functions-2/). But what about S3 buckets and Lambda? It's time for a closer look:

**A word of caution**: Be very careful when you test this. You could create large AWS bills.

TL;DR: Never ever use the same bucket for invoking a Lambda and write the output.

Some testing infrastructure:

1) A S3 bucket (CDK, GO)

```go
	bucky := awss3.NewBucket(stack, aws.String("incoming-ring"), &awss3.BucketProps{
		BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(),
		RemovalPolicy:     awscdk.RemovalPolicy_DESTROY,
	})
```

2) A Lambda Function Resource

```go
	ringFunction := awslambda.NewFunction(stack, aws.String("ring"),
		&awslambda.FunctionProps{
			Description:            aws.String("ring - test recursive loop stop"),
			FunctionName:           aws.String("ring"),
			LogRetention:           awslogs.RetentionDays_THREE_MONTHS,
			MemorySize:             aws.Float64(1024),
			Timeout:                awscdk.Duration_Seconds(aws.Float64(10)),
			Code:                   awslambda.Code_FromAsset(&lambdaPath, &awss3assets.AssetOptions{}),
			Handler:                aws.String("main"),
			Runtime:                awslambda.Runtime_GO_1_X(),
			DeadLetterQueueEnabled: aws.Bool(true),
			DeadLetterQueue:        dlq,
		})
```

3) A Lambda Function, which gets the PutObject Event and write *to the same bucket*:

```go
	someString := "hello world\nand hello go and more"
	myReader := strings.NewReader(someString)

	resp, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(s3input),
		Body:   myReader,
	})
```

4) An event, which triggers the Lambda function each time an object is Put to the bucket:

```go
	myHandler.AddEventSource(event.NewS3EventSource(bucky, &event.S3EventSourceProps{
		Events:  &[]awss3.EventType{awss3.EventType_OBJECT_CREATED,},
	}))
```

Now we have:


![Ring architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xjrjwuf0jyz5fe3ontyj.png)

And because I am very careful, I decide to create a recursive loop stopper inside the Lambda function:

```go
	GlobalCounter++
	if GlobalCounter > 20 {
		log.Fatal("Counter exceeded\n")
		os.Exit(1)
	}
```

And in the Console, we set the concurrency to 1:


![concurrency](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g4akfsj8lxvzc5yppl3t.png)

When we test the recursive loop, we *always* have a browser window with the concurrency open. Because: Setting the concurrency to **0** is the fastest way to stop an endless loop.

## The test 

Lets test whether S3-Lambda-S3 recursive loops are still possible:

We use [saw](https://github.com/TylerBrock/saw) to tail lambda log:

```bash
saw watch /aws/lambda/ring
```

And then start the recursive loop:

```bash
aws s3 cp README.md s3://ring-incomingring3eb9a9b9-1a8q8cp8qm2ch/readme1.md
upload: ./README.md to s3://ring-incomingring3eb9a9b9-1a8q8cp8qm2ch/readme1.md
```



Very fast you get the logs:

```txt
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) INIT_START Runtime Version: go:1.v18	Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:ccb68acb59818f9df9b10924cc6c83ca6eaf4067f70ba861c0e211b59e8af729
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) START RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9 Version: $LATEST
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Counter: 1
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Etag:  446cce24a7c945503232494e881150ec
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Seq:  0064B55F8E25E34383
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) END RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9
[2023-07-17T17:34:39+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) REPORT RequestId: 10fba020-0e93-4430-a9d7-728d34b146c9	Duration: 92.67 ms	Billed Duration: 93 ms	Memory Size: 1024 MB	Max Memory Used: 39 MB	Init Duration: 110.22 ms
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) START RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8 Version: $LATEST
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Counter: 2
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Etag:  a9c4c5f405a3e4cc6ba3b6b355da0e71
[2023-07-17T17:34:40+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) Seq:  0064B55F8FD24FBE14
[2023-07-17T17:34:41+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) END RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8
[2023-07-17T17:34:41+02:00] (2023/07/17/[$LATEST]e27ec4706f1a4fb0a6d3ca1fdb23bbc0) REPORT RequestId: 97cf261e-b272-437f-9637-b7573cfb0fa8	Duration: 27.20 ms	Billed Duration: 28 ms	Memory Size: 1024 MB	Max Memory Used: 40 MB
```

The interesting part is:

```txt
Counter: 1
Counter: 2
...
Counter: 19
Counter: 20
Counter exceeded
```

And because [AWS BLOG about recursive loops](https://aws.amazon.com/blogs/compute/detecting-and-stopping-recursive-loops-in-aws-lambda-functions-2/) says:
"Lambda now detects the function running in a recursive loop between supported services after exceeding 16 invocations. It returns a RecursiveInvocationException to the caller."

We are now sure that the loop detection for S3 is *not* working because we counted to 20!

And although I implemented a loop stopper  very fast, I started a recursive loop:

![Loop](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n9vsri4nlelm6glgb8pz.png)

So now I set the concurrency to 0 again and delete the Lambda.


## Summary

It's proven:

Recursive Loop detection does **not** stop S3 recursive loops, so do use *different buckets* for event trigger and Lambda output!

So be careful out there.
And checkout my website [go-on-aws](https://www.go-on-aws.com/) or - even better :) - use my new udemy course [GO on AWS - Coding, Serverless and Infrastructure as Code](https://www.udemy.com/course/go-on-aws-coding-serverless-and-iac/?referralCode=954E43527F32E22BB1C7)


Enjoy building!
Gernot