Event loops and idle connections: why is my lambda not returning and then timing out?
Tags: debugging, beginners, javascript, aws
Published: 2021-11-16 07:12:41

---
title: Event loops and idle connections: why is my lambda not returning and then timing out?
published: true
description:  debugging chronicle - what is the event loop, why does and idle connection makes a lambda time out?
tags: #debugging, #beginner, #javascript, #aws
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1nuhp2elgwe1x8j4afgj.png
---


Recently, we noticed that some of our AWS Lambdas started in some cases to **hang and time out**.
Checking last commits before the deployment we could not notice anything suspicious, no changes in the Lambda handler nor in the serverless configuration.
The logs showed no particular reason, no errors, no uncaught exceptions, no network issues. After adding some more logs here and there we were sure nothing was wrong in our code, **we even got the result from the underlying method, but the lambda handler was simply not returning it!**


### Fix the problem

One of our developers though, found out a parameter in the [Lambda nodejs context](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html) that would solve the issue: `callbackWaitsForEmptyEventLoop`.

I am not a fan of long variable names, but in this case I must admit that perfectly explains the purpose of such a property.

By setting it to `false`, the Lambda callback will wait for the next empty event loop.  If you are asking what an event loop is, I really suggest watching [this video](https://www.youtube.com/watch?v=8aGhZQkoFbQ), it dates back to 2014 - but it explains very well **what the heck is the event loop**! 

![what the heck is the event loop anyway?](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wrj4g16wir1ycyjw5fue.png)


Under `callbackWaitsForEmptyEventLoop` AWS documentation states: 
> For non-async handlers, function execution continues until the event loop is empty or the function times out. The response isn't sent to the invoker until all event loop tasks are finished. If the function times out, an error is returned instead.

**The response is not sent to the invoker (...)**. - that was clearly our case.  

Therefore we changed our handler so that it contained the following:

```
context.callbackWaitsForEmptyEventLoop = false;
```

Actually, since we are wrapping all our handlers with [Middy](https://github.com/middyjs/middy), we just went for [another middleware](https://www.npmjs.com/package/@middy/do-not-wait-for-empty-event-loop) to do that.

And the problem disappeared.  Everything was again super fast as before!


### Understand the solution 

But why?  we didn't change anything in the configuration, our handlers were always properly working with their [async and their await - we are not even using](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html#nodejs-handler-sync) **lambda callbacks** at all!).

> Why did that problem pop up so at once? Why does that parameter which we never needed, solved that problem now?


I googled and read [some more docs](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html) multiple times:


- The response isn't sent to the invoker until all event loop tasks are finished. 

- You can configure the runtime to send the response immediately.

What does that mean?
Our lambda was connecting to a [Aurora Serverless](https://aws.amazon.com/rds/aurora/serverless/), but our logs proved that we got our results on time, so it was not that hanging or timing out, it was really our lambda that was not sending back the results, waiting for something to be terminating in the event loop.  But what was that?

I spent some more time investigating latest commits and read some more documentation about [managing properly Aurora Serverless connections](https://www.jeremydaly.com/manage-rds-connections-aws-lambda/) and about [Sequelize, the ORM framework](https://sequelize.org/) we are using to connect to Aurora. 

I found out [this interesting article](https://www.serverlesslife.com/AWS_Serverless_Common_Mistakes_Communication_with_Other_Systems.html
) which explained a similar issue - a connection to a database, whose result was then idling to return because of the handler waiting for the next empty event loop.

This was pointing in the right direction.  Probably our DB connection was being kept open and caused the lambda to hang instead of returning immediately?
That was exactly the case, and indeed I saw some commits related to the [configuration of Sequelize instances pool](https://sequelize.org/v5/class/lib/sequelize.js~Sequelize.html#instance-constructor-constructor), and more precisely about how long they should be idle before being released. 

> **pool.idle**:  The maximum time, in milliseconds, that a connection can be idle before being released. Defaults to  10000 milliseconds.

Since our lambdas had a timeout of 5 seconds, that was exactly what was causing the lambda to hang and timeout!

After adjusting the idle timeout we did not need `callbackWaitsForEmptyEventLoop` anymore ( as we did before ) and we proved that **this solution would have just solved the symptom but not [the cause](https://dev.to/dvddpl/what-do-you-mean-with-there-is-no-root-cause-4gpa).**

Actually in the end we decided to keep the [middleware](https://www.npmjs.com/package/@middy/do-not-wait-for-empty-event-loop) on every lambda to avoid similar issues in the future, still, although [annoying and a bit time-consuming](https://dev.to/dvddpl/my-jobs-is-boring-and-everything-sucks-or-not-1lnp), this bug was a very interesting discovery.

I hope it helps


----
Photo by <a href="https://unsplash.com/@lieselot_dalle?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Lieselot. Dalle</a> on <a href="https://unsplash.com/s/photos/broken-clock?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>