Asynchronous Background Processing for Ruby or Rails using AWS Lambda Extensions. 
Tags: serverless, lambda, containers, ruby
Published: 2021-07-06 11:25:53

Ever since writing this post last year on [Using New Relic APM with Rails on AWS Lambda](https://dev.to/aws-heroes/using-new-relic-apm-with-rails-on-aws-lambda-51gi), I have always wanted to find a way to send APM data in a way that did not add extra milliseconds to the application's response times. Likewise, for smaller projects it would be nice to have a lightweight alternative to [Lambdakiq](https://github.com/customink/lambdakiq) for ActiveJob. Something similar to Brandon Hilkert's popular [SuckerPunch](https://github.com/brandonhilkert/sucker_punch) gem. Today we have both with the [LambdaPunch](https://github.com/customink/lambda_punch) gem. 

<a href="https://github.com/customink/lambda_punch"><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1yat4b33nybxm39mir0m.png" alt="LambdaPunch: Async Processing using Lambda Extensions"></a>

Since this is my first, dive into the now generally available [Lambda Extensions API](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html), I'd like to share a little bit of how LambdaPunch works. Huge credit to AWS Community Builder Duarte Nunes for this post [Building an AWS Lambda extension with Rust](https://dev.to/aws-builders/building-an-aws-lambda-extension-with-rust-3p81) which is a great in-depth read on how to author Lambda Extensions. Following is what makes our Ruby/Rails extension interesting.

## 🏗 Architecture

Because AWS Lambda [freezes the execution environment](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html) after each invoke, there is no "process" that continues to run after the handler's response. However, thanks to Lambda Extensions along with its "early return", we can do two important things. First, we leverage the [rb-inotify](https://github.com/guard/rb-inotify) gem to send the extension process a simulated `POST-INVOKE` event. We then use [Distributed Ruby](https://ruby-doc.org/stdlib-3.0.1/libdoc/drb/rdoc/DRb.html) (DRb) from the extension to signal your application to work jobs off a queue. Both of these are synchronous calls. Once complete the LambdaPunch extensions signals it is done and your function is ready for the next request.

![LambdaPunch Ruby Extension Sequence Diagrm](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dk0zgjy6ejz31pwm490t.png)

The LambdaPunch extension process is very small and lean. It only requires a few Ruby libraries and needed gems from your application's bundle. Its only job is to send signals back to your application on the runtime. It does this within a few milliseconds and adds no noticeable overhead to your function.

## 🎁 Installation

Add this line to your project's `Gemfile` and then make sure to `bundle install` afterward.

```ruby
gem 'lambda_punch'
```

Now, within your application's handler file, make sure to start the LambdaPunch DRb server outside of your handler method. Within the handler method, add an `ensure` section that lets the extension process know the request is done.

```ruby
LambdaPunch.start_server!

def handler(event:, context:)
  # ...
ensure
  LambdaPunch.handled!(context)
end
```

Within your project or [Rails application's](https://lamby.custominktech.com/docs/anatomy) `Dockerfile`, after you copy your code, add this `RUN` command to install the extension within your container's `/opt/extensions` directory.

```dockerfile
RUN bundle exec rake lambda_punch:install
```

If you are using `LambdaPunch` with a non-Rails project, add this to your Rake file

```ruby
spec = Gem::Specification.find_by_name 'lambda_punch'
load "#{spec.gem_dir}/lib/lambda_punch/tasks/install.rake"
```

## 🧰 Usage

Anywhere in your application's code, use the `LambdaPunch.push` method to add blocks of code to your jobs queue.

```ruby
LambdaPunch.push do
  # ...
end
```

For example, if you are using Rails with AWS Lambda via the [Lamby](https://lamby.custominktech.com/) gem along with [New Relic APM](https://dev.to/aws-heroes/using-new-relic-apm-with-rails-on-aws-lambda-51gi) here is how your handler function might appear to ensure their metrics are flushed after each request.

```ruby
def handler(event:, context:)
  Lamby.handler $app, event, context
ensure
  LambdaPunch.push { NewRelic::Agent.agent.flush_pipe_data }
  LambdaPunch.handled!(context)
end
```

### ActiveJob

You can use LambdaPunch with Rails' ActiveJob. **For a more robust background job solution, please consider using AWS SQS with the [Lambdakiq](https://github.com/customink/lambdakiq) gem.**

```ruby
config.active_job.queue_adapter = :lambda_punch
```

### Timeouts

Your function's timeout is the max amount to handle the request and process all extension's invoke events. If your function times out, it is possible that queued jobs will not be processed until the next invoke.

If your application integrates with API Gateway (which has a 30 second timeout) then it is possible your function can be set with a higher timeout to perform background work. Since work is done after each invoke, the LambdaPunch queue should be empty when your function receives the `SHUTDOWN` event. If jobs are in the queue when this happens they will have two seconds max to work them down before being lost.

**For a more robust background job solution, please consider using AWS SQS with the [Lambdakiq](https://github.com/customink/lambdakiq) gem.**

### Logging

The default log level is `error`, so you will not see any LambdaPunch lines in your logs. However, if you want some low level debugging information on how LambdaPunch is working, you can use this environment variable to change the log level.

```yaml
Environment:
  Variables:
    LAMBDA_PUNCH_LOG_LEVEL: debug
```

### Errors

As jobs are worked off the queue, all job errors are simply logged. If you want to customize this, you can set your own error handler.

```ruby
LambdaPunch.error_handler = lambda { |e| ... }
```

## 📊 CloudWatch Metrics

When using Extensions, your function's CloudWatch `Duration` metrics will be the sum of your response time combined with your extension's execution time. For example, if your request takes `200ms` to respond but your background task takes `1000ms` your duration will be a combined `1200ms`. For more details see the _"Performance impact and extension overhead"_ section of the [Lambda Extensions API
](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html)

Thankfully, when using Lambda Extensions, CloudWatch will create a `PostRuntimeExtensionsDuration` metric that you can use to isolate your true response times `Duration` [using some metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html). Here is an example where the metric math above is used in the first "Duration (response)" widget.


![CloudWatch Metric Math](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/91rxa6omap8o0dilhfu7.png)

![CloudWatch Durations](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lfot23dxuld5lp4u910g.png)