Ruby on Lambda: Integrating with ActiveStorage
Published: 2020-08-11 20:00:51

_This article is part of our Ruby on AWS Lambda blog series. A recent project had us migrating an existing PDF document processing system from Rails Sidekiq to AWS Lambda. The processing includes OCR, creating preview images, splicing the PDF, and more. Moving to Lambda reduced processing time by 300 times in some cases._
 
_This series of articles will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talk about creating a AWS Lambda Layer with Docker. In part two, we chatted about architecting a serverless app. In part three, we went through some strategies surrounding deployment. Here in part four, we'll investigate what's needed to integrate a Rails app that uses Active Storage with AWS Lambda. Check out the other posts in the series:_

- _[Layer Dependencies](/hint/ruby-on-aws-lambda-layer-dependencies-1ooe)_
- _[Planning and Architecting](/hint/ruby-on-aws-lambda-planning-architecting-3foi)_
- _[Package and Ship It](/hint/ruby-on-aws-lambda-package-ship-it-7ol)_
  
_Note: there are some assumptions made in this article regarding architecture and reading Part Two of this series, Planning & Architecting, may be useful before diving in here._
 
Performance was our main reason for moving from Sidekiq to AWS Lambda. We used Active Storage on a few models and wanted to continue using it after the migration. Specifically, we wanted to take advantage of Active Storage [variants](https://api.rubyonrails.org/classes/ActiveStorage/Variant.html) via a GraphQL API and React frontend. The app had multiple views that requested all of the variants at once, and because the variants are created at the time of the request, the user experience under Sidekiq was very poor. What if we could create all of those variants on Lambda as we processed the document? It sounded plausible to us, so we spiked on it.
 
Note that we'll be touching on specific pieces of code that directly relate to the integration; this won't be a tutorial on how to use Active Storage or setup AWS Lambda. We are using S3 as our storage service, and will be referencing it as a standalone entity. Let's dive in!
 
## Uploading
 
Instead of passing form params to a model, we’ll take more of a manual approach. In the controller:
 
```ruby
@document.file.upload(file)
```
 
We have a `Document` model with `has_one_attached :file`. This allows the file to be uploaded to S3 without creating the backing Active Storage record. Delaying that allows us to process the document on Lambda, create the needed variants, and respond back with data that will be used to create those records.
 
Lambda has no knowledge of the Rails app or Active Storage, but if we can point it to the file we just uploaded, it can reference and process it. After a `Document` has been created, we send `document.file.key` to the vent function (LINK TO PLANNING AND ARCHITECTURE POST) that starts the processing work. The document’s key is the name of the file on S3. On Lambda, the real fun begins.
 
## Variant Creation
 
Essentially, we needed to trick Active Storage. When requesting a variant, Active Storage will look for `variant.jpg` in a specific location on S3:
 
```ruby
"variants/#{key}/#{variant_key}"
```
 
`key` is the original file's key (`document.file.key`), and `variant_key` is a hash derived from the variant's transformation options (e.g. `{auto_orient: true, resize: '1000x1000'}`) and the Rails app's `ENV['SECRET_KEY_BASE']`. Let's look at the Lambda  code:
 
```ruby
def create_variant(image, key, transformation)
 # create message verifier and variant key
 verifier_key = verifier.generate(transformation, purpose: :variation)
 variant_key = Digest::SHA256.hexdigest(verifier_key)
 
 tmp = Tempfile.new('variant.jpg')
 
 MiniMagick::Tool::Convert.new do |convert|
   convert << image.path
   # apply transformations here, like:
   # convert.auto_orient
   # convert.resize transformation[:resize]
   convert << tmp.path
 end
 
 s3 = Aws::S3::Client.new
 s3.put_object(
   bucket:       "some-cool-bucket",
   key:          "variants/#{key}/#{variant_key}",
   body:         tmp,
   content_type: 'image/jpeg'
 )
end
```
 
`#create_variant` takes a `File`, `document.file.key`, and the transformation options as arguments. The `variant_key` is created (we'll look at what `verifier` is soon), and the actual image conversion is handled with ImageMagick. Note the actual ImageMagick conversions depend on the transformation options being passed in. For an example I used the options that were previously mentioned, `{auto_orient: true, resize: '1000x1000'}`. After the image is processed, we then place it on S3, using the specific path Active Storage expects.
 
Figuring out how to create the `verifier_key` was quite difficult. It took a late night fueled by coffee and spelunking into the bowels of Active Storage, but it proved fruitful. Let's take a look at `#verifier`:
 
```ruby
def verifier
 key_generator = ActiveSupport::CachingKeyGenerator.new(
   ActiveSupport::KeyGenerator.new(ENV['SECRET_KEY_BASE'], iterations: 1000)
 )
 key = key_generator.generate_key('ActiveStorage')
 ActiveSupport::MessageVerifier.new(key)
end
```
 
Active Storage was created by people much smarter than I, and I'll be the first to admit that I did not look at _all_ the implementation details surrounding `ActiveSupport::CachingKeyGenerator`,  than I, and I'll be the first to admit that I did not look at _all_ the implementation details surrounding `ActiveSupport::KeyGenerator`, `ActiveSupport::MessageVerifier`. However, their names are descriptive and give us a good bird's-eye view. A key is generated from your Rails app's `SECRET_KEY_BASE`, and that key is then turned into a digest that is used to create the path to the variant. A few notes:
 
- The `iterations` option when instantiating `ActiveSupport::KeyGenerator` does not have to be 1,000. I, of course, didn't test _every_ number, but the natural numbers I tested all had the same result. Active Storage uses 1,000, so I thought I would too.
- The argument passed to `#generate_key` is the salt, and for our intended purposes on Lambda it can be any string.
 
## Data for the `active_storage_attachments` record
 
Early on in the spike, we hit a lot of `ActiveStorage::IntegrityError`s. While you can find yourself raising this error on accident in many scenarios, the reason why is always the same: the hashed contents of the file don't match what's stored in the database. Since we uploaded the file straight to S3, we purposefully didn't create a record in `active_storage_attachments`, and don't have the reference to the file anymore without downloading it (which would be a waste). We had to create the digest on Lambda while we still had a reference to the file.
 
Here's the data we collect for each page in a document (note that `file` is a `File` object):
 
```ruby
{
 content_md5:  Digest::MD5.file(file).base64digest,
 byte_size:    File.size(file).to_s,
 key:          key,
 content_type: content_type
}
```
 
After every page is processed, our sink function collects all the data and sends a JSON response back to the Rails app.
 
## All together now
 
When the Rails app receives the JSON, Sidekiq workers are started that process the response. Part of the workers' responsibility is to create a record in `active_storage_attachments`:
 
```ruby
def create_blob(data)
 params = {
   key:          data['key'],
   filename:     data['key'],
   checksum:     data['content_md5'],
   byte_size:    data['byte_size'],
   content_type: data['content_type']
 }
 
 ActiveStorage::Blob.create_before_direct_upload!(params)
end
```
 
[`#create_before_direct_upload!`](https://api.rubyonrails.org/classes/ActiveStorage/Blob.html#method-c-create_before_direct_upload-21) allows us to create the record and _not_ upload the file to S3 as it already exists there. Tricking Active Storage, and all.
 
And with that, Active Storage works just as it normally would. With our implementation, however, we process _all_ variants in parallel, improving load times and user experience for everyone!