Using Action Mailbox in Rails 6 to Receive Mail
Tags: rails, ruby
Published: 2020-07-13 16:07:42

Sending mail from a Rails application has been covered by hundreds or thousands of articles, however, there is not a ton of articles about receiving, parsing and using the new `ActionMailbox`.

The following is an excerpt from my book [Build A SaaS App in Rails 6](https://buildasaasappinrails.com) and is used in a new product I am building [Mail Buffer](https://mailbuffer.io). Thus, I am standing by the tutorial here with real world usage!

Lastly, as this is an excerpt, this takes into account the chapter leading up to this section where the reader will have created an application, models for a `Standup`, `Todo`/`Did`/`Blockers` (which are `Tasks`) and a few mailers for outgoing mail.

Here goes nothing...

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user's responses to email or action items quicker.

At a high level, there are a few different layers to this. The topmost layer is the email service, and this book's case, Sendgrid. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it.

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Sendgrid's email routing service and ngrok, an HTTP tunneling service. `ngrok` is very useful for a few solutions throughout the remainder of this book. It allows you to have a web-accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a `http://somesubdomain.ngrok.com` that will forward to your computer and a specific port specified when you start `ngrok`. This allows you to test with external services such as Sendgrid, Stripe(later), Github(later), and more!

Let's get started with some setup:

- Download and use [ngrok](https://ngrok.com)
  - Download [ngrok](https://ngrok.com)
  - Unzip and move the executable file to where you would like.
  - In \*nix OS's, open ngrok with: `path/to/ngrok HTTP start 3000`. This will be dependent on the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.
  - Add `config.hosts << "yoursubdomain.ngrok.io"` to `config/environments/development.rb`
  - Optionally, if you upgrade to a paid version of `ngrok`, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart `ngrok`.
- Follow your domain's DNS provider's instructions for adding an MX record for the subdomain or domain name you have chosen to use for inbound email. The MX record will be `mx.sendgrid.net`, with a priority of `10`.

Now, before we head to the next part, we need to create and store a password for the incoming mail. You can use any method you like to create a password, and if you just can't think of any way to do so, you can run `SecureRandom.alphanumeric` in Rails console.

Now, to store this password you just came up with, we will need to introduce a new small, but powerful, feature, `Rails Credentials`. Since Rails 5.2, Rails provides a built-in way for storing secure credentials, that are encrypted.

To start using Rails Credentials, you will want to fire off the edit command:

```console
EDITOR="atom --wait" rails credentials:edit
```

The editor I am using in this example is Atom. However, you can use any editor you like, and it's command line caller. Also, `--wait` is used to make sure that the editor will wait for Rails to finish decrypting the file before opening it in the editor.

Now that we have the encrypted file open, add the following lines:

```yaml
action_mailbox:
  ingress_password: *password you just made*
```

Save the file and close the tab/editor to make sure Rails saves the file and encrypts it. You will notice when Rails first opened the file, there was some output text in your terminal. Which includes the master key, DO NOT LOSE THIS KEY. It will need to be used a the only text in a `config/master.key` to be able to encrypt and decrypt the credentials.

Now, with that out of the way, there is a little bit more setup we need to do for Action Mailbox.

First, two commands to get Rails and some tables set up for inbound email:

```console
bin/rails action_mailbox:install
bin/rails db:migrate
```

Lastly, add the following line to the `config/application.rb` file to let Rails know we're using Sendgrid:

`config.action_mailbox.ingress = :sendgrid`

Ok, back to Sendgrid's setup:

- In Sendgrid, go to the [Inbound Parse](https://app.sendgrid.com/settings/parse) link, under settings, to create the email route that will send the email. Enter the following settings:
  - Enter `app.yourdomain.com` for the domain field. You can optionally choose an additional subdomain to receive emails within the Inbound Parse.
  - Enter `http://actionmailbox:THEPASSWORDFROMBEFORE@yoursubdomain.ngrok.io/rails/action_mailbox/sendgrid/inbound_emails` as the destination URL.
  - Check `POST the raw, full MIME message`, before pressing the `submit` button.

Ok, we're getting nearly done with the setup here. Next, we will need to set up the applications inbound email route and run a command to generate that mailbox class. First, in the `application_mailbox.rb` file that was created just a few commands ago, you will add `routing /standups\.[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}@/i => :standups`. This says when an incoming mail matches that email address syntax, it will be routed to the `standups` mailbox class for processing.

After that, to create the mailbox file, you will just need to run a Rails generator:

```console
bin/rails generate mailbox forwards
```

A few changes will be needed to allow Action Mailbox and its files to capture text from incoming replies.

First, we will update the `EmailReminderMailer` to create a unique reply-to address and include that email address as part of the outgoing email:

```ruby
class EmailReminderMailer < ApplicationMailer
  def reminder_email(user, team)
    @user = user
    @team = team
    reply_to = "'Standups App' <standups.#{@user.id}@app.yourdomain.com>"
    make_bootstrap_mail(
      to: @user.email,
      subject: "#{team.name} Standup Reminder!",
      reply_to: reply_to
    )
  end
end

```

Here we are adding the `reply_to` string to so when someone replies to a reminder email, the reply will be routed through Sendgrid, to Action Mailbox, and then finally by the Standup Mailbox class.

Next, we will update the mailer template to have `##- Please type your reply above this line -##` and some text letting the email recipient know they can add a standup by replying:

```erb
<div class="container">
  <span class="text-secondary text-center" style="font-size:8px">##- Please type your reply above this line -##</span>
  <h1 class="text-center mb-4">
    Standup App
  </h1>
  <div class="card mb-4">
    <div class="card-body">
      <h3 class="text-center mb-4"><%= @team.name%> Reminder! </h3>
      <p class="mb-4">
        Just wanted to remind you to add your standup for
        the team: <%= @team.name %>
      </p>
      <%= link_to "Add Your Standup", new_standup_url(), {class: "btn btn-primary btn-lg mx-auto mt-2", style: "width:95%"} %>
    </div>
  </div>
</div>
```

Lastly, we will need to add an extra column to the `Standups` table to track the `Message-ID` coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

```
rails g migration AddMessageIdToStandups message_id
```

Next, before the end of the newly created migrations `change` method, you will want to add `add_index :standups, :message_id`. This index will allow quick lookups as the `Standups` table grows. Finally, migrate the actual change:

```
rails db:migrate
```

Lastly, to make sure sidekiq picks up the queues that Rails uses for Action Mailbox, we will need to adjust the `worker` in the `Procfile.dev` to:

```
worker: bundle exec sidekiq -q default -q mailers -q action_mailbox_routing -q active_storage_analysis
```

With those changes out of the way we can now add the new `StandupsMailbox` class that will parse the incoming email:

```ruby
class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  def process
    # Get a user id from reply-to or bail
    reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
    return if reply_user.blank?

    # Find a user by the id or bail
    user = User.find_by(id: reply_user)
    return if user.nil?

    # Bail if standup with incoming message-id exists
    return if Standup.exists?(message_id: inbound_email.message_id)

    # Bail if a standup for today exists
    today = Date.today.iso8601
    return if Standup.exists?(standup_date: today)

    # Get content or bail
    safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
    tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
    return if tasks_from_body.blank? || tasks_from_body.empty?

    build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: inbound_email.message_id
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end

  def mail_body
    @mail_body ||= begin
      if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
    end
  end
end
```

The class is relatively simple, but let's go over it section by section:

```ruby
class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  ...
```

Here we are setting up the `StandupsMailbox` to inherit from the `ApplicationMailbox` class, which handles the routing. Additionally, we are creating a hash to later use in the text content to `Task` type conversion.

```ruby
...

def process
  # Get a user id from reply-to or bail
  reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
  return if reply_user.blank?

  # Find a user by the id or bail
  user = User.find_by(id: reply_user)
  return if user.nil?

  # Bail if standup with incoming message-id exists
  return if Standup.exists?(message_id: inbound_email.message_id)

  # Bail if a standup for today exists
  today = Date.today.iso8601
  return if Standup.exists?(standup_date: today)

  # Get content or bail
  safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
  tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
  return if tasks_from_body.blank? || tasks_from_body.empty?

...
```

Here we are grabbing some information used in the parsing, as well as giving the `process` method chances to exit early if the incoming email is not sufficient for processing and `Standup` creation. In the first section, the incoming email address is parsed to find the user's `id`. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers, not guaranteeing "just once" delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the `current_user` and `current_date`.

To get the mail's content, we will want to do two things, make sure we get the correct body from the mail object as email clients can send both an HTML and plain text version. Then we will want to make sure we only grab the content above the `##- Please type your reply above this line -##` string by using the `split` method (which splits a string into as many parts based on the separator specified in the argument):

```ruby
def mail_body
    @mail_body ||= begin
      body = if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
      body.split('##- Please type your reply above this line -##').first
    end
  end
```

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation [here](https://regex101.com/r/Bg9EIy/2)) searches for lines that begin with `[d]`, `[t]` or `[r]`. If those are present, it captures the content to the end of the line. The `.scan` method on the content's body allows it to catch all occurrences of the above pattern. If the scan's output is empty, the `process` method exits.

```ruby
  build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: email.headers["Message-ID"]
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end
end
```

The last section here is a culmination of all of the stored information so far to be saved into a new `Standup`. The `user`, `tasks_from_body`, `today`, and `message_id` are all passed into a method that will hand the actual save. The `build_and_create_standup` method creates a new `Standup`, with the user's ID, date, and `message_id`. Once the object is created, the task's strings are iterated over to build the `Task` with a type and assigned as child objects with the `<<` syntax. Finally the new `Standup` object with children `Tasks` will be saved with `standup.save`

Lastly, you can test this all works if you reply back to an email(that was sent through Sendgrid SMTP and not letter_opener) with the following text:

```
[d] Did a thing
[d] And Another
[t] Something to do
[b] Something in the way. Some really long line about something or another
```

Testing this will require a new spec file with a few `it` blocks to test all the branches the `StandupMailbox` may encounter.

First, we will need to add one small change to the `rails_helper.rb` file to make sure RSpec has access to the Rails Action Mailbox test helpers. I'll add the new lines and a few lines above each, so you know where to put the new stuff:

```ruby
...

require 'support/system_macros'
include Warden::Test::Helpers
require 'action_mailbox/test_helper' # <-- new line

...

RSpec.configure do |config|
  config.include Devise::Test::ControllerHelpers, type: :controller
  config.include Devise::Test::ControllerHelpers, type: :view
  config.include ActionMailbox::TestHelper, type: :mailbox # <-- new line

...

```

...and now the actual spec file itself:

```ruby
require 'rails_helper'
include ActiveJob::TestHelper

RSpec.describe StandupsMailbox, type: :mailbox do
  let(:user) { FactoryBot.create(:user) }
  let(:to_email) { "standups.#{user.id}@app.buildasaasappinrails.com" }
  let(:body) do
    %Q(
      [d] Did a thing
      [d] And Another
      [t] Something to do
      [b] Something in the way. Some really long line about something or another

      ##- Please type your reply above this line -##
    )
  end

  subject do
    receive_inbound_email_from_mail(
      from: user.email,
      to: to_email,
      subject: 'Re: Reminder',
      body: body
    )
  end

  before do
    ActiveJob::Base.queue_adapter = :test
  end

  it 'saves standup record' do
    expect { subject }.to change(Standup, :count).by(1)
  end

  it 'saves task records' do
    expect { subject }.to change(Task, :count).by(4)
  end

  context 'fails on bad to email' do
    let(:to_email) {"standups@app.buildasaasappinrails.com"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on no user' do
    let(:to_email) {"standups.9a859af8-30ca-4473-b073-47105352d936@app.buildasaasappinrails.com"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on useless body' do
    let(:body) { 'asdj;oisaduaskd' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on empty body' do
    let(:body) { '' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end
end
```

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then checks each failing path that doesn't create a standup, in order as those paths appear in the `StandupsMailbox`'s `.process` method.