Receiving and Parsing Email in Rails 5
Tags: ruby, rails
Published: 2017-10-12 21:24:14

---
title: "Receiving and Parsing Email in Rails 5"
published: true
description: Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.
tags: ruby, rails, ruby on rails
---

![](https://cdn-images-1.medium.com/max/1600/1*Qmu6OFx86thnPRrL6pPHbw.png)

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.

*****

This is an excerpt from my book [Build a SaaS App in Ruby on Rails 5](http://buildasaasappinrails.com/). I have some sample chapters and discounts available for the pre-sale. The book is poised to be finished soon! With that being said, you may see some code blocks in this post that reference a model or object that is pertinent to the book, but does not detract from how you may interface with incoming mail in Rails.

*****

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Mailgun’s email routing service and ngrok, an HTTP tunneling service. `ngrok` is very useful for a few solution throughout the remainder of this book. It allows you to have a web accessible URL, which tunnels(connects) to ports within your local machine.

Meaning, that you will have a `http://somesubdomain.ngrok.com` that will forward to your computer, and a specific port specified when you start `ngrok`. This allows you to test with external services such as Mailgun, Stripe(later), Github(later) and more!

Let’s get started with some setup:

* Download [ngrok](https://ngrok.com/)
* Unzip and move the executable file to where you would like.
* In *nix OS’s, open ngrok with: `path/to/ngrok HTTP start 3000`. This will be
dependent of the port you are using for your Rails server. ngrok will now fire
up a tunnel service with a randomly generated URL.
* Optionally, if you upgrade to a paid version of `ngrok`, you can set a
subdomain, so you do not have to change your settings elsewhere every time you
restart `ngrok`.
* In Mailgun go to the Routes tab to create the email route(modeled after MVC
routes like Rails) that will send the email. Enter the following settings:
* Chose `Match Recipient`
* Enter `development.standup.*@app.yourdomain.com` for the recipient field.
Mailgun routes allow wildcard character matching, which will allow you input
extra characters in the email's `reply-to`. Meaning, adding something like a
user's `hash_id` to have identifiable information from the incoming email.
* Check `forward` and enter `http://yoursubdomain.ngrok.io/email_processor` as the
destination of that forward.
* You can leave the priority alone and give the route a name, before pressing the
`submit` button.

Now with that done, we can begin to modify the application. The application changes will consist of three main parts. First, a new gem(and a companion adapter gem) will be added. `griddler` is the main library to handle incoming email easily; `griddler-mailgun` is the Mailgun specific adapter that allows the `griddler` functionality to work with Mailgun as an incoming mail router. Next, Griddler will need to add a few parts of configuration application-wide to make sure some basic configuration is met. Lastly, an email processor file will be added to handle receiving and parsing the email.

The great part about adding Griddler to your current application is that if you are using the Gemfile from Chapter 3, you already have it installed. If not, just add `gem 'griddler'` and `gem 'griddler-mailgun'` to your Gemfile and run a `bundle install`.

Next to configure and setup Griddler in the application you will need to add a new file in the initializer folder setting a few configuration values. Then, add a quick line to add the default Griddler routing into the `routes.rb` file.

First the Griddler configuration:

```ruby
    Griddler.configure do |config|
      config.reply_delimiter = '-- REPLY ABOVE THIS LINE --'
      config.email_service = :mailgun
    end
```

This will set the text the griddler library will look for in the email and tell it that it will use the installed Mailgun adapter.

Next, add a line to mount the library based routes into your application. Adding the line right above the `root to:` is fine:

```ruby
    Rails.application.routes.draw do
      ...

      # mount using default path: /email_processor
      mount_griddler

      root to: 'activity#mine'
    end
```

By mounting Griddler with that syntax, it will automatically add a route to your application that will route from a specified endpoint to a Griddler based controller:

```ruby
    email_processor POST /email_processor(.:format)    griddler/emails#create
```

There are settings in Griddler’s GitHub documentation to change the defaults, but unless you want to get creative with route paths or email processor class names, it can be unnecessary.

The default settings expect a class `EmailProcessor` to exist and to handle parsing the incoming email with a method `process`. Griddler, however, does not care where the actual file is placed, but that the class exists and is loaded. Personally, I find that email processing fits most into the services definition and can be placed there.

To allow Griddler and its files to capture text from incoming replies, there are a few changes that will be needed.

First, we will update the `EmailReminderMailer` to create a unique reply-to address and include that email address as part of the outgoing email:

```ruby
class EmailReminderMailer < ApplicationMailer
  def reminder_email(user, team)
    @user = user
    @team = team
    reply_to = "'Standup App' <#{'development.' if Rails.env.development?}\
    standup.#{@user.hash_id}@app.yourdomain.com>"
    mail(
      to: @user.email,
      subject: "#{team.name} Standup Reminder!",
      reply_to: reply_to
    )
  end
end
```

Here we are using building the `reply_to` string by adding `development` if the current Rails environment is your local Rails application. This way, the separate Mailgun route made for development can route different from a production route you will add before deploying your application.

Next, we will update the mailer template to have `##- Please type your reply above this line -##` and some text letting the email recipient know they can add a standup by replying:

```slim
doctype html
html xmlns="http://www.w3.org/1999/xhtml"
  head
    meta content="width=device-width" name="viewport" /
    meta content=("text/html; charset=UTF-8") http-equiv="Content-Type" /
    title= "#{@team.name} Reminder!"
    css:
      | *{margin:0;padding:0;font-family:"Open Sans",Helvetica,Helvetica,Arial,
      sans-serif;box-sizing:border-box;font-size:14px}img{max-width:100%}body{-
      webkit-font-smoothing:antialiased;-webkit-text-size-adjust:none;width:100
      %!important;height:100%;line-height:1.6}table td{vertical-align:top}body{
      background-color:#f6f6f6}.body-wrap{background-color:#f6f6f6;width:100%}.
      container{display:block!important;max-width:800px!important;margin:0
      auto!important;clear:both!important}.content{max-width:800px;margin:0
      auto;display:block;padding:20px}.main{background:#fff;border:1px solid #e
      9e9e9;border-radius:3px}.content-wrap{padding:20px}.content-block{padding
      :0 0 20px}.header{width:100%;margin-bottom:20px}.footer{width:100%;clear:
      both;color:#999;padding:20px}.footer a{color:#999}.footer a,.footer
      p,.footer td,.footer unsubscribe{font-size:12px}h1,h2,h3,a,th,td{font-
      family:"Open Sans",Helvetica,Arial,"Lucida Grande",sans-serif;color:#
      000;margin:40px 0 0;line-height:1.2;font-weight:400}h1{font-size:32px;
      font-weight:500}h2{font-size:24px}h3{font-size:18px}h4{font-size:14px;
      font-weight:600}ol,p,ul{margin-bottom:10px;font-weight:400}ol li,p li,ul
      li{margin-left:5px;list-style-position:inside}a{color:##3c8dbc;text-decor
      ation:underline}.btn-primary{text-decoration:none;color:#
      FFF;background-color:##3c8dbc;border:solid ##3c8dbc;border-width:5px 10px
      ;line-height:2;font-weight:700;text-align:center;cursor:pointer;display:
      inline-block;border-radius:5px;text-transform:capitalize}.last{margin-
      bottom:0}.first{margin-top:0}.aligncenter{text-align:center}.alignright{
      text-align:right}.alignleft{text-align:left}.clear{clear:both}.alert{font
      -size:16px;color:#
      fff;font-weight:500;padding:20px;text-align:center;border-radius:3px 3px
      0 0}.alert a{color:#fff;text-decoration:none;font-weight:500;font-size:16
      px}.alert.alert-warning{background:#f8ac59}.alert.alert-bad{background:#e
      d5565}.alert.alert-good{background:##3c8dbc}.invoice{margin:40px
      auto;text-align:left;width:80%}.invoice td{padding:5px 0}.invoice .
      invoice-items{width:100%}.invoice .invoice-items td{border-top:#eee 1px
      solid}.invoice .invoice-items .total td{border-top:2px solid #
      333;border-bottom:2px solid #333;font-weight:700}@media only screen and (
      max-width:640px){h1,h2,h3,h4{font-weight:600!important;margin:20px 0 5px!
      important}h1{font-size:22px!important}h2{font-size:18px!important}h3{font
      -size:16px!important}.container{width:100% !important}
      .content,.content-wrap{padding:10px !important}.invoice{width:10
      0% !important}
  body
    div style="color: #b5b5b5;text-align:center;"
      | ##- Please type your reply above this line -##
    table.body-wrap style="width:100%"
      tr
        td
        td.container width="800"
          .content
            table.main cellpadding="0" cellspacing="0" width="100%"
              tr
                td.content-wrap
                  table cellpadding="0" cellspacing="0" style="width:100%"
                    tr
                      td.aligncenter
                        | Standup App
                    tr
                      td.content-block
                        h3= "#{@team.name} Reminder!"
                    tr
                      td.content-block
                        = "Just wanted to remind you to add your standup for \
                        the team: #{@team.name}"
                    tr
                      td.content-block.aligncenter
                        = link_to "Add Your Standup", new_standup_url(), \
                        {class:"btn-primary", style: "width:95%"}
                    tr
                      td.content-block
                        = "You can quickly submit your standup by replying to \
                          this email in the format:"
                        pre
                          pre
                          = "[d] This is a done item\n[t] This is a todo item\n\
                          [b] This is a blocker"
            .footer
              table width="100%"
                tr
                  td.aligncenter.content-block
        td
```

Lastly, we will need to add an extra column to the `Standups` table to track the `Message-ID` coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

```
    rails g migration AddMessageIdToStandups message_id
```

Next, before the end of the newly created migrations `change` method, you will want to add `add_index :standups, :message_id`. This index will allow quick lookups as the `Standups` table grows. Finally, migrate the actual change:

```
    bin/rails db:migrate
```

With those changes out of the way we can now add the new `EmailProcessor` class that will parse the incoming email:

```ruby
class EmailProcessor

    attr_reader :email

    def initialize(email)
     @email = email
    end

    TASK_TYPE_HASH = {
        '[d]' => 'Did',
        '[t]' => 'Todo',
        '[b]' => 'Blocker'
      }

    def process
        if Rails.env.development?
          Rails.logger.info '-----------EMAIL-------------'
          Rails.logger.info email.to.first[:token]
          Rails.logger.info email.body
          Rails.logger.info email.headers["Message-ID"]
          Rails.logger.info '-----------EMAIL-------------'
        end

    # Get a user hash_id from reploy-to or bail
        reply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.
        split('.')&.last
        return if reply_user.blank?

    # Find a user by the hash_id or bail
        user = User.find_by(hash_id: reply_user)
        return if user.nil?

    # Bail if standup with incoming message-id exists
        return if Standup.exists?(message_id: email.headers["Message-ID"])

    # Bail if a standup for today exists
        today = Date.today.iso8601
        return if Standup.exists?(standup_date: today)

    # Get content or bail
        tasks_from_body = email.body.scan(/(\[[dtb]{1}\].*)$/)
        return if tasks_from_body.blank? || tasks_from_body.empty?

    build_and_create_standup(
          user: user,
          tasks: tasks_from_body,
          date: today,
          message_id: email.headers["Message-ID"]
        )
      end

    private

    def build_and_create_standup(user:, tasks:, date:, message_id:)
        standup = Standup.new(
          user_id: user.id,
          standup_date: date,
          message_id: message_id
        )

    tasks.each do |task|
          task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
          standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
        end

    standup.save
      end
    end
```

The class is relatively simple, but let’s go over it section by section:

```ruby
class EmailProcessor

    attr_reader :email

    def initialize(email)   
     @email = email
    end

    TASK_TYPE_HASH = {
        '[d]' => 'Did',
        '[t]' => 'Todo',
        '[b]' => 'Blocker'
      }

    ...
```

Here we are initializing the object when the class is called by Griddler, setting the email to a local email variable. Additionally, we are creating a hash to later use in the text content to `Task` type conversion.

```ruby
    ...

    def process
        if Rails.env.development?
          Rails.logger.info '-----------EMAIL-------------'
          Rails.logger.info email.to.first[:token]
          Rails.logger.info email.body
          Rails.logger.info email.headers["Message-ID"]
          Rails.logger.info '-----------EMAIL-------------'
        end

    ...
```

This just adds some additional logging if the mail is processed in the local
development environment.

```ruby
    ...

    # Get a user hash_id from reply-to or bail
        reply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.
        split('.')&.last
        return if reply_user.blank?

    # Find a user by the hash_id or bail
        user = User.find_by(hash_id: reply_user)
        return if user.nil?

    # Bail if standup with incoming message-id exists
        return if Standup.exists?(message_id: email.headers["Message-ID"])

    # Bail if a standup for today exists
        today = Date.today.iso8601
        return if Standup.exists?(user_id: user.id, standup_date: today)

    # Get content or bail
        safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(email.body)
        tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
        return if tasks_from_body.blank? || tasks_from_body.empty?

    ...
```

Here we are grabbing some information used in the parsing, as well as giving the `process` method chances to exit early if the incoming email is not sufficient for processing and `Standup` creation. In the first section, the incoming email address is parsed to find the user's `hash_id`. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers not guaranteeing “just once delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current user and current time.

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation [here](https://regex101.com/r/Bg9EIy/2)) searches for lines that begin with `[d]`, `[t]` or `[r]`. If those are present, it captures the content to the end of the line. The `.scan` method on the content's body, allows it to catch all occurrences of the above pattern. If the scan's output is empty, the `process` method exits.

```ruby
    build_and_create_standup(
          user: user,
          tasks: tasks_from_body,
          date: today,
          message_id: email.headers["Message-ID"]
        )
      end

    private

    def build_and_create_standup(user:, tasks:, date:, message_id:)
        standup = Standup.new(
          user_id: user.id,
          standup_date: date,
          message_id: message_id
        )

    tasks.each do |task|
          task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
          standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
        end

    standup.save
      end
    end
```

The last section here is a culmination of all of the stored information so far to be saved into a new `Standup`. The `user`, `tasks_from_body`, `today` and `message_id` are all passed into a method that will hand the actual save. The `build_and_create_standup` method creates a new `Standup`, with the user's ID, date and `message_id`. Once the object is created, the tasks strings are iterated over to build the `Task` with a type and assigned as child objects with the `<<` syntax. 
Finally the new `Standup` object with children `Tasks` will be saved with `standup.save`

Lastly, you can test this all works if you reply back to an email(that was sent through Mailgun SMTP and not letter_opener) with the following text:

```
    [d] Did a thing
    [d] And Another
    [t] Something to do
    [b] Something in the way. Some really long line about something or another
```

Testing this will require a new spec file with quite a few `it` blocks to test all the branches the `EmailProcessor` may encounter.

First, it would be best if we create a factory to be able to quickly generate an email to be used within the `EmailProcessor`'s spec. This way the email can have defaults and then we can use the `FactoryGirl` `.build` commands to create a new email object with any different attributes when needed to test the processor.

The factory itself is pretty simple:

```ruby
FactoryGirl.define do
  factory :email, class: OpenStruct do
    # Assumes Griddler.configure.to is :hash (default)
    to [
      {
        full: 'to_user@email.com',
        email: 'to_user@email.com',
        token: 'to_user',
        host: 'email.com',
        name: nil
      }
    ]
    from(
      token: 'from_user',
      host: 'email.com',
      email: 'from_email@email.com',
      full: 'From User <from_user@email.com>',
      name: 'From User'
    )
    subject 'email subject'
    body '[d] Did a thing\n[t] Doing a thing\n[b] Blocked by a thing'
    headers {'Message-ID <98984d@local.mail>'}
  end
end
```

Now, with a factory available, the `email_processor_spec`, will be able to easily spin up new email objects as needed with specific changes to test all of the processors' conditional branches.

```ruby
require 'rails_helper'

describe EmailProcessor do
  subject(:email_processor) { EmailProcessor }
  let(:user) { FactoryGirl.create(:user) }
  let(:email) do
    FactoryGirl.build(:email,
      to: [
        {
          email: "standup.#{user.hash_id}@app.buildasaasappinrails.com",
          token: "standup.#{user.hash_id}@app.buildasaasappinrails.com"
        }
      ]
    )
  end

  describe 'processes incoming email' do

    it 'works as intended' do
      expect { email_processor.new(email).process }
        .to change(Standup, :count).by(1)
    end

    it 'fails on bad to' do
      bad_to = FactoryGirl.build(
        :email,
        to: [{ token: nil, email: 'standup@app.buildasaasappinrails.com' }]
      )
      expect { email_processor.new(bad_to).process }
        .to change(Standup, :count).by(0)
    end

    it 'fails on no user' do
      bad_to = FactoryGirl.build(
        :email,
        to: [
          {
            token: 'standup.o8yhiukj@app.buildasaasappinrails.com',
            email: 'standup.o8yhiukj@app.buildasaasappinrails.com'
          }
        ]
      )
      expect { email_processor.new(bad_to).process }
        .to change(Standup, :count).by(0)
    end

    it 'only saves one per message-id' do
      expect do
        email_processor.new(email).process
        email_processor.new(email).process
      end.to change(Standup, :count).by(1)
    end

    it 'only saves one per date' do
      email2 = FactoryGirl.build(:email, headers: { 'message-id': '123' })
      expect do
        email_processor.new(email).process
        email_processor.new(email2).process
      end.to change(Standup, :count).by(1)
    end

    it 'fails on empty or bad body' do
      email = FactoryGirl.build(:email, body: '90ioqwhdk.qhdu')
      email2 = FactoryGirl.build(:email, body: '')
      expect do
        email_processor.new(email).process
        email_processor.new(email2).process
      end.to change(Standup, :count).by(0)
    end
  end
end
```

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then tests each failing path that doesn’t create a standup, in order as those paths appear in the `EmailProcessor`'s `.process` method.

A quick run of the whole rspec suite should show no failing tests and nearly perfect test/code coverage:

```
rspec spec ........................................................................................................................................................

    Finished in 25.78 seconds (files took 10.16 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to standup_app/coverage. 428 / 431 LOC (99.3%) covered.
```