Serverless Backends With AWS Cloud: Email Lambda and DynamoDB
Tags: aws, lambda, mailgun, node
Published: 2017-04-24 19:06:39

---
title: "Serverless Backends With AWS Cloud: Email Lambda and DynamoDB"
published: true
description: Let's make some Lambda and email magic happen! AWS Serverless technologies 
cover_image: 
tags: AWS, Lambda, Mailgun, node.js
---

_This is part of a series of articles that comprise a tutorial of an end-to-end production AWS serverless system. If you're joining partway, please read this series from the intro article, available in its original formatting on my blog [J-bytes](http://robsherling.com/jbytes/index.php/2017/01/08/serverless-backends-with-aws-cloud-intro/)._

This is a very, very long piece.

# Email, Lambda, Mailgun, Dynamo, S3

There is going to be a lot coming at you all at once in this post. This is a large project with a lot of moving parts, and while I actually built this incrementally, we'll just be building pretty much everything final-stage here (with a few exceptions). Take your time and read everything. We'll be touching S3, Lambda, IAM, KMS, and DynamoDB services in this post.

## Email DynamoDB Tables

We're going to just go right into things and create a DynamoDB table to store all our registered info.

Navigate to DynamoDB in the AWS control panel. Click __create table.__ You should see this screen:

![dynamo table create screen](http://robsherling.com/jbytes/wp-content/uploads/2017/01/Dynamo-Create-Screen.png)

The primary key is simply a key that each database entry is guaranteed to have. There are other things to know about dynamo DB for different projects, but that's basically it for this one.

Let's call our table name __production_emails__ and our primary key __email__.

We're going to leave our __Use Default Settings__ checked every time. This will limit our table to 5 reads and 5 writes per second, and keep things nice and cheap. For production, your needs will obviously vary depending on your anticipated workload.

We do not require a sort key, so simply click __create.__

Create another table with table name __staging_emails__ and a primary key of __email__. Click __create__ again.

__A NOTE ON STAGES__: We would normally make separate versions of the tables for testing, staging, production, and dev but we'll only make two in this tutorial for brevity.

Give it a few minutes to make the tables.

## IAM

Let's make the role that our lambda will use next.

Open IAM
* Click __Roles__
* Click __Create new role__
* Name this role what you want (master_lambda is good)
* Role-Type is AWS Lambda
* Attach policy __AmazonDynamoDBFullAccess__
* Attach policy __AmazonS3ReadOnlyAccess__
* Attach policy __CloudWatchFullAccess__
  - Note: Even with this policy, it takes a long time sometimes to actually be able to see CloudWatch logs. See the end of this page.
* Save

##Lambda

Open up Lambda in your AWS control panel, and hit __get started now__.

I hit some major snags with Lambda here when I first did this project. A word of advice: do not choose the default Dynamo templates. They are a trap. As of the time I did this project, they used an old way of connecting to dynamo that isn't supported anymore, and I lost nearly a full day trying to troubleshoot what I thought were problems with my code. The old templates try to force to you specify the type of data you are saving and it all falls apart for some reason.

Select __Blank Function.__
You will now see __Configure triggers__ on this page to link your API. Do not do that. It is also a trap. It will create a new blank API resource for you, and we don't need that at the moment. Leave it blank and just hit next.

We're going to go whole hog, here. In reality, I made the basic email function first, then encryption, then I finished out with the ability to send mails. We, however, are going to benefit from my knowledge and just do it right the first time so we can write this one-and-done.

The following steps are slightly different depending on which mail provider you chose to send email with. I originally did the project with Sendgrid, but Mailgun is much easier to demo and really they are almost identical from an implementation standpoint. A little google will do you well here.

We'll need to upload a few files: index.js for sending the email, email.html for contents we want to send, and loader.js to load all our environment variables. We also need to upload the library that we'll use for Mailgun to Lambda. This was actually surprisingly easy, as  Mailgun has documentation that is great for the job we need ([docs](https://github.com/orliesaurus/nodemailer-mailgun-transport)).

In order to upload the files, minimize your browser window and do the following on your local system.

## The Lambda Index File

For the index file itself, create a folder called email_lambda in version control. Inside that, create a file named index.js and paste the following code:

[index.js](https://github.com/Rob117/serverless-twitter-email/blob/master/email_lambda/index.js)

On line 8, change __us-west-1__ to your actual dynamo location.

Read the comments if what is happening in the file is unclear.

The reason all code paths eventually throw an error is because that is the only way to redirect to a website from Lambda using the API gateway. This will be explained in more detail when we make the API.

## Loading our configs

This requires S3, which we will setup after. I will also explain after why we are using an S3 loader for environment variables and not the newly added environment variables function at the bottom of each lambda in the AWS console. For now, trust.

In reality, you would actually create this file outside of your individual lambda folders and put a hard system link into each lambda folder so that they all have the same version of this file. If you don't know how to do that, google or just put it directly inside the folder each time. Anyway, create a file next to index.js called loader.js and put the following code inside:

[loader.js](https://github.com/Rob117/serverless-twitter-email/blob/master/loader.js)

Edit the file as follows:
Replace __CONFIG BUCKET NAME__ on line 61 with your actual S3 bucket name that we'll make later. I strongly suggest something like `<project-name>-dev-config`.Please check [this](https://www.concurrencylabs.com/blog/configure-your-lambda-function-like-a-champ-sail-smoothly/) for the original guide. I made some modifications to make it more usable for our project.

## The Email Template 

Simply put a file called email.html in the folder as well, with whatever you want. Suggested contents (RECIPIENT will be replaced with the correct email by our code in the index.js file):

[email.html](https://github.com/Rob117/serverless-twitter-email/blob/master/email_lambda/email.html)

## Installing the Mailgun Library

Note: The library that Mailgun uses is huge and because it will take you over a designated size limit you will not be able to use the inline code editor in AWS lambda after you perform the next few steps. This can be a bit frustrating if you only need to make a small edit because you have to re-zip the file each time, but you get used to it quickly (and also can make an alias to do it for you to save you time).

Time to install the [relevant mailer node package]( https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/).


Run the following commands from inside the email_lambda folder. Copy-pasting always messes node installs up for me, so be careful that when you copy that it actually copied correctly.

```
npm install –prefix=./ nodemailer
npm install –prefix=./ nodemailer-mailgun-transport
```

`tree` should spit out:
```
.
|-- email.html
|-- etc
|-- index.js
|-- loader.js
`-- node_modules (with a ton of files in here, omitted for brevity)
```

Go to the Mailgun account that you made during installs and get an API key for your project (and verify a domain if you can, highly recommended but cumbersome). Set these aside for the environment variables portion.

Zip the files, __not the folder__. On Mac, if you just zip the folder itself it will create a folder within the zipped folder so your files will be __two levels down__ and not work. The zip should open to node_modules/, index.js, loader.js, email.html and etc/ directly. If you use the unix tool to zip, make sure to specify the -r option.

On the Lambda console screen that we had open, change __Name__ to __email_service__ (or whatever you'd like) and __Code entry type__ to __Upload a zip file.__ Then click upload and select the zip we just made. Leave environment variables empty, we'll talk about those in a minute.

Handler tells lambda what file name and what function to execute first when called. The format is filename.function. Leave it as is.

For Role, choose existing role, and choose the lambda role we made earlier (master_lambda if you followed the guide).

Leave memory at 128 mb/s, and set the timeout to something healthy like 15s. Remember, we get charged on the time we actually use, not the maximum that we set. DynamoDB also very occasionally has some weird delays when reading or writing, so you want this to be pretty long just in case.

Leave the rest as default, and click __Next__, then __Create Function__.

## Environment Variables

If you don't know what AES is or what an IV is, read roughly a page on it. Basically, AES is a series of encryption standards and an IV is a unique-per-data-item piece of information that makes the security on each item harder to crack.

On the AWS screen for your lambda under the Code tab (should be the screen you see after uploading the zip), you are going to see a space for Environment Variables. We are not going to use these.

Normally, Lambda will read these in with process.env.KEY_NAME. We can't really use these because while they are fantastic for a single lambda, they don't really work well for shared information like AES keys or email addresses across multiple lambdas, or for variables that differ by stage (production keys need to be different from all other stages). If you forget to change a single key on a single version of a lambda, it can break in really terrible and subtle ways.

So what we are instead going to do is load all of our environment variables from a JSON file that we will make for each stage and encrypt with KMS so that only our admins and our lambda can read it. The encryption happens when we store it in an S3 dev bucket.

## KMS

First, let's make a key. Head on over to KMS in the console (IAM -> Encryption Keys).

* Click __get started/create key.__
* For the alias, you can use anything (__S3-encrypt-key__) and click next step.
* Under Key Administrators, choose who you want to be able to rotate/edit/delete the key
* Under Key Usage Permissions, pick the __master_lambda__ role that we made earlier, as well as any console users/role that you want to be able to access the file.
* Click __Next Step__, then __Finish__


## S3

We need to make a JSON config object, then we need to upload it to our staging and production folders. We will encrypt it with the KMS key we just made, because encrypting your sensitive data at rest is just good practice.

First, go to S3 and make a bucket (this should have the same name that you set in your loader.js file, ideally `<project-name>-dev-config`).

![s3 bucket creation](http://robsherling.com/jbytes/wp-content/uploads/2017/01/Screen-Shot-2017-01-09-at-00.50.16-768x663.png)


Inside that bucket, make two folders, __staging__ and __production__.

![S3 bucket folders](http://robsherling.com/jbytes/wp-content/uploads/2017/01/Screen-Shot-2017-01-09-at-00.52.27.png)

At this point, we're ready to upload our environment configs. Some of these in the example file I'll link we actually don't need yet, but it's no problem to leave in dummy data and update when you need it. Please  keep backups of this data outside of S3 in case an angry admin deletes it and you lose your AES keys.

Remember: never commit environment variable config files to the repository. That would completely defeat the point of environment variables. This is only in the repo as an example.

Download the following file and configure it according to the instructions below.

[env-config.json](https://github.com/Rob117/serverless-twitter-email/blob/master/loader.js)

Explanations:

* site_callback
  
  This is where you will put the page that you want your user to be redirected to once they've registered their email or twitter. For example: http://robsherling.com/jbytes.

* email/twitter/session_table_name
  
  The table name that you want to store data in / read data from. Examples are __staging_emails__ and __production_twitter__.

* aes_password
  
  This is the password that you will use for your encryption key.
AES is very, very picky about keys. They have to be of a specific byte length. At least for testing, you can get keys from http://randomkeygen.com

Just go to CodeIgniter Encryption Keys and grab one you like and save it somewhere in addition to your config file, because if you lose it you lose access to all the information that it was guarding.

* from_email

  The email we want this to look like it was sent from.

* mail_api_key/mailgun_domain_name

  Use the API key and domain name you have in your Mailgun account settings.

Now that you know what the variables do, fill in the site_callback, email_table_name, aes_password, mail_api_key, mailgun_domain_name, and from_email  fields.

You will have to upload this file twice; once to the production folder and once to the staging folder. After you select the env-config file for upload, click the __Set Details__ button in the lower-right hand corner. Check __Use Server Side Encryption__, then __Use an AWS Key Management Service Master Key__, then our created key from the dropdown. Upload (don't worry about permissions, they work fine as is). If you do not see your key, check that the key region and bucket region are the same.

Please change your AES keys and DynamoDB tables for staging and production (staging_emails vs production_emails); I also would change the env-config.json in the staging folder to use a different __from_email__ address so you know staging is being called correctly.

## Lambda Alias

Finally, let's get our lambda aliases to correspond to our stages, edit our test action and get this bad-boy fired up!

First, let's set the test action in the lambda console. Click our __email_service__ lambda that we made, click the __Actions__ drop-down at the top of the page near the blue __Test__ button, and then click __Configure test event.__ Delete any code in the event and copy and paste the following, changing name and domain to the appropriate values for your receiving test email:

`{ "body-json": "email=<user>%40<domain.com>"}`

Please note that every time we manually test this, the above email is going to get an email, so be comfortable with that.

* Click __Save__, not __Save and test__

* Click __Actions__, then __publish new version.__ For the description, put __tested and ready!__ or whatever else you want there. Click __Publish__.

* Click __Actions__ again, then click __Create alias__. For the alias name, type __production__. For the version, select __1__ (the version we just made) and hit create. Now, repeat this again, but for the name type __staging__ and for the version select __$LATEST.__ This is how you can point aliases to different versions of your code.

Now, on the left hand side, click __qualifiers__, then __staging__ under alias.  Then hit the blue __Test__ button.

1) It __SHOULD__ say execution failed, with error message Email.MovedPermanently: Redirecting. This is because later we will catch that in AWS API Gateway and use it to redirect.

2) If you check your dynamo db __staging_emails__ table that we created and click on the items tab, there should be an item with email and email_iv, and it should be a bunch of gibberish. This is good.

3) Your email can take a few seconds to arrive. If it hasn't arrived, check your CloudWatch logs for errors and your Sendgrid/Mailgun logs.

4) If you have errors, after you try to fix them and re-upload any files, make sure to select __staging__ again from the alias list before running test. We haven't made a __$LATEST__ dev-config folder and json, so if you don't correctly set the right alias it won't load any environment configs.

## CloudWatch Logs

If your CloudWatch logs aren't showing up when you try to find them, _good luck_. The Amazon Support team's general take on this seems to be: __If the roles don't work, mess with them until they do.__ Seriously, I tried to put the CloudWatchFullAccess policy on my master_lambda role and it did nothing. I fiddled with roles for hours and it did nothing. So, I just put the CloudWatchFullAccess back on and walked away. I came back ~ five days later, and without changing anything it just magically started working.

This marks the completion of our first lambda. Next, we'll hook it up to an API so we can test it with Postman.