A crash course on Serverless with AWS - Building APIs with Lambda and Aurora Serverless
Tags: serverless, aws, lambda, showdev
Published: 2018-11-27 14:24:04

---
title: A crash course on Serverless with AWS - Building APIs with Lambda and Aurora Serverless
published: true
description: Ever since AWS made a serverless option of AWS RDS Aurora generally available, I've been on the edge of my seat, eagerly anticipating all the various possibilities.
cover_image: https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/header.jpeg
tags: serverless, aws, lambda, showdev
series: Serverless Crash Courses
---

Ever since AWS made a serverless option of [AWS RDS Aurora generally available](https://aws.amazon.com/blogs/aws/aurora-serverless-ga/), I've been on the edge of my seat, eagerly anticipating all the various possibilities. It would mean a breakthrough in building serverless architectures. No more managing connection pools with SQL. No more worrying about capping out available connections. This would make it viable to use serverless for database intensive operations.

I wrote an article a while back explaining [how to build a serverless API with MongoDB](https://hackernoon.com/building-a-serverless-rest-api-with-node-js-and-mongodb-2e0ed0638f47). But, I saw a pattern emerge. Higher throughput in a system, even though a database instance could take the load, would cause connections to drop because of a limited set of available connections.

### TL;DR

I've refactored the example with MongoDB to use SQL. It's agnostic regarding which database provider you use, but the example I'll show here is with [AWS Aurora Serverless](https://aws.amazon.com/rds/aurora/serverless/). Check out [the code here](https://github.com/adnanrahic/serverless-docker-image-resize), or if you want to follow along and learn how to set up everything yourself, keep reading.

## What'll we be doing?

This article will show you how to hook up a MySQL database as a service to a serverless API. We'll create a database cluster with AWS RDS and configure Aurora Serverless. What’s awesome with this set up is that you'll pay for the services based on how much you use them. Even the database! It's the ideal serverless scenario.

## What if you're new to this?

Let's break down the basics. We'll use [AWS Lambda](https://aws.amazon.com/lambda/) as the compute service. An AWS Lambda function is basically a container. Once the Lambda is invoked, the container spins up and runs the code. This is when we want to initialize the database connection, the first time the function is invoked, when the container is first initialized. Every subsequent request to the lambda function will use the existing database connection. Sounds quite simple once I break it down. Let's jump in.

## Configuring the project

I’ll assume you already have a basic understanding of [the Serverless Framework](https://serverless.com/framework/). I would also hope you have an AWS account set up. If you don’t, [please check this out](https://hackernoon.com/a-crash-course-on-serverless-with-node-js-632b37d58b44).

### 1. Creating a service

As always we need a fresh service to hold all our code.

```
$ sls create -t aws-nodejs -p web-api && cd web-api
```

Running this command, you'll get a nice boilerplate to begin developing your functions. It'll set up the basic functions and event triggers. You also see we're entering the `web-api` directory right away. We jumped into this directory to set up the required dependencies.

### 2. Installing modules

In total, we only need three dependencies. The MySQL ORM called Sequelize to handle connections and map relations, and `serverless-offline` for local development.

Make sure you’re in the **web-api** directory. First, install `serverless-offline`, then `mysql2` and `sequelize`.

```
$ npm init -y
$ npm i --save-dev serverless-offline
$ npm i --save mysql2 sequelize
```

That’s it, let’s take a break from the terminal and jump over to AWS to create a database cluster.

### 3. Creating a database on AWS RDS

I've always hated setting up resources through a web console, but it's a necessary evil. Bare with me, jump over to your AWS account and choose [RDS](https://aws.amazon.com/rds/). You'll land on a page similar to this one.

![rds-1](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-1.png)

Click the orange Create Database button and follow along.

![rds-select-engine](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-select-engine.png)

First, you'll choose the engine you want to use. Choose Amazon Aurora and click next.

![rds-create-1](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-create-1.png)

Once you've specified the engine, configure Aurora to be **Serverless**, add a name to your cluster and configure the master username and password.

![rds-create-2](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-create-2.png)

Only one more set of advanced settings to configure, and you should be set.

![rds-advanced-config-1](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-advanced-config-1.png)

Here you'll set up the capacity and network. Because Aurora Serverless lives inside a VPC you'll have to configure the access. Here you're choosing the default VPC.

![rds-advanced-config-2](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-advanced-config-2.png)

With the additional configuration, you can set up backups of you so please. Once you're happy with it all, press the orange **Create database** button.

Finally, it'll start provisioning the cluster. You'll get redirected and see some output looking like this.

![rds-cluster-created-1](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-cluster-created-1.png)

But, what's interesting to us is the details section.

![rds-cluster-created-2](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/rds-cluster-created-2.png)

Here you have access to security groups and subnets. You'll need to add them to the [AWS Lambda](https://aws.amazon.com/lambda/) functions in order for them to be able to communicate with the database. More about that in the next section.

*__Note__: If you want to access the AWS Aurora Serverless cluster follow [this tutorial](https://www.jeremydaly.com/aurora-serverless-the-good-the-bad-and-the-scalable/) to learn how to do it with the CLI, or [this official tutorial by AWS](https://aws.amazon.com/getting-started/tutorials/configure-connect-serverless-mysql-database-aurora/) to connect through [Cloud9](https://aws.amazon.com/cloud9/). You will need to connect to the cluster and create a database manually because Sequelize can't create them for you.*

The engineers over at [Statsbot](https://statsbot.co/) have shown me how awesome [AWS RDS](https://aws.amazon.com/rds/) can be with the huge amounts of data they're processing. Rumors are they're in the works of creating a completely [serverless open-source analytics framework](https://github.com/statsbotco/cubejs-client) called [Cube.js](https://statsbot.co/cubejs/). I'm eager to start using it myself. Hopefully, Aurora Serverless can be of help.

## Writing code

Configuring database resources on AWS is always a handful, especially the new serverless database offering. Luckily, the configuration part is over. Now we need to write the configuration for our serverless resources in the **serverless.yml** file and add the actual CRUD methods to the **handler.js**.

### 4. Configuring the YAML

The Serverless Framework helps a great deal with setting up the initial project structure. It can scaffold out pretty much everything you need when you're starting out. Jump back to the directory where you created the serverless project and open it up in your favorite code editor.

Open the **serverless.yml** file first and take a breather. Here you'll see a bunch of comments with sample configurations. I like clean code, so let's delete it all, and paste in this snippet instead.

```yaml
service: web-api

custom:
  secrets: ${file(secrets.json)}

provider:
  name: aws
  runtime: nodejs8.10
  timeout: 30
  stage: ${self:custom.secrets.NODE_ENV}
  environment: 
    NODE_ENV: ${self:custom.secrets.NODE_ENV}
    DB_NAME: ${self:custom.secrets.DB_NAME}
    DB_USER: ${self:custom.secrets.DB_USER}
    DB_PASSWORD: ${self:custom.secrets.DB_PASSWORD}
    DB_HOST: ${self:custom.secrets.DB_HOST}
    DB_PORT: ${self:custom.secrets.DB_PORT}
  vpc:
    securityGroupIds:
      - ${self:custom.secrets.SECURITY_GROUP_ID}
    subnetIds:
      - ${self:custom.secrets.SUBNET1_ID}
      - ${self:custom.secrets.SUBNET2_ID}
      - ${self:custom.secrets.SUBNET3_ID}
      - ${self:custom.secrets.SUBNET4_ID}

functions:
  healthCheck:
    handler: handler.healthCheck
    events:
      - http:
          path: /
          method: get
          cors: true
  create:
    handler: handler.create
    events:
      - http:
          path: notes
          method: post
          cors: true
  getOne:
    handler: handler.getOne
    events:
      - http:
          path: notes/{id}
          method: get
          cors: true
  getAll:
    handler: handler.getAll
    events:
     - http:
         path: notes
         method: get
         cors: true
  update:
    handler: handler.update
    events:
     - http:
         path: notes/{id}
         method: put
         cors: true
  destroy:
    handler: handler.destroy
    events:
     - http:
         path: notes/{id}
         method: delete
         cors: true

plugins:
  - serverless-offline
```

Let's break it down piece by piece. The `provider` section contains all the basic info about the cloud provider. But also data about the runtime, function timeouts, environment variables, and VPC configuration. The VPC setup is needed here because AWS Aurora Serverless can only run behind a VPC, so you need to put your [AWS Lambda functions in the same VPC](https://docs.aws.amazon.com/lambda/latest/dg/vpc.html) in order to access the database.

Moving on, there's a `custom` section at the top where we load environment variables and then pass them to the `environment` section.

Finally, let's talk about the **functions** section. We added a total of 6 functions: **healthCheck**, **create**, **getOne**, **getAll**, **update**, and **destroy**. They all point to identically named exported functions in the **handler.js** file. Their event triggers are hooked up to HTTP events which are handled by [AWS API Gateway](https://aws.amazon.com/api-gateway/). Meaning, an HTTP request to a specified path will trigger the referenced function.

That’s pretty much it, the last thing is to add a plugins section and **serverless-offline**. We installed this module above and we’ll use it for testing the service out before deploying to AWS. 

### 5. Adding secrets

You saw in the **serverless.yml** that we referenced a secrets file. Before moving on we need to create it and make sure it's loading our secrets correctly.

In the root of the project create a `secrets.json` and paste this snippet in.

```json
{
  "DB_NAME": "test",
  "DB_USER": "root",
  "DB_PASSWORD": "root",
  "DB_HOST": "127.0.0.1",
  "DB_PORT": 3306,
  "NODE_ENV": "dev",
  "SECURITY_GROUP_ID": "sg-xx",
  "SUBNET1_ID": "subnet-xx",
  "SUBNET2_ID": "subnet-xx",
  "SUBNET3_ID": "subnet-xx",
  "SUBNET4_ID": "subnet-xx"
}
```

This setup will work perfectly fine for a local development environment. Once you want to deploy it to AWS you'll have to add your own secrets. AWS Aurora Serverless will output everything like in the image above. The important secrets here are the database connection parameters, security group ID and subnet IDs. 

That's everything regarding the YAML. Let's play with the **handler.js** next. Finally, some real code!

### 6. Fleshing out the functions

To start out we'll define 6 functions. They'll describe the initial layout and behavior we want. Once that's done, we'll move and create the database connection and database interaction logic with [Sequelize]().

Switch over to the **handler.js** file. You’ll see the default boilerplate. Guess what? Delete it all and add the code below.

```javascript
module.exports.healthCheck = async () => {
  await connectToDatabase()
  console.log('Connection successful.')
  return {
    statusCode: 200,
    body: JSON.stringify({ message: 'Connection successful.' })
  }
}

module.exports.create = async (event) => {
  try {
    const { Note } = await connectToDatabase()
    const note = await Note.create(JSON.parse(event.body))
    return {
      statusCode: 200,
      body: JSON.stringify(note)
    }
  } catch (err) {
    return {
      statusCode: err.statusCode || 500,
      headers: { 'Content-Type': 'text/plain' },
      body: 'Could not create the note.'
    }
  }
}

module.exports.getOne = async (event) => {
  try {
    const { Note } = await connectToDatabase()
    const note = await Note.findById(event.pathParameters.id)
    if (!note) throw new HTTPError(404, `Note with id: ${event.pathParameters.id} was not found`)
    return {
      statusCode: 200,
      body: JSON.stringify(note)
    }
  } catch (err) {
    return {
      statusCode: err.statusCode || 500,
      headers: { 'Content-Type': 'text/plain' },
      body: err.message || 'Could not fetch the Note.'
    }
  }
}

module.exports.getAll = async () => {
  try {
    const { Note } = await connectToDatabase()
    const notes = await Note.findAll()
    return {
      statusCode: 200,
      body: JSON.stringify(notes)
    }
  } catch (err) {
    return {
      statusCode: err.statusCode || 500,
      headers: { 'Content-Type': 'text/plain' },
      body: 'Could not fetch the notes.'
    }
  }
}

module.exports.update = async (event) => {
  try {
    const input = JSON.parse(event.body)
    const { Note } = await connectToDatabase()
    const note = await Note.findById(event.pathParameters.id)
    if (!note) throw new HTTPError(404, `Note with id: ${event.pathParameters.id} was not found`)
    if (input.title) note.title = input.title
    if (input.description) note.description = input.description
    await note.save()
    return {
      statusCode: 200,
      body: JSON.stringify(note)
    }
  } catch (err) {
    return {
      statusCode: err.statusCode || 500,
      headers: { 'Content-Type': 'text/plain' },
      body: err.message || 'Could not update the Note.'
    }
  }
}

module.exports.destroy = async (event) => {
  try {
    const { Note } = await connectToDatabase()
    const note = await Note.findById(event.pathParameters.id)
    if (!note) throw new HTTPError(404, `Note with id: ${event.pathParameters.id} was not found`)
    await note.destroy()
    return {
      statusCode: 200,
      body: JSON.stringify(note)
    }
  } catch (err) {
    return {
      statusCode: err.statusCode || 500,
      headers: { 'Content-Type': 'text/plain' },
      body: err.message || 'Could destroy fetch the Note.'
    }
  }
}
```

I can understand you think this is a bit of a huge snippet. Don't worry. These are only 6 basic `async` functions. The `connectToDatabase()` function resolves to an object containing all the database models. We'll only implement the `Note` model in this tutorial. The logic behind the function will connect to the SQL database, cache the connection and make sure to never retry connecting if an existing connection is alive.

But wait, we haven’t defined or created any of this. Well, I did it on purpose, I first want you to see that this is not that complicated, nor any different from creating an API with Node.js and Express. 

### 7. Adding the database connection

Adding the database connection is our next step. When an AWS Lambda function is invoked for the first time, which is called a cold start, AWS will spin up a container to run the code. This is when we connect to the database. All subsequent requests will use the existing database connection. Conceptually, it’s rather easy to understand, but a handful when we need to wrap our heads around it in the code. Here goes nothing!

Create a new file in the root directory of the service, right alongside the **handler.js**. Give it a pretty logical name of **db.js**, and add the code below.

```javascript
const Sequelize = require('sequelize')
const NoteModel = require('./models/Note')
const sequelize = new Sequelize(
  process.env.DB_NAME,
  process.env.DB_USER,
  process.env.DB_PASSWORD,
  {
    dialect: 'mysql',
    host: process.env.DB_HOST,
    port: process.env.DB_PORT
  }
)
const Note = NoteModel(sequelize, Sequelize)
const Models = { Note }
const connection = {}

module.exports = async () => {
  if (connection.isConnected) {
    console.log('=> Using existing connection.')
    return Models
  }

  await sequelize.sync()
  await sequelize.authenticate()
  connection.isConnected = true
  console.log('=> Created a new connection.')
  return Models
}
```

At the top, we're requiring `Sequelize` and the `NoteModel`. 

*__Note__: We're yet to create the model, but hold on, we'll get to that right after this part.*

We then initialize `sequelize`, pass in the connection variables and establish a connection. Calling `NoteModel` will initialize the model, which we then pass to a `Models` constant. The arbitrary `connections` object is there just to use as a _cache_ for the connection. Making sure not to sync the database if it's not necessary. The second reason for `await`ing the `.sync()` and `.authenticate()` methods are to make sure the database connection is established during the initial function call, before handling any business logic.

With the **db.js** file created, let’s require it in the **handler.js**. Just add this snippet to the top of the handler.

```javascript
const connectToDatabase = require('./db') // initialize connection

// simple Error constructor for handling HTTP error codes
function HTTPError (statusCode, message) {
  const error = new Error(message)
  error.statusCode = statusCode
  return error
}
```

### 8. Adding a Note model

Jump back to the **handler.js**. You can see we’re calling `const { Note } = await connectToDatabase()` in the functions to retrieve data, but there’s no Note model defined. Well, now is as good a time as any.

Create a new folder in the service root directory and name it **models**. In it create another file and name it **Note.js**. This will be just a simple model definition.

```javascript
module.exports = (sequelize, type) => {
  return sequelize.define('note', {
    id: {
      type: type.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    title: type.STRING,
    description: type.STRING
  })
}
```

That’s it. Time to try it out.

*__Note__: Make sure to add your `secrets.json` to your `.gitignore` file.*

## How about some testing?

We’re ready to test the API. To be sure, let's start by running it all locally. One tiny thing to take into account is that there’s a flag we need to add while running `serverless-offline`. 

```
$ sls offline start --skipCacheInvalidation
```

*__Note:__ Serverless Offline invalidates the Node require cache on every run by default, we add this flag to disable it. In Node.js when you `require()` a module, it stores a cached version of the module, so that all subsequent calls to `require()` do not have to reload the module from the file system.*

Once you've run the command in the terminal, you should see something like this.

![sls-offline-start](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/sls-offline-start.png)

All our routes are up and running. Using [Insomnia](https://insomnia.rest/), I’ve created a POST request to `http://localhost:3000/notes`with a JSON body.

![insom-1](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/insom-1.png)

Checking the terminal you can see `=> using new database connection` get logged, meaning the initial database connection has been established. Send another POST request and you’ll see `=> using existing database connection` get logged instead.

Awesome, adding a new note works. Let’s retrieve all the notes we just added using the **getAll** method.

![insom-2](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/insom-2.png)

Try out the other endpoints yourself. Come back here once you're done playing. 

## Deploying to AWS

Now comes the tricky part. Make sure to write down all the database parameters you'll get in the AWS Aurora Serverless console, and of course the security group ID and subnet IDs as well. Usually, you'll have one security group and three subnets. But, it can vary, so don't worry. After you have the values, add them to your `secrets.json`. That's it! You're ready to deploy.

The Serverless framework makes deployments quick and painless. All you need to do is to run one command.

```
$ sls deploy
```

It will automagically provision resources on AWS, package up and push all code to S3 from where it will be sent to the Lambdas. The terminal should show output similar to this.

![sls-deploy](https://raw.githubusercontent.com/adnanrahic/cdn/master/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless/sls-deploy.png)

*__Note__: You can repeat the testing process from above with the endpoints provided. But, be patient with the cold-start times. AWS Aurora Serverless can take a decent 10 seconds to start. A similar issue happens with AWS Lambda in a VPC.*

That’s all there is to the deployment process. Easy right? This is why I love the Serverless framework so much.

## Wrapping up

This is an ever-evolving architecture. With the rise of a serverless relational database, the possibilities are endless in creating entirely serverless infrastructure. I’ve tried my best to explain the process of creating a proper API. Hopefully, you've learned a lot. Enjoy digging deeper into the possibilities of serverless architecture and all that comes with it!

And, of course, [here's the repo](https://github.com/adnanrahic/building-a-serverless-api-with-nodejs-and-aws-aurora-serverless) once again, give it a star if you want more people to see it on GitHub. If you want to read some of my previous serverless musings head over to [my profile](https://dev.to/adnanrahic) or [join my serverless newsletter!](https://upscri.be/b6f3d5/)

*I had an absolute playing with AWS Aurora Serverless. Hope you guys and girls enjoyed reading it as much as I enjoyed writing it. If you liked it, slap that tiny unicorn so more people here on dev.to will see this article. Don't forget to give the peeps at Cube.js some love if you need a [serverless SQL analytics framework](https://statsbot.co/)! Until next time, be curious and have fun.*

***

*Disclaimer: <a href="https://tracetest.io?utm_source=devto&utm_medium=blog&utm_campaign=adnans_blog" target="_blank">Tracetest</a> is sponsoring this blogpost. Use observability to reduce time and effort in test creation + troubleshooting by 80%.*

***