OpenAPI with Terraform on AWS API Gateway
Tags: aws, devops, serverless, testing
Published: 2020-01-26 16:58:16

# TL;DR

You just want to dig into the source code, look here on [Github](https://github.com/rpstreef/openapi-tf-example). The README.md contains instructions on how to run it. See [this](#a-brief-introduction-to-the-code-base) section for a brief solution break-down.

If you're interested in how I deployed this solution with Terraform, go  [here](#how-to-deploy-openapi-endpoints-with-terraform-to-aws).

If you're interested in the testing aspects, go [here](#how-to-setup-contract-testing-with-openapi).

Thanks for reading and till next week!

# Topic break-down

- [TL;DR](#tldr)
- [Topic break-down](#topic-break-down)
- [OpenAPI with Terraform on AWS API Gateway](#openapi-with-terraform-on-aws-api-gateway)
  - [A brief introduction to the code base](#a-brief-introduction-to-the-code-base)
- [What is the OpenAPI specification?](#what-is-the-openapi-specification)
  - [How to create a REST API endpoint?](#how-to-create-a-rest-api-endpoint)
  - [What about CORS?](#what-about-cors)
  - [Component schema specification](#component-schema-specification)
  - [How do we take care of secured endpoints?](#how-do-we-take-care-of-secured-endpoints)
- [How to deploy OpenAPI endpoints with Terraform to AWS?](#how-to-deploy-openapi-endpoints-with-terraform-to-aws)
  - [The API Gateway module](#the-api-gateway-module)
  - [The Services integration](#the-services-integration)
  - [The AWS Lambda handler function](#the-aws-lambda-handler-function)
- [How to setup contract testing with OpenAPI?](#how-to-setup-contract-testing-with-openapi)
- [What's next?](#whats-next)
- [Further reading](#further-reading)

# OpenAPI with Terraform on AWS API Gateway

[Last week](https://dev.to/rolfstreefkerk/why-document-a-rest-api-as-code-5e7p) we discussed the why's of OpenAPI and its tooling support. Today, I'll demonstrate and answer the following topics;

- __[What is the OpenAPI specification?](#what-is-the-openapi-specification)__. Our single source of truth. 
- __[How to deploy OpenAPI endpoints with Terraform to AWS?](#how-to-deploy-openapi-endpoints-with-terraform-to-aws)__
- __[How to setup contract testing with OpenAPI?](#how-to-setup-contract-testing-with-openapi)__

The key principles here are;
- __Automation__; we want to codify as much as possible, any manual interaction means we introduce possible errors due to inconsistency.
- __Single source of truth__; The fewer documents we need to maintain, the higher the likelihood the information is _current_ and _consistent_. 

## A brief introduction to the code base

If you like to follow along with the code, please look [here](https://github.com/rpstreef/openapi-tf-example) on Github. The README.md contains instruction on how to run it.

The structure is as follows:

- __./env/dev__ is where the environment specific [Terraform](https://www.terraform.io/) code resides. This is for the development environment. Similarly, you would duplicate these files for _test_ and _production_.
  - This way you can store the state files in their own AWS accounts specific to the environment, and you can create different configurations for each environment (e.g. in the _dev.tfvars_ file).
- __./modules__ contains all the base Terraform modules that are composed in the __./services__ folder 
- __./node__ contains;
  -  All the AWS Lambda handler/lib Node12 code in __./node/src__
  -  Node command-line programs in __./node/cmd__.
  -  JSON Schema file generated from the OpenAPI specification in __./node/schema__
-  __./services__ contains Terraform files that specify the services this solution is composed off using the base AWS Cloud services in __./modules__.
   -  The OpenAPI document is located in __./services/api__, and that is the base document for [AWS API Gateway](https://aws.amazon.com/api-gateway/) and the JSON Schema file discussed earlier.
-  __./gulpfile.js__ contains the code to create the Zip file artifacts from the __./node/src__ codebase and deploys that with Terraform.

# What is the OpenAPI specification?

Lets get right into it with the OpenAPI document structure before I explain the individual parts in more detail. Or [skip](#how-to-create-a-rest-api-endpoint) to the next subsection.

OpenAPI documents will at least have these major parts (top hierarchy);
- __openapi__, indicates the version,
- __info__, describes general information about the owner and license agreements (ToS),
- __servers__, where to connect to,
- __paths__, the available endpoints,
- __components__, specific models (JSON Schema) that apply to the endpoints and security definitions.
- __tags__, used to create a human readable structure (you can group endpoints together with the same tag) out of the document with descriptions.
- There's a concept of __extensions__ (e.g. ```x-``` custom parameters) in the specification, we'll see examples of that shortly, that allows for anyone to extend on to the official OpenAPI document without breaking the validation of the document. More on this [here](https://swagger.io/docs/specification/openapi-extensions/).

Now that we have a general sense of the document layout, let's look at how we create __paths__ (endpoints).

## How to create a REST API endpoint?

Lets consider the following endpoint; ```POST /identity/authenticate```
```yml
paths:
  /identity/authenticate:
    post:
      operationId: identityAuthenticate
      description: Authenticate user (either login, or continue session)
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Authenticate'
      x-amazon-apigateway-integration:
        uri: "arn:aws:apigateway:${region}:lambda:path/2015-03-31/functions/${lambda_identity_arn}/invocations"
        passthroughBehavior: "when_no_match"
        httpMethod: "POST"
        timeoutInMillis: ${lambda_identity_timeout}
        type: "aws_proxy"
      responses:
        200:
          $ref: '#/components/responses/lambda'
        500:
          $ref: '#/components/responses/500APIError'
      tags:
        - Identity
```

There's a lot to unpack here, lets start from the top:
- __operationId__ is used to differentiate between the API operations. This name has to be unique for this document. In this code base it's used to detect in AWS Lambda which operation is executed. More on this when I explain the custom AWS __x-__ parameter.
- __requestBody__ specifies which _POST_ object it expects and if it's required. 
  - The _$ref_ refers to the JSON Schema further down this document. This keyword allows you to reference other parts of the specification without duplicating the same content.
- __x-amazon-apigateway-integration__ is a custom AWS parameter that is used to define the integration with, in this case, AWS Lambda.
  - This parameter is required for each operation.
  - It specifies which AWS Lambda function it's integrated with via the __${lambda_identity_arn}__ parameter that is set by the Terraform scripting.
  - It specifies the maximum timeout of the integration with __${lambda_identity_timeout}__ parameter.

Lets say that we want to execute this API from another domain, how do we do that?

## What about CORS?

This is a typical nuisance with REST API's, how to get cross origin requests to work.

With OpenAPI and AWS API Gateway this is relatively simple, there are 2 steps:

- First, add an ```options``` to the endpoint paths that you want to execute cross domain:
```yml
    options:
      responses:
        200:
          $ref: '#/components/responses/cors'
        500:
          $ref: '#/components/responses/cors'
      x-amazon-apigateway-integration:
        responses:
          default:
            statusCode: "200"
            responseParameters:
              method.response.header.Access-Control-Max-Age: "'7200'"
              method.response.header.Access-Control-Allow-Methods: "'OPTIONS,HEAD,GET,POST,PUT,PATCH,DELETE'"
              method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
              method.response.header.Access-Control-Allow-Origin: "'*'"
        passthroughBehavior: "when_no_match"
        timeoutInMillis: 29000
        requestTemplates:
          application/json: "{ \"statusCode\": 200 }"
        type: "mock"
```

- Second, in the AWS Lambda handler code we need to make sure the response contains the following header

```javascript
{
  isBase64Encoded: false,
  statusCode: 200,
  headers: {
    'Access-Control-Allow-Origin': '*'
  },
  body: JSON.stringify(message)
}
```

Now we should be able to create an endpoint that is cross origin executable. How does the validation and specification of input and output work?

## Component schema specification

The __component__ section of the specification is all about the data, what comes in and what goes out.

We have 3 variants to specify;

- __parameters__: These are used in ```GET``` requests for instance, they appear as query parameters in the URL, e.g. ```/user/2```

```yml
  parameters:
    userID:
      description: User identifier
      in: query
      name: userID
      schema:
        type: string
      required: true
```
- __schemas__: Used in ```POST``` requests for example, and are often referenced in an API response. The ```required```, ```type```, and ```pattern``` definitions can all be used to validate your API input and output. 
  - More on this later when we cover JSON Schema validation

```yml
  schemas:
    Register:
      title: Register
      type: object
      description: Registration form
      required:
        - email
        - password
        - username
        - firstName
        - lastName
      properties:
        email:
          type: string
          example: "user@business.com"
          pattern: "^[_A-Za-z0-9-\\+]+(\\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\\.[A-Za-z0-9]+)*(\\.[A-Za-z]{2,})$"
        password:
          type: string
          example: "123$SFF22l"
        username:
          type: string
          example: "testUsername"
        firstName:
          type: string
          example: "Rolf"
        lastName:
          type: string
          example: "Streefkerk"
```

- __responses__: Below you can see the schema reference, or if we want we can explicity define the JSON Schema here like in __schemas__.

```yml
  responses:
    user:
      description: User
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/User'
```

## How do we take care of secured endpoints?

All of what we have seen so far can be accessed by the public at large, how do we secure an endpoint with authentication?

In this example I've used [AWS Cognito](https://aws.amazon.com/cognito/) as the authentication service and it integrates really well with API Gateway.

Specify the following details:
- __Name__ the security definition, e.g. _example-CognitoUserPoolAuthorizer_
- __Auth type__ is ```cognito_user_pools```
- __Authorizer__ is the actual AWS Cognito instance identified by its unique ARN (Amazon Resource Names) which is supplied by Terraform at deploy time.

```yml
securitySchemes:
  example-CognitoUserPoolAuthorizer:
    type: "apiKey"
    name: "Authorization"
    in: "header"
    x-amazon-apigateway-authtype: "cognito_user_pools"
    x-amazon-apigateway-authorizer:
      providerARNs:
      - "${cognito_user_pool_arn}"
      type: "cognito_user_pools"
```

Now to enable a secured endpoint, you will have to add the __security__ parameter as follows;

```yml
/user:
  get:
    operationId: getUser
    description: get User details by ID
    parameters:
      - $ref: '#/components/parameters/userID'
    security:
    - example-CognitoUserPoolAuthorizer: []
    ...
```

# How to deploy OpenAPI endpoints with Terraform to AWS?

> If you're new to Terraform please stay tuned for a more in-depth Terraform _how-to_.

Briefly we [discussed](#openapi-with-terraform-on-aws-api-gateway) the directory layout in the introduction. A good structure is crucial with Terraform to  ensure there's as little code duplication as possible, and the right information is stored in the right places.

My preferred structure is this:
- __env__ code and state that is specific to the environment we deploy to.
  - We do not want to mix in state from production in a development account for instance.
  - We definitely do not want to commit state to Git. Use an encrypted shared storage such as AWS S3.
- __modules__ to re-use the code, create modules for each of the major services used and preferably version these such that different projects can run different versions and upgrade independently. For this demo, they're just included in the same repo.
- __services__ here the infrastructure is composed with __modules__ and we store the OpenAPI definition file here as part of our solution.

## The API Gateway module

To build the OpenAPI integration, we need to feed the document into a __aws_api_gateway_rest_api__ resource. This is done in the __apigateway__ module as follows (an excerpt):

```terraform
data "template_file" "_" {
  template = var.api_template

  vars = var.api_template_vars
}

resource "aws_api_gateway_rest_api" "_" {
  name           = "${local.resource_name_prefix}-${var.api_name}"
  api_key_source = "HEADER"

  body = data.template_file._.rendered
}
```

When this module is called in __./env/dev/main.tf__ Terraform file. We need to specify the template variables that we've already seen in the OpenAPI document (e.g. __${cognito_user_pool_arn}__)

```terraform
module "apigateway" {
  source            = "../../modules/apigateway"
  resource_tag_name = var.resource_tag_name
  namespace         = var.namespace
  region            = var.region

  api_name                   = local.api_name
  api_throttling_rate_limit  = var.api_throttling_rate_limit
  api_throttling_burst_limit = var.api_throttling_burst_limit
  api_template               = file("../../services/api/${local.api_name}.yml")
  api_template_vars = {
    region = var.region

    cognito_user_pool_arn = module.cognito.cognito_user_pool_arn

    lambda_identity_arn     = module.identity.lambda_arn
    lambda_identity_timeout = var.lambda_identity_api_timeout

    lambda_user_arn     = module.user.lambda_arn
    lambda_user_timeout = var.lambda_user_api_timeout
  }

  lambda_zip_name = local.lambda_zip_name
  dist_file_path  = local.dist_file_path
}
```

## The Services integration

You'll notice that there is mention of __module.identity.lambda_arn__ and __module.user.lambda_arn__. These are the two services that have been integrated with the Identity and User endpoints.

When __GET /user__ is called, the AWS Lambda defined in the module __module.user__ will be executed.

Furthermore, any access a Lambda function requires to an AWS service such Cognito, it will need appropriate permissions to be set. These are defined in their own json files. E.g. for the User service, they're located in __./services/user/policies/lambda.json__

## The AWS Lambda handler function

Now that the Terraform side of it has been setup, it still needs executable code. Handlers are AWS Lambda entry-points in our Node code. They look like this; (see __./node/src/handlers__)

```javascript
const middy = require('middy')
const { httpErrorHandler, httpSecurityHeaders } = require('middy/middlewares')
const { ErrorResponse } = require('../../lib/response')
const standards = require('../../lib/standards')
const getUser = require('./operations/getUser')

const handler = middy(async (event, context) => {
  const params = standards.getParams(event)
  const operation = standards.getOperationName(event)

  switch (operation) {
    case 'getUser': {
      return getUser.handler(params, operation)
    }

    default: {
      console.error('Unsupported operationName: ' + operation)
      return new ErrorResponse('Unsupported operationName: ' + operation)
    }
  }
}).use(httpErrorHandler()).use(httpSecurityHeaders())

module.exports = { handler }
```

This is where we can use the JSON schema to validate our REST endpoint input data.

The source file is (re)created by running this node script:
```npm run convert``` this will take the OpenAPI yml file and convert it to a JSON Schema V4 compatible file. 

In turn this is used by the lib __./node/src/lib/jsonSchema__, and initialized in a handler function as follows:

```javascript
const JsonSchema = require('../../../lib/jsonSchema') // our library
const schema = require('../../../schema/example.json') // our JSON Schema
const input = new JsonSchema(schema['/identity/register'].post.body) //the Path in that JSON Schema we want to validate against

const results = input.validateInput(params) // validate the input
 
if (results.valid) {
  return new SuccessResponse('Success, user with email address: ' + params.email + ' registered successfully')
}
```

# How to setup contract testing with OpenAPI?

We have a working OpenAPI specification, we deployed the solution with Terraform. Now we can also generate the basis of our contract test suite with [Postman](https://www.getpostman.com/).

Here's how to do that:

- Open Postman, go to APIs tab on the left side of the application, hit __+New API__.
- __Add Schema > Import file__, navigate to  __\openapi-tf-example\services\api\example.yml__.
- Replace in the imported document, ( servers: - url: http://example.com/ ), with the URL that's output at the command line when deployment of the Example REST API has finished, it's called _api_url_.
  - Recommended, create a domain name and link that to your REST API and have it automatically filled into the OpenAPI yml file with Terraform.
- Click on __Generate Collection__, give it a name (e.g. _Example_) and then add to __Contract Test__.
- When you navigate to your Collections (left hand side) you should see it listed there.
- Go to your _Example_ collection and open  __identity > register> POST identity register__, hit send
- You should see error messages coming in at the bottom panel;
  - e.g. "message": "should match pattern \"^[_A-Za-z0-9-\\+]+(\\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\\.[A-Za-z0-9]+)*(\\.[A-Za-z]{2,})$\""
- If you get the message, that confirms the API is working correctly and you can design your contract tests with documentation under the Tab __Test__ in the __APIs__ tab.
  
You can use this as the base to create test cases. Look at [this](http://www.adeveloperdiary.com/java/webservice/rest/how-to-automate-rest-api-schema-validation-testing-using-postman/) article for instance that discusses more in detail how you can create test cases and run them from the Postman CLI with [Newman](https://github.com/postmanlabs/newman).

# What's next?

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/fyyjkttqskkrek6kr8tn.jpg)

We covered a lot of ground in this article, let me know in the comments if it was useful and what you'd like to see more of.

Any questions and idea's. Shoot!

Thanks for reading and till next week!

# Further reading

- [OpenAPI 3.0.2 specification](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md)
- [OpenAPI2Schema project](https://github.com/mikunn/openapi2schema)
- [How to automate REST API schema validation testing using Postman](http://www.adeveloperdiary.com/java/webservice/rest/how-to-automate-rest-api-schema-validation-testing-using-postman/)
- [AJV JSON Schema validation](https://ajv.js.org/) 
- [Middy AWS Lambda middleware](https://github.com/middyjs/middy)
- [Official JSON Schema](https://json-schema.org/)