How to monitor your serverless (Open)API?
Tags: aws, beginners, devops, serverless
Published: 2020-02-02 11:28:50

# TL;DR

You can monitor your Serverless (Open)API with the help of CloudWatch Alarms. Please refer to [this](https://github.com/rpstreef/openapi-tf-example) Github repository for the code.

CloudWatch Alarms for API Gateway are detailed in directory __./modules/cloudwatch-alarms-apigateway__ and for AWS Lambda in __./modules/cloudwatch-alarms-lambda__

Next week i'll discuss making sense of logs and failures with AWS CloudWatch, and enable tracing with AWS X-Ray.

Thanks for reading!

# Topic break-down
- [TL;DR](#tldr)
- [Topic break-down](#topic-break-down)
- [How to monitor your serverless (Open)API?](#how-to-monitor-your-serverless-openapi)
- [How to setup CloudWatch Alarms for API Gateway?](#how-to-setup-cloudwatch-alarms-for-api-gateway)
  - [Which metrics are we tracking?](#which-metrics-are-we-tracking)
  - [Prerequisites](#prerequisites)
  - [The Terraform details](#the-terraform-details)
- [How to setup CloudWatch Alarms for AWS Lambda?](#how-to-setup-cloudwatch-alarms-for-aws-lambda)
  - [Which metrics are we tracking?](#which-metrics-are-we-tracking-1)
  - [The Terraform details](#the-terraform-details-1)
- [The result](#the-result)
- [What's next](#whats-next)
- [Further reading](#further-reading)

# How to monitor your serverless (Open)API?

In previous articles we've discussed the advantages of OpenAPI and how we can utilize OpenAPI for deployment on AWS API Gateway, and applying input/output verification & validation. This week we're tackling a piece of the _serverless observability_ problem, and that's _monitoring_.

The service on AWS to do just that is [AWS CloudWatch](https://aws.amazon.com/cloudwatch/). AWS CloudWatch collects metrics and logs across all services such that we can program actions or simply create dashboards to display that data as useful graphs. This gives us insights into the real-time status of our  API's.

This blog will show you how you can implement & configure monitoring for; [AWS API Gateway](#how-to-setup-cloudwatch-alarms-for-api-gateway), and [AWS Lambda](#how-to-setup-cloudwatch-alarms-for-aws-lambda). 

Please refer to [this](https://github.com/rpstreef/openapi-tf-example) Github repository to view the code and get a more in-depth understanding of the setup.

# How to setup CloudWatch Alarms for API Gateway?

## Which metrics are we tracking?

The metrics that are available for API Gateway are summarized in great detail [here](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html). We will cover these metrics (quoted in part from AWS docs):

- __Latency__ The time between when API Gateway receives a request from a client and when it returns a response to the client. The latency includes the integration latency and other API Gateway overhead. Specifically, we want to monitor these:
  - __P95/P99__ this measures the 95th and 99th percentile, the outliers are what we're interesting in measuring and will give us the best possible measurement of real world performance of our application for end users.
- __4XX Errors__ is a count of all 4XX errors (average or a sum total).
- __5XX Errors__ is a count of all the 5XX errors, also given as either an average or a sum total.

## Prerequisites

Prerequisites for per API Method alarms is to enable __Detailed CloudWatch Metrics__. This will incur extra [costs](https://aws.amazon.com/cloudwatch/pricing/)!

We do this in the __aws_api_gateway_method_settings__ resource in the __apigateway__ module, set __metrics_enabled__ to true:

```terraform
resource "aws_api_gateway_method_settings" "_" {
  rest_api_id = aws_api_gateway_rest_api._.id
  stage_name  = aws_api_gateway_stage._.stage_name
  method_path = "*/*"

  settings {
    throttling_burst_limit = var.api_throttling_burst_limit
    throttling_rate_limit  = var.api_throttling_rate_limit
    metrics_enabled        = var.api_metrics_enabled
  }
}
```

In your AWS API Gateway console under __Stages > Logs/Tracing__ you should see a tick next to __Enable Detailed CloudWatch Metrics__

## The Terraform details

The configuration for CloudWatch Alarms for API Gateway is located in a new module; __cloudwatch-alarms-apigateway__. You can find this in the directory; _./modules/cloudwatch-alarms-apigateway_

In our deployment _main.tf_ file, call the module with all the API resources you want to enable Alarms for under the _api_resources_ variable:

```terraform
module "cloudwatch_alarms_apigateway" {
  source = "../../modules/cloudwatch-alarms-apigateway"

  namespace         = var.namespace
  region            = var.region
  resource_tag_name = var.resource_tag_name

  api_name  = module.apigateway.api_name
  api_stage = module.apigateway.api_stage

  resources = var.api_resources
}
```

The _api_resources_ variable is defined in __dev.tfvars__ this specifies which API's we want monitored:

```terraform
api_resources = {
  "/identity/authenticate" = "POST",
  "/identity/register"     = "POST",
  "/identity/reset"        = "POST",
  "/identity/verify"       = "POST",
  "/user"                  = "GET"
}
```

# How to setup CloudWatch Alarms for AWS Lambda?

## Which metrics are we tracking?

Summarized in great detail [here](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html). We will cover these metrics (quoted in part from AWS docs):

- __Errors:__ measures the number of invocations that failed due to errors (4XX)
- __Throttles:__ Measures the number of Lambda function invocation attempts that were throttled due to invocation rates exceeding the customer’s concurrent limits (error code 429). 
- __IteratorAge:__ Emitted for stream-based invocations only (functions triggered by an Amazon DynamoDB stream or Kinesis stream). Measures the age of the last record for each batch of records processed. Age is the difference between the time Lambda received the batch, and the time the last record in the batch was written to the stream. 
- __DeadLetterErrors:__ Incremented when Lambda is unable to write the failed event payload to your function's dead-letter queue. This could be due to the following: Permissions errors, Throttles from downstream services, Misconfigured resources, Timeouts

## The Terraform details

In each of the __Services__ directories (e.g. __identity__ and __user__) we apply the configuration as shown below to enable the AWS Lambda CloudWatch alarms for _Errors_ and _Throttles_.

> Since we do not have a dead letter queue, nor are we integrated with a stream (DynamoDB or Kinesis) we do not need to create these alarms.

Everything else we do not explicitly need to set, there are sensible defaults configured in the __cloudwatch-alarms-lambda__ module _variables.tf_ file

```terraform
locals {
  resource_name_prefix = "${var.namespace}-${var.resource_tag_name}"
  lambda_function_name = "identity"
}

module "cloudwatch-alarms-lambda" {
  source = "../../cloudwatch-alarms-lambda"

  namespace         = var.namespace
  region            = var.region
  resource_tag_name = var.resource_tag_name

  create_iteratorAge_alarm     = false
  create_deadLetterQueue_alarm = false

  function_name = "${local.resource_name_prefix}-${local.lambda_function_name}"
}
```

# The result

Once both AWS Lambda and API Gateway alarms have been configured, we can deploy this with __npm run dev-infra__. If you need to initialize Terraform again, run: __npm run dev-init__. 

Once deployed, in our CloudWatch console we'll see the Alarms listed like below:

> Note: most will read _insufficient data_, over time they will turn to _OK_.

![CloudWatch Alarms](https://dev-to-uploads.s3.amazonaws.com/i/m6oyt24o3g00uj3h7edm.png)

# What's next

We've created all the alarms in CloudWatch, next logical additions to this would be to integrate this with your own team tooling such as a channel on Slack, or perhaps a dedicated Administrator portal/mobile application to increase visibility of your API status.

Next week I'll attempt to showcase how errors & failures can be made more transparent with AWS CloudWatch and [AWS X-Ray](https://aws.amazon.com/xray/) with these goals in mind:

- How to make sense of logs & failures,
- and how can we trace execution across multiple services. 

Thanks for reading!

# Further reading
- [OpenAPI Example Github repository](https://github.com/rpstreef/openapi-tf-example)
- [Lumigo CloudWatch Alarms Macro](https://github.com/lumigo/SAR-cloudwatch-alarms-macro)
- [Why is my Lambda IteratorAge metric increasing?](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-iterator-age/)
- [Lambda monitoring function metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html)
- [API Gateway metrics and dimensions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html)
- [AWS Lambda best practices](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html)
- [Understanding latency](https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/)
- [What is Serverless Observability?](https://www.youtube.com/watch?v=Nc_T0iDzJpM)