Testing Serverless Applications on AWS
Published: 2023-11-02 19:28:17

## Introduction

The topic of Serverless testing is a hot one at the moment - there are many different approaches and opinions on how best to do it. In this post, I'm going to share some advice on how _we_ tackled this problem, what the benefits are of our approach and how things could be improved.

The project in question is [Stroll Insurance](/case-studies/stroll-insurance/), a fully Serverless application running on AWS. In [previous posts](/blog/zero-to-serverless-car-insurance-part-1/), we have covered some of the general lessons learnt from this project but in this post we are going to focus on testing.

For context; the web application is built with [React](/courses/introduction-to-react-course/) and [TypeScript](/courses/introduction-to-typescript-course/) which makes calls to an [AppSync](https://aws.amazon.com/appsync/) API that makes use of the [Lambda and DynamoDB datasources](https://docs.aws.amazon.com/appsync/latest/devguide/data-source-components.html). We use [Step Functions](https://aws.amazon.com/step-functions/) to orchestrate the flow of events for complex processing like purchasing and renewing policies, and we use [S3](https://aws.amazon.com/s3) and [SQS](https://aws.amazon.com/sqs/) to process document workloads.

## The Testing 'Triangle'

When the project started, it relied heavily on unit testing. This isn't necessarily a bad thing but we needed a better balance between getting features delivered and maintaining quality. Our [testing ~pyramid~](https://martinfowler.com/articles/practical-test-pyramid.html) triangle looked like this:

![](https://instil.co/static/a1bde71065edebb133573adba4cafb6e/a5d4d/testing-serverless-applications-triangle.webp)

Essentially, we had an abundance of unit tests and very few E2E tests. This worked really well for the initial stages of the project but as the product and AWS footprint grew in complexity, we could see that a number of critical parts of the application had no test coverage. Specifically, we had no tests for:

* **Direct service integrations** used by AppSync resolvers + StepFunctions
* **Event driven flows** like document processing

These underpin critical parts of the application. If they stop working, people will unable to purchase insurance!

One problem that we kept experiencing was unit tests would continue to pass after a change to a lambda function but subsequent deployments would fail. Typically, this was because the developer had forgotten to update the permissions in CDK. As a result, we created a rule that everyone had to deploy and test their changes locally first, in their own AWS sandbox, before merging. This worked but it was an additional step that could easily be forgotten, especially when under pressure or time constraints.

## Balancing the Triangle 

So, we agreed that it was time to address the elephant in the room. Where are the integration tests?

Our motivation was simple:

> There is a lack of confidence when we deploy to production, meaning we perform a lot of manual checks before we deploy and sometimes these don't catch everything.
>  
> This increases our lead time and reduces our deployment frequency. We would like to invert this.

The benefits for our client were clear:

* **Features go live quicker**, reducing time to market but while still maintaing quality
    * Gives them a competitive edge
    * Reduces feedback loop, enabling iteration on ideas over a shorter period of time
        
* **Critical aspects of the application are tested**
    * Issues can be diagnosed quicker
    * Complex bugs can be reproduced
    * Ensures / Reduces no loss of business
        
Integration testing can mean a lot of different things to different teams, so our definition was this:

> An integration test in this project is defined as one that validates integrations with AWS services (e.g. DynamoDB, S3, SQS, etc) but _not_ third parties. These should be mocked out instead.

## Breaking down the problem

We decided to start small by first figuring out how to test a few critical paths that we had caused us issues in the past. We made a list of how we “trigger” a workload:

*   S3 → SQS → Lambda
*   DynamoDB Stream → SNS → Lambda
*   SQS → Lambda
*   StepFunction → Lambda
    

The pattern that emerged was that we have an event that flows through a messaging queue, primarily SQS and SNS. There are a number of comments we can make about this:

*   There's no real business logic to test until a Lambda function or a State Machine is executed but we still want to test that everything is hooked up correctly.
*   We have most control over the Lambda functions, it will be easier to control the test setup in there.
*   We want to be able to put a function or a State Machine into “test mode” so that it will know when to make mocked calls to third parties.
*   We want to keep track of test data that is created so we can clean it up afterwards.
    

## Setting the Test Context

One of the most critical parts of the application is how we process insurance policy documents. This has enough complexity to be able to develop a good pattern for writing our tests so that other engineers could build upon it in the future. This was the first integration test we were going to write.

![](https://instil.co/static/d4f8d55540e8b2e166b202134a695887/a5d4d/testing-serverless-applications-flow.webp)


The flow is like this:

*   File is uploaded to S3 bucket
*   This event is placed onto an SQS queue with a Lambda trigger
*   The Lambda function reads the PDF metadata and determines who the document belongs to.
*   It fetches some data from a third party API relating to the policy and updates a DynamoDB table.
*   File is moved to another bucket for further processing.
    

We wanted to assert that:

1.  The file no longer exists in the source bucket
2.  The DynamoDB table was updated with the correct data
3.  The file exists in the destination bucket
    

This would be an incredibly valuable test. Not only does it verify that the workload is behaving correctly, it also verifies that the deployed infrastructure is working properly and that it has the correct permissions.

For this to work, we needed to make the Lambda Function _aware_ that it was running as part of a test so that it would use a mocked response instead. The solution that we came up with was to attach some additional [metadata to the object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html) when it was uploaded at the start of test case - an `is-test` flag:

![](https://instil.co/static/4c4d96be0d6fba9d0e11dd2aa49cf489/a5d4d/testing-serverless-applications-metadata.webp)

If the S3 object is moved to another bucket as part of its processing then we also copy its metadata. The metadata is never lost even in more complex or much larger end-to-end workflows. 

## The Middy Touch

Adding the `is-test` flag to our object metadata gave us our way of passing some kind of test context into our workload. The next step was to make the Lambda Function capable of _discovering_ the context and then using that to control how it _behaves_ under test. For this we used [Middy](https://middy.js.org).

If you're not familiar, Middy is a middleware framework specifically designed for Lambda functions. Essentially it allows you to wrap your handler code up so that you can do some `before` and `after` processing. I'm not going to do a Middy deep dive here but the [documentation](https://middy.js.org/docs/intro/how-it-works) is great if you haven't used it before.

We were already using Middy for various different things so it was a great place to do some checks before we execute our handler. The logic is simple:

* **In the before** phase of the middleware, check for the `is-test` flag in the object's metadata and if `true`, set a global test context so that the handler is aware it's running as part of a test. 

* **In the after** phase (which is triggered after the handler is finished), clear the context to avoid any issues for subsequent invocations of the warmed up function:

```typescript
export const S3SqsEventIntegrationTestHandler = (logger: Logger): middy.MiddlewareObj => {
  
  // this happens before our handler is invoked.
  const before: middy.MiddlewareFn = async (request: middy.Request<SQSEvent>): Promise<void> => {
    const objectMetadata = await getObjectMetadata(request.event);
    const isIntegrationTest = objectMetadata.some(metadata => metadata["is-test"] === "true");

    setTestContext({isIntegrationTest});
  };
  
  // this happens after the handler is invoked.
  const after: middy.MiddlewareFn = (): void => {
    setTestContext({isIntegrationTest: false});
  };

  return {
    before,
    after,
    onError
  };
};
```

Here's the test context code. It follows a simple TypeScript pattern to make the context read-only:

```typescript
export interface TestContext {
  isIntegrationTest: boolean
}

const _testContext: TestContext = {
  isIntegrationTest: false
};

export const testContext: Readonly<TestContext> = _testContext;

export const setTestContext = (updatedContext: TestContext): void => {
  _testContext.isIntegrationTest = updatedContext.isIntegrationTest;
};
```

I think this is the hardest part about solving the Serverless testing “problem”. I believe the correct way to do this is in a real AWS environment, not a local simulator and just making that deployed code _aware_ that it is running as part of a test is the trickiest part. Once you have some kind of pattern for that, the rest is straightforward enough.

We then built upon this pattern for each of our various triggers, building up a set of middleware handlers for each trigger type.

![](https://instil.co/static/7166d8833df84a2d525f13a7e64f085d/a5d4d/testing-serverless-applications-code1.webp)

For our S3 middleware we pass the `is-test` flag in an object's metadata, but for SQS and SNS we pass the flag using [message attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes). 

## A note on Step Functions

By far the most annoying trigger to deal with was Lambda Functions invoked by a State Machine task.

There is no easy way of passing metadata around each of the states in a State Machine - a global state would be really helpful (but would probably be overused and abused by people). The only thing that is globally accessible by each state is the [Context Object](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html). 

Our workaround was to use a specific naming convention when the State Machine is executed, with the execution name included in the Context Object and therefore available to every state in the State Machine.

![](https://instil.co/static/a9c841b4365b0ecf37aa604fbd35002a/a5d4d/testing-serverless-applications-code2.webp)

For State Machines that are executed by a Lambda Function, we can use our `testContext` to prefix all State Machine executions with "IntegrationTest-". This is obviously a bit of a hack, but it does make it easy to spot integration test runs from the execution history of the State Machine.

We then make sure that the execution name is passed into each Lambda Task and that our middleware is able to read the execution name from the event. (Note that `$$` provides access the Context Object).

![](https://instil.co/static/9047faa695418f05005b7c4ff191d575/a5d4d/testing-serverless-applications-code3.webp)


Another difficult thing to test with Step Functions is error scenarios. These will often be configured with retry and backoff functionality which can make tests too slow to execute. Thankfully, there is a way around this which my colleague, Tom Bailey, has [covered in a great post](https://instil.co/blog/testing-step-functions-locally/). I would recommend giving that a read.

## Mocking third party APIs

We're now at a point where a Lambda Function is being invoked as part of our workload under test. That function is also aware that its running as part of a test. The next thing we want to do is determine how we can mock the calls to our third party APIs.

There are a few options here:

* **Wiremock**: You could host something _like_ wiremock in the AWS account and call the mocked API rather than the real one. I've used wiremock quite a bit and it works really well, but can be difficult to maintain as your application grows. Plus it's another thing that you have to deploy and maintain.

  
* **API Gateway**: Either spin up your own custom API for this or use the built in [mock integrations](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-mock-integration.html).

  
* **DynamoDB**: This is our current choice. We have a mocked HTTP client that instead of making an HTTP call, queries a DynamoDB table for a mocked response, which has been seeded before the test has run.

Using DynamoDB gave us the flexibility we needed to control what happens for a given API call without having to deploy a bunch of additional infrastructure.

## Asserting that something has happened

Now it's time to determine if our test has actually passed or failed. A typical test would be structured like this:

```typescript
it("should successfully move documents to the correct place", async () => {
    const seededPolicyData = await seedPolicyData();
    
    await whenDocumentIsUploadedToBucket();

    await thenDocumentWasDeletedFromBucket();
    await thenDocumentWasMovedToTheCorrectLocation();
  });
```

With our assertions making use of the [`aws-testing-library`](https://github.com/erezrokah/aws-testing-library/blob/main/src/jest/README.md):

```typescript
async function thenDocumentWasMovedToTheCorrectLocation(): Promise<void> {
  await expect({
    region,
    bucket: bucketName,
  }).toHaveObject(expectedKey);
}
```

The `aws-testing-library` gives you a set of really useful assertions with built in delays and retries. For example:

Checking an item exists in DynamoDB:
```typescript
await expect({
  region: 'us-east-1',
  table: 'dynamo-db-table',
}).toHaveItem({
    partitionKey: 'itemId',
});
```


Checking an object exists in an S3 bucket:
```typescript
await expect({
  region: 'us-east-1',
  bucket: 's3-bucket',
}).toHaveObject(
  'object-key'
);
```


Checking if a State Machine is in a given state:
```typescript
await expect({
  region: 'us-east-1',
  stateMachineArn: 'stateMachineArn',
}).toBeAtState('ExpectedState');
```

It's important to note that because you're testing in a live, distributed system, you will have to allow for cold-starts and other non-deterministic delays when running your tests. It certainly took us a while to get the right balance between retries and timeouts. While at times it has been flakey, the benefits of having these tests far outweigh the occasional test failure.


## Running the tests

There are two places where these tests get executed, developer machines and on CI.

Each developer on our team has their own AWS account, they are regularly deploying a full version of the application and running these integration tests against it.

What I really like to do is get into a test driven development flow where I will write the integration test first and make my code changes, which will be [hot swapped using CDK](https://instil.co/blog/improving-cdk-development-cycle/) and then run my integration test until it turns green. This would be pretty painful if I was waiting on a full stack to deploy each time, but hot swap works well at reducing the deployment time.

On CI we run these tests against a development environment after a deployment has finished.

## It could be better

There are a number of things that we would like to improve upon in this approach

1.  **Temporary Environments** \- We would love to run these tests against temporary environments when a Pull Request is opened.
    
2.  **Test data cleanup** \- Sometimes tests are flaky and don't clean up after themselves properly. We have toyed with the idea of setting a TTL on DynamoDB records when data is created as part of a test.
    
3.  **Run against production** \- We don't run these against production yet, but that is the goal.
    
4.  **Open source the middleware** \- I think more people could make use of the middleware than just us, but we haven't got round to open sourcing it _yet._
    
5.  **AWS is trying to make it better** - Serverless testing is a hot topic at the moment, AWS have responded with some great resources you can find here - [https://github.com/aws-samples/serverless-test-samples](https://github.com/aws-samples/serverless-test-samples)
    

## Summary

While there are still some rough edges to our approach, the integration tests really helped with the issues we have already outlined and can be nicely summarised with three of the four key DORA metrics:

*   **Deployment Frequency** - The team's confidence increased when performing deployments, which increased their frequency.
    
*   **Lead Time for Changes** - Less need for manual testing reduced the time it takes for a commit to make it to production.
    
*   **Change Failure Rate** - Permissions errors no longer happen in production and bugs are caught sooner in the process. The percentage of deployments causing a failure in production reduced.

{% embed https://www.youtube.com/watch?v=Z_jhircdAPE %}