Benchmarking .NET on AWS Lambda
Tags: aws, dotnet, serverless
Published: 2022-08-29 17:50:11

My motivation for benchmarking all the compiler, deployment, and runtime options has been to feed my curiosity. I've seen various blog posts recommending one setting over another, but they never provided a justification.

Unfortunately, we can't use the outstanding [BenchmarkDotNet](https://benchmarkdotnet.org/) tool with AWS Lambda. So, I built a [benchmarking harness](https://github.com/LambdaSharp/LambdaSharp.Benchmark) to collect data for all the deployment options, and hopefully, determine an "optimal" combination. 

As the first blog post of this series mentioned, there are two distinct cases we can optimize for: "Minimize Cold Start Duration" or "Minimize Operating Cost". Now that the groundwork has been laid, we can formally capture what that means.

## Optimal Strategies

1. To _Minimize Cold Start Duration_, we need to minimize the INIT and the first INVOKE phase. The optimal configuration yields the lowest duration, measured in milliseconds (ms), to process the request. For this measurement, we rely on the data reported by AWS Lambda in the logs. This is the same data that is used for billing and is the most accurate we have access to.

2. To _Minimize Execution Cost_, we need to minimize the sum of all INVOKE phases (cold and warm) while taking into account the Lambda memory configuration and CPU architecture. The optimal configuration yields the lowest execution cost for 1 cold start followed by 100 warm invocations. AWS Lambe execution has an extremely low unit cost. To make the number more intuitive, I opted to report execution cost in millionth of a dollar, or micro-Dollars (µ$).

## Benchmarked Options

To leave no stone unturned, the benchmarking harness executes all possible permutations of the following options.

* Tiered Compilation: On and Off
* ReadyToRun: On and Off
* Lambda Memory: 128 MB, 256 MB, 512 MB, 1024 MB, 1769 MB, and 5120 MB
* CPU Architecture: x86-64 and ARM64
* .NET Runtime: .NET Core 3.1 and .NET 6
* Pre-JIT .NET: On and Off

These options produce 192 unique combinations that are benchmarked.

## Benchmarking Approach

The Lambda function for each project is measured performing 100 cold starts. Each cold start is followed by 100 warm invocations. The results are then averaged.

I debated using the median or a percentile value instead of the average value. The challenge is when these values are further combined. For example, summing the p99 value for the INIT phase with p99 value of the first INVOKE phase doesn't seem right. Furthermore, outliers happen in real life. I'm hoping that 100 cold and warm invocations are sufficient to fairly represent what should be expected in real world situations.

That said, the [raw measurements](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Data) have been captured for each benchmark. That way, alternative analyses can be done without having to collect the data again.

## Benchmarked Projects

My interest was in studying how the options are impacting the compute aspect of Lambda functions. Therefore, I opted to only benchmark projects that do not perform I/O operations.

### Minimal Baseline

The [Minimal project](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/Minimal/) establishes a baseline for all projects. It has no business logic and only includes required libraries.

### JSON Serializers

JSON serialization is necessary for virtually all Lambda functions. With .NET 6, there are 3 common approached to handle this task.

1. [NewtonsoftJson](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/NewtonsoftJson/): using Newtonsoft JSON.NET

1. [SourceGeneratorJson](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/SourceGeneratorJson/): using .NET 6 source generators for JSON parsing

1. [SystemTextJson](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/SystemTextJson/): using System.Text.Json

### AWS SDK

Most Lambda functions will interact with other AWS services via the AWS SDK. The [AwsSdk project](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/AwsSdk/) is used to benchmark the cost of initializing the SDK.

### Top-Level Statements

Available starting in .NET 6, Lambda functions can use top-level statements instead of declaring a class. What is the performance impact when doing so?

1. [AwsNewtonsoftJson](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/AwsNewtonsoftJson/): using AWS .NET SDK and Newtonsoft JSON.NET

1. [SampleAwsNewtonsoftTopLevel](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/SampleAwsNewtonsoftTopLevel/): using AWS .NET SDK, Newtonsoft JSON.NET, and Top-Level Statements

1. [SampleAwsSystemTextJsonTopLevel](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/SampleAwsSystemTextJsonTopLevel/): using AWS .NET SDK, System.Text.Json, and Top-Level Statements

### Minimal API

Also new in .NET 6 is a new approach to express ASP.NET routes using top-level statements. This sample was taken from the [.NET 6 support on AWS Lambda](https://aws.amazon.com/blogs/compute/introducing-the-net-6-runtime-for-aws-lambda/) announcement blog post.

1. [SampleMinimalApi](https://github.com/LambdaSharp/LambdaSharp.Benchmark/tree/main/Projects/SampleMinimalApi/): using ASP.NET Core Minimal API

## Benchmark Explorer

The results from the benchmarks have been compiled into an [interactive Google spreadsheet](https://docs.google.com/spreadsheets/d/1pOVJQi9Q2COj0amqV06BT8Nbgacf0nuyHpFj_P7sYH4/edit?usp=sharing). Feel free to explore the data anyway you like and draw your own conclusions. Any feedback on improving the analysis or visualization is welcome.

## What's Next

Finally, we can dive into the results and see what new insights we can gain! First up is the baseline performance measurement. While this is not critical for production code, it gives us a foundation to work on.