Two lesser-known ways to increase observability in AWS Lambda based applications
Tags: serverless, aws, observability, cdk
Published: 2021-06-06 11:09:47

---
title: Two lesser-known ways to increase observability in AWS Lambda based applications
published: true
canonical_url: https://cloudash.dev/blog/improve-lambdas-observability
description: 
tags: serverless, aws, observability, cdk
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oqx3cz0zqf6vtj6uecz5.jpeg
---
If you have ever worked in a microservices environment, you probably are aware of how vital monitoring and observability are. Every outage can turn into a "murder mystery", where developers spend a significant amount of time uncovering what exactly went wrong and how it could have been prevented.

This blog post aims to show you two lesser-known techniques you can employ to turn an _AWS Lambda_ related "murder mystery" into a simple logs check.

## Logging SDK operations

Logging every SDK operation that you perform might be an easy way of ensuring you have a clear picture of precisely what is going on within your _Lambda_ function. This method should be used cautiously though. Given high enough traffic, you might incur significant _CloudWatch_ costs. In such situations, you might want to look into sampling your logs.

The AWS-SDKs I'm familiar with (the Node.js and Go ones) allow for enabling such logging without us having to pollute our code with log statements everywhere. All we have to do is to pass a logger to a given SDK instance.

Here is an example of instrumenting the `DocumentClient` of the Node.js AWS-SDK with a logger.

```js
import { DocumentClient } from "aws-sdk/clients/dynamodb";
import pino from "pino";

const logger = pino();

const db = new DocumentClient({
  logger: {
    /**
     * The implementation is dependant on the logger itself.
     * In a real-world scenario you most likely want to also log the `requestId` or `X-Ray traceId`.
     */
    log: message => {
      logger.info({}, message);
    }
  }
});
```

And here is the sample log message produced by adding an item to the DynamoDB table.

![The log message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hodzz9aobwgvapqlpbts.png)

In a production setting, you most likely should redact fields that might contain sensitive information. In such a case, look into the `redact` configuration option of `pino`.

## Turning on Client Side Monitoring

I've stumbled upon the [_Client Side Monitoring_](https://summitroute.com/blog/2020/05/25/client_side_monitoring/) feature only recently. Enabling the _CSM_ will cause the SDK calls to push metadata about that given call to a UDP endpoint ([IP is configurable](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/metrics.html#metrics-update-cw)). The metadata concentrates not on the call's content but on the operation itself - what kind of API call was performed, latency, and similar information.

The following is an example event produced by the _CSM_.

```json
{
  "Version": 1,
  "ClientId": "",
  "Type": "ApiCallAttempt",
  "Service": "S3",
  "Api": "ListObjectsV2",
  "Timestamp": 1590430065174,
  "AttemptLatency": 243,
  "Fqdn": "s3.amazonaws.com",
  "UserAgent": "aws-cli/1.18.51 Python/3.7.6 Darwin/19.4.0 botocore/1.16.1",
  "AccessKey": "ASIAXXXXXXXXXXXXXXXX",
  "Region": "us-east-1",
  "SessionToken": "XXX",
  "HttpStatusCode": 301,
  "XAmzRequestId": "0000000000000000",
  "XAmzId2": "XXX",
  "AwsException": "PermanentRedirect",
  "AwsExceptionMessage": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."
}
```

As you can see, the event is very different than the logs produced by the SDK calls. 

Why should you bother enabling _CSM_?

### Scoping IAM policies down

Applying the principles of least privilege access in the context of Lambdas (and other compute) is crucial from the security perspective. From my perspective, the _CSM_ is an ideal mechanism for ensuring that these principles are followed in the context of AWS-SDK calls.

You could enable _CSM_, run your _AWS Lambda_ through a couple of workloads (ideally, those would be end-to-end or integration tests), and then based on the collected _metadata_ deduce the least privileged permissions for your function.

It turns out most of the work in that area has already been done for you. [Ian Mckay](https://twitter.com/iann0036) released [iam-live](https://github.com/iann0036/iamlive) and the [iam-live lambda extension](https://github.com/iann0036/iamlive-lambda-extension). These tools make the process I described earlier a breeze. All you have to do is to hook them up to your existing stack.

### Security monitoring

_CSM_ could be used to monitor for any unusual behavior that your application might perform. While the _AWS CloudTrial_ is a vital option in most cases, **not every AWS event lands there**. Having both _CSM_ and _AWS CloudTrial_ in place, you gain much more visibility on the security axis than with _AWS CloudTrial_ alone.

If you are curious how could you leverage both _AWS CloudTrial_ and _CSM_ so that they complement each other, [checkout this great article on cloudonaut.io](https://cloudonaut.io/record-aws-api-calls-to-improve-iam-policies/)

## Summary

These were, in my opinion, two lesser-known ways to increase observability in your _AWS Lambda_ based AWS applications.
I hope you learned something new as I did while writing this blog.

You can find me on twitter - [@wm_matuszewski](https://twitter.com/wm_matuszewski)

Thank you for your time.