AWS CloudWatch Contributor Insights. Is it useful?
Tags: aws, serverless, observability, devops
Published: 2021-10-27 20:58:20

---
title: AWS CloudWatch Contributor Insights. Is it useful?
series: [Lambda Observability]
published: true
description: Check AWS Cloudwatch Contributor Insights
tags: #AWS #Serverless #Observability #DevOps
cover_image: https://www.pawelpiwosz.net/assets/lambdaObservability/sherlock-holmes.jpg
---

## Check logs with Contributor Insights

Right! So we have logs. We can use CloudWatch Logs Insights to query it, we can observe it through CloudWatch logs, and we can do many, many things with them now. Finally, we have them!

AWS gives us another tool, called Contributor Insights. This service can be used to analyze logs, create time series and display this data. That's quite powerful!

![First look on Contributor Insights](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights1.jpg)

Go to CloudWatch service, Contributor insights. Click `Create rule`. In the dropdown `Log group(s)` select our API logs. For the first time, select `Sample rule`, `API Gateway` and `Calls by route and HTTP method (CLF)`. In fact, it doesn't matter if you select JSON or CLF (in this case).

![First sample rule](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights2.jpg)

Keep rest of configuration as it is.


```txt
Log format: CLF (or JSON if you selected it earlier)  
Contribution: ip  
Aggregation: Count
```

Click `Next` and give the rule name. I named mine as `APIGW_byCount`.

After you click `Create rule`, you have to wait a few minutes (AWS claims 5) to see any data.

![The result](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights3.jpg)

This will build the analysis and aggregation. On the screen you see four different IPs (are they IPs, really? We'll see). 

The example above is autogenerated.

Let's see, if we can to create custom one.

Go to editor, click `Syntax` to change the screen to text editor.

![Switch editor type](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights4.jpg)

Our example looks like this:

```JSON
{
	"Schema": {
		"Name": "CloudWatchLogRule",
		"Version": 1
	},
	"LogGroupNames": [
		"API-Gateway-Execution-Logs_9fcv1s573l/Prod"
	],
	"LogFormat": "JSON",
	"Contribution": {
		"Keys": [
			"$.ip"
		],
		"Filters": []
	},
	"AggregateOn": "Count"
}
```

Here you can start playing with your query.

## Let's visualise what we queried

It is time to visualise our work. Click `Actions`, `View in CloudWatch Metrics` and select `Unique contributors`. 

![Render metrics](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights5.jpg)

In time I received a few more requests from different IPs, and below you can see the visualisation.

![Rendered metrics](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights6.jpg)

## Let's make it useful

Well... What we see now is... not really useful, right? Right. This visualisation is not really about IPs. It is some mix of many elements. You know what? Forget this. Let's do it right.

What we need to do, is work on the samples provided by AWS and rewrite them, or create our own. Or, and this is what we will do, we will rewrite the logs which API is generating. Well, we should do it, anyway.

Ok. Let's go to the API service, select the proper API, again, got to `Stages`, `Prod` and select the `Logs/Tracing` tab.

![Update API logs](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights7.jpg)

Click `Enable Access Logging`, and use `Insert Example` for JSON. You will have default log (which actually is different than default default log ;) ). Let's add one more field, like in example below.

```JSON
{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "caller":"$context.identity.caller", "user":"$context.identity.user","requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath", "status":"$context.status","protocol":"$context.protocol", "responseLength":"$context.responseLength","traceId":"$context.xrayTraceId" }
```
On the end of the string you can find `traceId` field.

I also slightly modified the `Log Destination ARN` in order to differentiate the behavior between the logs. It will be more readable for you.

Ok, now let's be sure, that the syntax for rule is like this:

```JSON
{
	"AggregateOn": "Count",
	"Contribution": {
		"Filters": [],
		"Keys": [
			"$.ip"
		]
	},
	"LogFormat": "JSON",
	"LogGroupNames": [
		"API-Gateway-Execution-Logs_9fcv1s573l"
	],
	"Schema": {
		"Name": "CloudWatchLogRule",
		"Version": 1
	}
}
```

Be also sure, you use the proper (new one!) log group (remember, yours will be different than mine!)

First of all, you can see totally different log. Shorter, more clearly structurized.

![API logs](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights8.jpg)

And how it looks in Contributor Insights? Well, great!

![API logs visualization](https://www.pawelpiwosz.net/assets/lambdaObservability/contributorinsights9.jpg)

## SAM template

It is time to modify the template.

```yaml
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: simple Lambda

Resources:
  accessLogGroup:
    Type: AWS::Logs::LogGroup
  
  lambdaDemoApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Description: 'Prod stage'
      TracingEnabled: true
      MethodSettings:
      - HttpMethod: '*'
        LoggingLevel: INFO
        ResourcePath: '/*'
        MetricsEnabled: true
        DataTraceEnabled: true
      AccessLogSetting:
        DestinationArn: !GetAtt accessLogGroup.Arn
        Format: >-
          '{ "requestId":"$context.requestId", 
          "ip": "$context.identity.sourceIp", 
          "caller":"$context.identity.caller", 
          "user":"$context.identity.user", 
          "requestTime":"$context.requestTime", 
          "httpMethod":"$context.httpMethod", 
          "resourcePath":"$context.resourcePath", 
          "status":"$context.status", 
          "protocol":"$context.protocol", 
          "responseLength":"$context.responseLength", 
          "traceId":"$context.xrayTraceId" }'

  lambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: simplefunction.handler
      CodeUri: lambdafunction/
      Runtime: python3.8
      Policies:
        - CloudWatchLambdaInsightsExecutionRolePolicy
      AutoPublishAlias: live
      Description: Simple demo function
      MemorySize: 128
      Timeout: 10
      Tracing: Active
      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"
      Events:
        simpleApi:
          Type: Api
          Properties:
            RestApiId: 
              !Ref lambdaDemoApi
            Path: /
            Method: get
```

As you can see, I added one resource - `AccessLogGroup` and `AccessLogSettings` in Properties of `lambdaDemoApi`.

***

Now you can go crazy. There is many possibilities to build API Logs and to visualize it. Now you know :)

***

Cover image by <a href="https://pixabay.com/users/422737-422737/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=462957">Hebi B.</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=462957">Pixabay</a>