A brief summary of our AWS productivity tools
Published: 2020-09-12 22:29:37

## Table of content
* [Background](#background) - What brought us here?
* [cfn-diagram](#cfn-diagram) - CloudFormation visualizer
* [evb-cli](#evb-cli) - EventBridge pattern generator and debugging suite
* [cfn-resource-actions](#cfn-resource-actions) - VS Code extension that lets you interact with deployed resources from the template
* [sam-policies-cli](#sam-policies-cli) - CLI UI for browsing and injecting SAM Policy Templates
* [iam-policies-cli](#iam-policies-cli) - CLI UI for building complex IAM policies
* [cwlogs-cli](#cwlogs-cli) - Tool to quickly launch CloudWatch Logs Insights with multiple log groups pre-selected


## A background <a name="background"></a>

When we in January 2017 made the decision to rewrite Sweden's leading online grocery store, [MatHem.se](https://mathem.se), from scratch into a serverless microservices architecture we found ourselves in an extremely promising yet premature tech segment with a lack of tooling due to a yet quite small community.

This led us to adopting a mindset of 'when tasks get repetitive, find a tool for it or write one yourself'.

Much of our tooling solves bespoke in-house procedural bottlenecks, but when we find solutions to tasks that can benefit the community we make sure to publish it in the open.

This post is a summary of the AWS/serverless related productivity tools we have authored.

## cfn-diagram <a name="cfn-diagram"></a>
NPM: https://www.npmjs.com/package/@mhlabs/cfn-diagram
GitHub: https://github.com/mhlabs/cfn-diagram

**Installation**
`npm i -g @mhlabs/cfn-diagram`

**Features**
Parses CloudFormation/SAM JSON or YAML and renders a diagram in either draw.io or vis.js network format.

**Examples**
![Cfn-diagram-drawio](https://raw.githubusercontent.com/mhlabs/cfn-diagram/master/demo.gif)

![Cfn-diagram-html](https://raw.githubusercontent.com/mhlabs/cfn-diagram/master/demo-html.gif)

**Reason for existing**
Our application is made up of hundreds of microservices, each defined with its own CloudFormation/SAM template. No engineer knows about every stack. Our engineering team has been going through a rapid growth phase which has required onboarding of new developers to be efficient.

To ease the getting to know a stack, we had an initiative that every repository was required to have an up-to-date diagram over the service's resources in the readme file. No matter how good the intentions were, those sort of 'musts' will never be followed and these diagram will be something that 'might be true' and therefore never trusted.

With `cfn-diagram` any developer new to a stack can quickly pull up a visualisation of how it all hangs together. It's also useful to render in planning meetings instead of drawing on the whiteboard

## evb-cli <a name="evb-cli"></a>
NPM: https://www.npmjs.com/package/@mhlabs/evb-cli
GitHub: https://github.com/mhlabs/evb-cli

**Installation**
`npm i -g @mhlabs/evb-cli`

**Features**
Pattern generator and debugging tool for EventBridge with features including:
* Event pattern builder 
* Input transformation builder
* Event rules browser
* Generate a diagram over the events flow in a region
* Local debugging of events where actual events are sent to the console over websockets. These can be forwarded into sam-local for some advanced debugging.

**Examples**
![Demo1](https://raw.githubusercontent.com/mhlabs/evb-cli/master/demo.gif)
*Pattern generation*

![Demo2](https://raw.githubusercontent.com/mhlabs/evb-cli/master/demo-diagram.gif)
*Events flow diagram generator*

![Demo2](https://raw.githubusercontent.com/mhlabs/evb-cli/master/demo-local.gif)

*Local debugging*


**Reason for existing**
Historically we have been heavily invested in pub/sub using SNS/SQS, but we were always annoyed with the tight coupling this created between services and teams, so when EventBridge was announced we got very excited. 

One of the most powerful features of EventBridge is its content based filtering where you only let through events matching a part of the payload that interests the consumer of the event. Together with EventBridge's Schema Registry an engineer can build event patterns without knowing anything about the producing service.

These event patterns often get quite complex and we found ourselves spending a lot of time first writing, then deploying, then debugging them. Typos were a common mistake both when composing event patterns and input transformations.

Our developers also reported that they find it cumbersome to debug their rules as well as finding it difficult to see the whole picture of how all events hang together, so from that feedback we added local debugging features

## cfn-resource-actions <a name="cfn-resource-actions"></a>
Marketplace: https://marketplace.visualstudio.com/items?itemName=ljacobsson.cfn-resource-actions
GitHub: https://github.com/mhlabs/cfn-resource-actions

**Installation**
`ext install ljacobsson.cfn-resource-actions`

**Features**
Interact with your deployed CloudFormation/SAM templates directly from the template. Turns your template in to an interface into the AWS console
* Tail lambda log
* Query DynamoDB tables
* Copy Physical Id of any resource
* Consume EventBridge events directly in vs code
* Send to and Poll SQS
* Visualise template
* etc

**Examples**

![Demo1](https://raw.githubusercontent.com/mhlabs/cfn-resource-actions/master/images/example-lambda.gif)

*Invoke lambda + tail its logs*

![Demo2](https://raw.githubusercontent.com/mhlabs/cfn-resource-actions/master/images/example-f12.gif)

*CTRL+click from template to Lambda handler*

![Demo3](https://raw.githubusercontent.com/mhlabs/cfn-resource-actions/master/images/example-dynamodb.gif)

*Query DynamoDB*

![Demo4](https://raw.githubusercontent.com/mhlabs/cfn-resource-actions/master/images/example-visualize.gif)

*Visualise stack requires [Draw.io Integration](https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio) extension*


**Reason for existing**
Context switching is a huge cost in productivity loss. Leaving VS Code to go and do stuff in the AWS console is a distraction we want to avoid

## sam-policies-cli <a name="sam-policies-cli"></a>
NPM: https://www.npmjs.com/package/@mhlabs/sam-policies-cli
GitHub: https://github.com/mhlabs/sam-policies-cli

**Installation**
`npm i -g @mhlabs/sam-policies-cli`

**Features**
CLI UI to quickly inject SAM policy templates into your `AWS::Serverless::Function` resources

**Example**

![Demo](https://raw.githubusercontent.com/mhlabs/sam-policies-cli/master/demo.gif)



**Reason for existing**
We strive to follow the Principle of Least Privilege and finding the correct policy templates requires googling

## iam-policies-cli <a name="iam-policies-cli"></a>
NPM: https://www.npmjs.com/package/@mhlabs/iam-policies-cli
GitHub: https://github.com/mhlabs/iam-policies-cli

**Installation**
`npm i -g @mhlabs/iam-policies-cli`

**Features**
UI to quickly build simple to complex IAM policies based on resources in a CloudFormation template

**Example**

![Demo](https://raw.githubusercontent.com/mhlabs/iam-policies-cli/master/demo.gif)


**Reason for existing**
We strive to follow the Principle of Least Privilege and composing granular IAM policies is time consuming and prone to human errors

## cwlogs-cli <a name="cwlogs-cli"></a>
NPM: https://www.npmjs.com/package/@mhlabs/cwlogs-cli
GitHub: https://github.com/mhlabs/cwlogs-cli

**Installation**
`npm i -g @mhlabs/cwlogs-cli`

**Features**
Create groups of log groups from resource tags or log group prefixes and launch CloudWatch Logs Insights with those groups predefined

**Example**

![Demo](https://raw.githubusercontent.com/mhlabs/cwlogs-cli/master/demo/demo-tag.gif)


**Reason for existing**
CloudWatch Insights Logs is a great way to search your logs. Often you want to search across many log groups to, for example, tracing a correlation id across multiple services. CloudWatch Logs Insights lets you search across up to 20 log groups in one query, but adding log groups from the dropdown is a bit cumbersome.