Using AWS SAM Cookiecutter Project Templates
Tags: aws, python, ruby, javascript
Published: 2020-06-18 11:54:38

For the upcoming [Lamby](https://lamby.custominktech.com/) work, we really want to improve our "getting started" experience. Creating a new Rails app on AWS Lambda should feel just as natural as running the `$ rails new` command line tool. One option to explore could be AWS' [Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/). Commonly called SAR (czar, not es•ay•are) for short, it offers re-usable applications similiar to Heroku's [deploy button](https://blog.heroku.com/heroku-button).

Learning SAR is definitely on my list. However, initial research showed that it was not well suited for generator style project templates. Nor did it appear to put an emphasis on leaving the user with some local version-controllable artifact to move the application forward using various CI/CD tools. However, I have known that the SAM CLI tool [does support](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-init.html) a popular Python project called [Cookiecutter](https://github.com/cookiecutter/cookiecutter) which might be perfect for Lamby. So I set out to learn more about it.


## 👟 Want to Skip Ahead?

Like learning from the outside in? I took all the work (🚧) below and made a simple demo Lambda cookiecutter publicly available at [customink/cookiecutter-ruby](https://github.com/customink/cookiecutter-ruby) on GitHub. Feedback on your experience or how we can do this better is [very much appreciated](https://github.com/customink/cookiecutter-ruby/issues/1)!

If you already have [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) installed, run this command.

```shell
$ sam init --location "gh:customink/cookiecutter-ruby"
```

If you want to avoid installing SAM CLI alltogether, you can use this Docker command.

```shell
$ docker run \
  --interactive \
  --volume "${PWD}:/var/task:delegated" \
  lambci/lambda:build-ruby2.7 \
  sam init --location "gh:customink/cookiecutter-ruby"
```

Commit your newly generated Lambda project to GitHub and the project's README.md for usage and next steps.


## 🚧 Learning Cookiecutter

Abstract learning without a goal to apply it, rarely helps me explore a technology. So for this exercise, I set out to build a suite of Lambda starters to help our Custom Ink teams adopt a "serverless-first" mindset by starting on the small/medium sized workload needs. Here is a checklist of features I think we needed.

* Mimic Rails' CLI "app_name" argument to customize file, folder, and class namespaces.
* Expose a configuration option for simple event-driven workload or API responses.
* Integrate a [GitHub Actions](https://technology.customink.com/blog/2019/09/02/from-travis-ci-to-github-actions/) workflow template to run tests and deploy.
* Leverage Docker containers to make development, test, and deployment easy.
* Remove the need to copy our SAM `bin` script conventions to new projects.
* Explore the new [HTTP API](https://aws.amazon.com/blogs/compute/announcing-http-apis-for-amazon-api-gateway/) version of API Gateway.

While learning, I made heavy use of the great documentation by the Cookiecutter team, notably the "Advanced Usage" section. Searching GitHub issues is also a fine art. Especially when you lack the proper project-specific keywords to use. Sharing both below.

* [Cookiecutter Docs](https://cookiecutter.readthedocs.io)
* [Cookiecutter GitHub Repo](https://github.com/cookiecutter/cookiecutter)

## 🚧 Inflecting a Single Input

Cookiecutter's input parameters are driven by a `cookiecutter.json` file at the root of the repository. The user can customize these via the CLI prompt after running `sam init`. Most Cookiecutter projects have a `project_name` variable and this is the one we want to mimic Rails' `new` command which drives all naming conventions for the newly created project. So if someone typed "my_new_service" it would be converted to "MyNewService" and ideally used via Cookiecutter's template code like `{{cookiecutter.class_name}}`. These are the derived project name variables we needed:

* `MyNewService` - Class name or module namespace.
* `my_new_service` - File or folder namespace.
* `my-new-service` - Other places like function/stack names.

The problem is that Cookiecutter was not built to directly support this. Sure we could use some inline, possibly verbose, Python to transform a single variable. Especially since Cookiecutter does not natively support succinct rails-like ActiveSupport inflector methods. Nor the ability to import ad-hoc code. The solution is somewhat hacky, but involves these steps:

1. Leverage Cookiecutter's "pre" and "post" gen hooks.
2. Add [some inflector](https://github.com/jpvanhal/inflection/blob/master/inflection.py) code to the `pre_gen_project.py` file.
3. Write individual text files, one for each derived property.
4. Include those values within in your templates.
5. Clean up the project's build directory using `post_gen_project.py` file.

So instead of being able to use `{{cookiecutter.class_name}}` we have to use something like `{% include "_cctmp/class_name.txt" %}`. It also means that file renaming now needs to be done in the post gen hook via Python's `os.rename` method. A small price to keep the outside user experience clean.

How can Cookiecutter make this better? A little would go a long way. These two issues on their site are good ones to watch if you want to advocate for improving things.

* [Allow Jinja Extensions to be Bundled](https://github.com/cookiecutter/cookiecutter/pull/944)
* [Help generating GUID/UUID values for Context Variables](https://github.com/cookiecutter/cookiecutter/issues/1101)

## 🚧 GitHub Actions for CI/CD

Your newly created Lambda project includes a simple test suite with some events to get you started. It even includes a `.github/workflows/cicd.yml` file to run those tests with GitHub Actions and even deploys your Lambda when changes land in master. We leverage the [Configure AWS Credentials](https://github.com/aws-actions/configure-aws-credentials) action which requires you to provide both a `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` via their [managed secrets](https://github.blog/2020-02-06-manage-secrets-and-more-with-the-github-actions-api/) interface. To survive the Cookiecutter templating process we had to quote this literal `${{ secrets.AWS_ACCESS_KEY_ID }}` format like so.

```yaml
aws-access-key-id: {{ "${{ secrets.AWS_ACCESS_KEY_ID }}" }}
aws-secret-access-key: {{ "${{ secrets.AWS_SECRET_ACCESS_KEY }}" }}
```

One easy way to make a "deploy" user with limited IAM capabilities is to do the first deployment yourself in a role with elevated permissions, and then create a user with more restricted permissions to do the updates. For example, assuming your local machine has the AWS CLI configured, this would deploy your Lambda:

```shell
$ STAGE_ENV=production ./bin/deploy
```

To create a simple "deploy" user with keys, we recommend getting started with the `AWSLambdaFullAccess` managed policy. To do so, in the "AWS Console → IAM → Users → Add User":

1. Check "Programmatic access" option.
2. Select "Attach existing policies directly" option.
3. Select "AWSLambdaFullAccess" policy.
4. Copy the "Access key ID" and "Secret access key"

Now within your GitHub repo page. Click Settings → Secrets → Add a new secret

* Name: `AWS_ACCESS_KEY_ID` Value: Value from the step above.
* Name: `AWS_SECRET_ACCESS_KEY` Value: Value from the step above.


## 🚧 BYO Events or HTTP API?

Our demo Cookiecutter project has an option to bring your own events or to make a simple HTTP microservice using the newly released HTTP API for API Gateway. I recommend [Yan Cui's "_HTTP API GOES GA TODAY!_"](https://lumigo.io/blog/http-api-goes-ga-today/) post if HTTP API is new to you.

To enable the event options, we simply added this `http_api` variable to our `cookiecutter.json` file. It defaults to `yes` because it is the first option in the array. In our template code we can check for this using simple conditions like `{%- if cookiecutter.http_api == "yes" %}`.

```json
{
  "project_name": "my_awesome_lambda",
  "http_api": ["yes", "no"]
}

```

I was overjoyed to find that implementing HTTP API via SAM was super terse and easy. Every [AWS::Serverless::HttpApi](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-httpapi.html) property like `DefinitionBody` was optional and the default behavior was a full proxy. So this would be the conditional resource added in the template:

```yaml
MyNewServiceHttpApi:
  Type: AWS::Serverless::HttpApi
  Properties:
    StageName: !Ref StageEnv
```

And this would be the conditional event added to the Lambda resource.

```yaml
Events:
  MyNewServiceHttpApiProxy:
    Type: HttpApi
    Properties:
      ApiId: MyNewServiceHttpApi
```

Amazing! This is the first time I felt like HTTP events in SAM was well into the convention-over-configuration camp. Well done, team! That said, they do have some work ahead of them to enable all the features of [AWS::ApiGatewayV2::Api](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-api.html). For example, tags are not passed down via CloudFormation and you have no way to use them with `AWS::Serverless::HttpApi`. Also, our `bin/server` script does not work since SAM CLI [has yet to add support](https://github.com/awslabs/aws-sam-cli/issues/1641) via the `start-api` command. But I'm sure that is coming soon and not technically needed in this demo project.

## 🚧 Everything else

I could go on and on but for brevity's sake, we can stop here. If you want to learn how to build your own Cookiecutter Lambda starter, use our [customink/cookiecutter-ruby](https://github.com/customink/cookiecutter-ruby) project for inspiration. Our internal ones span multiple languages (Ruby, Node, & Python) and do a lot more. SAM Cookiecutters are a great way to remove process or boilerplate burdens for your team. Here are some things we put an emphasis on:

#### Docker Images

By using `docker-compose`, we are able to provide a shared development, testing, and deployment process. Not only does this allow a unified developer experience, but it effectively commoditizes the deployment process, making it more portable. We do this for Lambda by leveraging [@hichaelmart's](https://twitter.com/hichaelmart) community [lambci/docker-lambda](https://github.com/lambci/docker-lambda) project. These are the same images that AWS SAM CLI uses too.

#### Strap Scripts

Following ideas like this [scripts to rule them all](https://github.com/github/scripts-to-rule-them-all) project, we make heavy use of `bin` script conventions at Custom Ink. Every project no matter the language or implementation should have a unified interface to `bootstrap`, `setup`, and `test` the project. Our adoption of Lambda and Infrastructure as Code extends these concepts to `deploy` as well.

#### Multi-Account Deploys

From development to staging and production. Using a combination of environment variables like `STAGE_ENV` or `RAILS_ENV` along with our script conventions above, we make it easy to deploy to any AWS account. Docker compose also makes this easy by leveraging its `environment` option. It even support defaults. For example, this line will use the host's `STAGE_ENV` or default to `development`; combine this with `AWS_PROFILE` and you can get some really neat results:

```yaml
- STAGE_ENV=${STAGE_ENV-development}
```

## Resources

As always, thanks for reading! Below are some quick links for reference. If you have any ideas or feedback, leave a comment. I'd love to hear from you.

* [Demo AWS SAM Lambda Cookiecutter Ruby Project](https://github.com/customink/cookiecutter-ruby)
* [Yan Cui's HTTP API Rundown](https://lumigo.io/blog/http-api-goes-ga-today/)
* [AWS Serverless Application Repository: Lambda and Beyond](https://medium.com/panther-labs/aws-serverless-application-repository-lambda-and-beyond-fa4e652b9c75)
* [The Cookiecutter GitHub Project](https://github.com/cookiecutter/cookiecutter)
* [LambCI Docker images used by SAM](https://github.com/lambci/docker-lambda)