Build & deploy all AWS resources as easy as: npm run deploy-prod (AWS Sam worflow)
Tags: aws, sam, serverless, devops
Published: 2020-06-23 21:07:25

After manually managing my AWS resources I finally decided to use Sam templates and put everything in Git. Easy deploying to prod and test stages was also a requirement. Here is a short overview of some issues I ran into and how I resolved them. It is by no means a detailed step by step guide, but just a quick overview with some problems I ran into and solutions.

Disclaimer: This is my first experience with AWS Sam. I probably did some things sub-optimal or wrong. Idea is to share and get comments from those with more experience. It may also help those starting with AWS Sam, to speed up the process and save a week I spend investigating.
Also note that current Sam version I'm using is 0.52.0 and I'm on Linux.

## sam-cli installation

First requirement is sam-cli. I'm not getting into details of installing and configuring it, but you can find it [here](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-linux.html).

You will also have to add sam-cli user an permissions on AWS IAM. This can be a slow process of trial and error. Depending on your resources, and motivation to close down permissions, settings will vary. Basically you try to publish, read error, add permission, rinse and repeat. Many tutorials just tell you to use admin account but I don't like that. Here's my IAM role policy but you'll need to adapt it:
```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DeleteUserPool",
                "cognito-idp:AddCustomAttributes",
                "iam:UntagRole",
                "iam:TagRole",
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "cloudformation:CreateChangeSet",
                "apigateway:DELETE",
                "cognito-idp:DescribeUserPool",
                "iam:PassRole",
                "iam:DetachRolePolicy",
                "cloudformation:DescribeStackEvents",
                "iam:DeleteRolePolicy",
                "apigateway:PATCH",
                "cloudformation:DescribeChangeSet",
                "apigateway:GET",
                "cloudformation:ExecuteChangeSet",
                "iam:GetRole",
                "apigateway:PUT",
                "cognito-idp:UpdateUserPoolClient",
                "iam:DeleteRole",
                "cognito-idp:ListTagsForResource",
                "cloudformation:DescribeStacks",
                "cognito-idp:CreateUserPoolClient",
                "apigateway:POST",
                "cognito-idp:UpdateUserPool",
                "iam:GetRolePolicy"
            ],
            "Resource": [
                "arn:aws:cognito-idp:<region>:<AccountID>:userpool/*",
                "arn:aws:iam::<AccountID>:role/<MyApp>*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/jsbenchme*/*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/aws-sam-cli-managed-default/*",
                "arn:aws:cloudformation:<region>:aws:transform/Serverless-2016-10-31",
                "arn:aws:apigateway:<region>::/restapis/*",
                "arn:aws:apigateway:<region>::/restapis"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:CreateUserPool",
                "cloudformation:GetTemplateSummary"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "events:*",
            "Resource": "arn:aws:events:iam::<AccountID>:rule/MyApp-*"
        },
        {
            "Effect": "Allow",
            "Action": "lambda:*",
            "Resource": "arn:aws:lambda:iam:<region>:<AccountID>:function:<MyLambdaFunction>-*"
        }
    ]
}
```


## Organizing folders and files
I decided to put my AWS Serverless app templates into `AWS` subfolder of my project. Sam will create folder structure like this.

Run `sam init` and choose AWS Quick Start Templates, runtime (mine was Node) and Hello World Example. Folder structure like this will be created by Sam:
```
AWS
 |-- events -> where events for testing go
 |-- hello-world -> where your code goes (Lambda in my case)
 `- template.yaml -> Sam template to deploy resources through CloudFormation
```
First, I renamed hello-world with `Lambda` and put my Lambda code there, deleted some files too.
Next step, which might not be as obvious at start, is splitting up `template.yaml` in different files. If you have many services (especially API), file becomes unmanageable very fast. so I created folder `./templates` for my templates expecting to merge them easily when the time comes to build.

### Issue #1 - merging AWS Sam templates
It turns out there were several issues here. First about ability of Sam to merge files, then with some references requiring resources to be in the same template and last (but not least for sure) a lot of issues I had in my API Gateway definition yaml with references to other properties. More on that later. Enough to say for now that I solved all those issues with a small tool `json-refs`. So, I just did `npm i -D json-refs`, split template in multiple files, put them in `./templates` folder and now my entry point template `./templates/main.yaml` looks something like this:
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    My App

Globals:
  Function:
    Timeout: 10

Resources:
  MyAppLambdaRole:
    $ref: 'LambdaAPIhandler.yaml#/Role'
  MyAppLambdaAPIhandler:
    $ref: 'LambdaAPIhandler.yaml#/Handler'
  MyAppDynamoDB:
    $ref: 'DynamoDB.yaml'
  MyAppAPIGateway:
    $ref: 'APIGateway.yaml#/APIGateway'
  MyAppFunctionLambdaPermission:
    $ref: 'APIGateway.yaml#/LambdaInvocationPermission'
  MyAppCognitoUserPool:
    $ref: 'CognitoUserPool.yaml#/Pool'
  MyAppCognitoClient:
    $ref: 'CognitoUserPool.yaml#/Client'
```
This is much neater and manageable. But to make it work, we need to setup script to use `json-refs` and merge those files into `./template.yaml`. I decided to stick with Bash...

## json-refs build script
We need to point json-refs to `./templates/main.yaml` and it will render all those `$ref`s and produce merged `./template.yaml` for Sam to use.
I started by creating another folder `./templates/build`, (also put it in `.gitignore`). Then created a Bash script `./build-deploy.sh` which started like this:
```bash
    echo '[build] deleting build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml
```
It's all quite self-explanatory, except maybe the last line where I remove `T00:00:00.000Z` which gets appended to version-dates of the Sam template. This might be unnecessary,  but I like to keep thing as they are in AWS docs and versions don't have timestamps there.

## Writing AWS resource definition yaml files
Again, relatively simple. AWS Docs are your best friend. It goes slow but steady. Some issues along the way though...

### Issue #2 - json-refs complains about custom tags like !Ref, !Sub...
Very easy fixable once you find out that `!Ref` can be replaced with `Ref:`, `!Sub` with `Fn:Sub` and `!GetAtt` with `Fn::GetAtt`. They are identical but your new build script will not complain. Just don't forget colon (`:`) after those.

### Issue #3 - API Gateway definition gets huge and has repetitive entries,  in some cases (inside `DefinitionBody`) Ref does not work
My biggest fear was to write yaml for all API Gateway paths and methods, knowing how much time I spent on it in AWS Gateway UI. Exporting Swagger and pasting it in template required some changes. Some references were broken and I failed to make them work. It might be possible, but since I already had json-refs in place, I decided to use that, it seemed easier. At the end, it turned out writing API in yaml was easier. So much stuff can be reused, just define bits and pieces within `components` in your yaml and reuse them with `$ref`. I've put individual method responses to components, lambda invocation properties, even complete repetitive response objects for many paths. Just as an illustration, it can look like this now (shortened!):
```yaml
APIGateway:
  Type: AWS::Serverless::Api
  Properties:
    Name: MyAPIGateway
    DefinitionBody:
      openapi: "3.0.1"
      path:
        "/me":
          get:
            x-amazon-apigateway-integration:
              $ref: "#/components/x-amazon-apigateway-integrations/default-integration"
            responses:
              $ref: "#/components/responses/defaultResponseGroup"

# components parsed by json-refs
components:
  responses:
    defaultResponseGroup:
      "404":
        description: "404 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
      "200":
        description: "200 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
  headers:
    default:
      Access-Control-Allow-Origin:
        schema:
          type: string
  content:
    default:
      application/json:
        schema:
          $ref: "#/components/schemas/Empty"
```
Of course, my definition has many paths, methods and responses, but putting some default properties and grouping them into `components` made it really easy to write API, easier than using API Gateway UI!

### Issue #4 - circular references
Well, I let Sam create unique names for my resources so their `Arn`s are created when deployed. But I need LambdaFunction to reference  LambdaFunctionRole.Arn and LambdaFunctionRole to reference LambdaFunction.Arn. Sam cannot create both at the same time, they depend on each other. AWS docs have a good [article](https://aws.amazon.com/blogs/infrastructure-and-automation/handling-circular-dependency-errors-in-aws-cloudformation/) on this and on more complex scenarios, but my solution was to name my LambdaFunction manually, so I can refer to it by name. It can be useful later in deployment process for other things too. Read on. Also check `DependsOn` CloudFormation attribute docs, this will help ordernig creation of resources to avoid similar issues in referrences.

## Automating build & deployment to separate stages (test and prod)
My initial impression was that I'll use `stageName` and `alias` properties of some resources to build `prod` and `test` stages but it turns out best practice is to separate stages into different stacks. Actually, many people suggest using different accounts but that's not always practical. So, I decided that each stage will have it's own stack. I did use `stageName` too to make it even more clear which resource belongs where, although it seems unnecessary because we will make `stackName` also contain stage name to differentiate different stacks.
General idea is to use single code base for both prod and test stage, but before the build process I need to be able to decide which stage I'm going to build and publish. So there must be a way to make template(s) dynamic, to make certain properties changeable from command line. CloudFormation parameters and Sam's `--parameter-overrides` argument is what we need. First, I added parameterss to my `./templates/main.yaml`:
```yaml
Parameters:
  parAccountId:
    Type: String
    Description: AWS Account Id
    Default: ""
  parRegion:
    Type: String
    Description: AWS Region to deploy to
    Default: ""
  parStackName:
    Type: String
    Description: The name of the stack
    Default: "myStack-test"
  parStage:
    Type: String
    Description: "The name of the stage, must me one of: test, prod"
    AllowedValues:
    - test
    - prod
    Default: "test"
```
... then those parameters need to be used in templates in place of hard-coded ones (i.e. `${parStackName}`).
Now is time to (maybe) make a step back. I'm actually editing this article after running into problems because I haven't done so. So, that step is to make naming convetion for your resources. Trust me, it will make your life easier if you just name all the resources manually, following that naming convention. Mine is:
`<project-name>-<resource-name>-<stack>-<stage>`

Go through all templates and implement this dynamic naming. Don't foget your sam-cli IAM role that you had to create at the beginning. This naming convention is going to make it a whole lot easier to avoid issues and resorting to '*'. Remember that Lambda function? Now part of its definition will look like:
```yaml
  Type: AWS::Serverless::Function
  Properties:
    FunctionName:
      Fn::Sub: "MyProject-MyAppLambdaAPIhandler-${parStackName}-${parStage}"
```

Now, we can use `sam build` and `sam deploy` to build and deploy our resources. We will use --parameter-overrides argument to send those parameters to CloudFormation and they will affect template there (note! local `template.yaml` will NOT be changed based on params! This will happen on CloudFormation):
```bash
    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"

    sam deploy --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM
```
Of course, we will put this in our `./build-deploy.sh` script which will take arguments and build those parameters with them, from command line. You can check it in the final script source at the end of the article.

### Issue #5 - Sam requires S3 bucket to deploy
Well, actually this S3 bucket is made automatically, but ONLY in "guided" deploy mode which is activated by `--guided` argument of `sam-build`. In this mode, sam asks questions about some deployment parameters and creates S3 bucket auto-magically, then stores your choices to `./samconfig.yaml` for later (add this file to .gitignore!). So, in our final `./build-deploy.sh` script we will check for existence of this file and if it is not there, we will start deploy in "guided" mode. Just check the final code for details.

### Issue #6 - Sam ignores --capabilities argument when in "guided" mode
We need `--capabilities CAPABILITY_NAMED_IAM` in my implementation and we do pass it to `sam build` as an argument. However, if it is in "guided" mode, it ignores `--capabilities` argument and asks you to enter the value manually (or uses wrong default). "Workaround" is also implemented in final `./build-deploy.sh`.

## Outputs
You will want to know some parameters created by AWS, which are not known before deploy is done. In example your API Gateway URI. We don't want anything hard-coded so there must be a way to fetch those after build & deploy. `Options` to the rescue. Again in `./main.yaml` I added this:
```yaml
Outputs:
  apiURL:
    Description: "production stage API URL"
    Value:
      Fn::Sub: "https://${MyAppAPIGateway}.execute-api.${parRegion}.amazonaws.com/${parStage}/"
```
This will create `apiURL` output visible in AWS CloudFormation UI. But how to access it from our build script and make it available to other parts of our app (in my case frontend React app)? Again, with Sam:
```bash
aws cloudformation describe-stacks --stack-name ${stackName} \
    --query 'Stacks[0].Outputs[?OutputKey==`apiURL`].OutputValue' \
    --output text
```
All we need to do now is to include this in our `./build-deploy.sh` and add some code to save those values into a config file readable to our app (don't forget to .gitignore it too). Also don't forget to somehow differentiate same variables for different stages! Because ApiURL for prod stage and test stage should both be saved. You can see it all in final `./build-deploy.sh` code where I also added more code to offer some help and arguments management:
```bash
#!/bin/bash

###################################################################################
#
# Bash script do build & deploy to different stages
#
###################################################################################

show-help () {
    echo 
    echo "--------------------------------"
    echo " AWS deployment script "
    echo "--------------------------------"
    echo 
    echo "usage: deploy.sh [options] <action>"
    echo 
    echo "ACTION:"
    echo "    build           Builds AWS app and stores template to ./template.yaml (Default action)"
    echo "    deploy          Deploys AWS app to CloudFormation using ./template.yaml"
    echo "    build-deploy    Builds and deploys in single step"
    echo 
    echo "OPTIONS:"
    echo "  mandatory arguments:"
    echo "    --accountId=   AWS account id to deploy to CloudFormation with"
    echo "    --region=       AWS region to deploy to"
    echo "    --stackName=   stack name"
    echo "    --stage=STAGE  stack stage name to deploy to"
    echo "                   STAGE is one of: prod, test"
    echo 
}

check-arguments () {
    # action arg default
    if [ -z "$action" ]
        then action = "build"
    fi

    # check empty mandatory args
    if [ -z "$accountId" -o -z "$region" -o -z "$stackName" -o -z "$stage" ]
        then echo "Error! Mandatory arguments missing"
    else
        # check format of args
        if [ "$stage" != "prod" -a "$stage" != "test" ]
        then
            echo "Error! --stage needs to be one of: prod, test"
            exit 3
        else
            return
        fi
    fi
    # output error specs
    if [ -z $accountId ]
        then echo " --accountId is a mandatory argument"
    fi
    if [ -z $region ]
        then echo " --region is a mandatory argument"
    fi
    if [ -z $stackName ]
        then echo " --stackName is a mandatory argument"
    fi
    if [ -z $stage ]
        then echo " --stage is a mandatory argument"
    fi
    exit 2
}

aws-build () {
    echo '[build] deleteing build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml
}

sam-build () {
    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"
}

sam-deploy () {
    # check if samconfig.toml exists. If not, run --guided to autocreate S3 bucket and remember it
    if [ -f "./samconfig.toml" ]
        then isGuided=""
        else
            isGuided="--guided"
            printf "This is the first deploy, Sam will be started in --guided mode.\n \
                    NOTE! \n\
                    Please accept default params except those: \n\
                        Allow SAM CLI IAM role creation [Y/n]:   Select N! \n\
                        Capabilities [['CAPABILITY_IAM']]:   Enter: CAPABILITY_NAMED_IAM \n\
                    This is to avoid a bug(?) in Sam which (only on guided run) uses default capabilities instead of ones passed as an argument. \n\
                    You can also leave defaults and (after erroring out) rerun deployment again. Secontd time it will use passed arguments and run ok. \n\
                    \n"
            read -p "Press ENTER to continue."
    fi

    sam deploy "${isGuided}" \
              --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM
}

save-outputs () {
    echo "getting outputs and storing them in configs..."
    apiURL=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`apiURLprod`].OutputValue' \
        --output text)

    cognitoClientId=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`cognitoClientId`].OutputValue' \
        --output text)

    if [ -z "$apiURL" -o -z "$cognitoClientId" ]
    then
        echo "Error! Error getting output params $apiURL $cognitoClientId"
        exit 5
    else
        echo '{"apiURL": "'"$apiURL"'", "cognitoClientId": "'"$cognitoClientId"'"}' > ./options-${stage}.json
    fi
}


while [ $# -gt 0 ]; do
    case "$1" in
        --accountId=*) accountId="${1#*=}"
        ;;
        --region=*) region="${1#*=}"
        ;;
        --stackName=*) stackName="${1#*=}"
        ;;
        --stage=*) stage="${1#*=}" #allowed stage names are limited in template.yaml param definition, not here
        ;;
        build) action="build";;
        deploy) action="deploy";;
        build-deploy) action="build-deploy";;
        *)  echo 
            echo "Error: invalid argument: $1"
            echo "Use --help for list of arguments"
            show-help
            exit 1
    esac
    shift
done

check-arguments
echo "Running $action command of stack: $stackName, stage: $stage to region: $region using AWS accountID: $accountId"


if [ "$action" == "build" ]
    then
        aws-build
        sam-build
fi

if [ "$action" == "deploy" ]
    then
        sam-deploy
        save-outputs
fi

if [ "$action" == "build-deploy" ]
    then
        aws-build
        sam-build
        sam-deploy
        save-outputs
fi
```
Now all that is left is to use our script in package.json to build&deploy (and/or other scenarios) like:
`./build-deploy.sh --accountId=012345678 --region=eu-west-1 --stackName=MyApp-test --stage=test build-deploy`