Creating a CI/CD pipeline for a Python3 Lambda Function through AWS CloudFormation
Published: 2020-03-20 07:51:27

Creating a [AWS Lambda](https://aws.amazon.com/lambda/features/) function is often as straightforward as selecting the language and pasting your code in the [console](https://aws.amazon.com/console/). However, you'd still make use of a version control system if you're not simply playing around with the functions but rather plan to run them with serious purposes -just like any good-old server applications. 

I would not surprise you if I know you prefer using [Git](https://git-scm.com) for that purpose. Taking one step further, I also know you're probably using Github as your remote codebase. So far so good, in the following lines we are going to make your code published as a Lambda function every time you commit to your preferred Git branch.

![Matrix Reloaded source room with a wall of screens](https://dev-to-uploads.s3.amazonaws.com/i/4fygco1msqu99l7pnmj4.jpg)

## Part #1: Preparing Source Code Repo

We are going to discuss the content of a desired source code repo for our function. You can read through or check the sample repo here:

https://github.com/vahdet/python-lambda-template

As the name suggests, it expects some code in `Python`. The version for it is `3.8`.

### Write Your Code

The default Python Lambda function in AWS consists of a single file named `lambda_function.py` and it has a single function `lambda_handler`. Assuring we update the `handler` field in [`template.yaml`](the-template.yaml-file) we can choose any name for both, but for the sake of simplicity we are going to stick to the defaults.

So, basically, add your repo a file named `lambda_function.py` and initialize its content like the console-generated one:

```py
# lambda_function.py
""" Lambda function module """
import os
import json
import boto3

def lambda_handler(event, context):
    """ Handles function event and context """

    # resolve backend api key from the secrets manager
    sm_client = boto3.client('secretsmanager')
    sm_resp = sm_client.get_secret_value(os.getenv('BACKEND_SERVICE_API_KEY_SECRET_ARN'))
    backend_api_key = json.dumps(sm_resp.get('SecretString')).get('key')

    # TODO implement further business logic
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }
```

So, writing down all your business logic here is not an urgent matter. Enjoy implementing right away or defer it. Here are some point you may consider:

- You can add other functions, closures and call them in our *main handler* function here (i.e. `lambda_handler`)

- The two arguments of the main function `event` and `context`. To have a wider understanding of their contents, please refer to the [official docs](https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html)

### List Your Required Libraries in `requirements.txt`

The more you write your stuff, the more likely you will need some third-party libraries.

When developing local scripts you can simply run 

`pip install somelibrary`

However, as the number of them grows, it becomes a burden keeping track of the libs. That's when you consider using some collection, like a file called [`requirements.txt`](https://pip.readthedocs.io/en/1.1/requirements.html). We also adopted this approach for centralizing the dependencies and fetching them before **bundling** our code.

### The `buildpec.yaml` file

By **bundling** I mean **bundling**. Really! Generally, we make use of Python as an [interpreted language](https://en.wikipedia.org/wiki/Interpreted_language) in our daily lives. So, compiling into an executable out of our Python project is not generally the case. But we are expected to collect the dependencies of our code, put it in a [deployment package](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html) and then push the result as a Lambda function -we cannot give requirements list to Lambda and want it to download.

So, we keep those preparation logic in a file called `buildspec.yaml`. Just like `lambda_function.py` and `requirements.txt`, it must be located in the root level of the repo.

The name `buildspec` is not an outcome of our fancy marketing/brainstorming sessions but a default convention from [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html). We are going to dig into CodeBuild when we are defining the pipeline, but we should just be aware that a `buildspec.yml` has [pre-defined keys & syntax](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax) and something like follows will help us collect out requirements and state that an [aws-cli command (package)](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/package.html) will be used to package (or bundle :) our project:

```yaml
# buildspec.yml
version: 0.2

phases:
  install:
    commands:
      # Install/upgrade pip and AWS CLI 
      - pip install --upgrade pip awscli
      # Install the packages required
      - pip install -r requirements.txt -t .
  build:
    commands:
      # LAMBDA_ARTIFACT_STORE_BUCKET should be an environment variable in AWS::CodeBuild::Project
      - aws cloudformation package --s3-bucket $LAMBDA_ARTIFACT_STORE_BUCKET --template-file template.yaml --output-template-file output-template.yaml
      
artifacts:
  type: zip
  files:
    - template.yaml 
    - output-template.yaml
```

Here, LAMBDA_ARTIFACT_STORE_BUCKET is defined as an environment variable. To see where it is defined, we are again bound to wait for our CI/CD pipeline section.

### The `template.yaml` file

The `aws cloudformation package` command in the previous section takes an argument like `--template-file`. Now, we are going to create a file to act as a CloudFormation template for the function itself (**not the pipeline** yet). You can check [`AWS::Serverless::Function`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html) documentation to deep diver, but with this following example we can declare our function will:

- have permission to logs,

- make use of an environment variable called `BACKEND_SERVICE_BASE_URL`. It is just a dummy value to depict how environment variables are specified when creating a Lambda function not by console, by local AWS CLI commands, but through Cloudformation.

```yaml
AWSTemplateFormatVersion : "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: A sample SAM template for deploying Lambda functions.
Parameters:
  FunctionName:
    Type: String
  BackendServiceBaseURL:
    Type: String
  BackendServiceApiKeySecretArn:
    Type: String
  Environment:
    Type: String
  Application:
    Type: String
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Ref FunctionName
      CodeUri: ./
      Runtime: python3.8
      Handler: lambda_function.lambda_handler
      Environment:
        Variables:
          BACKEND_SERVICE_BASE_URL: !Ref BackendServiceBaseURL
          BACKEND_SERVICE_API_KEY_SECRET_ARN: !Ref BackendServiceApiKeySecretArn
      Policies:
        - Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - "logs:*"
                - "secretsmanager:GetSecretValue"
              Resource: "*" 
      Tags:
        Environment: !Ref Environment
        Application: !Ref Application
```

The `Parameters` will make more sense when we head for CI/CD pipeline. Which is, indeed, the next!

![A stack of files. Blurred.](https://dev-to-uploads.s3.amazonaws.com/i/woxc5k659m4cfb5th2bi.jpg)

## Part #2: Preparing Cloudformation templates

We are going to use [nested stacks](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-nested-stacks.html) to have a slightly easier to maintain structure.

The templates can be found here https://github.com/vahdet/func-cicd-pipeline-cfn-template . Again, you can fork it or copy the following functional ones in it:

- `main.yaml` (parent)

- `iam.yaml` (child)

- `storage.yaml` (child)

- `pipeline.yaml` (child)

### Prework

Create a S3 Bucket via the console (https://aws.amazon.com/console/). Make sure it is in the same region you want your resources and the Lambda function reside.

Upload those four files to this bucket.

Note down the name of the bucket, we are going to use it in the path to find the child stacks (Yes, from the inside, Cloudformation is [not aware](https://stackoverflow.com/a/52750111/4636715) of the bucket it gets the stack template files from).

### Create stack

- Navigate to Cloudformation page in AWS console.

- Create stack selecting the full URL to the `main.yaml` (e.g. `https://${TemplateS3BucketName}.s3.${AwsRegion}.amazonaws.com/main.yaml`) in your S3 bucket in the previous section.

## Part #3: A CI/CD pipeline built with Cloudformation for the Cloudformation templates of the function 🐍🐍🐍

Nah, We are not building an Ouroboros here. But, if you know a wise way to publish our Cloudformation templates to their S3 buckets; do not hesitate to post it.

Cheers!