Trigger step functions from lambda after uploading files to S3 using Serverless Framework and Python
Tags: aws, serverless, stepfunction, architecture
Published: 2020-08-14 23:07:48

In this post, I'm gonna show you how to execute a step function from a lambda function after a file is uploaded to an S3 bucket.

You can watch the video version here:
{% youtube KcoLDAhLpbg %}

## Intro
In the image, we can see we have three AWS services, S3, Lambda Functions, and Step Functions.

![Trigger step function](https://dev-to-uploads.s3.amazonaws.com/i/dvj2xtaews4e91a6qb7x.png)
After a file is uploaded to an S3 bucket, AWS will send an event to a lambda function passing the bucket and file name as part of the event.

## Steps
Let's start by creating a serverless application:
```
serverless create --template aws-python3 --path my-state-machine
```
Now let's go inside the created folder **my-state-machine**
```
cd my-state-machine
```
And open it up in your favorite IDE. I'm using [vscode](https://code.visualstudio.com/)
With vscode you can open your project directly from the terminal with the command:
```
code .
```
Now let's open the serverless.yml file and remove all comments for clarity.
After removing all comments we'll have a file similar to this:
```yaml
service: my-state-machine

provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    handler: handler.hello
``` 
Our next step is to create and configure our Lambda function. 
Let's replace the **functions** section with the following snippet:
```yml
functions:
  StateMachineTrigger:
    handler: handler.upload
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-upload-bucket
          event: s3:ObjectCreated:*
    environment:
      MY_STATE_MACHINE_ARN: ${self:resources.Outputs.MyStateMachine.Value}          
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - states:*
        Resource: "*"

```
The **events** section is declaring that every time a file is uploaded (**ObjectCreated**) into our bucket **${self:service}-${self:provider.stage}-upload-bucket** AWS will send a message to our lambda function **lib/trigger.handler** with the bucket and file information.
You can see we are declaring an environment variable **MY_STATE_MACHINE_ARN**. This is for our lambda function to know what is the step function's ARN.

Now we need to create the lambda handler we just declared so lets
replace the file handler.py with the following:
```python
import os
import json
import boto3

my_state_machine_arn = os.environ['MY_STATE_MACHINE_ARN']
client = boto3.client('stepfunctions')


def upload(event, context):
    print(event)
    for record in event['Records']:
        response = client.start_execution(
            stateMachineArn=my_state_machine_arn,
            input=json.dumps(record['s3'])
        )
```
We are using [AWS boto3](https://github.com/boto/boto3) to trigger our step function that we are going to declare in just a second, and so we are creating a **client** for that. We are also declaring a variable **my_state_machine_arn** to access the environment variable we declared in our serverless.yml. And finally, we are using the **start_execution** method to execute our lambda function passing the event's s3 (**event_record['s3']**) object as argument.

Our final step is to declare the step function. 
To do that we are going to use a plugin named [serverless step functions](https://github.com/serverless-operations/serverless-step-functions) so let's add the following to the end of our serverless.yml:
```yml
stepFunctions:
  stateMachines:
    MyStateMachine:
      id: MyStateMachine
      definition:
        StartAt: ValidateImage
        States:
          ValidateImage:
            Type: Pass
            Result: "Valid"
            Next: GenerateThumbnail
          GenerateThumbnail:
            Type: Pass
            Result: "Thumbnail created"
            Next: NotifyUser
          NotifyUser:
            Type: Pass
            Result: "Notification sent"
            End: true

resources:
  Outputs:
    MyStateMachine:
      Description: The ARN of the example state machine
      Value:
        Ref: MyStateMachine            


plugins:
  - serverless-step-functions
  - serverless-iam-roles-per-function

package:
  individually: true
  exclude:
    - '**/*'
  include:
    - handler.py
```
This is just a dummy state machine to simulate a use case when we have do the following tasks 
* ValidateImage: you could validate if the image is from a celebrity, or cats, dogs, etc 
* GenerateThumbnail: generate a thumbnail of the image
* NotifyUser: send an email to the user with validation feedback

The **Outputs** section is to get the step function's ARN we are passing in the lambda environment variable
> It is recommended and a best practice to always use the plugin [serverless-iam-roles-per-function](https://github.com/functionalone/serverless-iam-roles-per-function) for all your serverless applications

Before we deploy our application we need to install our serverless plugins and for that, we need to create a package.json file first, so let's do:
```bash
npm init -f
```
And now we install the plugins:
```bash
npm install serverless-step-functions
npm install serverless-iam-roles-per-function
```
After that, we are ready to deploy our application so let's go ahead and run:
```bash
serverless deploy
```

Now let's go to the AWS console, and upload a file in the bucket created and you will see that your state machine is executed 

if you liked this post don't forget to click on :heart:

# About me
[Raisel Melian’s Twitter](https://twitter.com/raiselmelian)
[Raisel Melian’s Linkedin](https://www.linkedin.com/in/raisel-melian-78831895/)
[Raisel Melian’s YouTube channel](https://www.youtube.com/channel/UCOJei3jcJXhtAuzPLOLtDBg?view_as=subscriber)