AWS Step Functions - Refactor to native integrations with AWS API
Tags: aws, stepfunctions, lowcode, lambda
Published: 2022-01-10 10:18:08

In this article, I will describe how I got rid of two Lambda functions in my Data Lake loading process.

All thanks to the change announced in October 2021, Step Functions received [native API integrations with over 200 AWS services] (https://aws.amazon.com/blogs/aws/now-ws-step-functions-supports-200-AWS-Services-to-enable-Easier-Workflow-Automation). Yep, I call them *native API integrations* because for me, it's _cloud-native_ low-code solution. AWS refers to them as *AWS SDK Service Integrations*, which is also fine 😉

Anyhow, I must admit that I waited for this particular update since summer.

### What has changed in Step Functions?

Some prelude to the update has been available for a long time. I'm talking about this new way of calling Lambda functions from Step Functions. The difference is only visible in a language defining the machine. In the old configuration, we defined a Lambda call like that:

```YAML
StateName:
  Type: Task
  Resource: !GetAtt functionName.Arn
```
and in the new way like that:
```YAML
StateName:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  Parameters:
    FunctionName: !GetAtt functionName.Arn
    Payload.$: $
```
At first glance, you can notice differences. Instead of your own resource, we use *a universal* client in the `Resource` field, and provide the function name as a parameter.

This change to us - users - had no significance, both excerpts are equivalent. However, I'm eager to speculate that it was a milestone step on the way to providing the native integration of Step Functions with other AWS services via API.

**From now on, we have access to 200 different integrations** that we call similarly:

```YAML
StateName:
  Type: Task
  Resource: arn:aws:states:::aws-sdk:<service>:<action>
  Parameters:
    ParametrApi1: value
    ParametrApi2: value
```
AWS SDK - feels like home 😃

In this way, you can copy a file from S3 (`arn:aws:states:::aws-sdk:s3:copyObject`), run the EC2 instance(`arn:aws:states:::aws-sdk:ec2:startInstances`), and so on. There are myriad possibilities.
 
## But how much does it cost?
And here is the biggest deal: **it costs nothing**!

I repeat: **it's free** 😃  

I'm talking about the SDK API call (if it is free, and most of them are). For the state machine and the operation of the invoked service, of course, we pay as we used to be.

If it does not cost anything, then **it is worth replacing Lambda functions that invoke other services with native integration**. Thanks to that, we're going to save on Lambda cost, but also in terms of implementation (developer's time).
Because, as always: *No code is the best code*.
 
### And how much does it actually take to refactor?
The over mentioned developer's time-savings, I must admit, was not so obvious. When I refactored two functions to native integrations, it took me 3 hours. A lot of time, but we all know that:

![Googling can save you...](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nf9b7mofulr4y16zk7gh.png)

That's how it looked in my case because AWS API does not response with exactly what I needed at the moment. Variable names returned by AWS API must be processed, which is difficult, as there is no Lambda function at your disposal. And I didn't read the documentation 🤣

## Refactoring

In my case, **I changed the left implementation to the right one**. I remind you, the entire process loads data to a Data Lake, hence references to the Glue service. In your case, this may be any other AWS service.

![State machines](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0pwxj09e2wirsq7eywwd.png)

The entire state machine is called with a Payload, where the Glue Crawler name is given. The former Lambda function expected a variable named `crawlerName`, while API method named `StartCrawler` requires `Name`.

That was not so difficult to solve:

```YAML
StartCrawler:
  Type: Task
  Resource: arn:aws:states:::aws-sdk:glue:startCrawler
  Parameters:
    Name.$: $.crawlerName
```
However, a big problem for me was to embrace the output data from this step because `startCrawler` does not return any data, and by default, Step Functions relays the current result to the next step. Two steps further, `GetCrawler` will need a Crawler name to check a condition (*Has Crawler finished running?*).

I solved the problem by reaching for a **global context** of the state machine and using the `ResultSelector` filter to build my own object at the state exit.

```YAML
StartCrawler:
  Type: Task
  Resource: arn:aws:states:::aws-sdk:glue:startCrawler
  Parameters:
    Name.$: $.crawlerName
  ResultSelector:
    crawlerName.$: $$.Execution.Input.crawlerName
```
*This is just an illustrative example and does not include error handling and mapping them to the exit state.*

That mapping realized what I required, an exit state with a variable `crawlerName` and proper value.

The next step was to get the current state of the Crawler using the `GetCrawler` method in the loop, that lasts until it is done. I used a similar solution here as before. The value of `state` is used by the condition (*Has Crawler finished running?*) in a loop, and `crawlerName` is used as an input parameter for the next state `GetCrawler`. `Crawler.State` comes from the results returned by the API `getCrawler`.

```YAML
ResultSelector:
  crawlerName.$: $$.Execution.Input.crawlerName
  state.$: $.Crawler.State
```

## Was it worth changing?

Of course, it was!

Firstly, ***no code is the best code***. Here, we replace the custom Lambda with declarative definition of states. This means that any person in the world knowing Step Functions will understand how it works, without deep-diving into our *beautiful* Lambda code.

Secondly, _practice makes perfect_. ~~Juggling~~ Mapping values on the output is only difficult at the beginning. When you figure out how this works, you get more efficient in writing them. *I hope this article will help you avoid many problems I have encountered* 🙂

Thirdly, **native integrations are so versatile**, that from now on, practically every state machine you will use them. This means that the investment in mastering them will definitely pay off in the future projects.

Fourthly, the declarative code of the states is easier to reuse (*copy-paste*) than the Lambda code.