Using Parameter Store for Workflow on AWS Step Functions
Tags: awsstepfunctions, awsparameterstore
Published: 2019-06-08 09:10:21

---
title: Using Parameter Store for Workflow on AWS Step Functions
published: true
description: 
tags: aws-step-functions, aws-parameter-store
---


Workflow in AWS Step Functions is represented by a state machine which consists of a number of states (steps), their relationships and their input and output. When we are constructing the state machine using [Amazon States Language](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html), we might define the application secrets as part of the state machine code. 

One of the possible situations is that we are using a `Pass` state to prepare an input JSON for the next state. In the following example, we are passing the remote FTP username and password to the next state, which is a Lambda function to download files from remote FTP server.
 

```
"ftp-download-lambda-config": {
      "Type": "Pass",
      "Result": {
          "ftpHost" : "ftp-server.com",
          "ftpUsername" : "username",
          "ftpPassword" : "password",
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},
```

This will raise a concern if we are to version control the state machine code into any version control system like Git. It is because the application secrets(the FTP username and password) will be exposed to the users who have access to the codebase. As a good security practise, it is advisable to store the secrets in a separate environment, which will ensure that it is safe with limited access and possible encrypted where they are stored.  

There are few prominent services in market: **Vault**(from Hashicorp), **Secrets Manager** and **Parameter Store** (from AWS). Since a managed service is preferred, Vault will not be taken into consideration. Comparing Secrets Manager with Parameter Store, Secrets Manager costs $0.40 per secret per month $0.05 per 10,000 API calls and Parameter Store costs nothing for usage. Although Secrets Manager comes with a secret rotation feature which allows you to automatically rotate API keys, passwords and more, it is not required for our requirement. Therefore, AWS **Parameter Store** is more preferable for this use case. 


So, once we have decided to store our secrets in the AWS Parameter Store, we have to modify our application(Lambda function in this case) to accept the reference of secrets stored in Parameter Store and retrieve the secrets from Lambda function. The change in the state machine code is showed as below:

```
"ftp-download-lambda-config": {
      "Type": "Pass",
      "Result": {
          "ftpHost" : "ftp-server.com",
          "ftpUsername" : "<Reference to username in Parameter Store>",
          "ftpPassword" : "<Reference to password in Parameter Store>",
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},
```

Or we can have additional Lambda function to retrieve the secrets from Parameters Store and pass the output to the next state in the state machine. Thus, the existing Lambda functions/tasks can remain as it is with the usage of new Lambda functions. The usage of new Lambda function is showed as below:

```
"ssm-parameters-getter-lambda-config": {
      "Type": "Pass",
      "Result": {
          "param": [
            "parameters-store-ftpUsername",
            "parameters-store-ftpPassword"
          ],
          "withDecryption": true
      },
      "ResultPath": "$",
      "Next": "ssm-parameters-getter-lambda"
},
"ssm-parameters-getter-lambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:xxx:xxx:function:ssm-parameters-getter-lambda",
      "Next": "ftp-download-lambda"
},
"ftp-download-lambda": {
      "Type": "Task",
      "Parameters" : {
        "ftpHost": "ftp-server.com",
        "ftpUsername.$": "$.parameters-store-ftpUsername",
        "ftpPassword.$": "$.parameters-store-ftpPassword",
      },
      "Resource": "arn:aws:lambda:xxx:xxx:function:ftp-download-lambda",
      "Next": "next-step"
},
```

The `ssm-parameters-getter-lambda` accepts the following as input:
```
{
  "param": [
    "random-input-a",
    "random-input-b"
  ],
  "withDecryption": true
}
```
And produce an object with param name as attribute and value from parameter store. The value can be 
```
{
  "random-input-a": "<Value from Parameter Store>"
  "random-input-b": "<Value from Parameter Store>"
}
```


The Lambda function is open sourced on Github. It can be deployed manually to your AWS account using the source code: index.js or deploy directly from the AWS **Serverless Application Repository** with name: `ssm-parameters-getter-lambda`. Remember to check **Show apps that create custom IAM roles or resource policies** as if this Lambda requires additional role.

{% github mengjiann/ssm-parameters-getter-lambda %}


References:
- https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
- https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html
- https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html