Manage Airflow connections with Terraform and AWS SecretsManager
Tags: airflow, terraform, infrastructureascode, encryption
Published: 2023-08-15 17:57:42


Managing infrastructure as code brings speed, consistency and it makes the software development lifecycle more efficient and predictable. Infrastructure for your ETL/orchestration tool is managed with code - why not manage the secrets that your tool uses with code as well?

This article shows a proof-of-concept implementation of how to manage Airflow secrets through Terraform and keep them committed to a code repository.

Caveats/assumptions:
* Users (developers) on the AWS account don't have permissions to retrieve secret values from SecretsManager  
* Terraform is using a remote state with [appropriate security measures](https://developer.hashicorp.com/terraform/language/state/sensitive-data) in place
* IAM role used by Terraform has relevant permissions to manage a wide range of AWS services and resources

### TL;DR
[Repository with example code](https://github.com/ivica-k/encrypted-airflow-secrets/tree/main).

#### Architecture
![Solution architecture diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w6ezwrv8xytvh205zhcz.png)

#### How it works

![How it works](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zolmel5bay3ns8b2wviq.png)

## Intro to Airflow Connections

Airflow can connect to various systems, such as databases, SFTP servers or S3 buckets. To connect, it needs credentials. Connections are an Airflow concept to store those credentials. They are a great way to configure access to an external system once and use it multiple times.

[More on Connections](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/connections.html#).

A Connection can be expressed as a string; for example, a connection to a MySQL database may look like this:

```
mysql://username:password@hostname:3306/database
```

Airflow understands this format and can use it to connect to the database for which the connection was configured.

Connections can be configured through environment variables, in an external secrets backend (our use case) and in the internal Airflow database.

A centralized way of managing connections becomes a necessity as soon as your proof-of-concept goes live or you are working in a team.

## External secrets backends

Airflow supports multiple [external secrets backends](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/secrets-backends.html), such as AWS SecretsManager, Azure KeyVault and Hashicorp Vault.

Connection details are read from these backends when a connection is used. This keeps the sensitive part of the connection, such as a password, secure and minimizes the attack surface.

### AWS SecretsManager backend

Configuring your Airflow deployment to use AWS SecretsManager is well explained on [this page](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/secrets_manager/index.html#airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend).

Creating AWS SecretsManager secrets with Terraform is done in a simple way:

```
resource "aws_secretsmanager_secret" "secret" {
  name = "my-precious"
}

resource "aws_secretsmanager_secret_version" "string" {
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = "your secret here"
}
```

but committing this to a code repository is a cardinal sin! 

So how do you manage Airflow Connections in such a way that:
* sensitive part of a connection is hidden
* users can manage connections through code and commit them to a repository
* Airflow can use these connections when running DAGs

Read on!

## Encryption (is the) key

> Encryption is a way to conceal information by altering it so that it appears to be random data. [Source](https://www.cloudflare.com/en-gb/learning/ssl/what-is-encryption/)

The AWS Key Management Service (KMS) allows us to create and manage encryption keys. These keys can be used to encrypt the contents of many AWS resources (buckets, disks, clusters...) but they can also be used to [encrypt and decrypt](https://docs.aws.amazon.com/cli/latest/reference/kms/encrypt.html#examples) user-provided strings.

![User encrypts a string](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nimulmt7hp7vurqqkseb.png)

Your users (developers) need a developer-friendly way of encrypting strings without having access to the KMS key. Developers love APIs. Keep your developers happy and give them an API.

In this case, we have an API built with [Powertools for AWS Lambda (Python)](https://docs.powertools.aws.dev/lambda/python/latest/) and [Lambda Function URLs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html).

A custom [Lambda function](https://github.com/ivica-k/encrypted-airflow-secrets/blob/main/functions/kms_encrypt/lambda_function.py) can be used to encrypt or generate and encrypt random strings. This covers two use cases:
* Administrator of an external system has created credentials for us and we are now using them to create an Airflow connection
* We are creating credentials to a system we manage and will use those credentials to create an Airflow connection

Give a string to this Lambda
```
POST https://xxxxxx.lambda-url.eu-central-1.on.aws/encrypt
Accept: application/json

{
"encrypt_this": "mysql://username:password@hostname:3306/database"
}
```

and it returns something like this:

```
AQICAHjTAGlNShkkcAYzHludk...IhvcNAQcGoIGOMIGLAgEAMIGFBg/AluidQ==
```

Completely unreadable to you and me and safe to commit to a repository. (If you recognized it, yes, it is `base64` encoded. [Try decoding it](https://www.base64decode.org/); even less readable!)

## Create secrets with Terraform

That unreadable "sausage" from before can be used with Terraform, given that it has the permission to decrypt using the key that encrypted the original string. 

```
data "aws_kms_secrets" "secret" {
  secret {
    name    = "secret"
    payload = "AQICAHjTAGlNShkkcAYzHludk...IhvcNAQcGoIGOMIGLAgEAMIGFBg/AluidQ=="
  }
}

resource "aws_secretsmanager_secret" "secret" {
  name = var.name
}

resource "aws_secretsmanager_secret_version" "string" {
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = data.aws_kms_secrets.secret.plaintext["secret"]
}
```

Code above will happily decrypt the encrypted string using a KMS key from your AWS account and store the decrypted value in SecretsManager. 

> **Warning:**
>
> It **will store the secret** in the Terraform state - take the [necessary precautions](https://developer.hashicorp.com/terraform/language/state/sensitive-data) to secure it. **Anyone with access** to the KMS key that encrypted the string **can decrypt it**.

Keep your Terraform state secure and your KMS keys secure-er(?).

## Using it in practice

### Encrypt a MySQL connection string that Airflow will use:
```
http -b POST https://xxxxxx.lambda-url.eu-central-1.on.aws/encrypt encrypt_this='mysql://mysql_user:nb_6qaAI8CmkoI-FKxuK@hostname:3306/mysqldb'
{
    "encrypted_value": "AQICAHjTAGlNShkkcAYzHl8C2qXs7f...zaxroJDDw==",
    "message": "Encrypted a user-provided value."
}
```

#### Use the `"encrypted_value"` value with a Terraform module to create a secret

```
module "db_conn" {
  source = "./modules/airflow_secret"

  name             = "airflow/connections/db"
  encrypted_string = "AQICAHjTAGlNShkkcAYzHl8C2qXs7f...zaxroJDDw=="
}
```

after which you get a nice AWS SecretsManager secret.

![AWS SecretsManager secret](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/67kdz6vvp7m83moj63sp.png)

> **Warning:**
>
> Anyone with the `secretsmanager:GetSecretValue` permission will be able to read the secret. Keep access to your AWS SecretsManager secrets secured.


## Configure Airflow to use the AWS SecretsManager backend

One of the great features of Airflow is the possibility to set (and override) configuration parameters [through environment variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html). We can leverage this to configure MWAA so that it uses a different secrets backend:

```
resource "aws_mwaa_environment" "this" {
  airflow_configuration_options = {
    "secrets.backend"               = "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend",
    "secrets.backend_kwargs"        = "{\"connections_prefix\": \"airflow/connections\",\"variables_prefix\": \"airflow/variables\"}"
  }
... rest of the config...
}
```

With the secret in SecretsManager and Airflow configured to use SecretsManager as the backend, we can finally use the secret in a default way.

[Example DAG](https://github.com/ivica-k/encrypted-airflow-secrets/blob/main/infra/dags/example_dag.py) shows that fetching the secret works.

![Airflow logs showing the secret value](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27310rqq45buipptlxil.png)

---

With this proof-of-concept solution we were able to achieve the following:

* sensitive part of a connection is hidden
* users can manage connections through code and commit them to a repository
* Airflow can use these connections when running DAGs

One obvious downside of having encrypted strings in Git is that you can't understand what actually changed:

```
diff --git a/infra/secrets.tf b/infra/secrets.tf
index fe53e5f..7e85d90 100644
--- a/infra/secrets.tf
+++ b/infra/secrets.tf
@@ -9,5 +9,5 @@ module "db_conn" {
   source = "./modules/airflow_secret"
 
   name             = "airflow/connections/db"
-  encrypted_string = "AQICAHjTAGlNShkkcAYzHl8C2qXs7fs5x9gByXim/PPuwt+TuwH8pYZHik8Cx0AZDM+ECML8AAAAnzCBnAYJ...XwF2a8zaxroJDDw=="
+  encrypted_string = "AQICAHjTAGlNShkkcAYzHl8C2qXs7fs5x9gByXim/PPuwt+TuwGhmhBNcePnQmhjrTgozm6rAAAAnTCmgYJK...nLU8TVWkLDUsSfDs="
 }
```

This encrypted approach also doesn't work well for connections that have no secrets in them, for example AWS connections that use IAM roles:

```
aws://?aws_iam_role=my_role_name&region_name=eu-west-1&aws_account_id=123456789123
```

--- 

If you would like to improve the cost-effectiveness of your MWAA setup, give [this article](https://medium.com/apache-airflow/setting-up-aws-secrets-backends-with-airflow-in-a-cost-effective-way-dac2d2c43f13) by Vince Beck a read.

Nice things I said about MWAA in [my thoughts on MWAA](https://medium.com/aws-in-plain-english/my-thoughts-on-aws-managed-workflows-for-apache-airflow-10df7dd6fb4d) 18 months ago are still valid to this day, and the speed of development on the [aws-mwaa-local-runner](https://github.com/aws/aws-mwaa-local-runner) has increased.

Until next time, keep those ETLs ET-elling!