How to deploy to multiple providers with dynamic credentials at once in Terraform CDK for Python
Tags: terraform, python, aws, cdktf
Published: 2024-03-13 23:55:59


_My organization primarily uses AWS, so this post's code predominantly targets AWS providers but the principles for supporting multiple configurations for dynamic provider credentials in the Terraform CDK for Python should apply to all providers, just consult their specific documentation for details._

---

### Table of contents

- [Terraform Cloud dynamic provider credentials](#terraform-cloud-dynamic-provider-credentials)
- [Configuring dynamic provider credentials for multiple providers](#configuring-dynamic-provider-credentials-for-multiple-providers)
- [Using dynamic credentials for multiple providers in the Terraform CDK for Python](#using-dynamic-credentials-for-multiple-providers-in-the-terraform-cdk-for-python)

---

<a name="terraform-cloud-dynamic-provider-credentials"></a>
## Terraform Cloud dynamic provider credentials

If you use Hashicorp's Terraform and Terraform Cloud products, you may be familiar with a feature they recently added known as **dynamic provider credentials**. If not, it's a mechanism which allows runners in Terraform Cloud (TFC) which execute your infrastructure changes to authenticate to your target cloud platform with scoped, temporary credentials. It uses the OpenID Connect (OIDC) protocol which defines a trust relationship, and in the context of AWS it allows the TFC runners to assume a configured role. This post assumes a familiarity with this authentication method. More information can be found here:

- [Dynamic provider credential documentation](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials)
- [Tutorial about setting up dynamic provider credentials](https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials)

For those unfamiliar, the gist of this setup for AWS is:
1. [Create an identity provider in the IAM service of each of your target AWS accounts.](https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials#configure-trust-with-your-cloud-provider)


2. [Create a role with sufficient permissions that Terraform Cloud runners can assume within your AWS account and specify this role in the trust relationship.](https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials#apply-configuration)


3. [Declare these two environment variables in the remote configuration for your Terraform Cloud workspace](https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials#configure-trust-variables):
    - `TFC_AWS_PROVIDER_AUTH` = `true`
    - `TFC_AWS_RUN_ROLE_ARN` = ARN of the IAM role you created in the above step, e.g. `arn:aws:iam::123456789010:role/TerraformCloudDynamicProviderCredentialsRole`


Here's a diagram I made showing the high-level process for how the TFC runner assumes the given role to execute infrastructure changes:

![Dynamic provider credentials authflow diagram](https://media.dev.to/cdn-cgi/image/fit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fw0xc188q2gxiva0zzb4x.png)
<center><sub><em>Here's <a href="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w0xc188q2gxiva0zzb4x.png">a link to a full-sized version of this image</a> if the preview is too small.</em></sub></center>


<a name="configuring-dynamic-provider-credentials-for-multiple-providers"></a>
## Configuring dynamic provider credentials for multiple providers

An early limitation with this authentication setup was that it only supported a single instance of a given provider, which meant Terraform workspaces could only target a single AWS account at a time using this authentication method. Hashicorp eventually implemented support for configuring dynamic provider credentials for multiple providers simultaneously[^1]. This means it's possible to now design workspaces that manage infrastructure state in multiple AWS accounts for example, instead of having to split these deployments into separate workspaces.

This is a useful feature in my organization namely because we use the AWS IAM Identity Center (formerly AWS Single Sign-On) to allow our customers to authenticate to AWS with their organizational identity using SAML. As part of our workflow, we preferred to provision a client's infrastructure in a new dedicated account in our AWS organization, and then create corresponding Single Sign-On (SSO) roles that they use to login to that account. These SSO roles must be managed in the AWS organizational root account[^2], meaning we can now design workspaces that:
1. Create the dedicated AWS organizational account
1. Create infrastructure in this new account
1. Generate policies to access this infrastructure in this account
1. **_And_** now create an SSO permission set in the root account and associate the aforementioned customer managed IAM policies in the new AWS account to this permission set

**And all this in a single invocation of `apply`!**
This greatly simplified our workspace design and coordinating state for a single customer project.

### TFC workspace configuration for multiple dynamic provider credentials

Here are Hashicorp's official docs regarding this setup: <https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/specifying-multiple-configurations>

Configuring dynamic credentials for multiple providers still uses the workspace environment variables mentioned above, however you will now create multiple pairs of these environment variables that share a common underscore (`_`) delimited suffix called a "tag" in the [documentation](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/specifying-multiple-configurations). In my testing I found it was necessary to make these "tags" match identically to the provider aliases I specified in my code.

For the following example, let's say I have two target AWS accounts that are pre-configured to support dynamic provider credentials[^3]. The account with ID 012345678910 will be labelled "A" and the account with ID 109876543210 will be labelled "B".

I would likely use the following AWS provider blocks in my Terraform HCL:
```terraform
provider "aws" {
  alias = "A"
  allowed_account_ids = ["012345678910"]
  ...
}

provider "aws" {
  alias = "B"
  allowed_account_ids = ["109876543210"]
  ...
}
```
Corresponding to these provider blocks, you would extend the TFC workspace environment variables accordingly:

| :information_source: Note that these must be *environment* variables, not *Terraform* variables |
|-----------------------------------------|

<br/>

| Variable name | Variable value |
|---------------|----------------|
|  <code>TFC_AWS_PROVIDER_AUTH<b>_A</b></code> | `true` |
|  <code>TFC_AWS_RUN_ROLE_ARN<b>_A</b></code>  | <code>arn:aws:iam::<b>123456789010</b>:role/TerraformCloudDynamicProviderCredentialsRole</code> |
|  <code>TFC_AWS_PROVIDER_AUTH<b>_B</b></code> | `true` |
|  <code>TFC_AWS_RUN_ROLE_ARN<b>_B</b></code>  | <code>arn:aws:iam::<b>109876543210</b>:role/TerraformCloudDynamicProviderCredentialsRole</code> |

Another important configuration change is also needed:

**You must also add a specific Terraform variable to your workspace code to reference which set of provider credentials your resources should use.**

Hashicorp calls this out in their docs here, and below I copy-paste verbatim the variable configuration for AWS providers on this page: <https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration#required-terraform-variable>

```terraform
variable "tfc_aws_dynamic_credentials" {
  description = "Object containing AWS dynamic credentials configuration"
  type = object({
    default = object({
      shared_config_file = string
    })
    aliases = map(object({
      shared_config_file = string
    }))
  })
}
```

And then by adapting their [sample AWS provider definitions using this variable](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration#example-usage), we would edit the above AWS provider codeblock like so:

```diff
provider "aws" {
  alias = "A"
  allowed_account_ids = ["012345678910"]
+ shared_config_files = [var.tfc_aws_dynamic_credentials.aliases["A"].shared_config_file]
  ...
}

provider "aws" {
  alias = "B"
  allowed_account_ids = ["109876543210"]
+ shared_config_files = [var.tfc_aws_dynamic_credentials.aliases["B"].shared_config_file]
  ...
}
```

And voilà! With this configuration, we can define resources which specify `provider = aws.A` or `provider = aws.B` and can coordinate infrastructure changes in both accounts with the same call to `terraform plan` and `terraform apply`. 

Here's a diagram I made which illustrates the high-level process for how this new configuration coordinates multiple dynamic provider credentials:

![Multiple dynamic provider credentials authflow diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mstl6c6oqrmg4uekch7n.png)
<center><sub><em>Here's <a href="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mstl6c6oqrmg4uekch7n.png">a link to a full-sized version of this image</a> if the preview is too small.</em></sub></center>


<br/>

<a name="using-dynamic-credentials-for-multiple-providers-in-the-terraform-cdk-for-python"></a>
## Using dynamic credentials for multiple providers in the Terraform CDK for Python

#### Backgound

My organization was previously using the AWS CDK for Python so we ended up building many supporting modules with battle-hardened business logic also in Python. Naturally we were hesitant to discard these mature libraries during our migration to using Terraform, so we had an interest in using the [Terraform CDK for Python](https://developer.hashicorp.com/terraform/cdktf) (also called `cdktf`) so that we could still use these Python libraries for common infrastructure patterns we support among our customers.

In the process of adapting the configuration for dynamic  credentials for multiple providers to the Python `cdktf`, I personally found Hashicorp's documentation on this subject to be somewhat underdeveloped, at least as of writing. There are many examples in HCL, but the Python `cdktf` docs seem to still mostly be auto-generated and don't currently have a lot of examples to reference.

Given this, to hopefully spare others the trial and error I endured, I wrote this post to share a sample of how to configure dynamic credentials for multiple providers in the Python `cdktf` for anyone else hoping to take advantage of these technologies.

#### Python `cdktf` Code


First, let's translate the above AWS providers for our 2 accounts into Python `cdktf` code since the translation is relatively straighforward:

```python
import cdktf
from constructs import Construct
from cdktf_cdktf_provider_aws.provider import AwsProvider
  
class AppStack(cdktf.TerraformStack):
  def __init__(self, scope: Construct, stack_id: str):
    super().__init__(scope, stack_id)
  
    aws_account_A_id    = '012345678910'
    aws_account_A_alias = 'A'
    
    aws_account_B_id    = '109876543210'
    aws_account_B_alias = 'B'

    aws_provider_A = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_A_alias}",
                                       alias = aws_account_A_alias,
                                       allowed_account_ids=[aws_account_A_id])
    
    aws_provider_B = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_B_alias}",
                                       alias = aws_account_B_alias,
                                       allowed_account_ids=[aws_account_B_id])
```

And now for the trickiest part, we will need to translate the above HCL definition for the required Terraform variable `tfc_aws_dynamic_credentials` into `cdktf` code. Since the variable is only defined at runtime on a TFC runner, it's not possible to use Python-native value-accessing syntax. Instead, you must use the `cdktf` built-in remote state management functions.

Here's the fully translated variable declaration:

```python
tfc_aws_dynamic_credentials =(
  cdktf.TerraformVariable(
    self,
    "tfc_aws_dynamic_credentials",
    type = cdktf.VariableType.object(
      {
        "default": cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING}),
        "aliases": (
          cdktf.VariableType.map(
            cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING})
          )
        )
      }
    )
  )
)
```

If you look closely this translation is still mostly straightforward, with the main difference being the use of `cdktf` classes to represent keywords in HCL like `object`, `map`, etc.

Next we will need to use the `cdktf.Fn` submodule to get execution-delayed accessors for the variable to be applied once it's defined at runtime.

For example, for the above providers in HCL above the syntax to access the `shared_config_file` attribute of the `tfc_aws_dynamic_credentials` variable is:

```terraform
var.tfc_aws_dynamic_credentials.aliases["B"].shared_config_file
```

In Python we can't directly access the `aliases` attribute of this variable (including any of the remaining nested attributes) since the Python interpreter is only ever running during "synthesis" time, meaning the variable's contents are not populated for Python to manage. In other words, `tfc_aws_dynamic_credentials.aliases` will just yield an `AttributeError`.

Instead we need the `cdktf.Fn.lookup` function, or even better the `cdktf.Fn.lookup_nested` function which will delay the `alias` attribute lookup until runtime on the TFC runner. Using that function and adapting the syntax yields:

```python
cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                       ["aliases", "B", "shared_config_file"])
```

This syntax is a bit of a "line-full" so I chose to make a helper function which accepts the AWS provider alias as the only argument:

```python
def get_aws_dynamic_config_file_from_variable(provider_alias: str):
  return cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                                ["aliases", provider_alias, "shared_config_file"])
```
<sub><em>Note that the `shared_config_files` property of AWS providers expects a list object so you will need to emplace the return value of this function as a list element in the provider declaration.</em></sub>


Putting this all together in one `cdktf` Stack, we get the following code:

```python
import cdktf
from constructs import Construct
from cdktf_cdktf_provider_aws.provider import AwsProvider

class AppStack(cdktf.TerraformStack):
  def __init__(self, scope: Construct, stack_id: str):
    super().__init__(scope, stack_id)
    
    tfc_aws_dynamic_credentials = (
      cdktf.TerraformVariable(
        self,
        "tfc_aws_dynamic_credentials",
        type = cdktf.VariableType.object(
          {
            "default": cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING}),
            "aliases": (
              cdktf.VariableType.map(
                cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING})
              )
            )
          }
        )
      )
    )
    
    def get_aws_dynamic_config_file_from_variable(provider_alias: str):
      return cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                                    ["aliases", provider_alias, "shared_config_file"])


    aws_account_A_id    = '012345678910'
    aws_account_A_alias = 'A'
    aws_account_A_shared_config_file = get_aws_dynamic_config_file_from_variable(aws_account_A_alias)

    aws_provider_A = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_A_alias}",
                                       alias               = aws_account_A_alias,
                                       allowed_account_ids = [aws_account_A_id],
                                       shared_config_files = [aws_account_A_shared_config_file])
    
    aws_account_B_id    = '109876543210'
    aws_account_B_alias = 'B'
    aws_account_B_shared_config_file = get_aws_dynamic_config_file_from_variable(aws_account_A_alias)

    aws_provider_B = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_B_alias}",
                                       alias               = aws_account_B_alias,
                                       allowed_account_ids = [aws_account_B_id],
                                       shared_config_files = [aws_account_B_shared_config_file])
    
    ...
```

And with this, you may now freely declare more infrastructure targeting either of these 2 AWS accounts specifically by simply adding the keyword argument `provider = aws_provider_A` or `provider = aws_provider_B`.

Happy Python-wrapped Terraforming! Feel free to post any questions below and I'll be happy to answer the best I can.





[^1]: Hashicorp's announcement for multiple dynamic provider credentials configurations: <https://www.hashicorp.com/blog/terraform-cloud-now-supports-multiple-configurations-for-dynamic-provider-credent>

[^2]: Technically it's possible to [delegate SSO administration to a different organizational account](https://aws.amazon.com/blogs/security/getting-started-with-aws-sso-delegated-administration/) besides the root account (a.k.a. "management" account), making it possible to manage IAM Identity Center principals and role assignments from that account. However, this doesn't remove the requirement for deploying to multiple AWS accounts.

[^3]: In my AWS organization I have written a CloudFormation StackSet which automatically creates a role named _TerraformCloudDynamicProviderCredentialsRole_ with sufficient permissions, along with an OIDC trust connection to our Terraform Cloud organization which grants assume-role access to that role for any new AWS organizational accounts we create. This means I can safely assume that this role already exists in these accounts and that Terraform Cloud can use OIDC to assume it for TFC runs, but you should take care to verify your AWS account(s) have this pre-existing infrastructure.