Python – AWS Secrets Manager: Remote env vars
Tags: python, env, aws, secretsmanager
Published: 2023-05-18 14:33:43

Environment variables play a crucial role in configuring and customizing software applications. They allow developers to store sensitive information securely and adjust the behavior of their code without modifying the underlying source files. In Python, retrieving environment variables is typically a straightforward process.

However, when dealing with remote environments or secrets management, the process can become more complex. This article presents a way to retrieve environment variables, with a particular focus on integrating with AWS Secrets Manager.

### Content
- [Code to be discussed](#code-to-be-discussed)
- [The env function](#the-env-function)
- [The get_env function](#the-get_env-function)
- [The remote_environment_variable function](#the-remote_environment_variable-function)
- [Usage examples](#usage-examples)

### Code to be discussed

The code provided down below consists of three main functions: `env`, `get_env`, and `remote_environment_variable`, read carefully each function's implementation, try to understand its purpose and its role in retrieving environment variables.

```python
import json
import os
from base64 import b64decode

from botocore.exceptions import (ClientError, NoCredentialsError,
                                 ParamValidationError)

from configs import aws
from configs.logging import logger

"""
Retrieves the value of an env variable from a remote source or a .env file.

:type key: str
:type default_value: str
:return: str
"""


def env(key: str, default_value: str = ""):
    value = get_env(key, default_value)

    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False

    return value


"""
Retrieves env variable from .env and AWS Secrets Manager, giving precedence to
remote value. If both sources exist, the remote value is returned; otherwise,
.env value or default_value is used.

:type key: str
:type default_value: str
:return: str
"""


def get_env(key: str, default_value: str = "") -> str:
    local_value = str(os.getenv(key, default_value))
    remote_value = str(remote_environment_variable(key))
    return remote_value if remote_value else local_value


"""
Retrieves env variable from AWS Secrets Manager.

:type key: str
:return: str
"""


def remote_environment_variable(key: str) -> str:
    secret_id = os.getenv("AWS_SECRETS_MANAGER_SECRET_ID")

    if not secret_id:
        return ""

    try:
        response = aws.secrets_manager.get_secret_value(SecretId=secret_id)
    except (ClientError, NoCredentialsError, ParamValidationError) as error:
        if isinstance(error, (NoCredentialsError, ParamValidationError)):
            logger.debug("AWS Secrets Manager: %s", error)
        else:
            message = f"{error.response['Error']['Code']} to secret"
            logger.error(f"{message} {secret_id}: {error}")

        return ""

    if "SecretString" in response:
        secret_dictionary = json.loads(response["SecretString"])
    else:
        secret_dictionary = json.loads(b64decode(response["SecretBinary"]))

    return secret_dictionary.get(key)
```

Now, that the code has been read, let's explore the code in detail.

### The env function

This function is designed to retrieve the value of an environment variable from a remote source or a `.env` file. It provides a practical interface for accessing environment variables, with additional handling for boolean values. By utilizing the `get_env` function internally, it ensures consistent retrieval of variables from the appropriate source.

```python
def env(key: str, default_value: str = "") -> str:
    value = get_env(key, default_value)

    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False

    return value
```

The env function takes two parameters: `key`, which represents the name of the environment variable to retrieve, and an optional `default_value` that is used if the variable is not found.

Within the function, it calls the `get_env` function, passing the key and `default_value` parameters. The return value is assigned to the `value` variable.

To provide additional flexibility, the function checks if the retrieved value is either “true” or “false”. If the value is “true” (case-insensitive), it returns a boolean `True`. If the value is “false” (case-insensitive), it returns a boolean `False`. This allows for more practical handling of boolean environment variables.

If the value is not a boolean, the function returns the original value as a string.

Overall, the `env` function simplifies the retrieval of environment variables by utilizing the `get_env` function internally and providing a consistent and flexible interface for accessing and interpreting their values.

### The get_env function

This function serves as a central piece in retrieving environment variables. It handles the logic to fetch variables from either a local .env file or AWS Secrets Manager. It gives precedence to remote values when available, and falls back to local values or default values if necessary.

```python
def get_env(key: str, default_value: str = "") -> str:
    local_value = str(os.getenv(key, default_value))
    remote_value = str(remote_environment_variable(key))
    return remote_value if remote_value else local_value

```

The `get_env` function is responsible for retrieving the environment variable from either the local `.env` file or AWS Secrets Manager.

Within the function, it first attempts to retrieve the value from the local environment using `os.getenv(key, default_value)`. The `os.getenv()` function retrieves the value of the environment variable with the provided `key`. The retrieved value, whether from the environment or the default value, is then converted to a string using the `str()` function and stored in the local_value variable.

Next, the function calls the `remote_environment_variable(key)` function, which retrieves the value from AWS Secrets Manager. The result is stored in the `remote_value` variable, also converted to a string representation.

Finally, the function returns the `remote_value` if it exists (i.e., is not an empty string) or falls back to the `local_value` if the remote value is not available. This ensures that the function gives precedence to the remote value when both sources exist.

In summary, the `get_env` function provides a unified approach to retrieve environment variables, combining values from a local `.env` file and AWS Secrets Manager. It prioritizes remote values when available, offering flexibility and consistency in handling environment variable retrieval.

### The remote_environment_variable function

This function is responsible for retrieving environment variables from AWS Secrets Manager. It uses the `AWS_SECRETS_MANAGER_SECRET_ID` environment variable to identify the secret in AWS Secrets Manager and fetches the corresponding value.

```python
def remote_environment_variable(key: str) -> str:
    secret_id = os.getenv("AWS_SECRETS_MANAGER_SECRET_ID")

    if not secret_id:
        return ""

    try:
        response = aws.secrets_manager.get_secret_value(SecretId=secret_id)
    except (ClientError, NoCredentialsError, ParamValidationError) as error:
        if isinstance(error, (NoCredentialsError, ParamValidationError)):
            logger.debug("AWS Secrets Manager: %s", error)
        else:
            message = f"{error.response['Error']['Code']} to secret"
            logger.error(f"{message} {secret_id}: {error}")

        return ""

    if "SecretString" in response:
        secret_dictionary = json.loads(response["SecretString"])
    else:
        secret_dictionary = json.loads(b64decode(response["SecretBinary"]))

    return secret_dictionary.get(key)
```

The `remote_environment_variable` function retrieves the value of an environment variable from AWS Secrets Manager. It takes one parameter, key, which represents the name of the environment variable to retrieve.

First, the function retrieves the value of the `AWS_SECRETS_MANAGER_SECRET_ID` environment variable using `os.getenv()`. This value represents the ID of the secret in AWS Secrets Manager that contains all the desired environment variables stored as a JSON.

If the `secret_id` is not found or empty, indicating that the required environment variable is not available, the function returns an empty string.

Next, the function attempts to fetch the secret value using the `aws.secrets_manager.get_secret_value()` method, passing the `SecretId` parameter with the retrieved `secret_id`. Any potential exceptions, such as `ClientError`, `NoCredentialsError`, or `ParamValidationError`, are caught and handled within the `except` block.

If the retrieval is successful, the function checks if the response contains a “SecretString” key. If present, it assumes the secret value is in string format and parses it using `json.loads()`, storing the resulting dictionary in the `secret_dictionary` variable. Otherwise, if the response contains a “SecretBinary” key, it assumes the secret value is in base64-encoded binary format and decodes it using `b64decode()` before parsing it as a dictionary.

Finally, the function attempts to retrieve the value associated with the provided `key` from the `secret_dictionary` using the `get()` method. If the `key` is found in the dictionary, the corresponding value is returned. If not found, `None` is returned.

In summary, the `remote_environment_variable` function provides a way to retrieve environment variables from AWS Secrets Manager by utilizing the `AWS_SECRETS_MANAGER_SECRET_ID` environment variable to identify the secret. It handles potential errors and exceptions during the retrieval process, ensuring robustness and reliable access to sensitive configuration values.

### Usage examples

This function demonstrates the usage of the env function in different scenarios.

```python
from example_module import env

def example_function():
    # Example 1
    database_url = env('DATABASE_URL')
    print(f"The database URL is: {database_url}")

    # Example 2
    api_key = env('API_KEY', default_value='DEFAULT_API_KEY')
    print(f"The API key is: {api_key}")

    # Example 3
    debug_mode = env('DEBUG_MODE')
    if debug_mode:
        print("Debug mode is enabled.")
    else:
        print("Debug mode is disabled.")
```

In the first example it's possible to see a basic environment variable retrieval, it retrieves the value of the 'DATABASE_URL' environment variable and prints it. In the second example, retrieves the value of the 'API_KEY' environment variable with a default value and prints it. The third and last example handles boolean environment variables, it retrieves the value of the 'DEBUG_MODE' environment variable as a boolean and prints the debug mode status accordingly.

### Final thoughts

Using this approach combined with the appropriated AWS EC2 ↔ AWS IAM configuration, it’s possible to centralize all environment variables into AWS Secrets Manager, eliminating the need to deal with an .env file during the deployment.

Therefore, this solution presents two upsides to the application lifecycle; 1) simplification of the deployment due to the elimination of the .env file outside the local environment; 2) better security since you can establish IAM access polices in a way that just specific members of the team can add and modify the Secrets Manager entries.


---

I'm Jack Miras, a Brazilian engineer who works mostly with infrastructure and backend development, for more of my stuff [click here](https://dev.to/jackmiras).

Happy coding!