Automatic API Key rotation for Amazon Managed Grafana
Tags: aws, grafana, cicd, terraform
Published: 2023-02-26 04:15:00

---
title: Automatic API Key rotation for Amazon Managed Grafana
published: true
description: Amazon Managed Grafana has an unfortunate limitation where API keys created have a maximum expiration of 30 days - making continuous deployments difficult without manual intervention. This post will show you how to write some simple Terraform to automatically rotate an API key that can be used in your CI/CD pipeline.
canonical_url: https://devopstar.com/2023/02/25/automatic-api-key-rotation-for-amazon-managed-grafana
tags: aws, grafana, cicd, terraform
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/69ykbfy6ia2p2bmbd3dq.jpg
published_at: 2023-02-26 04:15 +0000
---

> Please reach out to me on [Twitter @nathangloverAUS](https://twitter.com/nathangloverAUS) if you have follow up questions!

*This post was originally written on [DevOpStar](https://devopstar.com/)*. Check it out [here](https://devopstar.com/2023/02/25/automatic-api-key-rotation-for-amazon-managed-grafana)

Amazon Managed Grafana has an unfortunate limitation where API keys created have a maximum expiration of 30 days. This limitation is quite frustrating if you were trying to automate the deployment of Grafana dashboards and datasources as part of your CI/CD pipeline - as you would need to manually update the API key every 30 days or your deployments would fail.

This problem is exacerbated by the fact that Amazon Managed Grafana API keys are billed out at the cost of a full user license - so you cannot simply create a new API key every time you deploy your dashboards either. I found this out the expensive way when I didn't read the pricing guide properly and created an API key for each deployment ($8 a key).

**Hopefully, Amazon will address this limitation in the future** - but in the meantime, I've written a simple pattern that can be used to automatically rotate an API key every 30 days and store it for use in AWS Secrets Manager. At the end of this post, I outline my plea to Amazon to address this limitation along with some suggestions on how they could do it.

## Overview of the solution

I've opted to build this example in terraform as it is most likely you will be wanting to deploy Grafana dashboards as part of your CI/CD pipeline. Terraform is arguably the best tool for this - however, there is no reason why the code in this post couldn't be adapted to work with other tools such as CloudFormation or AWS CDK.

![Architecture Diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i5bv7p47tcdh74c3qapi.png)

The solution is made up of two components:

1. AWS Secret is created with a rotation lifecycle policy that will trigger a Lambda function every 30 days
2. AWS Lambda Function that will create a new API key in Amazon Managed Grafana and update the AWS Secret with the new key

> It is expected that you will have already created an Amazon Managed Grafana instance (though you can copy the terraform from this post side by side with your existing terraform and it will work).

The source code for this example can be found at [t04glovern/amazon-managed-grafana-api-key-rotation-terraform](https://github.com/t04glovern/amazon-managed-grafana-api-key-rotation-terraform).

## Solution Walkthrough

We'll begin by looking at the python code that is in charge of rotating the Managed Grafana API keys - as understanding how that works will help when we look at the terraform code.

Look at [src/rotate.py](https://github.com/t04glovern/amazon-managed-grafana-api-key-rotation-terraform/blob/main/src/rotate.py) in the source code for this example

The function is going to expect three environment variables to be present

```python
grafana_secret_arn = os.environ['GRAFANA_API_SECRET_ARN']
grafana_api_key_name = os.environ['GRAFANA_API_KEY_NAME']
grafana_workspace_id = os.environ['GRAFANA_WORKSPACE_ID']
```

> **NOTE**: While you don't technically need to pass the secret ARN as it is available in the Lambda context when the function is invoked by secrets manager, I've opted to do so as it makes it easier to understand.

Next, we attempt to delete any existing API keys with the same name as the one we are about to create. This is to ensure that we clean up old API keys and we don't get billed for duplicates (even though you cannot have multiple API keys with the same name).

```python
try:
    grafana_client.delete_workspace_api_key(
        keyName=grafana_api_key_name,
        workspaceId=grafana_workspace_id
    )
except grafana_client.exceptions.ResourceNotFoundException:
    pass
```

Following cleanup, we create a new API key with a 30-day expiration.

```python
try:
    new_api_key = grafana_client.create_workspace_api_key(
        keyName=grafana_api_key_name,
        keyRole='ADMIN',
        secondsToLive=2592000,
        workspaceId=grafana_workspace_id
    )['key']
except botocore.exceptions.ClientError as error:
    logger.error(error)
    return {
        'statusCode': 500,
        'message': 'Error: Failed to generate new API key'
    }
```

The last step is to update the AWS Secret with the new API key.

```python
try:
    secretmanager_client.update_secret(
        SecretId=grafana_secret_arn,
        SecretString=new_api_key
    )
except botocore.exceptions.ClientError as error:
    logger.error(error)
    return {
        'statusCode': 500,
        'message': 'Error: Failed to update secret'
    }
```

Now that we've seen how the Lambda function works, let's look at the terraform code that will deploy it.

### variables.tf

There are two expected variables for this solution to function - You can however substitute these with hardcoded values or references to your own terraform resources instead.

```hcl
variable "name" {
  type        = string
  description = "Named identifier for the workspace and related resources"
}

variable "grafana_workspace_id" {
  type        = string
  description = "The ID of the Grafana workspace to manage"
}
```

### main.tf

The [main.tf](https://github.com/t04glovern/amazon-managed-grafana-api-key-rotation-terraform/blob/main/main.tf) file is where the bulk of the solution is defined. The first thing we do is create an AWS Secret that will store the API key.

```hcl
resource "aws_secretsmanager_secret" "api_key" {
  name = "${var.name}-api-key"
}
```

The next part looks complicated but is required to bundle the python code into a zip file that can be deployed to Lambda. The code is zipped up and stored in a `zip` directory alongside `src` and is not checked into source control.

```hcl
resource "random_uuid" "lambda_src_hash" {
  keepers = {
    for filename in setunion(
      fileset("${path.module}/src/", "*.py"),
      fileset("${path.module}/src/", "requirements.txt"),
    ) :
    filename => filemd5("${path.module}/src/${filename}")
  }
}

data "archive_file" "lambda_zip" {
  depends_on = [
    null_resource.install_dependencies
  ]

  type       = "zip"
  source_dir = "${path.module}/src/"
  excludes = [
    "__pycache__"
  ]
  output_path = "${path.module}/zip/${random_uuid.lambda_src_hash.result}.zip"
}
```

Unfortunately, because AWS Lambda python runtime is not running the most up-to-date boto3 core version - we must force the terraform to install and zip a more recent version of boto3 which complicates this solution quite a lot. If you are reading this post in the future, check out the [AWS Lambda Python Runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) page to see if this is still required. As of writing this, version `boto3-1.20.32` is the latest version available and `boto3-1.26.65` is needed.

```hcl
resource "null_resource" "install_dependencies" {
  provisioner "local-exec" {
    command = "pip install -r ${path.module}/src/requirements.txt -t ${path.module}/src/  --upgrade"
  }

  triggers = {
    dependencies_versions = filemd5("${path.module}/src/requirements.txt")
  }
}
```

Skipping over the IAM role and policy terraform (which I won't explain in this post, but you can find in the source code on GitHub), we can see that the Lambda function is created and provided the environment variables we defined earlier.

```hcl
resource "aws_lambda_function" "api_key_rotation" {
  function_name = "${var.name}-api-key-rotation"

  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256

  handler = "rotate.lambda_handler"
  runtime = "python3.9"
  environment {
    variables = {
      GRAFANA_API_SECRET_ARN = aws_secretsmanager_secret.api_key.arn
      GRAFANA_API_KEY_NAME   = "${var.name}-mangement-api-key"
      GRAFANA_WORKSPACE_ID   = var.grafana_workspace_id
    }
  }

  role = aws_iam_role.api_key_rotation_lambda_role.arn
}
```

With both the lambda and secret created, a Secret manager rotation schedule is created to invoke the lambda function every 29 days.

```hcl
resource "aws_lambda_permission" "secrets_manager_api_key_rotation" {
  statement_id  = "AllowExecutionFromSecretsManager"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.api_key_rotation.function_name
  principal     = "secretsmanager.amazonaws.com"
}

resource "aws_secretsmanager_secret_rotation" "api_key" {
  secret_id           = aws_secretsmanager_secret.api_key.id
  rotation_lambda_arn = aws_lambda_function.api_key_rotation.arn

  rotation_rules {
    automatically_after_days = 29
  }
}
```

If you only need the terraform to deploy an API key - but do not necessarily use it straight away, then you could get away with not including the final part of the terraform code. However, if there is a requirement to use the API key immediately after deployment in the same Terraform stack, then you will need to add a `null_resource` to delay the terraform execution until the secret has been rotated. 

> I set an arbitrary 20 second delay, but if you wanted to be safe you could increase that.

```hcl
resource "null_resource" "api_key_delay" {
  provisioner "local-exec" {
    command = "sleep 20"
  }
  triggers = {
    after = aws_secretsmanager_secret_rotation.api_key.id
  }
}
```

The following terraform can be used to retrieve the API key from the newly created and updated secret.

```hcl
data "aws_secretsmanager_secret" "api_key" {
  depends_on = [
    null_resource.api_key_delay
  ]
  arn = aws_secretsmanager_secret.api_key.arn
}

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = data.aws_secretsmanager_secret.api_key.id
}
```

## Plea to Amazon

In this post, we have seen how to use Terraform to deploy an AWS Secret and Lambda function that will rotate the API key for an Amazon Managed Grafana workspace. We did this to get around some frustrating limitations with the way that Amazon Managed Grafana provides API keys that hopefully will be addressed in the future.

My request to Amazon is for them to make this solution I outlined redundant by doing the following:

1. Provide a way to vend short-lived session tokens for API access against Managed Grafanas API for use in CI/CD pipelines.
2. If this is not possible, provide a way to create API keys that are not tied to a specific user account - but manage the storage and rotation of these keys in Secrets manager similar to RDS: [https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-rds-integration-aws-secrets-manager/](https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-rds-integration-aws-secrets-manager/)
3. If this is not possible, reduce the price of API keys to $0.01 per month so we can use them in CI/CD pipelines without worrying about the cost.

If you've had this same problem and can think of a better way to solve it, please let me know on Twitter [@nathangloverAUS](https://twitter.com/nathangloverAUS) or in the comments below.