How to update your terraformed RDS / Aurora certificates
Tags: aws, terraform, rds, devops
Published: 2020-01-15 00:42:22

This post was prompted by Jeff Barr's recent [Urgent & Important blog post](https://aws.amazon.com/blogs/aws/urgent-important-rotate-your-amazon-rds-aurora-and-documentdb-certificates)

If you use RDS or Aurora you have probably received an email with this subject recently:

**Important Reminder: Update Your Amazon RDS SSL/TLS Certificates by February 5, 2020**

If you operate many AWS accounts or have put this change on the back burner your inbox might look like this:

![Lots of emails](https://thepracticaldev.s3.amazonaws.com/i/1e3g5alim676ge8ery2t.png)

I was a bit concerned when I first saw this communication in October. A _lot_ more people are now concerned as the deadline is looming and the AWS social media / email campaign has ramped up. You even have to dismiss a modal upon logging in to AWS RDS Console at the minute to acknowledge these changes!

## Does this affect you?

The first thing you have to do is figure out if this issue affects you. If you don't make SSL connections to your database this issue probably doesn't affect you.

If you use [IAM Database Authentication](https://aws.amazon.com/premiumsupport/knowledge-center/users-connect-rds-iam/) this definitely affects you!

The first thing to do is to update your application to support the new certificate.

## Update your Application

Note: _If you do not have some kind of sandbox / staging environment it is worth spinning up an RDS / Aurora cluster with your existing terraform code so you can test all this safely_

There are 100s of possible languages / clients you may be using to interact with RDS, so this part is up to you. I will give a lowest common denominator example in Bash towards the end of this post. But you need to update the certificate you use to connect to RDS.

AWS have updated their combined ca bundle to include both the old and new certificate to avoid a chicken and egg scenario. i.e the 

- `rds-ca-2015-root.pem` will only work with RDS instances configured to use the old 2015 certificate
- `rds-ca-2019-root.pem` will only work with RDS instances configured to use the new 2019 certificate

By using the `rds-combined-ca-bundle.pem` certificate in our application we can update the underlying RDS certificate without issue. Those certificates are available in S3. Links as well as additional documentation are available in the [official AWS Documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)


## Terraform Changes

Typically when I get a notification like this from AWS my initial thought is "I hope I can achieve this with Terraform" I have found when you start using Infrastructure as Code you want every single change to be orchestrated by that tool. Out of bounds changes feel dirty and you want everything to be driven by version controlled code. 

Both the `aws_db_instance` and `aws_rds_cluster_instance` resources support the `ca_cert_identifier` argument. You can update each resource to have the `ca_cert_identifier` argument. You probably want to expose this as a variable so that other consumers of this module have a choice. You may still be responsible for this code in 5 years when the 2024 cert is rolled out!


```hcl
resource "aws_db_instance" "db" {
  # truncated for brevity
  tags                    = "${var.tags}"
  apply_immediately       = "${var.apply_immediately}"
  ca_cert_identifier      = "${var.ca_cert_identifier}"
}
```

Let's update our variable to `rds-ca-2019` so that our `aws_rds_cluster_instance` resource is configured to use the new certificate.

```hcl
variable "ca_cert_identifier" {
  default     = "rds-ca-2019"
  description = "Allows you to modify the underlying RDS certificate"
}
```

We can now run a `terraform plan`

At the point you may get an error like this:


```hcl
Error: module.your_module.aws_db_instance.db: "ca_cert_identifier": this field cannot be set
```

Thats because you write best practices terraform and have pinned your provider version! 


```hcl
provider "aws" {
  version = "~> 1.52"
  region  = "${var.aws_region}"
}
```

Your current provider does not have support for updating the `ca_cert_identifier`. We pin terraform providers for the same reason we pin software libraries. To avoid breaking changes and other surprises from upstream. Typically you can head to the GitHub repo for the AWS provider and find out A) If what you are trying to do is supported. B) What provider version it was introduced in.

In this case (and most cases) the provider maintainers are on top of things and support for updating the certificate was introduced in `v2.37.0` of the provider. We can simply bump our minimum provider version

```hcl
provider "aws" {
  version = "~> 2.37"
  region  = "${var.aws_region}"
}
```

Lets do another plan. 

```hcl
~ module.your_module.aws_db_instance.db
    ca_cert_identifier:   "rds-ca-2015" => "rds-ca-2019"

Plan: 0 to add, 1 to change, 0 to destroy.
```

Now we get the expected plan. 

However this change would not be applied immediately. Many changes to RDS resources are applied at the next maintenance window. Depending on the workload this may be the desired approach. We can run `terraform apply`
and make the change. 

You may notice that if you do another `plan` it looks like our change did nothing. That's because the certificate won't be changed until a reboot during the maintenance window. To verify our work we can use the aws cli

```
aws rds describe-db-instances --db-instance-id your-db-identifier | jq '.DBInstances[].PendingModifiedValues'


{
  "CACertificateIdentifier": "rds-ca-2019"
}
```

Here we can see this instance has pending modifications.

### Applying immediately

In this particular case I want to make the change immediately so I can make a connection from my client and ensure the certificate chain works as expected

I recommend exposing an `apply_immediately` variable on modules that create RDS resources.

```hcl
variable "apply_immediately" {
  default     = false
  description = "Should modifications be applied immediately? Otherwise they are applied at next maintenance window"
}
```

It has a sensible default of `false`, but we still have the flexibility to apply changes immediately. In a production environment maybe somebody else on your team has to approve a Pull Request to make this change and your PR body would explain your rationale for the change. After the change is made you would set `apply_immediately` back to `false`


After setting our `apply_immediately` variable to `true` our plan looks like this:


```hcl
~ module.your_module.aws_db_instance.db
    apply_immediately:    "false" => "true"
    ca_cert_identifier:   "rds-ca-2015" => "rds-ca-2019"

Plan: 0 to add, 1 to change, 0 to destroy.
```

For me this apply took 90 seconds. The RDS instance rebooted and now has the new certificate.

Bash example of connecting to an RDS instance with SSL and IAM authentication. The following code should work before and after your certificate upgrade

```bash
wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
TOKEN="$(aws rds generate-db-auth-token --hostname the_host --port 3306 --region eu-west-1 --username db_user)"
mysql -h the_host --port=3306 --ssl-ca=rds-combined-ca-bundle.pem --enable-cleartext-plugin -u db_user --password=$TOKEN"
```

I hope this helps anyone that has terraformed RDS workloads.