Idempotently Create/Update AWS RDS Instances in Python
Tags: aws, database, python
Published: 2018-10-24 22:01:44

---
title: Idempotently Create/Update AWS RDS Instances in Python
published: true
description: 
tags: AWS,database,Python
---

There are several ways to manage databases via code, for Amazon Web Service's [Relational Database Service(RDS)](https://aws.amazon.com/rds/), you can use [Terraform](https://www.terraform.io/docs/providers/aws/r/db_instance.html), [Cloudformation Templates](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html), or API libraries in your favorite language. While Terraform's `statefiles` are a hassle to managed and sometimes it will delete your database and start from scratch. And CloudFormation templates are too verbose to be managed properly. In this blog post, I'm talking specifically about the Python AWS API library: [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html).


#### Creating a Postgres database
Firstly, we need to set up the `boto` client:
```python
import boto3
from botocore.config import Config

boto_config = Config(retries=dict(max_attempts=20))
client = boto3.client(
    'rds', region_name='us-east-1', config=boto_config
)
```
then let's invoke the [`create_db_instance()`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Client.create_db_instance) method with a set of configurations:
```python
db_vars = {
    "DBName": "db_name",
    "DBInstanceIdentifier": "instance_name",
    "AllocatedStorage": 20,
    "DBInstanceClass": "db.m3.medium",
    "Engine": "postgres",
    "MasterUsername": "username",
    "MasterUserPassword": "password",
    "VpcSecurityGroupIds": [
        "sg-0007c6489efbd9bca",
    ],
    "DBSubnetGroupName": "my-subnet",
    "DBParameterGroupName": "my-parameter-group",
    "BackupRetentionPeriod": 7,
    "MultiAZ": True,
    "EngineVersion": "10.0.1",
    "PubliclyAccessible": False,
    "StorageType": "gp2",
}
client.create_db_instance(**db_vars)
```

#### Updating existing database

Updating database is a bit more complicated because it involves handling [idempotency](https://stackoverflow.com/a/1077421). In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. Why is it important? We don't want to update the same set of configurations on the database again and again, this causes confusion from `boto` and will result in errors.

Part of handling idempotency is reconciliation with existing state. This usually involve computation of differences between desired state and current state. For `boto`, this means whatever we give [`modify_db_instance`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Client.modify_db_instance) must represent this difference. In `Python`, this can simply be done with a Dictionary comprehension:

```python
updatable_boto_params = (
    "AllocatedStorage",
    "DBInstanceClass",
    "DBParameterGroupName",
    "DBSubnetGroupName",
    "EngineVersion",
    "Iops",
    "StorageType",
)

updated_params = {
    param: new_params.get(param)
    for param in updatable_boto_params
    if not current_params.get(param) == new_params.get(param)
}
```
where `updatable_boto_params` is parameters that you allow to update, you can obviously lock this list down to a very small set of parameters for compliance or security purposes. See the complete list of updatable configurations [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Client.modify_db_instance).

Now that we have the change set, we can invoke the `modify_db_instance`:
```python
client.modify_db_instance(**updated_params)
```

#### Waiting for it to finish
Database operations are usually long-running, when performing a series of steps sequentially (ie. create database and create a read replica), you need to wait for the synchronous process to finish. One way to tell if database has finished updating itself is by looking at the `DBInstanceStatus` returned by the ['describe_db_instances'](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Client.describe_db_instances) method. Status of `available` means database has finished whatever it's doing and you can proceed to the next operation.

A better way is to use `boto`'s [RDS waiter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Waiter.DBInstanceAvailable). It polls `describe_db_instances()` every 30 seconds until a successful state is reached by default. But you can customize how frequently you want it to pool database status, and how long before you give up and declare the database not ready.

We can also use Python's [context manager](http://book.pythontips.com/en/latest/context_managers.html) as a nice little wrapper of the waiter, we can also throw whatever exception handling logic into the context manager just to clean up the code:
```python
from contextlib import contextmanager
from botocore.exceptions import WaiterError

@contextmanager
def wait_for_availability(instance_id, delay=120, retries=60):
    " Yield back control after database is available "
    yield True
    try:
        waiter = self.db_client.get_waiter("db_instance_available")
        waiter.wait(
            DBInstanceIdentifier=instance_id,
            WaiterConfig={"Delay": delay, "MaxAttempts": retries},
        )
    except WaiterError as exception:
        LOG.error("DBInstance: %s is not available.", instance_id)
        raise exception
    else:
        LOG.info("DB instance: %s is now available.", instance_id)
    break


with wait_for_availability(instance_id):
    client.modify_db_instance(**updated_params)
```
In the code snippet above, we modify our database and pool for its status every 2 minutes and for a total of 60 times. After that the database is either available and we exit, or still unavailable and we raise an exception.

Database operations are usually time consuming, and guarantee idempotence is a major complexity to deal with. Fortunately, `boto` provides a bunch of very useful APIs to satisfy our needs.