database-artifact-factory 0.1.0
An RDS database factory for rapidly creating database copies
License: MIT
Requires Python: >=3.6

## Background

A "database artifact" is a bundle of metadata and code/automation for creating or
recreating another database.

In the case of creating a database, it includes: * specifications for the location of the new database - networking, etc.
* specifications for the hardware of the cluster instances - cpu, memory etc
* specifications for the maintenance of the cluster - backup windows, etc
* the schema for a relational database

In the case of re-creating a database, it includes:
* a locator for the source database
* specifications for the location of the new database - networking, etc.
* specifications for the hardware of the cluster instances - cpu, memory etc
* specifications for the maintenance of the cluster - backup windows, etc

In both cases, a new cluster is created and automation either does the copy or the create.

In the case of Aurora RDS - the copy can be a "copy-on-write" which means even a massive 
production database can be "copied" in a matter of minutes.  The commonality is shared between
the source and target, while deltas to the target are recorded separately.

The "database artifact" produced by this module is a Docker image.  The image
contains all the necessary metadata and automation to do the necessary.  A Docker image
was used because the automation (originally) contains a fairly diverse mix of software components.

## Installing the Database Artifact Factory

### Prerequisites
You will need a Docker engine installed locally.  For more information on how to do this, please
see: https://docs.docker.com/engine/install/

You will need a python 3.7 or greater environment with pip installed.  For more information on installing
python, please see: https://www.python.org/about/gettingstarted/

### Install via pip
```
pip install db-artifact-factory
```


## Build the Database Artifact

### Configuration
In order to build the artifact, a configuration INI file needs to be created with all the necessary
information for the source and target databases.

An example INI file follows:

```
[source_db]
source_db_cluster_id = dbartifact-prod-create-cluster-1k326eivoq3i0

[target_db]
subnet_ids = source
instance_type = db.r4.large
```

The singular published artifact contains all the information to do both the create or the clone.  
This makes for a mix of arguments in the source/target sections - some values are only necessary
for create, and some are only necessary for clone.

* *source_db_cluster_id*
  This is the Aurora RDS database cluster id to clone.  Even when creating a fresh database cluster,
  there should be a "source" database that it is re-creating - just without data.
* *subnet_ids*
  This can be a comma-delimited list of two or more subnet ids to create the target db in, e.g. subnet-1234, subnet-456
  or it can be "source" and discover the subnet-ids from the source db cluster.

  Make sure these subnet_ids are in different AZ.

* *instance_type*
  The instance type of the DB instance created in the cluster, e.g. db.r4.xlarge
* *database_name*
  In the create case, the name of a database to create and apply the liquibase changelog to.  This can be dontcare
  if you only want the artifact to be able to clone.


### Building
To invoke the build process, first make sure there are ambient AWS credentials.

Then run `db-artifact-builder` against the created INI file:
```
db-artifact-builder artifact-config.ini
```

If the build process is successful, the build process will appear in stdout and a Docker image will be created
with the tag `db-artifact`

### Publishing
Once the db-artifact is in the local registry, from here it can be "published" to DockerHub or the internal registry
of choice (e.g. GitHub, Artifactory, etc.).  It is up to the operator to publish the image (and this tool doesn't 
do anything special to support that).

## Converging a Database Artifact
If running locally after a "build", then "convergence" (i.e creating or cloning) is just a matter of running the Docker image with a few execution settings.  If the Docker image has been published in a registry, then a docker pull will be necessary to
make the image accessible.

Ultimately `docker run` is being executed but there is a wrapper CLI script `db-artifact-converge` installed as part of database-artifact-factory that provides a nicer user experience (no fussing with the filesystem mounts and environment variables)

To clarify - installing database-artifact-factory contains both the tools to build and to converge, but an operator v. an end-user
might use only one or the other.

### Clone an Existing Database
The following is the minimal command to clone an existing database.  It presumes a few things:
* that AWS credentials are stored in ~/.aws/credentials
* AWS profile is "default".
* the Docker image db-artifact:latest contains the artifact to converge

BEWARE - you must specify the password, but the username follows from the source cluster
```
db-artifact-converge --db_password thisisafakepassw0rd
```

If any of those defaults are inappropriate, please invoke the `db-artifact-converge -h` to see the other options.

## Future Direction
* Parameterize support for mysql v postgresql
* Add support for other operational configuration items for the RDS cluster
* better progress reporting and troubleshooting mechanisms
* control over stack names