Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)
Tags: aws, lambda, dynamodb, serverless
Published: 2020-09-22 01:56:36

## aws-auto-cleanup
### Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

#### Functional Requirements

 - Reduce operational run-time of resources used for a testing\development, or for temporary spin-up of resources
 - Reduce cost\operational expenses
 - Ability to whitelist AWS resources that need to be retained

#### Operating Cost
< $2.00/mo for the following:

 - AWS::Events::Rule 
 - AWS::Lambda::Function 
 - AWS::Lambda::Permission
 - AWS::Logs::LogGroup 
 - AWS::IAM::Role 
 - AWS::S3::Bucket 
 - AWS::S3::Bucket
 - AWS::DynamoDB::Table 
 - AWS::DynamoDB::Table

|Node                                                       |Type                  |Unit                                               |Usage            |Total  |
|-----------------------------------------------------------|----------------------|---------------------------------------------------|-----------------|-------|
|auto-cleanup-settings-prod (DynamoDB-Table)                |provisioned read      |$0.00013 per hour for units of read capacity       |1 units per month|$0.10  |
|auto-cleanup-settings-prod (DynamoDB-Table)                |provisioned write     |$0.00065 per hour for units of write capacity      |1 units per month|$0.48  |
|auto-cleanup-settings-prod (DynamoDB-Table)                |storage               |$0.25 per GB-month                                 |1 GB             |$0.25  |
|auto-cleanup-whitelist-prod (DynamoDB-Table)               |provisioned read      |$0.00013 per hour for units of read capacity       |1 units per month|$0.10  |
|auto-cleanup-whitelist-prod (DynamoDB-Table)               |provisioned write     |$0.00065 per hour for units of write capacity      |1 units per month|$0.48  |
|auto-cleanup-whitelist-prod (DynamoDB-Table)               |storage               |$0.25 per GB-month                                 |1 GB             |$0.25  |
|auto-cleanup-prod (Lambda-Function)                        |1000                  |128MB \* AvgTime \* Invocations per month (Usage)  |10000 invocations|$0.02  |
|ServerlessDeploymentBucket (S3-Bucket)                     |Standard              |$0.023 per GB - first 50 TB / month of storage used|10 GB            |$0.23  |
|auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket)|Standard              |$0.023 per GB - first 50 TB / month of storage used|10 GB            |$0.23  |
|Dynamo Storage                                             |Discount              |First 25GB Free (-$0.25 per GB)                    |\-2              |($0.50)|
|Dynamo Backup                                              |Point In Time Recovery|$0.20 per GB-month (Continuous backups)            |0                |$0.00  |
|Dynamo Backup                                              |OnDemand              |$0.10 per GB-month (On Demand)                     |0                |$0.00  |
|Dynamo Network Outbound Traffic                            |Transfer              |$0.09 per GB (Data Transfer Out)                   |1                |$0.09  |
|Lambda Service                                             |Number of invocations |Invocation call for a Lambda function              |10000            |$0.00  |

#### Diagram
![diagram](https://www.troydieter.com/images/aws_cleanup_diagram.png)

#### Stack

1. [Serverless Stack](https://serverless.com/)
`Framework Core: 2.1.1
Plugin: 4.0.4
SDK: 2.3.2
Components: 3.1.3`
2. [AWS CLI](https://aws.amazon.com/cli/) 
3. [NPM](https://www.npmjs.com/get-npm)

#### Preferred configuration
After you clone the aws-auto-cleanup repository in the next step, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

1. Change the serverless.yml file line 4, to your company name or project name.
2. Within the `auto_cleanup/data/auto-cleanup-settings.json` file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:

            "ttl": {
              "N": "120"
            }

#### Deploy
Following the directions listed on the README, over [at the aws-auto-cleanup Github page!](https://github.com/servian/aws-auto-cleanup)

#### Useful commands

    serverless deploy --region region-example --aws-profile profile-example
    serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

#### Outputs

 - Under CloudWatch, you'll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

`/aws/lambda/auto-cleanup-prod`

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

> | 1600638849212 | [INFO] S3 Bucket
> 'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
> 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849212 | [INFO] S3 Bucket
> 'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
> 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849212 | [INFO] S3 Bucket
> 'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
> 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849212 | [INFO] S3 Bucket
> 'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
> 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849212 | [INFO] S3 Bucket
> '46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346' was created
> 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849268 | [INFO] S3 Bucket '46346-artifacts' was created 505
> days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)   
> | | 1600638849269 | [INFO] S3 Bucket '46346-cpds-infra' was created
> 494 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849269 | [INFO] S3 Bucket '46346-sree52-testbkt' was
> created 430 days ago and has been deleted. (s3_cleanup.py, buckets(),
> line 168)                                                             
> | | 1600638849269 | [INFO] S3 Bucket 'python-myservice-bucket' was
> created 158 days ago and has been deleted. (s3_cleanup.py, buckets(),
> line 168)                                                             
> | | 1600638849269 | [INFO] S3 Bucket 'python-poc-deployment-bucket'
> was created 166 days ago and has been deleted. (s3_cleanup.py,
> buckets(), line 168)                                                  
> | | 1600638849269 | [INFO] S3 Bucket 's3-to-sns' was created 264 days
> ago and has been deleted. (s3_cleanup.py, buckets(), line 168)        
> | | 1600638849269 | [INFO] S3 Bucket
> 'serverless-telegram-bot-serverlessdeploymentbuck-46346' was created
> 236 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849269 | [INFO] S3 Bucket 'swagger-bucket-1' was created
> 199 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)                                                                  
> | | 1600638849269 | [INFO] S3 Bucket 'terraform-bucket-lokesh' was
> created 243 days ago and has been deleted. (s3_cleanup.py, buckets(),
> line 168)                                                             
> | | 1600638849270 | [INFO] S3 Bucket
> 'twilio-voice-test-dev-serverlessdeploymentbucket-46346' was created
> 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
> 168)

#### Performing clean-up

 1. Change the `dry_run` item in the DynamoDB table (`auto-cleanup-settings-prod` - if you kept the default settings name) to false to perform clean-up:

	   `{  "dry_run" : { "BOOL" : false }}`
2. Invoke using:

   ` serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event`

#### Recap
[aws-auto-cleanup](https://github.com/servian/aws-auto-cleanup) helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!