AWS Lambda replicate RDS snapshot
Tags: aws, serverless, devops, go
Published: 2023-07-28 11:59:09

A few days ago, I encountered an issue with AWS RDS services, specifically with SQL server databases that could not be backed up using the AWS Backup service. Currently, there are limitations for requesting RDS to backup databases. However, it is still possible to create backup requests for our database using the RDS API.

Therefore, I want to remove the activity of engineers having to create backups manually through API or web console, and instead develop Serverless services to automate this process. This approach can also be extended to solve other issues in the future.

Let’s go from this guideline

## Preparations

Before creating any Lambda function, we need to read a document that explains how to develop them. You can find it [here](https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html). Then, you should have the following.

1. `Golang` version 1.16 or higher
2. `Pre-commit` that use to ensure you are pushing a verified source code to the SCM
3. `Taskfile` the optional, that use to reduce routine step in your development.

## Overview solution is…

We focus on events generated by RDS when a snapshot is created. These events are triggered by AWS EventBridge, which provisions a function in AWS Lambda to execute the necessary actions.

![Lambda with EventBridge](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cuywv0j8ck40hxqelf81.png)

## First thing

You can provide a folder structure for develop a Lambda function in your workspace.

1. Create new Github repository and clone it into your local machine.
    
    ```bash
    git clone <your_repo_url> 
    ```
    
2. Go to your folder and init a new Golang project. it will create a new `go.mod` file.
    
    ```bash
    go mod init <your_repo_url>
    #Example
    go mod init github.com/StartloJ/lambda-rds-utils
    ```
    
3. (Optional) Create a pre-commit policy file to help you improve development quality.
    
    ```yaml
    #.pre-commit-config.yaml
    repos:
    - repo: https://github.com/dnephin/pre-commit-golang
      rev: master
      hooks:
      - id: go-fmt
      - id: go-vet
      - id: go-imports
      - id: go-mod-tidy
    ```
    

## Create a first function

Now, we are ready to develop the Lambda function to handle the event. You can follow the steps below to create a real use case function that replicates the RDS snapshot across regions.

### Remember the event structure from AWS EventBridge

You can see more event on official docs [here](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html).

```json
{
	"version": "0",
	"id": "844e2571-85d4-695f-b930-0153b71dcb42",
	"detail-type": "RDS DB Snapshot Event",
	"source": "aws.rds",
	"account": "123456789012",
	"time": "2018-10-06T12:26:13Z",
	"region": "us-east-1",
	"resources": ["arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24"],
	"detail": {
	  "EventCategories": ["creation"],
	  "SourceType": "SNAPSHOT",
	  "SourceArn": "arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24",
	  "Date": "2018-10-06T12:26:13.882Z",
	  "SourceIdentifier": "rds:snapshot-replica-2018-10-06-12-24",
	  "Message": "Automated snapshot created",
	  "EventID": "RDS-EVENT-0091"
	}
}
```

### Define a configuration for functions

I will define parameters to reuse this function below.

- `OPT_SRC_REGION` is a source region of RDS snapshots for copy.
- `OPT_TARGET_REGION` is a target region that we need to store snapshots.
- `OPT_DB_NAME` is a identity of RDS DB name that we need to copy snapshots.
- `OPT_OPTION_GROUP_NAME` is a target option group to attachment to replicate snapshot in the target region.
- `OPT_KMS_KEY_ID` is a KMS key that use to encrypt snapshot on target region.

### Define main func

```go
package main

import (
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/sirupsen/logrus"
)

func main() {
	// Make the hangler available for remote procedure call by Lambda
	logrus.Info("we starting handle lambda...")
	lambda.Start(HandleEvents)
}
```

### Create a function handle by lambda

```go
// Define func to receive a variable from `BridgeEvent`
// and it should return `error` if any mistake.
func HandleEvents(events BridgeEvent) error {
	return nil
}
```

### Define a new AWS session

```go
// define AWS session for source and target region to copy snapshot
func HandleEvents(event BridgeEvent) error {
	// start copy code
	des_sess := session.Must(session.NewSessionWithOptions(session.Options{
		Config: aws.Config{
			Region: aws.String(viper.GetString("target_region")),
		},
	}))
	src_sess := session.Must(session.NewSessionWithOptions(session.Options{
		Config: aws.Config{
			Region: aws.String(viper.GetString("src_region")),
		},
	}))
	// end copy code
	return nil
}
```

### Create func to get all existing snapshot

```go
// This func input session of RDS service. Then, it get all list of database
// snapshot that already in your resources in List of Snapshot name.
func ListAllSnapshot(svc *rds.RDS) ([]string, error) {

	db := viper.GetString("DB_NAME")

	// This line will return a list of snapshots included snapshot name.
	out, err := svc.DescribeDBSnapshots(&rds.DescribeDBSnapshotsInput{
		DBInstanceIdentifier: &db,
	})
	if err != nil {
		panic(err)
	}

	var snapShotName []string

	// this loop is extract a snapshot name only.
	for _, b := range out.DBSnapshots {
		logrus.Infof("We have %s", ConvertToString(*b.DBSnapshotArn))
		snapShotName = append(snapShotName, ConvertToString(*b.DBSnapshotArn))
	}

	return snapShotName, nil
}
```

### Get a list of RDS snapshot name from source and target region

```go
func HandleEvents(event BridgeEvent) error {
	...
	// start copy code
	targetSvc := rds.New(des_sess)
	rep_dbSnapshots, err := ListAllSnapshot(targetSvc)
	if err != nil {
		return fmt.Errorf("we can't get any Snapshot name")
	}

	srcSvc := rds.New(src_sess)
	src_dbSnapshots, err := ListAllSnapshot(srcSvc)
	if err != nil {
		return fmt.Errorf("we can't get any Snapshot name")
	}
	// End copy code
	...
}
```

### Create a func to remove duplicate snapshot as source and target

```go
// this func use to remove duplicate name source if found in target
// you shouldn't switch position input to this func.
// `t` is mean a target that use double check to `s`
// it will remove a value in `s` if found it in `t`
func GetUniqueSnapShots(t, s []string) ([]string, error) {
	//Create a map to keep track a unique strings
	uniqueMap := make(map[string]bool)

	//Iterate over `s` and add each string to map
	for _, str := range s {
		uniqueMap[str] = true
	}

	//Iterate over `t` and remove any string that are already in uniqueMap
	for _, str2 := range t {
		delete(uniqueMap, str2)
	}

	//Convert the unique string from Map to slice string
	result := make([]string, 0, len(uniqueMap))
	for str := range uniqueMap {
		result = append(result, str)
	}

	if len(result) == 0 {
		return nil, fmt.Errorf("not any Snapshot unique between source and target region")
	}

	return result, nil
}
```

### Define a unique snapshot

```go
func HandleEvents(event BridgeEvent) error {
	...
	// start copy code
	dbSnapShots2Copy, err := GetUniqueSnapShots(rep_dbSnapshots, src_dbSnapshots)
	if err != nil {
		logrus.Warnf("it doesn't any task copy snapshot to %s", viper.GetString("target_region"))
		return nil
	}
	// End copy code
	...
}
```

### Create a func copy RDS snapshot across region

```go
// Request to AWS RDS API for create event copy a specific snapshot
// into across region and encrypt it by KMS key multi-region
func CopySnapshotToTarget(svc *rds.RDS, snap string) (string, error) {
	targetSnapArn := strings.Split(snap, ":")
	targetSnapName := targetSnapArn[len(targetSnapArn)-1]

	// Copy the snapshot to the target region
	copySnapshotInput := &rds.CopyDBSnapshotInput{
		OptionGroupName:            aws.String(viper.GetString("option_group_name")),
		KmsKeyId:                   aws.String(viper.GetString("kms_key_id")),
		CopyTags:                   aws.Bool(true),
		SourceDBSnapshotIdentifier: aws.String(snap),
		TargetDBSnapshotIdentifier: aws.String(targetSnapName),
		SourceRegion:               aws.String(viper.GetString("src_region")),
	}

	_, err := svc.CopyDBSnapshot(copySnapshotInput)
	if err != nil {
		logrus.Errorf("Copy request %s is failed", snap)
		return "", err
	}

	logrus.Infof("Copy %s is created", snap)
	return fmt.Sprintf("Copy %s is created", snap), nil
}
```

### Define loop to execute all snapshot value

```go
func HandleEvents(event BridgeEvent) error {
	...
	// start copy code
	for s := range dbSnapShots2Copy {
		logrus.Infof("trying to copy DBSnapshot to %s...", viper.GetString("target_region"))
		_, err := CopySnapshotToTarget(targetSvc, dbSnapShots2Copy[s])
		if err != nil {
			logrus.Error(err.Error())
		}
	}
	// End copy code
	...
}
```

> If all workflow is complete, it will normal end function from AWS Lambda controller.

## Finally steps

1. If you finished to develop function, you will build and pack it with ZIP format.
    
    ```bash
    export BINARY_NAME="lambda-rds-util" #you can change this name.
    go build -o $BINARY_NAME main.go
    zip lambda-$BINARY_NAME.zip $BINARY_NAME
    ```
    
2. You can upload your compress file(also ZIP file) to S3.
    
    ```bash
    #you can push it with below command
    export S3_NAME="<your_s3_bucket_name>"
    aws s3api put-object --bucket $S3_NAME --key lambda-$BINARY_NAME.zip --body ./lambda-$BINARY_NAME.zip
    ```
    
3. In the web console of AWS Lambda, you can define a new resource and select source from S3.
4. In the web console, you can test function in AWS Lambda to ensure your code is work!.

## Conclusion

You can automate certain activities on AWS or other cloud providers using the `Serverless` service, such as AWS Lambda. It can enhance your development experience and enable you to explore new possibilities. You can even extend this idea to develop a new workflow in your work.

I hope this blog has inspired you to consider using `Serverless` in your development journey.