Crib-Notes: Offline Delta-Syncs of S3 Buckets
Tags: aws, offline, replication, s3
Published: 2019-05-07 21:56:26

---
title: Crib-Notes: Offline Delta-Syncs of S3 Buckets
published: true
tags: AWS,offline,replication,s3
canonical_url: https://thjones2.blogspot.com/2019/05/crib-notes-offline-delta-syncs-of-s3.html
---

In the normal world, synchronizing two buckets is as simple as doing `aws s3 sync <SYNC_OPTIONS> <SOURCE_BUCKET> <DESTINATION_BUCKET>`. However, due to the information security needs of some of my customers, it's occasionally necessary to perform data-synchronizations between two S3 buckets, but using methods that amount to "offline" transfers.  
  
To illustrate what is meant by "offline":

1. Create a transfer-archive from a data source
2. Copy the transfer-archive across a security boundary
3. Unpack the transfer-archive to its final destination
  
Note that things are a bit more involved than the summary of the process – but this gives you the gist of the major effort-points.  
  
The first time you do an offline bucket sync, transferring the entirety of a bucket is typically the goal. However, for a refresh-sync – particularly for a bucket of greater than a trivial content-size, this can be sub-ideal. For example, it might be necessary to do monthly syncs of a bucket that grows by a few Gigabytes per month. After a year, a full sync can mean having to move tens to hundreds of gigabytes. A better way is to only sync the deltas – copying only what's changed between the current and immediately-prior sync-tasks (a few GiB rather than tens to hundreds).  
  
The AWS CLI tools don't really have a "sync only the files that have been added/modified since `<DATE>`". That said, it's not super difficult to work around that gap. A simple shell script like the following works a trick:  
  

```
for FILE in $( aws s3 ls --recursive s3://<SOURCE_BUCKET>/ | \
   awk '$1 > "2019-03-01 00:00:00" {print $4}' )
do
   echo "Downloading ${FILE}"
   install -bDm 000644 <( aws s3 cp "s3://<SOURCE_BUCKET>/${FILE}" - ) \
     "<STAGING_DIR>/${FILE}"
done

```
  
To explain the above:  
  

1. Create a list of files to iterate:
  1. Invoke a subprocess using the `$()` notation. Within that subprocess...
  2. Invoke the AWS CLI's S3 module to recursively list the source-bucket's contents (`aws s3 ls --recursive`)
  3. Pipe the output to `awk` – looking for any date-string that's newer than the value in `s3 ls`'s first output-column (the file-modification date column) and print out only the fourth column (the S3 object-path)
The output from the subprocess is captured as an iterable list-structure
2. Use a `for` loop-method to iterate the previously-assembled list, assigning each S3 object-path to the `${FILE}` variable
3. Since I hate sending programs off to do things in silence (I don't trust them to not hang), my first looped-command is to say what's happening via the `echo "Downloading ${FILE}"` directive.
4. The `install` line makes use of some niftiness within both BASH and the AWS CLI's S3 command:
  1. By specifying "`-`" as the "destination" for the file-copy operation, you tell the S3 command to write the fetched object-contents to `STDOUT`.
  2. BASH allows you take a stream of output and assign a file-handle to it by surrounding the output-producing command with `<( )`.
  3. Invoking the `install` command with the `-D` flag tells the command to "create all necessary path-elements to place the source 'file' in the desired location within the filesystem, even if none of the intervening directory structure exists, yet."
 
    Putting it all together, the `install` operation takes the streamed `s3 cp` output, and installs it as a file (with mode 000644) at the location derived from the `STAGING_DIR` plus the S3 object-path ...thus preserving the `SOURCE_BUCKET`'s content-structure within the `STAGING_DIR`

Obviously, this method really only works for additive/substitutive deltas. If you need to account for deletions and/or moves, this approach will be insufficient.