How to sync EFS and S3 every 5 min in AWS
Tags: aws, efs, s3, python
Published: 2021-11-04 11:10:54

**DataSync** is a powerful tool to move data between different AWS storage options like S3, EFS, and EFx. However, there is a catch, you can only run a scheduled task [every hour](https://docs.aws.amazon.com/datasync/latest/userguide/task-scheduling.html) (Nov 2021), you cannot create a custom `cron` expression for a lower time like `*/5 * * * *`. My guessing about this restriction is that this feature was planned for Data Warehousing, not for active synchronization.

My challenge started when I had to read some XMLs using _RDS for SQL Server_. RDS for SQL Server can read the files from [S3 natively](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/User.SQLServer.Options.S3-integration.html), but my files came from several micro-services running in Fargate that have only access to EFS as a volume (for S3, you must use a [3rd party plugin](https://github.com/s3fs-fuse/s3fs-fuse) that requires your container to run as _privileged_ and this is [not authorized](https://github.com/aws/containers-roadmap/issues/1000#issuecomment-920408603)). These files came from external services at different times of the day and represented several gigas to transfer.

In the beginning, I was trying to find a way to read the EFS from SQL Server but it didn't work. RDS doesn't have an option to read EFS because it runs in Windows, and there is not a Linux option available yet, which could potentially give us access to EFS.

After several failed attempts, I created a workaround that involves:

1. A **DataSync** task for creating the basic task and synchronizing the data.
2. A **Lambda** function for running the task.
3. An **EventBridge** rule for triggering the Lambda function every 5 min.

## DataSync

### Step 1:

Configure your data source (EFS, for instance):

![Step 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9vcjl3pzlvfku7zj4ybo.jpg)

### Step 2:

Choose the destination (S3, for instance):

![Step 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x21vc3cd1c3vilj38tk5.jpg)

### Step 3:

Configure what you want to move.

### Step 4:

Review your new task and create it.

***

## Lambda

This is the Python script that I wrote:

```python
import boto3

client = boto3.client('datasync', region_name='YOUR_REGION')

def lambda_handler(event,context):

	response = client.start_task_execution(
		TaskArn='arn:aws:datasync:YOUR_REGION:YOUR_USER_ID:task/YOUR_TASK_ID'
	)
```
 
Where,

- **YOUR_REGION** is the location where you want to run it like _eu-west-1_.
- **YOUR_USER_ID** is the user that is going to run the task.
- **YOUR_TASK_ID** is the task ID created in the DataSync.

***

## EventBridge

Create a new rule that runs in your expected schedule.

### Step 1:

Create a new rule:

![New rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sdiehup6n5ir9yqzxsri.png)

### Step 2:

Configure your schedule:

![Step 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/86t9q5qs4z0gc26fel8w.png)

### Step 3:

Choose your lambda function:

![Step 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o7imosoub1yqeb51ieka.png)

### Step 4:

Review your new rule and create it.

And that's all. Now, you can run your task `DataSync` in your required schedule.

### Follow me on:

|  Personal  |  LinkedIn  |YouTube|Instagram|Cyber Prophets|Sharing Your Stories|
|:----------|:----------|:------------:|:------------:|:------------:|:------------:|
|[![Personal](https://raw.githubusercontent.com/FANMixco/federiconavarrete/master/img/favicons/favicon.png)](https://federiconavarrete.com)|[![LinkedIn](https://i.stack.imgur.com/idQWu.png)](https://www.linkedin.com/in/federiconavarrete)|[![YouTube](https://i.stack.imgur.com/CFPMR.png)](https://youtube.com/c/FedericoNavarrete)|[![Instagram](https://i.stack.imgur.com/PIfqY.png)](https://www.instagram.com/federico_the_consultant)|[![RedCircle Podcast](https://i.stack.imgur.com/4XICF.png)](https://redcircle.com/shows/cyber-prophets)|[![RedCircle Podcast](https://i.stack.imgur.com/4XICF.png)](https://redcircle.com/shows/sharing-your-stories)|

[![sponsor me](https://raw.githubusercontent.com/FANMixco/Xamarin-SearchBar/master/bmc-rezr5vpd.gif)](https://www.buymeacoffee.com/fanmixco)