Unlock the Power of Enhanced Filtering with S3 Batch Operations
Tags: aws, s3, batch
Published: 2024-01-31 07:13:22

## What is Amazon S3 Batch Operations?

- It allows you to perform batch operations on large sets of objects in Amazon S3.
- You can select objects based on prefixes, tags, and metadata.
- Actions that can be performed include copying, archiving, applying access controls, and adding object tags.
- The goal is to simplify and automate repetitive tasks.

<br/>&nbsp;

## Any news on Batch Operations?

- You can now manage objects within an entire bucket, based on prefix, suffix, creation date, or storage class.
- You can quickly apply the operation to all the matching objects.


<br/>&nbsp;

## Benefit of Batch Operations

- You can easily perform one-time or batch workloads:
- Copying objects between staging and production buckets
- Invoking an AWS Lambda function to convert file types
- Restoring archived backups from S3 Glacier storage


<br/>&nbsp;

## Create a replication rule
Set the filtering: file name prefix

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ygeerf4afhe5irkin8fv.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ahwlddexamry3z2yjbf.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/quohe0fh6mw03eu4nwx5.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hc4hire10drfis7ofo7x.png)



<br/>&nbsp;

## Create a CSV inventory configuration for batch operation
Manifest.csv contains all objects' names for the batch operation


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6einsa8od1wbszh7yawp.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ti2py2kz98in5crwz7i.png)



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/47zobi10n92rxah6lgja.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nkuwuki2fjq2rk9sz6wa.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3uidlhq7fq8hftohejwp.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4p5ixgendv5sws9x8ezs.png)

```
generative-ai-cities,aws_team_1.png
generative-ai-cities,aws_team_2.png
generative-ai-cities,aws_team_3.png
generative-ai-cities,aws_team_1_large.png
generative-ai-cities,aws_team_2_large.png
generative-ai-cities,aws_team_3_large.png
```



<br/>&nbsp;

## Create an S3 batch operation using a CSV inventory configuration
Select "copy" to replicate objects to the destination bucket.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m7n4hbvhkhhpm0bdd10s.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/md614qtu9dvdxyyf3w0u.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ozlenu6u0577fvasy2kd.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hlkovuca74pb7x7n4uqp.png)




<br/>&nbsp;

## Create an S3 batch operation using an S3 replication configuration
Filter objects by creation date
You can specify filters to reduce the scope of replicated objects.
These filters work in conjunction with existing filters in your replication configuration.
If no filters are specified, all objects defined by the replication configuration will be replicated.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9tpz31f1b2137nrwdesq.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kgr3hybxnw7bc924t78c.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ltmjikjok1c6gzu7d4k2.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yolgti71b5w8908alnrv.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/86frtvhm4vp7z1uqy5tv.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ed7kvn82a7z8xg5emv42.png)



<br/>&nbsp;

## Full visibility
Monitor the running time and percentage of objects completed.
Receive a detailed completion report with the status of each object.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hqwnmjentaztqhv3sjku.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vr3n0xdpm2koggcpdkq2.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/10im9rbagqdekmbu4m9d.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5n7pc7zwzob9lcgcn5pc.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ngvqmr3qbmkltqhp7pww.PNG)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cxqnay1mipr5e4xx9rdd.png)


<br/>&nbsp;

## Reference

[Amazon S3 Batch Operations now manages buckets or prefixes in a single step](https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-batch-operations-buckets-prefixes-single-step/)

[Replication configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-add-config.html)

[Replication configuration hand-on](https://aws.amazon.com/getting-started/hands-on/replicate-data-using-amazon-s3-replication/#:~:text=Amazon%20S3%20Replication%20is%20an,%2C%20or%20different%2C%20AWS%20Regions.)