Find and Count files on AWS S3 by size
Tags: aws, shell, productivity, bestpractices
Published: 2021-10-22 07:06:46

---
title: Find and Count files on AWS S3 by size
published: true
description: search, filter and count files contained in a AWS S3 Bucket using command-line, JQ JMSE path and s3api!
tags: #aws, #shell, #productivity, #bestpractices
---

## TL;DR

If you want to search an AWS S3 Bucket for all the files that are empty take advantage of the [aws s3api](https://docs.aws.amazon.com/cli/latest/reference/s3api/) cli and j[JMESpath queries](https://jmespath.org/) for a simple and effective one-liner:

```
aws s3api list-objects --bucket BUCKET_NAME --output json 
--query 'Contents[?Size==`0`]' > bucket_content-not-empty.json
```

---

While working on a legacy project recently I realised that an AWS S3 Bucket that had been used for years as a recipient for files generated by different **ETLs** ( **extract, transform, load** scripts), was in a very bad shape.

Literally tons of files,  some of them very old, and some of them even empty.

I wanted to understand what we were keeping in that Bucket, how and when it was used in order to do some clean up,  optimise the costs of storage and simplify access and usage of those files, but **navigating the files via UI console was really a pain.**

> How can I get a list of all the files in the bucket and immediately know what are empty?

After some reading in AWS Docs and StackOverflow here are some approaches.


### Recursively list files in the bucket

```
aws s3 ls s3://BUCKET_NAME  --recursive --summarize
```
This solution was the first, a classic, in my case rather slow but was enough to get a list of all the files.

Unfortunately the possibility of filtering and searching where limited, then I discovered that alongside s3,  another CLI exists: s3api which thanks to a **JSON output and JMESPath queries** grants more flexibility and granularity.  

Check this article for more information --> [Leveraging S3 and S3API Commands](https://aws.amazon.com/it/blogs/developer/leveraging-the-s3-and-s3api-commands/)

### Enter S3API

```
aws s3api list-objects --bucket BUCKET_NAME 
--output json --query "[length(Contents[])]"
```

Counts all files in the bucket, while 

```
aws s3api list-objects --bucket BUCKET_NAME 
--output json --query 'Contents[].{Key: Key, Size: Size}' 
```

nicely gives you a json list consisting of Name and Size of the files.

I was just happy with that and from there, I moved to my own territory,  [JQ](https://stedolan.github.io/jq/manual)! 

I just added to the above script the command to save the result to a file ( `> bucket_content.json` ) and then started working with JQ filters (found the [Cookbook](https://github.com/stedolan/jq/wiki/Cookbook#filter-objects-based-on-the-contents-of-a-key) very useful!).


```
cat bucket_content.json | jq -c '.[] 
| select(.Size != 0 )' > content-not-empty.json
```

Now just extract the names only and count them:

```
cat bucket_content-not-empty.json| jq -r '.[] | .Key' | wc -l
```

### Improve the solution

Once I was done with JQ though, I was very curious about the query language used by s3api and [a comment on Stack Overflow](https://stackoverflow.com/a/55910828) blew my mind, [JMESpath queries](https://jmespath.org/) are very powerful ( and very similar to JQ btw).

Here the command that returns all the files that are not empty without piping multiple times and using JQ:

```
aws s3api list-objects --bucket BUCKET_NAME --output json 
--query 'Contents[?Size!=`0`]' > bucket_content-not-empty.json
```

### Where to go from here

What I discover was satisfying and unpleasant at the same time: Out of hundreds of thousands files in the Bucket only just a few thousands had actual content.  

That opens up huge room for improvement and clean up.
- remove all empty files
- change where possible the ETLs to avoid saving files to S3 if after Extract and Transform we realize there is no data
- Or alternatively add some Expiration policy, or rules/lambdas checking and cleaning up empty files regularly
- move very old files which are not used / accessed anymore, but for which we still need some backup to other less expensive solutions like [AWS Glacier](https://aws.amazon.com/it/s3/glacier/). But this is a topic for different post. 


Hope this helped