Getting S3 bucket size different ways
Tags: aws, s3
Published: 2019-12-28 01:14:01

---
title: Getting S3 bucket size different ways
published: true
description: How to get AWS S3 bucket size in most suitable way
tags: AWS, S3
---

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/soown2nxay87qlrboxhx.png)

In this post I will show a few methods of how to check AWS S3 bucket size for bucket named `my-docs`. You can change this name to any existing bucket name which you have access to.

## S3 CLI
All you need is [AWS CLI](https://aws.amazon.com/cli/) installed and configured.

```
aws s3 ls --summarize --human-readable --recursive s3://my-docs
```

It prints output like this one:


```

2019-03-07 12:11:24   69.7 KiB 2019/01/file1.pdf
2019-03-07 12:11:20  921.4 KiB 2019/01/file2.pdf
2019-03-07 12:11:16  130.9 KiB 2019/01/file3.pdf

Total Objects: 310
Total Size: 121.7 MiB
```

The output looks similar to bash `ls` command.

You can than cut total size from the output, e.g.:

```
aws s3 ls --summarize --human-readable --recursive s3://my-docs \
| tail -n 1 \
| awk -F" " '{print $3}'
```

`tail` will get last line of the output
`awk` will tokenize line by space and print third token which is bucket size in MB.


## CloudWatch metric 

```
aws cloudwatch get-metric-statistics --namespace AWS/S3 \
| --start-time 2019-07-07T23:00:00Z \
| --end-time 2019-10-31T23:00:00Z \
| --period 86400 \
| --statistics Sum \
| --region us-east-1 \
| --metric-name BucketSizeBytes \
| --dimensions Name=BucketName,Value="my-docs" Name=StorageType,Value=StandardStorage \
| --output text \
| sort -k3 -n | tail -n 1 | cut -f 2-2
```

What is happening in the above command?
 
AWS CLI CloudWatch method `get-metric-statistics` prints metric data points from range `start-time` till `end-time` with period every 86400 sec (which is 24h, the `period` depends on the time frame, see [docs](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) for more info). 

The bucket name is `my-docs` and we want output to be printed in plain text (we could as well choose json).

The output printed would look like: 
```
DATAPOINTS	127633754.0	2019-10-17T23:00:00Z	Bytes
DATAPOINTS	127633754.0	2019-08-13T23:00:00Z	Bytes
DATAPOINTS	127633754.0	2019-07-07T23:00:00Z	Bytes
DATAPOINTS	127633754.0	2019-10-03T23:00:00Z	Bytes
```

The third column is a timestamp. Data is unordered and we are interested in the most current size of the bucket so we should sort output by this column: `sort -k3 -n` will sort output by 3rd column.

Finally, we want to take second column which is bucket size in bytes.

`tail -n 1` takes last line of the output
`cut -f 2-2` will cut the line from 2nd to 2nd column, in other words it takes only the column we are interested in.


This method of fetching bucket size is error prone because data points are present only for time frames when data was actually changed on s3 so if you have not modified data through last month and you request metrics from this period you won't get any.

##  AWS S3 job (inventory)
This is a feature provided by AWS - an inventory report. It allows to configure a scheduled job which will save information about S3 bucket in a file in another bucket. The information, among others, can contain the size of objects in source bucket.


[This](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-inventory.html) instruction explains how to configure AWS S3 inventory manually on AWS console.


To wrap up, the first option looks like the easiest one from command line, but other options are worth to know too. 

They may serve better for particular use case. 

E.g. if you wanted to see how bucket size changed over time period the 2nd method would be more suitable, but if you'd like to get a report with bucket size on regular basis the third seems easier to implement. You could listen on new report object in the second bucket and trigger lambda function on object-created event to process the report (maybe notify user by email).