Exploring the "Requester Pays" Feature for AWS S3 Buckets. Use Cases and Cost Analysis
Tags: aws, cloudcomputing, cloudcost, s3
Published: 2024-05-08 01:55:30

As infrastructure architects, we need to manage cloud resources with cost in mind, so understanding features like _Requester Pays_ for AWS S3 is useful for cost optimization.

The _Requester Pays_ feature of AWS S3 buckets is a good option to explore in "_some cases_" because it can potentially help with optimizing the cost of AWS S3 buckets.

Note: Please keep in mind that I used the term "_some cases_" because different use cases will likely require different resources and configurations.

**Examples of such practical cases include:**

**Large Data Sets**: Companies provide large datasets used for training machine learning (ML) models.

**Benefit** from enabling Requester Pays because they can reduce the high costs associated with accessing training datasets for AI and ML.

**Commercial Distribution**: Platforms that offer video editing directly in the cloud, where users stream large video files during the editing process (an example of such an app mostly likely already installed on your phone :) )

**Benefit**: Users will pay for the data they consume, and the platform will provide high-performance editing tools, saving on costs with usage for scalable service offerings.

**Cross-Account Access**: This is when users from other AWS accounts frequently access S3 objects.

**Benefit**: Users will pay the cross-account data transfer costs

## How to enable the Requester Pays feature

Using Terraform, we can create the `aws_s3_bucket_request_payment_configuration` resource for an AWS S3 bucket. For the payer attribute (described later in this blog), we can select `BucketOwner or Requester`.

`BucketOwner` is the default setting for all new S3 buckets if no specific request payment configuration is applied.

Simple terraform code snippet ( of course, not **production-friendly**):

```hcl
provider "aws" {
  region = "us-east-1"
}

variable "request_payer" {
  description = <<-EOD
    (Optional) Specifies who should bear the cost of Amazon S3 data transfer. 
    It can be either BucketOwner or Requester. By default, the owner of the S3 
    bucket would incur the costs of any data transfer. See Requester Pays 
    Buckets developer guide for more information.
  EOD
  type        = string
  default     = "Requester" 
}

resource "aws_s3_bucket" "this" {
  bucket        = "in-n-out-editme"
  force_destroy = false
  tags = {
    foo = "bar"
  }
}

resource "aws_s3_bucket_request_payment_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  payer  = var.request_payer
}

output "bucket_name" {
  value = aws_s3_bucket.this.bucket
}

output "requester_pays_status" {
  value = aws_s3_bucket_request_payment_configuration.this.payer
}
```

Note: `aws_s3_bucket_request_payment_configuration` resource cannot be used with S3 directory buckets.

## View in the AWS Console UI

The result of the above code (**Enabled** feature) can be found in the AWS console under the S3 Bucket overview and navigating to the _Requester Pays UI_ view (refer to the screenshot below)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fnxwsi8ap2xa2rwyz6ty.png)

## Understanding the Cost breakdown

Charges are handled differently in AWS S3 when using the Requester Pays feature compared to the standard S3 pricing model.

Normally, the `bucket owner` pays for all data transfer and request charges associated with the bucket. However, in the Requester Pays model, these costs are shifted to the person or service accessing the data (`requester`).

**Bucket Owner Costs**:

**Storage**: the data stored in the S3 bucket regardless of access patterns.

**Requester Costs**:

**Data Transfer Out**: When data is transferred from the S3 bucket to the internet or another AWS region, the requester pays for the data transfer costs.

**S3 Requests**: for initiated operations, such as PUT, GET, POST, and LIST.

Including when request _authentication_ fails and the _request is anonymous_, both resulting in an HTTP 403 error.

## How are Requesters billed?

When accessing the _Requester Pays_ S3 buckets, **requesters** must include billing details: `x-amz-request-payer: requester` in the request header. This indicates that they agree to pay for the data transfer and request costs. AWS utilizes this header to ensure that costs are billed to the **requester**.

This setup prevents unauthorized **requesters** from being charged without their consent.

**Requesters** also need IAM permissions to ensure that only authorized users can access the data.

## How to View Payment Split in AWS Console

Here are the steps to check how the payment is split for _Requester Pays_ S3 buckets:

**Billing and Cost Management Dashboard
**

- Navigate to the Billing and Cost Management Dashboard.
- Use the **Cost Explorer** to get detailed information about S3.
- Go to **Cost Explorer**: filter your data by service ( which in this case would be Amazon S3)

**Detailed Billing Reports:**

Enable **Detailed Billing Reports** with Resources and Tags to see detailed information about your S3 usage. If you have _Requester Pays_ enabled, these reports will include information on requester-initiated transactions.

Reports can be set to show which bucket incurred which costs so you can see the split between what you pay as the **bucket owner** (for storage and owner-initiated data transfers) and what **requesters** pay (for their data transfers).

**Tagging**:

Add cost allocation tags on your S3 buckets. Once tagged, activate these tags in the **Cost Allocation Tag** section of the Billing Dashboard to include them in your reports.

**Visualizing with AWS QuickSight Integration**:

It is possible to download **Detailed Billing Reports** reports from the S3 bucket specified and analyze them using tools like Amazon QuickSight.

## Summary

When implementing the _Requester Pays_ model on an S3 bucket, **it's important to ensure that potential data users are aware that they will incur charges for their data access**.

It is important to Monitor Access to your S3 buckets at all times. Use S3 access logs or AWS CloudTrail to **monitor who is accessing your S3 buckets** to manage data transfer and associated costs (maybe one day I will write a blog about this setup)

Understanding the _Requester Pays_ model allows all of us, AWS users, to:

- deploy cost-effective data-sharing solutions in AWS

- manage costs associated with data stored in AWS S3

- help to build cost-effective services aligned with organizational or project goals.

**Official documentation link**:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html?icmpid=docs_amazons3_console