Storage-first pattern using Serverless
Tags: serverless, aws, lambda, s3
Published: 2024-06-07 15:30:48

Sometimes we have **sensitive requests** which we need to prepeare our systems to avoid losing it. Usually losing these requests mean that the company/product **loses money** or introduce issues in the product/application


## Sensitive requests

- Payment requests
- Delivery requests
- Webhooks with external information


## Non-sensitive requests

- List of users
- List of user-roles
- Endpoints for internal systems

The main difference between **sensitive** and **non-sensitive** request is that sensitive requests in general, you can not re-send it or it's more difficult in comparison with non-sensitive request


The idea behind of storage-first pattern, is basically to **store** the request information **first** ( in a high availability storage ) and **process** it **later** .



## Why serverless is usefull in these cases ? 

Using Serverless technologies like s3, sqs, kinesis, dynamodb gives you high availability without needing to pay for the provisioning. Using serverless storage you only pay per use.

we're going to explain here different use-cases and pattern, and talk a bit about pricing


## Use cases

Here, there are 3 kind of implementation of storage-first pattern : 

### SQS-based Queueing


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/59v9zouzy8pbszpb7ptc.png)



**Ideal for**: Payment processing, order fulfillment, webhook handling.

**How it works**:
- Request is immediately placed in an SQS queue.
- A separate process (Lambda function, ECS task, etc.) continuously polls the queue, processes messages, and performs necessary actions.

**Benefits**
- Decouples request ingestion and processing.
- Ensures high availability and fault tolerance.
- Scales automatically to handle spikes in requests.
- Retry mechanism

**Pricing example** 

Processing per 1 M of requests 

- SQS : $0.4
- API-GATEWAY (http): $1.00
- **Total : $1.4***

* This is an estimate of the pricing because we're not taking care of Data Transfer


### DynamoDB

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/256hov1sbx5kgmkm8mi9.png)


**Ideal for**: Storing and retrieving sensitive request data quickly (e.g., payment details, delivery addresses)

**How it works**
- Critical request data is stored in DynamoDB with a unique identifier.
- Subsequent processes can look up and update data in DynamoDB with low latency using DynamoDB streams.

**Benefits**
- Fast and reliable data access.
- Serverless, scales automatically.
- Supports fine-grained access control.

**Pricing example** 

Processing per 1 M of requests 

- DynamoDB (write) : $1.25
- API-GATEWAY (http): $1.00
- **Total : $2.25***

* This is an estimate of the pricing because we're not taking care of Data Transfer and Storage


### S3

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1g75e2zhagx7xveukhkt.png)

**Ideal for**: Storing raw request data for auditing, compliance, and analysis.

**How it works:**
- Requests are stored as objects in S3.
- S3 triggers events (e.g., object creation) to notify other services for further processing or analysis.

**Benefits**
- Durable and scalable storage.
- Cost-effective for long-term storage.
- Integrates with other AWS services.

**Pricing example** 

Processing per 1 M of requests 

- S3 put(standard) : $5
- API-GATEWAY (http): $1.00
- **Total : $6***

* This is an estimate of the pricing because we're not taking care of Data Transfer and Storage


If you want to go deeper on AWS storage first pattern I highly recommend the following [link](https://aws.amazon.com/blogs/compute/building-storage-first-applications-with-http-apis-service-integrations/)