Implement a Batch API using FastAPI
Tags: python, architecture, api, webdev
Published: 2022-11-19 13:52:24

In this blogpost I would like to present you a possible implementation for a batch route using FastAPI. FastAPI is a great framework for creating REST API's, unfortunately there is currently no native support for batching.

## Why batching?
Batching is the process of combining multiple API requests into a single request with a single response. Without batching, the whole thing looks like this:

[![Without batch requests](https://mermaid.ink/img/pako:eNrNkMEKwjAQRH8l7Ln-QA4F0Ys3sddc1mbUYrOtm0SQ0n83tfQfvO0MbwZ2JmoHD7IU8cqQFseO78rBieGcBsnhCi3i0HeQZHZ1bRroG2rN_nwylyUVUwFW9wes7AbEcZCIv6kwVFGABu58-XpaHEfpgQBHtpye9enIyVy4ZYHmIy3ZpBkV5dFz2hYie-M-Yv4C4eBsfQ?type=png)](https://mermaid.live/edit#pako:eNrNkMEKwjAQRH8l7Ln-QA4F0Ys3sddc1mbUYrOtm0SQ0n83tfQfvO0MbwZ2JmoHD7IU8cqQFseO78rBieGcBsnhCi3i0HeQZHZ1bRroG2rN_nwylyUVUwFW9wes7AbEcZCIv6kwVFGABu58-XpaHEfpgQBHtpye9enIyVy4ZYHmIy3ZpBkV5dFz2hYie-M-Yv4C4eBsfQ)

Each request between client and server has a certain network latency, so the processing of several consecutive requests can take some time.

With a batch request, the roundtrip between client and server and thus the number of requests can be reduced, which in turn rapidly improves performance:

[![With batch requests](https://mermaid.ink/img/pako:eNq9kEEOwjAMBL8S-QwfyKEH4AX0motJFlrRuMVJkFDVv5OqQogPcLN3x9KuZ_JjAFlKeBSIx6nnm3J0YrjkUUq8QOtyHHpINvumMS30CbXmwNl35rzepVyRTf9B_mluCb-x0jRKQmUM7ShCI_eh9pxXxVHuEOHI1jGw3h05WSq3dm5f4slmLdhRmQLnz0_IXnlIWN6xGmo5?type=png)](https://mermaid.live/edit#pako:eNq9kEEOwjAMBL8S-QwfyKEH4AX0motJFlrRuMVJkFDVv5OqQogPcLN3x9KuZ_JjAFlKeBSIx6nnm3J0YrjkUUq8QOtyHHpINvumMS30CbXmwNl35rzepVyRTf9B_mluCb-x0jRKQmUM7ShCI_eh9pxXxVHuEOHI1jGw3h05WSq3dm5f4slmLdhRmQLnz0_IXnlIWN6xGmo5)

## Technical consideration of the batch route
In the course of my research, I looked at the batch routes of other major API's, including:

- [Microsoft Graph API Batch Route](https://learn.microsoft.com/en-us/graph/json-batching)
- [Microsoft Sharepoint API Batch Route](https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/make-batch-requests-with-the-rest-apis)
- [Google Cloud Storage API Batch Route](https://cloud.google.com/storage/docs/batch)

Based on the API's listed above, there are two different ways to submit batch requests:
- Submit batch request with MIME type multipart/mixed
 - The advantage of this approach is that requests can be sent with different MIME types (e.g. application/json, application/http, application/octet-stream etc.).
 - The body of the request is divided into parts that are separated from each other by a boundary string that is specified in the header of the request.
- Submit batch request with MIME type application/json
 - The batch request is submitted in JSON format. This limits the API to the MIME type application/json.

I personally prefer the JSON batch approach, assembling the batch request is slightly easier with this approach. Of course this approach is limited to the MIME type application/json only, but since most FastAPI routes implement exactly this MIME type this is acceptable for me. The choice therefore varies simply on the technical requirements.

Our batch route is therefore inspired by the Microsoft Graph API, the batch request will have the following structure:
```http
POST /batch HTTP/1.1
Content-Type: application/json

{
  "requests": [
    {
      "id": "1",
      "url": "/products?skip=0&limit=1",
      "method": "GET"
    },
     {
      "id": "2",
      "url": "/products/1",
      "method": "GET"
    },
    {
        "id": "3",
        "url": "/products",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "title": "Test Product"
        }
    }
  ]
}
```

|Property     |Mandatory|Description  |
|-------------|---------|-------------|
|id           |Yes      |A correlation value to associate individual responses with requests. This value allows the server to process requests in the batch in the most efficient order.|
|url          |Yes      |The relative resource URL the individual request would typically be sent to.|
|method       |Yes      |The HTTP method (e.g. GET, POST etc.).|
|headers      |No       |A JSON object with the key/value pair for the headers.|
|body         |No       |JSON body.|

The next step is to consider how the individual requests should be processed on the server side. Behind each FastAPI route there is a corresponding Python function that can be called directly. At first this sounds like a good idea, but then FastAPI is no longer able to determine the dependencies via depency injection, so we would have to do this manually. 

Determining the dependencies manually could be difficult and probably often leads to unexpected behavior of the API route. So I did some research and came across this repository: 
{% embed https://github.com/dtkav/flask-batch %}

Flask-Batch performs batching on the HTTP layer. This means that on the server side a HTTP client executes all batch requests. The network latency is much lower on the server side which implies that the requests can be processed faster on the server side. Additionally, since the requests are processed on the HTTP layer, FastAPI is also able to resolve the dependencies via dependicy injection. Therefore I also decided to do batching on HTTP level.

## Let's get started
### Defining the batch models
{% embed https://gist.github.com/manukanne/59f279439124b0853c998fcdeac67865 %}

#### HTTPVerbs
This enum defines the allowed HTTP methods.

#### BatchRequest
This model represents a single API request. This model also ensures that the referenced endpoint meets our requirements, in this case that no absolute URL is given and that the route starts with a leading slash. Especially the check that it is not an absolute URL is extremely important to prevent potential improper usage of our batch route.

#### BatchResponse
This model represents the result of the corresponding request. Here also the ID of the request is returned so that the client can map request and response (theoretically the order of processing could vary). Besides the ID, the HTTP status code, the headers and the JSON body are also returned.

#### BatchIn
This is the container that includes all our batch requests. Here it is also checked if all ID's are unique and the allowed maximum amount of requests has not been exceeded.

#### BatchOut
This is the container that holds all the responses, this model is the one that is passed back to the caller.

### Implement the batch route
The batch requests are processed using the Python package [aiohttp](https://docs.aiohttp.org/en/stable/index.html). This allows the requests to be processed in an asynchronous manner. One blog post that helped me a lot with the implementation is the post [Making 1 million requests with python-aiohttp](https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html) which I would definitely like to recommend to you!

{% embed https://gist.github.com/manukanne/b0b2e876dc91c65635eee3c1a5bde98d %}

In the next step we implement the actual batch route, here we just have to plug everything together:

{% embed https://gist.github.com/manukanne/c6c0b1caf3e339c350923aad7e6af7e7 %}

The repository, which contains all the code including a small test API, can be found here:

{% embed https://github.com/manukanne/fastapi-batch-requests-demo %}

## Conclusion
Bathing is an important component to combine many requests to a single API call and thus increase the performance rapidly. 
I hope you enjoyed my blog, thanks for reading! If you have any recommendations on how to improve my implementation, please let me know in the comments section.