Can Appwrite handle 25 Million documents?
Tags: performance, opensource, programming, database
Published: 2022-01-07 17:06:33

With the release of [Appwrite](https://appwrite.io) 0.12 come many new features, most notable being a completely overhauled database service.

{% link appwrite/its-here-announcing-the-release-of-appwrite-012-5c8b %}

The database didn't scale well with many documents in previous Appwrite versions. This was noticeable in collections containing upwards of ten thousand documents.

Since Appwrite uses the same database for internal resources like **Users** and **Files**, this limitation impacted every other service indirectly. The more the data, the slower the requests.

With 0.12, the database API introduces [Indexes](https://appwrite.io/docs/database#indexes), which allow you to improve queries for specific documents by a lot. Complimentary, we also added a new and more powerful way of filtering with [different query operators](https://appwrite.io/docs/database#querying-documents).

These improvements along with a complete rewrite of the database service enhanced Appwrite in a way that we were able to effortlessly query a collection of **25 Million documents** with excellent response times. 

## 🏁 Benchmark

[k6](https://k6.io/) was used with 500 **Virtual Users (VUs)** over 30 seconds, with each request performing an `EQUALS` query against an attribute that is indexed with a **Key index**.

```js
import http from 'k6/http'
import { Rate } from 'k6/metrics'

const config = {
    headers: {
        'X-Appwrite-Key': '[API_KEY]',
        'X-Appwrite-Project': '[PROJECT_ID]',
        'Content-Type': 'application/json'
    }
};

const myFailRate = new Rate('failed requests');

export let options = {
    vus: 500,
    discardResponseBodies: true,
    duration: '30s',
    thresholds: {
        'failed requests': ['rate<0.05'],
    },
};

export default function () {
    const res = http.get(
        `http://[APPWRITE_URL]/v1/database/collections/[COLLECTION_ID]/documents?queries[]=number.equal(${Math.floor(Math.random() * 25000000 + 1)})`,
        config
    );

    myFailRate.add(res.status !== 200);
}
```

## 📝 Collection

Here is a representation of what the collection looks like used for the benchmark, the `$id` is automatically generated, and `number` is a unique random value up to 25 million.

| $id                  | number   |
|----------------------|----------|
| 61d7fe9f91d6fed26fc0 | 1        |
| 61d7fe73eb0142ab50ec | 2        |
| 61d7fe73eb5142c4b5e5 | 3        |
| ...                  | ...      |
| 61d86657b6179ea91483 | 25000000 |

## ⚙️ Request

Requests were performed against the [list documents endpoint](https://appwrite.io/docs/client/database?sdk=web-default#databaseListDocuments) with an `EQUALS` query against the `number` attribute and a random number from 1 - 25,000,000. On top of the query itself, each request goes through authentication, authorization, and permissions validation.

> `/v1/database/[COLLECTION]/documents?queries[]=number.equal([RANDOM_NUMBER])`

## 🔬 Results

![Benchmark Result 0.12.0](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l9l6xfnfly9sgxwdt0hg.png)

The results speak for themselves. With **40k successful requests** and **1,334 per second**, we reached numbers that were very difficult to achieve with previous versions.

Not only is the new Database better at scale with simple Queries - but it also improves overall performance with more complex queries.

![htop](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tk49ptzk5bywl1dusy11.png)

The Server were Appwrite was hosted had 8-Cores with 16GB memory and it's CPU usage during the benchmark peaked around 80%. 

> k6 was run on a Mac Mini M1 with 16 GB memory.

This performance boost is part of a series of steps we take to ensure developers can take full advantage of their Appwrite servers. We plan to share more data and insights from both the development process and Appwrite benchmarks with upcoming new versions.

## 📚 Learn more
You can use following resources to learn more and get help:

- [🚀 Appwrite](https://appwrite.io)
- [🚀 Github](https://github.com/appwrite/appwrite)
- [📜 Appwrite Docs](https://appwrite.io/docs)
- [💬 Discord Community](https://appwrite.io/discord)