Simple KV storage on top of indexedDB
Tags: javascript, webdev, typescript, programming
Published: 2024-05-30 16:25:53

If your client-side application ever needs to persist a larger portion of data, it's no longer suitable to put it inside a `localStorage` entry. The first thing that comes into mind is to use indexedDB. But then you have to manage transactions, versioning, etc.

Sometimes all you need may be a straightforward key-value store, that hides those intricacies inside, like:

```javascript
const kv = await openKV("kv");

await kv.set(key, value);

const data = await kv.get(key);
```

What is stopping you from building one?
Let's get to work!

## Opening up

Working with indexedDB API is not especially pleasant. It's taken straight from the primal age of JavaScript. If you're looking for a more civilized API to digest - check out [`idb`](https://github.com/jakearchibald/idb) - it enhances the indexedDB API with promises and shortcuts for common operations

But in this post, we're not afraid of the tears and pain of the past

### Opening up (the database)

First, we need to open a new connection request. Then attach handlers for `success` and `error` events. Everything is enclosed within a compact Promise:

```typescript
const STORE_NAME = "store";
const openKVDatabase = (dbName: string) =>
  new Promise<IDBDatabase>((resolve, reject) => {
    const request = indexedDB.open(dbName);

    request.onsuccess = () => {
      resolve(request.result);
    };
    request.onerror = () => {
      reject("indexedDB request error");
    };
    request.onupgradeneeded = () => {
      request.result.createObjectStore(STORE_NAME, { keyPath: "key" });
    };
  });
```

`upgradeneeded` event will be fired once the database is created (or when its version gets updated). Inside this handler, we can create our one and only store - the KV store. I've put `STORE_NAME` in a constant as we'll need to use it in multiple places later on

## First ~~blood~~ methods

Let's scaffold a basic shape for `get`, `set`, and `delete` methods. They will correspond to indexDB **_objectStore_** operations consequently: `get`, `put`, and `delete`

```typescript
export async function openKV<T = unknown>(dbName: string) {
  const db = await openKVDatabase(dbName);

  const openStore = () => {
    return db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME);
  };

  return {
    async get(key: string) {},

    async set(key: string, value: T) {},

    async delete(key: string) {},
  };
}
```

`openStore` helper function opens up a new transaction and returns the handle for our KV store

### Requests as promised

One more thing needs to be done before implementing the methods. `objectStore` methods return [IDBRequest](https://developer.mozilla.org/en-US/docs/Web/API/IDBRequest) object. This object achieves the same goal as a promise (it's like a goofy version of it). Let's create a utility that will map them into promises - so we can `await` them:

```typescript
function idbRequestToPromise<T>(request: IDBRequest<T>) {
  return new Promise<T>((resolve, reject) => {
    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
}
```

### Methods

```typescript
  async get(key: string): Promise<T | undefined> {
      const pair: Pair | undefined = await idbRequestToPromise(
        openStore().get(key)
      );

      return pair?.value as T | undefined;
    },

    async set(key: string, value: T) {
      const pair: Pair = { key, value };
      await idbRequestToPromise(openStore().put(pair));
    },

    delete(key: string) {
      return idbRequestToPromise(openStore().delete(key));
    },
```

The `Pair` type used here is just:

```typescript
type Pair<T = unknown> = { key: string; value: T };
```

## You got to pump it up

As you probably noticed opening a new transaction every time we perform a single key value operation is suboptimal. Consider this snippet:

```javascript
for (const item of arr) {
	kv.set(item.id, item);
}
```

To handle an array of 1000 items, we need to open 1000 transactions. If the operations are executed in a single task (triggered synchronously) as in the example above, grouping them into a single transaction (_aka_ batching) could improve efficiency. Let's verify if this assumption holds true

### Batching

To implement batching, we need to update the `openStore` function a little bit

```typescript
const db = await openKVDatabase(dbName);
// Create 'store' variable to share it between calls
let store: IDBObjectStore | null;

const openStore = () => {
  if (!store) {
    store = db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME);

    queueMicrotask(() => {
      // Finish the transaction after the current task ends
      store?.transaction?.commit();
      store = null;
    });
  }

  return store;
};
```

> `queueMicrotask` allows running code after the current task has been executed (microtasks are run between regular tasks). Learn more [here](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API/Microtask_guide).

#### Testing

I used [`tinybench`](https://github.com/tinylibs/tinybench) to prepare a basic test case like so:

```typescript
Promise.all(arr.map((v) => kv.set(v, v)));
```

Where `arr` is a 1000-element array of strings

##### Results

Unsurprisingly there is a small improvement over the 1000 transaction version:

```markdown
| 1000 transactions | batching     |
| ----------------- | ------------ |
| 7 (ops/sec)       | 32 (ops/sec) |
```

### Transactions

Okay, so when I run queries synchronously, they will be put into a single transaction. But what about the original reason for inventing database transactions? It was to group queries together into one to ensure consistency. Check out this code:

```typescript
async function inc() {
	await kv.set("x", (await kv.get("x")) + 1);
}
```

It would only make sense if both `set` and `get` operations would form a single transaction

#### Async / await tracking

Unfortunately, APIs like `AsyncLocalStorage` that are available in server runtimes including _Node.js,_ _Deno_, and _Bun_, that would allow us to track async context are not (yet) available in the browsers. However, we can hook into async / await by leveraging custom `Thenables` and microtasks scheduling...

> If you are interested in learning more about tracking asynchronous contexts, you can check out [proposal-async-context](https://github.com/tc39/proposal-async-context#async-context-for-javascript) - the official ECMAScript proposal that addresses this particular issue

##### ...and then

Userland `Thenables` can be awaited just like `Promises`. The key difference if it comes to `Thenables` is that “_then_” method is always executed when used in async / await code. This allows us to intercept the async execution of the code and inject hooks before and after the continuation of the async / await block. Here's my attempt at doing that:

```typescript
export class Thenable<T> {

  constructor(
    private promise: Promise<T>,
    private hooks?: {
      before?: () => void;
      after?: () => void;
    }
  ) {}

  then<U>(onFulfilled: (value: T) => U): Thenable<U> {
    return new Thenable(
      this.promise.then((value) => {
        this.hooks?.before?.();
        const result = onFulfilled(value);
        queueMicrotask(() => this.hooks?.after?.());
        return result;
      }),
      this.hooks
    );
  }
}
```

> Notice how _after_ hook is pushed into the microtask queue. It's because calling `onFulfilled` will push continuation to the queue itself - this way _after_ hook is called after the _continuation_ microtask

#### Sharing current transaction

Taking up _before_ and after _hooks_, we can now make the current transaction accessible from within adjacent queries. Here's the type of the transaction object that will be shared:

```typescript
type Transaction = {
  store: IDBObjectStore;
  committed?: boolean;
  lastQueried?: boolean;
};
```

`commited` and `lastQueried` flags will be used to implement auto-committing of the transaction. All queries will now be wrapped in a `query` function to handle sharing.

```typescript
  const query = <R>(
    fn: (transaction: Transaction<T>) => Promise<R>
  ): Thenable<R> => {
    const transaction = (currentTransaction ??= {
      store: db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME),
    });

	// Clear current transaction after current task
    queueMicrotask(() => {
      currentTransaction = null;
    });

    const result = fn(transaction);

    return new Thenable(result, {
      before() {
	    // Resume transaction before the continuation
        currentTransaction = transaction;
      },
      after() {
        currentTransaction = null;
      },

    });

  };
```

And the example of usage:

```typescript
    set(key: string, value: T) {
	  // Wrap handler with query
      return query(async ({ store }) => {
        const pair = { key, value };
        await idbRequestToPromise(store.put(pair));
      });

    },
```

#### Auto-commiting

After the series of queries, it would be great to handle commiting automatically. The `lastQueried` flag will indicate if the queries were executed last microtasks:

```typescript
  const query = <R>(
    fn: (transaction: Transaction<T>) => Promise<R>
  ): Thenable<R> => {
    const transaction: Transaction<T> = (currentTransaction ??= {
      store: db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME),
    });


	// Running `query` will reset the flag
    transaction.lastQueried = true;

    queueMicrotask(() => {
      currentTransaction = null;
    });

    const result = fn(transaction);

    return new Thenable(result, {

      before() {
        currentTransaction = transaction;
        transaction.lastQueried = false;
      },

      after() {
	    // If there were no new queries during the last microtask
        if (!transaction.lastQueried && !transaction.committed) {
          transaction.store.transaction.commit();
          transaction.committed = true;
        }
        currentTransaction = null;
      },
    });
  };
```

#### Admiring the results

Look at that and think:

```typescript
async function inc() {
	await kv.set("x", (await kv.get("x")) + 1);
}
```

The function above now forms a single ACID transaction!

## Summing Up

The indexedDB is not the easiest API to work with. It's not the fastest horse in the stable either. It's probably a good idea to use some of the popular wrapper libraries like `idb` or [`Dexie.js`](https://dexie.org/).   That will simplify and streamline the process of working with it. There is also [`idbkeyval`](https://github.com/jakearchibald/idb-keyval?tab=readme-ov-file) - **super**-simple key-value store (but without automatic batching and transactions  🙊). Still, implementing your own wrapper may be great fun and will definitely help you understand better how it works