Getting Started with IndexedDB for Big Data Storage
Tags: bigdata, javascript
Published: 2021-06-03 05:14:39

_by author [Craig Buckler](https://blog.asayer.io/authors/craig-buckler)_

It's not always necessary to send a user's data to the server: you could choose to store some information in the browser. It's ideal for device-specific settings such as interface configurations (e.g. light/dark modes) or private data which should not be transmitted across the network (e.g. an encryption key).

Consider the [Performance API](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api) for timing network and page events. You *could* upload all data the moment it's available but that's a considerable volume of information which, ironically, would hit page performance. A better option would be to store it locally, process statistics using a [Web Worker](https://developer.mozilla.org/docs/Web/API/Web_Workers_API), and upload the results when the browser is less busy.

Two cross-browser client-side storage APIs are available:

1. [Web Storage](https://developer.mozilla.org/docs/Web/API/Web_Storage_API)
   Synchronous name-value pair storage saved permanently (`localStore`) or for the current session (`sessionStore`). Browsers permit up to 5MB of Web Storage per domain.
1. [IndexedDB](https://developer.mozilla.org/docs/Web/API/IndexedDB_API)
   An asynchronous NoSQL-like name-value store which can save data, files, and blobs. At least 1GB should be available per domain and it can reach up to 60% of the remaining disk space.

> Another storage option, [WebSQL](https://www.w3.org/TR/webdatabase/), is available in some editions of Chrome and Safari. However, it has a 5MB limit, is inconsistent, and was deprecated in 2010.

In this tutorial we'll store timing data for the page and all assets. Web Storage space can be slow and too limiting so IndexedDB is the best option.

[**All example code is available on Github**](https://github.com/craigbuckler/asayer-indexeddb) should you want to try IndexedDB on your own site.


## What is IndexedDB?

IndexedDB was first implemented in 2011 and became a W3C standard in January 2015. It has [good browser support](https://caniuse.com/indexeddb) although its callback and events-based API seems clunky now we have ES2015+. This article demonstrates how to write a Promise-based wrapper so you can use chaining and `async`/`await`.

Note the following IndexedDB terms:

* **database** -- a top-level store. A domain can create any number of IndexedDB databases but it's unusual to see more than one. Only pages within the same domain can access the database.

* **object store** -- a name/value store for related data items. It's similar to a collection in MongoDB or tables in a relational database.

* **key** -- a unique name used to reference every record (value) in the object store. It can be generated using an **autoIncrement** number or set to any unique value within the record.

* **index** -- another way to organize data in an object store. Search queries can only examine the key or an index.

* **schema** -- the definition of object stores, keys, and indexes.

* **version** -- a version number (integer) assigned to a schema. IndexedDB offers automated versioning so you can update databases to the latest schema.

* **operation** -- database activities such as creating, reading, updating, or deleting records.

* **transaction** -- a set of one or more operations. A transaction guarantees all its operations either succeed or fail. It cannot fail some and not others.

* **cursor** -- a way to iterate over records without having to load all into memory at once.


## Developing and debugging a database

In this tutorial, you will create an IndexedDB database named `performance`. It contains two object stores:


### 1. `navigation`

This stores [page navigation timings](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api#navigation-timing) information (redirects, DNS lookups, page loading, file sizes, load events, etc). A date will be added to use as the **key**.


### 2. `resource`

This stores [resource timings](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api#resource-timing) information (timings for other resources such as images, stylesheets, scripts, Ajax calls etc.) A date will be added, but two or more assets could load at the same time so an auto-incrementing ID will be used as the **key**. Indexes will be created for the date and the name (the resource's URL).


All Chrome-based browsers have an **Application** tab where you can examine storage space, artificially limit the capacity, and wipe all data. The **IndexedDB** entry in the Storage tree allows you to view, update, and delete object stores, indexes, and individual records. Firefox's panel is named **Storage**.

You can also run your application in incognito mode so all data is deleted once the browser window is closed.


## Connecting to an IndexedDB database

A wrapper class created in `indexeddb.js` checks for IndexedDB support using:

```js
if ('indexedDB' in window) // ...
```

It then opens a database connection using [`indexedDB.open()`](https://developer.mozilla.org/docs/Web/API/IDBFactory/open) by passing:

1. the database name, and
1. an optional version integer.

```js
const dbOpen = indexedDB.open('performance', 1);
```

Three important event handler functions must be defined:

1. `dbOpen.onerror` runs when an IndexedDB connection cannot be established.

1. `dbOpen.onupgradeneeded` runs when the version required (`1`) is greater than the current version (`0` when the database is not defined). A handler function must run IndexedDB methods such as [`createObjectStore()`](https://developer.mozilla.org/docs/Web/API/IDBDatabase/createObjectStore) and [`createIndex()`](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/createIndex) to create the storage structures.

1. `dbOpen.onsuccess` runs when the connection has been established and any upgrades have completed. The connection object in `dbOpen.result` is used in all subsequent data operations. It is assigned to `this.db` in the wrapper class.

The wrapper constructor code:


```js
// IndexedDB wrapper class: indexeddb.js
export class IndexedDB {

  // connect to IndexedDB database
  constructor(dbName, dbVersion, dbUpgrade) {

    return new Promise((resolve, reject) => {

      // connection object
      this.db = null;

      // no support
      if (!('indexedDB' in window)) reject('not supported');

      // open database
      const dbOpen = indexedDB.open(dbName, dbVersion);

      if (dbUpgrade) {

        // database upgrade event
        dbOpen.onupgradeneeded = e => {
          dbUpgrade(dbOpen.result, e.oldVersion, e.newVersion);
        };

      }

      // success event handler
      dbOpen.onsuccess = () => {
        this.db = dbOpen.result;
        resolve( this );
      };

      // failure event handler
      dbOpen.onerror = e => {
        reject(`IndexedDB error: ${ e.target.errorCode }`);
      };

    });

  }

  // more methods coming later...

}
```

A `performance.js` script loads this module and instantiates a new IndexedDB object named `perfDB` after the page has loaded. It passes the database name (`performance`), version (`1`), and an upgrade function. The `indexeddb.js` constructor calls the upgrade function with the database connection object, the current database version, and the new version:

```js
// performance.js
import { IndexedDB } from './indexeddb.js';

window.addEventListener('load', async () => {

  // IndexedDB connection
  const perfDB = await new IndexedDB(
    'performance',
    1,
    (db, oldVersion, newVersion) => {

      console.log(`upgrading database from ${ oldVersion } to ${ newVersion }`);

      switch (oldVersion) {

        case 0: {

          const
            navigation = db.createObjectStore('navigation', { keyPath: 'date' }),
            resource = db.createObjectStore('resource', { keyPath: 'id', autoIncrement: true });

          resource.createIndex('dateIdx', 'date', { unique: false });
          resource.createIndex('nameIdx', 'name', { unique: false });

        }


      }

  });

  // more code coming later...

});
```

At some point it will become necessary to change the database schema -- perhaps to add new object stores, indexes, or data updates. In that situation, you must increment the version (from `1` to `2`). The next page load will trigger the upgrade handler again so you can add a further block to the `switch` statement, e.g. to create an index named `durationIdx` on the `duration` property in the `resource` object store:

```js
case 1: {
  const resource = db.transaction.objectStore('resource');
  resource.createIndex('durationIdx', 'duration', { unique: false });
}
```

The usual `break` at the end of each `case` block is omitted. When someone accesses the application for the first time, the `case 0` block will run followed by `case 1` and all subsequent blocks. Anyone already on version `1` would run the updates starting at the `case 1`. IndexedDB schema update methods include:

* [`createIndex()`](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/createIndex)
* [`deleteIndex()`](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/deleteIndex)
* [`createObjectStore()`](https://developer.mozilla.org/docs/Web/API/IDBDatabase/createObjectStore)
* [`deleteObjectStore()`](https://developer.mozilla.org/docs/Web/API/IDBDatabase/deleteObjectStore)
* [`deleteDatabase()`](https://developer.mozilla.org/docs/Web/API/IDBFactory/deleteDatabase)

Everyone who loads the page will be on the same version -- *unless they have the app running in two or more tabs*. To avoid conflicts, the database connection `onversionchange` handler can be added to `indexeddb.js` which prompts the user to reload the page:

```js
// version change handler
dbOpen.onversionchange = () => {

  dbOpen.close();
  alert('Database upgrade required - reloading...');
  location.reload();

};
```

You can now add the `performance.js` script to a page and run it to check that object stores and indexes are created (DevTools **Application** or **Storage** panels):

```html
<script type="module" src="./performance.js"></script>
```


## Record performance statistics

All IndexedDB operations are wrapped in a transaction. The following process is used:

1. Create a database transaction object. This defines one or more object stores (single string or array of strings) and the access type: `"readonly"` for fetching data, or `"readwrite"` for inserts and updates.

1. Create a reference to an [`objectStore()`](https://developer.mozilla.org/docs/Web/API/IDBTransaction/objectStore) within the scope of the transaction.

1. Run any number of [`add()`](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/add) (inserts only) or [`put()`](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/put) methods (inserts and updates).

Add a new `update()` method to the `IndexedDB` class in `indexeddb.js`:

```js
  // store item
  update(storeName, value, overwrite = false) {

    return new Promise((resolve, reject) => {

      // new transaction
      const
        transaction = this.db.transaction(storeName, 'readwrite'),
        store = transaction.objectStore(storeName);

      // ensure values are in array
      value = Array.isArray(value) ? value : [ value ];

      // write all values
      value.forEach(v => {
        if (overwrite) store.put(v);
        else store.add(v);
      });

      transaction.oncomplete = () => {
        resolve(true); // success
      };

      transaction.onerror = () => {
        reject(transaction.error); // failure
      };

    });

  }
```

This adds or updates (if the `overwrite` parameter is `true`) one or more values in the named store and wraps the whole transaction in a Promise. The `transaction.oncomplete` event handler runs when the transaction auto-commits at the end of the function and all database operations are complete. A `transaction.onerror` handler reports errors.

> IndexedDB events bubble up from the operation to the transaction, to the store, and to the database. You could create a single `onerror` handler on the database which receives all errors. Like DOM events, propagation can be stopped with `event.stopPropagation()`.

The `performance.js` script can now report page navigation metrics:

```js
  // record page navigation information
  const
    date = new Date(),

    nav = Object.assign(
      { date },
      performance.getEntriesByType('navigation')[0].toJSON()
    );

  await perfDB.update('navigation', nav);
```

and resource metrics:

```js
  const res = performance.getEntriesByType('resource').map(
    r => Object.assign({ date }, r.toJSON())
  );

  await perfDB.update('resource', res);
```

In both cases, a `date` property is added to the cloned timing objects so it becomes possible to search for data within a specific period.


## Frontend Monitoring
[OpenReplay](https://github.com/openreplay/openreplay) is a developer-oriented, open-source session replay alternative to FullStory, LogRocket and Hotjar. 
It’s like having your browser’s DevTools open while looking over your user’s shoulder.

[OpenReplay](https://github.com/openreplay/openreplay) is self-hosted so you have complete control over your data and costs.

![OpenReplay](https://raw.githubusercontent.com/openreplay/openreplay/main/static/replayer.png)

Happy debugging, for modern frontend teams - [Start monitoring your web app for free](https://github.com/openreplay/openreplay).


## Reading performance records

IndexedDB searching is rudimentary compared to other databases. You can only fetch records by their key or an indexed value. You cannot use equivalents of SQL `JOIN` or functions such as `AVERAGE()` and `SUM()`. All record processing must be handled with JavaScript code; a background [Web Worker](https://developer.mozilla.org/docs/Web/API/Web_Workers_API) thread could be a practical option.

You can retrieve a single record by passing its key to an object store or index's [`.get()` method](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/get) and defining an `onsuccess` handler:

```js
// EXAMPLE CODE
const

  // new readonly transaction
  transaction = db.transaction('resource', 'readonly'),

  // get resource object store
  resource = transaction.objectStore('resource'),

  // fetch record 1
  request = resource.get(1);

// request complete
request.onsuccess = () => {
  console.log('result:', request.result);
};

// request failed
request.onerror = () => {
  console.log('failed:', request.error);
};
```

Similar methods include:

* [`.count(query)`](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAllKeys) -- the number of matching records
* [`.getAll(query)`](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAll) -- an array of matching records
* [`.getKey(query)`](https://developer.mozilla.org/docs/Web/API/IDBIndex/getKey) -- a matching key (rather than the value assigned to that key)
* [`.getAllKeys(query)`](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAllKeys) -- an array of matching keys

The *query* can also be a [KeyRange](https://developer.mozilla.org/docs/Web/API/IDBKeyRange) argument to find records within a range, e.g. `IDBKeyRange.bound(1, 10)` returns all records with key between 1 and 10 inclusive:

```js
request = resource.getAll( IDBKeyRange.bound(1, 10) );
```

KeyRange options:

* [`IDBKeyRange.lowerBound(X)`](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/lowerBound) -- greater than or equal to `X`
* [`IDBKeyRange.upperBound(X)`](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/upperBound) -- less than or equal to `Y`
* [`IDBKeyRange.bound(X,Y)`](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/bound) -- between `X` and `Y` inclusive
* [`IDBKeyRange.only(X)`](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/only) -- a single key matching `X`.

The lower, upper, and bound methods have an optional exclusive flag, e.g. `IDBKeyRange.bound(1, 10, true, false)` -- keys greater than `1` (but not `1` itself) and less than or equal to `10`.

Reading a whole dataset into an array becomes impossible as the database grows larger. IndexedDB provides *cursors* which can iterate through each record one at a time. The [`.openCursor()` method](https://developer.mozilla.org/docs/Web/API/IDBIndex/openCursor) is passed a KeyRange and optional direction string (`"next"`, `"nextunique"`, `"prev"`, or `"preunique"`).

Add a new `fetch()` method to the `IndexedDB` class in `indexeddb.js` to search an object store or index with upper and lower bounds with a callback function that is passed the cursor. Two further methods are also required:

1. `index(storeName, indexName)` -- returns either an object store or index on that store, and
1. `bound(lowerBound, upperBound)` -- returns an appropriate KeyRange object.

```js
  // get items using cursor
  fetch(storeName, indexName, lowerBound = null, upperBound = null, callback) {

    const
      request = this.index(storeName, indexName)
        .openCursor( this.bound(lowerBound, upperBound) );

    // pass cursor to callback function
    request.onsuccess = () => {
      if (callback) callback(request.result);
    };

    request.onerror = () => {
      return(request.error); // failure
    };

  }


  // start a new read transaction on object store or index
  index(storeName, indexName) {

    const
      transaction = this.db.transaction(storeName),
      store = transaction.objectStore(storeName);

    return indexName ? store.index(indexName) : store;

  }


  // get bounding object
  bound(lowerBound, upperBound) {

    if (lowerBound && upperBound) return IDBKeyRange.bound(lowerBound, upperBound);
    else if (lowerBound) return IDBKeyRange.lowerBound(lowerBound);
    else if (upperBound) return IDBKeyRange.upperBound(upperBound);

  }
```

The `performance.js` script can now retrieve page navigation metrics, e.g. return all `domContentLoadedEventEnd` during June 2021:

```js
  // fetch page navigation objects in June 2021
  perfDB.fetch(
    'navigation',
    null, // not an index
    new Date(2021,5,1,10,40,0,0), // lower
    new Date(2021,6,1,10,40,0,0), // upper
    cursor => { // callback function

      if (cursor) {
        console.log(cursor.value.domContentLoadedEventEnd);
        cursor.continue();
      }

    }
  );
```

Similarly, you can calculate the average download time for a specific file and report it back to [OpenReplay](https://github.com/openreplay/openreplay):

```js
  // calculate average download time using index
  let
    filename = 'http://mysite.com/main.css',
    count = 0,
    total = 0;

  perfDB.fetch(
    'resource', // object store
    'nameIdx',  // index
    filename,   // matching file
    filename,
    cursor => { // callback

    if (cursor) {

      count++;
      total += cursor.value.duration;
      cursor.continue();

    }
    else {

      // all records processed
      if (count) {

        const avgDuration = total / count;

        console.log(`average duration for ${ filename }: ${ avgDuration } ms`);

        // report to OpenReplay
        if (asayer) asayer.event(`${ filename }`, { avgDuration });

      }

    }

  });
```

In both cases, the `cursor` object is passed to the callback function where it can:

1. obtain the record value with `cursor.value`
1. advance to the next record with [`cursor.continue()`](https://developer.mozilla.org/docs/Web/API/IDBCursor/continue)
1. move forward `N` records with [`cursor.advance(N)`](https://developer.mozilla.org/docs/Web/API/IDBCursor/advance)
1. update the record with [`cursor.update(data)`](https://developer.mozilla.org/docs/Web/API/IDBCursor/update), or
1. delete the record with [`cursor.delete()`](https://developer.mozilla.org/docs/Web/API/IDBCursor/delete)

`cursor` is `null` when all matching records have been processed.


## Check Remaining Storage Space

Browsers allocate a significant volume of storage to IndexedDB but it will eventually run out. The new Promise-based [StorageManager API](https://developer.mozilla.org/docs/Web/API/StorageManager) can calculate the space remaining for the domain:

```js
(async () => {

  if (!navigator.storage) return;

  const
    estimate = await navigator.storage.estimate(),

    // calculate remaining storage in MB
    available = Math.floor((estimate.quota - estimate.usage) / 1024 / 1024);

    console.log(`${ available } MB remaining`);

})();
```

The API is not supported in IE or Safari. As the limit is approached, you could choose to remove older records.


## Conclusion

IndexedDB is one of the older and more complex browser APIs but you can add wrapper methods to adopt Promises and `async`/`await`. Pre-built libraries such as [idb](https://www.npmjs.com/package/idb) can help if you'd rather not do that yourself.

Despite its drawbacks and some unusual design decisions, IndexedDB remains the fastest and largest browser-based data store.