Simplified Firestore with Redis
Tags: typescript, redis, firebase, node
Published: 2020-06-03 16:11:52

> Library available on [github](https://github.com/ehacke/simple-cached-firestore) and [npm](https://www.npmjs.com/package/simple-cached-firestore)

I've used either Firestore or Datastore on four different large projects now (including my [onboarding app](https://roleup.io) RoleUp and [uptime testing service](https://asserted.io) asserted), and over time I've been able to refine and improve my own wrapper. 

## Isn't this better?

{% gist https://gist.github.com/ehacke/f75c2762095547eb20589586b380ea74.js  %}

## Features

*simple-cached-firestore* offers a number of key features:

- transparent, no-effort redis caching to improve speed and limit costs
- model validation (optional, suggest using [validated-base](https://github.com/ehacke/validated-base))
- simplified API to reduce boilerplate
- still have access to the underlying firestore client if you need custom functionality

## Why build an API when using Firestore?

Obviously one of the biggest and most popular features of Firebase/Firestore is that it can be used entirely serverless. With the correct configuration it can be securely accessed directly from the web or a native app without having to write your own API.

But that comes with a few big sacrifices that I wasn't willing to make.

### Validation

You can't easily validate your data models without an API. There is a capability to [write rules](https://firebase.google.com/docs/rules/data-validation), but I really don't want to spend hours writing complicated validation logic in thisr DSL:

{% gist https://gist.github.com/ehacke/9269cd6c577d306ec04391735a513d40.js %}

Furthermore, in some cases it's just not possible. If you have any kind of complicated validation logic, or even something as simple as wanting to use constants from a library, you're out of luck.

### Sanitization

Additionally, the rules merely determine whether or not to allow a write to occur. 

What if the properties you are checking are valid, but the user has messed with the Javascript and is saving extra arbitrary properties within the same object? Or much more likely, what if you accidentally attach properties you don't mean to save? In either case you only have limited control over what gets written to your db.

### Caching

Caching can serve both as a circuit breaker, and insurance against malice or bugs. Which is why it's unfortunate that caching also cannot be implemented in a serverless setup without a lot of complexity. 

When implemented well, caching provides significant benefits in terms of cost-reduction and responsiveness. 

> With simple-cached-firestore wrapper, I regularly see cache hit rates of 95-98%. Amounting to a huge reduction in Firestore READ costs.

## Usage

Moving on to the subject at hand, we'll look at how I've addressed the shortcomings above with an API and *simple-cached-firestore*. 

Each instance of *simple-cached-firestore* is responsible for all reads and writes to a specific collection, and it's assumed that all elements of that collection can be represented by the same model.

To create an instance of *simple-cached-firestore*, we must first create the model that will exists in the collection. 

### Create a Model

At minimum, the model has to fulfill the following interface:

{% gist https://gist.github.com/ehacke/b7f38e609aeb694e717e306b94c3999d.js %}

The easiest way to do this is to just extend [validated-base](https://www.npmjs.com/package/validated-base) (the subject of the post on [validated models](https://asserted.io/posts/type-safe-models-in-node)) and use that.

{% gist https://gist.github.com/ehacke/226ca5b3943ead991c988447583eeaef.js %}

Now that we have a model to work with, let's create an instance of *simple-cached-firestore*.

### Create simple-cached-firestore

As mentioned above, a single instance is responsible for reading and writing to a specific Firestore collection. 

Reads are cached for the configured TTL, and writes update the cache. Because all reads and write pass through this layer, cache invalidation is not an issue. We have perfect knowledge of what is written, so the only real limit on the cache TTL is how big of a Redis instance you want to pay for.

{% gist https://gist.github.com/ehacke/2f1ba832e65b706f925b02a32559b1e3.js %}

You may not want to do all of these operations in one place like this, but this is the general idea.

The validated class we created above serves as both validation of anything that's passed to it, and a way to translate the object to and from the db (and the cache) into a class instance with known properties.

## Basic CRUD Operations

You can see the breakdown of the basic operations [here](https://github.com/ehacke/simple-cached-firestore/tree/82507ebd021e1d2bf3ee3cebd79debf47242abf5#crud-api), but included the expected *create*, *get*, *patch*, *update*, and *remove*. 

To give you an idea of how these CRUD operations are implemented, here is an example of how simple-cached-firestore implements the get operation. It's actually more complicated than this, but this is just to show the major details.

{% gist https://gist.github.com/ehacke/e06b3208c40870f60d204d91a16fdfc2.js %}

The full implementation is [here](https://github.com/ehacke/simple-cached-firestore/blob/82507ebd021e1d2bf3ee3cebd79debf47242abf5/src/firestore.ts#L288), and includes some extra work with timestamps to avoid race conditions contaminating the cache. But basically the process is:

- Check cache and return if cache exists
- Otherwise get snapshot and convert into a model instance
- Update cache before returning if a value is found

Pretty straight-forward, and you can imagine write operations working in a similar way. 

Depending on the problem you're solving, and if you're careful about how you design all of the data models for your project, you can actually do a large portion of the regular tasks with just the basic CRUD operations. 

This is great if you can manage it because it not only minimizes costs in normal operation, but thanks to the cache, means that you'll almost never have to hit the Firestore itself.

## Query Operations

At some point, some type of query operation is usually required in most projects, even if it's just a list operation with a single filter. In Firestore this is done by chaining operations, often in a specific order. In order to abstract and simplify this, I created a simpler query abstraction that looks like this:

{% gist https://gist.github.com/ehacke/fe76946c4dfc2bb8ae9d390fa31fe8ab.js %}

In use, the query objects look like this:

{% gist https://gist.github.com/ehacke/c6ed6d710d90614c3d98aa7a4d0b07f1.js %}

One important thing to note is that while queries are cached, due to the complexity of the query logic, accurate invalidation is hard. As a result, the cache for queries within a given collection is invalidated on every write to that collection. This makes it not very useful by default, so if you want effective caching of queries, that should be implemented on a case-by-case basis.

## Custom Functionality

If the crud and query functionality don't work for you in a specific case, you can always access the underlying Firestore client or cache instance with:

{% gist https://gist.github.com/ehacke/04410c212d0307d57cb907272bf7059e.js %}

But keep in mind, that any modifications you make directly to objects in Firestore will not be captured by the cache unless you update it manually, and can result in inconsistencies if you don't do it properly.

## Next

From here I'll next describe how the validated models and simple-cached-firestore can be integrated together in a dependency-injected Node microservice architecture.