Firestore Many-to-Many: Part 5 - Scalable Arrays
Tags: firebase, firestore, backend, database
Published: 2021-09-04 15:05:11

## Update 3/24/24
This post is out-of-date. I would not recommend this method and my package is no longer maintained. I would use the Fan-out method depending on the use case.
https://code.build/p/building-a-scalable-follower-feed-with-firestore-wCeklv
___
## Original Post
___

## Scalable Arrays Now Possible

In order to get to the scalable follower feed that I want to make possible in Firestore, I first need scalable arrays. I added this feature today (9/4/21) to my [adv-firestore-functions](https://github.com/jdgamble555/adv-firestore-functions/blob/master/AGGREGATION.md) npm package. Now you can have a scalable array on any document in any collection for searchable purposes.

To install the package in your firebase functions, go to your functions directory and run `npm i adv-firestore-functions`.

### Products Purchased by a User

So, let's say you have purchased many items from a company. If you're like me, you hate Amazon for their monopolistic practices, but still buy stuff from em all the time :(

Let's say I have bought more than 10,000 items. I cannot store 10,000 items in an array (I don't know the exact limit because it depends on how big your document is, but the limit is close). How can I model this so that I can find:

1. All products I have purchased

This is easy. Either use a sub-collection with:
```typescript
users/{userID}/products/ID --> {
  productID: 12jsk3,
  userID: 23929
  createdAt: timestamp,
  updatedAt: timestamp
}
```
or the compound index on a root collection:
```typescript
products/productID__userID --> {
  productID: 12jsk3,
  userID: 23929
  createdAt: timestamp,
  updatedAt: timestamp
}
```
**Note:** The createdAt / updatedAt can be good for sorting in certain circumstances.

2. Get all users who purchased a product (productID)

```typescript
db.collectionGroup('products')
.where('productID', '==', productID)
.where('userID', '==', userID);
```
OR
```typescript
db.collection('products')
.where('productID', '==', productID)
.where('userID', '==', userID);
```
But, you still have to grab the user document on the frontend with rxjs (we love rxjs, but hate using it):

```typescript
this.afs.collection('products',
    ref => ref.where('productID', '==', productID)
  ).valueChanges({ idField: 'id' }).pipe(
    switchMap((r: any) => {
      const docs = r.map(
        (d: Post) => this.afs.doc(`users/${d.userID}`).valueChanges()
      ) as Observable<any>[];
      return combineLatest(docs).pipe(
        map((d: any[]) => {
          return d.map(
            (doc: any, i: number) => {
              return { ...r[i++], user: doc };
            }
          );
        })
      );
    }),
  );
```
But then we still have the docID that is not joined! We could technically do it by using more combinedLatest (why we saved createdAt on the compound document), but that is a mess!

## Scalable Arrays Function

So, I created `array-index` that basically lets you have scalable arrays by creating indexes on multiple documents. You must, however, create a Firebase Function that gets triggered on the products onWrite.

Example:
```typescript
import { arrayIndex } from "adv-firestore-functions";

functions.firestore
  .document("users/{userId}/products/{productId}")
  .onWrite(async (change: any, context: any) => {

    await arrayIndex(change, context);

  });
```
This will generate a `users/{userID}/products_index` collection that will be searchable as:

```typescript
db.collectionGroup('products_index')
.where('products', 'array-contains', productID);
```
There are also options for maps, pre-sorting, etc.  See the [docs](https://github.com/jdgamble555/adv-firestore-functions/blob/master/AGGREGATION.md).  

The products_index collection will look like this:

```typescript
users/{userID}/product_index/{product_index_id} --> {
  products: [
    3k2lsle,
    2k3k2l,
    62221,
  ],
  user: {
    ... all user doc info here
  },
  createdAt: timestamp,
  updatedAt: timestamp
```
The beauty of this, is it automatically scales. It creates a new index document after 10,000 items (you can set this with max, see the [docs](https://github.com/jdgamble555/adv-firestore-functions/blob/master/AGGREGATION.md)). It will automatically remove a product from the index if the product doc is delete from the collection. It automatically adds. I have id sorting and value sorting options that I spoke about in [Part 3](https://dev.to/jdgamble555/firestore-many-to-many-part-3-maps-23nh).

There are no joins to get the user info. If you need the product info, you need to do a frontend join.  However, since you're searching by Product ID, it assumes you have that info.

One final caveat, is you may need to update the index when the user doc gets updated.  Luckily, I had already written this code with my `updateJoinData` function.

Usage:
```typescript
import { updateJoinData } from 'adv-firestore-functions';

functions.firestore
  .document("users/{userId}")
  .onWrite(async (change: any, context: any) => {

    const userID = context.params.userId;
    const queryRef = db.collectionGroup('products_index')
    .where('userId', '==', userID)
    const joinFields = ['displayName', 'photoURL'];
    await updateJoinData(change, queryRef, joinFields, 'user');

  });
```

This will make sure the `products_index` always stays up-to-date. For more info on `updateJoinData` see the [docs](https://github.com/jdgamble555/adv-firestore-functions/blob/master/AGGREGATION.md). There are many options here as well.

### Get all products a user purchases...

Obviously you can just query the subcollection as normal, or use a collecitionGroup query.

**IN SUM**

There are so many options and customizations for these arrays, but this allows you to basically create scalable, automatic arrays (or maps) of any type that grow and can be searched for, easily.

Try it out, let me know bugs on github etc. Pretty [complex code](https://github.com/jdgamble555/adv-firestore-functions/blob/master/src/shared/joins.ts#L229), but I wanted to handle every situation automatically.

I also realize there is now Firebase 9. You could translate these frontend functions easily. That is on you as I do not have infinite time.

You should also note that this alone makes Fireship.io's [follower feed function](https://fireship.io/courses/firestore-data-modeling/models-social-feed/), scalable. Get those courses, as I find them invaluable.

The final touch on my scalable follower feed, will be to create an index using this index. You can already get mostly there with these two functions alone, but there are some caveats I will need to address, write more code, etc.

The biggest problem will be updating some of these documents that change...

To be continued more on the follower feed...

J