Firestore Pagination / Infinite Scrolling + Snapshot Listeners (in React / useEffect)
Published: 2021-03-10 10:58:57

---
title: Firestore Pagination / Infinite Scrolling + Snapshot Listeners (in React / useEffect)
published: true
description: 
tags: 
//cover_image: https://direct_url_to_image.jpg
---

I would like to share with you something that I noticed many developers have been trying to implement in their applications and that is reactive pagination with onSnapshot listeners, i.e. loading documents from collection in batches/bulks/pages and also listening when someone changes, adds or removes a document in the collection.

First of all, before I share my code I would like to emphasize that this is probably not what you want to do generally. As Firebase Consultant  
[Doug Stevenson said in one of his comments on stackoverflow posts](https://stackoverflow.com/questions/55384694/android-firestore-infinite-pagination-with-listeners#comment97491338_55384694)

> "You can't really combine paging with listening. You have to choose one or the other in order to maintain the sanity of your code."

It's good to understand trade-offs and use cases of this. I can't think of application which uses realtime listening on a big feed with pagination / infinite scrolling and handles in realtime actions of editing and removing an item. Even in chats, if you remove a message, you just change its state to `removed=true` and don't immediately delete it from database completely. It's sort of heavy thing to maintain and have in your application and it is good to evaluate which functionality do you actually need.

But, for the sake of theoretical exercise, let's try to implement it!

This isn't completely trivial as you have to set up `startAfter` and `endAt` firestore cursors to prevent document(s) from one bulk/batch/page to load in another one.

You therefore need two queries.

One for initial load of documents with `limit(PAGE_LIMIT)` (it still needs `startAfter` - that is in `query` variable)
```javascript
query.limit(PAGE_LIMIT).get()
```
And another one which listens for document changes in between `startAfter` and `endAt` changes (yup, `startAfter` is in `query` variable already)
```javascript
query.endAt(newPosts[newPosts.length-1].timestamp).onSnapshot()
```

I don't really think you can make it just with one query, unless you can predict temporal scale of your posts and ignore explicit PAGE_LIMIT.

So here's the code:

```javascript
import { useEffect, useRef, useState } from 'react'

const PAGE_LIMIT = 5;

const useInfiniteScroll = ({ fetching: fetchingInit = false, hasMore: hasMoreInit = false, threshold = 200 }) => {
  const [fetching, setFetching] = useState(fetchingInit);
  const [hasMore, setHasMore] = useState(hasMoreInit);

  useEffect(() => {
    window.addEventListener('scroll', handleScroll);
    return () => window.removeEventListener('scroll', handleScroll);
  }, [fetching, hasMore]);

  function handleScroll() {
    const offsetHeight = document.documentElement.offsetHeight, innerHeight = window.innerHeight,
          scrollTop = window.pageYOffset || document.documentElement.scrollTop || document.body.scrollTop || 0;

    if (!hasMore || fetching || innerHeight + scrollTop + threshold <= offsetHeight) return;
    setFetching(true);
  }

  return [fetching, setFetching, setHasMore];
}

const PostFeed = ({}) => {
  const [pages, setPages] = useState([[]]),
        [fetching, setFetching, setHasMore] = useInfiniteScroll({ fetching: true, hasMore: true }),
        listeners = useRef([]);

  useEffect(() => () => listeners.current.map(l => l()), []);

  useEffect(() => {
    if (!fetching) return;
    const l = pages.length, page = pages[l - 1] || [], postLast = page[page.length - 1];
    let query = firebase.firestore().collection('posts').orderBy("timestamp", "desc");
    if (0 < l) query = query.startAfter(postLast?.timestamp || new Date(253402300799999));

    let mounted = true;
    query.limit(PAGE_LIMIT).get().then(snapshot => {
      if (mounted) {
        const posts = snapshot.docs.map(doc => doc.data());
        setPages(ps => ps.concat([posts]));
        const unsubscribe = query.endAt(posts[posts.length-1]?.timestamp || new Date()).onSnapshot(snapshot => {
          const postsUpdated = snapshot.docs.map(doc => doc.data());
          setPages(ps => ps.map((b, i) => i === l ? postsUpdated : b));
        });
        listeners.current = listeners.current.concat([unsubscribe]);
        if (posts.length < PAGE_LIMIT) setHasMore(false);
        setFetching(false);
      }
    });

    return () => mounted = false;
  }, [fetching]);

  return (
    <>
      {pages.map(page => page.map(post =>
        <PostCard key={post.id} post={post} />
      ))}
      { fetching && <SomePostSkeleton /> }
    </>
  )
}

export default PostFeed;
```

Some caveats: 
- `new Date(253402300799999)` is the maximum possible date for [firestore Timestamp object](https://firebase.google.com/docs/reference/js/firebase.firestore.Timestamp) and that is 9999-12-31T23:59:59.999999999Z. We use this to reuse the same query and listen for new future posts in the first page. (I changed some stuff, see the last point and I believe we do not need it anymore - If `0 < l` ->  `last(batch)?.[orderBy]` has to exist, only if the last batch was empty, but in such case `setHasMore` would be `true` and we wouldn't be able to set `fetching` to `true` anymore and that code would never run.)
- `useInfiniteScroll` is traditional `useInfiniteScroll` hook, nothing that special about it, maybe it's important to notice that we set its value of `fetching` to true immediately in `PostFeed`, because we use the same queries for initial load and then for scroll / page load.
- The notion of page... I used the terms bulk/batch before as well. At this point I am unsure how to name these 'clusters' of posts. But yes, as you can notice, our post array is an array of arrays really and it is called 'pages'.
- The variable `l` is preserved within the scope of `useEffect` so each time a document in batch/page is updated, the change will map into the right batch/page.
- Don't forget to unsubscribe your listeners. Those we store in `useRef` because `useState` should be used only if it affects the render and unsubscribe all of them at once in the return function of the other `useEffect`.
- we use `mounted` variable to handle out-of-order responses. I.e., it can happen that firestore promise is still pending, but component has been unmounted. If promise finishes, setting a state on unmounted component will result in React warning. See https://reactjs.org/docs/hooks-faq.html#how-can-i-do-data-fetching-with-hooks and search for 'out-of-order' responses.
- We also use `endAt` instead of `endBefore` because `startAfter` excludes the end of previous batch/page (open interval) and `endAt` includes the end of current batch/page (closed interval).
- we use `if (0 < l)` condition to include also documents which do not have `timestamp` property. We expect every document to have `timestamp` property, but... actually, when updating a document, there is a brief moment of time when the `timestamp` is not indexed yet - such document is not included in `startAfter`-`endAt` interval and it can cause annoying flicker of modified document, therefore we do not have `startAfter` when adding the first batch.

Please, feel free to ask any questions or add remarks! I might not have managed to explain everything in clarity.

Cheers and enjoy