How I Manage a Ten Thousand Push Notifications a Minute (CloudTask)
Tags: pushnotification, firebase, cloudfunction, cloudtask
Published: 2021-12-25 05:15:04

---
title: How I Manage a Ten Thousand Push Notifications a Minute (CloudTask)
published: true
description: Send massive notifications is hard, this is how I manage to do that using Google CloudTask, Firebase, CloudFunction
tags: pushnotification, firebase, cloudfunction, cloudtask
//cover_image: https://direct_url_to_image.jpg
---

I'm working on an app that we need to send personalized notifications to every user on schedule (9 AM, 3 PM, 8 PM) in their timezone.

Each user will receive a unique notification related to them

### Simple architecture:
![image](https://user-images.githubusercontent.com/7245986/147377268-1a270917-041b-4dae-9408-8bc9cb087240.png)

> As the number of users grows, we run into a bottleneck because only one cloud function is too slow to process 10k users.

### Solution
![image](https://user-images.githubusercontent.com/7245986/147377387-aa34fedb-c47c-4fd3-8635-cd73be6fc5ad.png)

> The hint here is to split the workload into smaller chunks (2k users task). Utilize Cloud Task as a queue to slowly distribute tasks and it will trigger multiple cloud functions to send notifications.

### Main Cloud Function: 
The main function will run at the scheduled time, it will get the total number of users (we store it in a separate document) and create multiple tasks


```typescript
const pages = []
const batchSize = 2000;
let currentIndex = 0;
while (currentIndex < totalUsers) {
  pages.push({ fromIndex: currentIndex, toIndex: currentIndex + batchSize });

  currentIndex += batchSize;
}
    
await PromisePool.for(pages)
      .withConcurrency(10)
      .process(async (page) => {
        await createHttpTaskWithToken({
          projectId,
          queue: 'YOUR-CLOUD-TASK-QUEUE-NAME',
          url: `https://us-central1-${projectId}.cloudfunctions.net/sendNotifications`,
          email: `YOUR-SERVICE-EMAIL@${projectId}.iam.gserviceaccount.com`,
          payload: {
            fromIndex: page.fromIndex,
            toIndex: page.toIndex,
          },
        });
      });
```

### Send Notifications Function: 
This function only processes 2k users (maximum) at a time. 
1. Get 2k users by using pagination cursor (fromIndex, toIndex)
2. Filter out only users at the correct timezone
3. Prepare user notification message (title, content, image)
4. Call firebase messaging API to send notifications.

```typescript
export const sendNotifications = functions
  .runWith({
    timeoutSeconds: 540,
    memory: '1GB',
  })
  .https.onRequest(async (req, res): Promise<void> => {
    const { fromIndex, toIndex } = req.body;

    const usersDataResponse = await db.users.orderBy('userIndex').startAt(fromIndex).endAt(toIndex).get();
    const userToSendNotifications = usersDataResponse.docs.filter(onlyUserAtTimezone)

    try {
      await PromisePool.for(userToSendNotifications)
        .withConcurrency(100)
        .process(async (user) => {
          try {
            const userData = user.data();
            const tokens = userData.tokens || [];
            const notificationData = prepareNotification(userData);

            if (notificationData && tokens.length > 0) {
              const response = await messaging().sendMulticast({
                tokens: tokens,
                notification: notificationData,
              });

              if (response.failureCount > 0) {
                const failedTokens: string[] = [];
                response.responses.forEach((resp, idx) => {
                  // Cleanup the tokens who are not registered anymore.
                  if (
                    resp.error?.code === 'messaging/invalid-registration-token' ||
                    resp.error?.code === 'messaging/registration-token-not-registered'
                  ) {
                    failedTokens.push(tokens[idx]);
                  }
                });

                // remove failed tokens
                if (failedTokens.length > 0) {
                  await db.users.doc(user.id).update({
                    tokens: firestore.FieldValue.arrayRemove(...failedTokens),
                  });
                }

                console.log('List of tokens that caused failures: ' + failedTokens);
              }
            }
          } catch (error) {
            functions.logger.error('error', error);
          }
        });
      res.send('OK');
    } catch (error) {
      res.sendStatus(500);
    }
  });

```

## Read more
- https://www.gojek.io/blog/how-we-manage-a-million-push-notifications-an-hour
- https://cloud.google.com/tasks/docs/creating-http-target-tasks
- https://firebase.google.com/docs/firestore/query-data/query-cursors