How LinkedIn Uses Kafka, Samza and RocksDB to Smartly Route Notifications?
Tags: eventdriven, aws, kafka, programming
Published: 2024-07-06 07:11:55

LinkedIn understands the importance of staying connected within your professional network, even when you're not actively using the app. To achieve this, they've developed a sophisticated system called [Air Traffic Controller (ATC)](https://www.linkedin.com/blog/engineering/messaging-notifications/air-traffic-controller-member-first-notifications-at-linkedin) that manages all outgoing notifications. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kq8gn0g1z27zhvc2d29v.png)

Learn all about smart channel routing for notifications. 

`Original Content here: https://www.suprsend.com/post/how-linkedin-smartly-route-their-unseen-notifications-from-in-app-inbox-to-your-email`

## TABLE OF CONTENTS
- [What is LinkedIn’s Notifications Channel Routing?](#what-is-linkedin’s-notifications-channel-routing)
- [LinkedIn’s ATC Smartly Routing In-app Notifications to Emails](#linkedins-atc-smartly-routing-in-app-notifications-to-emails)
  1. [In-App Alerts and Tracking](#in-app-alerts-and-tracking)
  2. [Prioritization and Inactivity Triggers Email Routing](#prioritization-and-inactivity-triggers-email-routing)
  3. [Personalization Through Data and Machine Learning](#personalization-through-data-and-machine-learning)
  4. [Keeping Your Email Inbox Clean & Manageable](#keeping-your-email-inbox-clean--manageable)
- [Implementing Smart Notifications Channel Routing](#implementing-smart-notifications-channel-routing)
- [How Does SuprSend’s Smart Channel Routing Node Work in the Notification Workflow?](#how-does-suprsends-smart-channel-routing-node-work-in-the-notification-workflow)
  1. [Identifying Your Delivery Channels](#identifying-your-delivery-channels)
  2. [User Preferences: Opting In/Out](#user-preferences-opting-inout)
  3. [Prioritizing the Channel: Cost vs. Engagement](#prioritizing-the-channel-cost-vs-engagement)
  4. [Sequential Delivery with Time Intervals](#sequential-delivery-with-time-intervals)
  5. [Mission Accomplished (or Not!)](#mission-accomplished-or-not)
- [Closing Remarks](#closing-remarks)

---

### What is LinkedIn’s Notifications Channel Routing?

As a software company, you already understand the importance of good communication with your users. Notifications serve as the crucial lifeline, keeping users informed, engaged, and ultimately driving product adoption.

Often, companies go multi-channel with notifications to provide an omnichannel experience to their users and reach out to them promptly. However, going multi-channel is tricky and costly.

At first glance, it might seem like a good idea to simultaneously send notifications on all available channels. This way, users can see the notification quickly, no matter which platform they’re on.

While the cost of sending notifications on two additional channels may seem negligible for a single user, the expenses escalate significantly at scale. For platforms sending upwards of 1 million notifications monthly (which is very common), the costs can multiply by 2x to 4x with the addition of new channels. This can quickly become a substantial burden.

> A potential solution is to stagger multi-channel notifications. This approach can range from simple timed delays to complex systems that detect if a user hasn't seen a notification or is active on a specific channel. However, building and perfecting such a system is challenging and often requires a dedicated engineering team.

That’s why LinkedIn developed smart channel routing. This intelligent system takes multi-channel notifications to the next level. It decides the best way to deliver a notification—whether it’s a push notification, email, in-app alert, or even SMS—making it efficient, effective, and economical.

Let’s consider an actual example from LinkedIn.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fv0ka7kkgc5mhhdaegzb.png)

This is a sample LinkedIn member’s notification preference for ‘Mentions-related notifications’ on LinkedIn. The member has enabled receiving notifications in LinkedIn’s in-app notification center, push, and email.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qxk3uh4h1fwkgxxu7rb8.png)

On priority, the member receives an in-app notification in the app. After some time, when the member has not viewed the notifications on the in-app notification center, they receive an email notification informing them about the same ‘Mention’ notifications. The best part is that if you see the message in the notification center during the message window (like 3 hours), LinkedIn automatically cancels the email notification, reducing notification clutter and saving costs.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gj5rpg87bbfa72m2nk4s.png)

Now that you know about smart routing, let’s dive deep into the technicals.

### LinkedIn’s ATC Smartly Routing In-app Notifications to Emails

#### 1. In-App Alerts and Tracking

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uusbyltnt6rz2j9zr4mw.png)

Let's say you receive a notification about a new job opportunity or a connection request. This notification likely appears on your phone's screen or within the app's dedicated notifications section called the in-app notification center. Behind the scenes, ATC leverages several technologies:
- LinkedIn uses Apache Kafka message queue to send notification requests to ATC, potentially containing details about the notification type and recipient.
- An unseen notification counter or flag within the app's data structures is used to identify unseen notifications.

#### 2. Prioritization and Inactivity Triggers Email Routing

Unseen notifications don't disappear into the void. ATC, powered by Apache Samza (a stream processing framework), constantly processes data to maintain the notification state based on user feedback. Here's where email routing comes into play:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/39sjkslow2c1p4z3oypx.png)

- **Inactivity Period:** On LinkedIn, Samza pipelines are designed to analyze timestamps associated with notifications stored in RocksDB (a key-value store used by ATC). If a notification remains unseen for a certain period (e.g., 24 or 48 hours) based on these timestamps, it could be flagged for email routing.
- **Notification Priority:** Data from the Kafka request or RocksDB also includes notification priority. High-priority unseen notifications (e.g., important job alerts) are routed to email sooner than lower-priority ones (e.g., connection requests).

#### 3. Personalization Through Data and Machine Learning

Here's where things get interesting:
- **User Data in RocksDB:** RocksDB not only stores timestamps but also user data relevant to unseen notification routing, such as notification settings that you can configure within the LinkedIn app (preferences for email routing). It's very similar to what we have built in both embeddable and hosted manner. [Preferences](https://docs.suprsend.com/docs/preferences?_gl=1*i70eo3*_gcl_au*OTg3Mjc0NTAwLjE3MjAyNDg3NDk.*_ga*NDUwNTAxNTM1LjE3MjAyNDg3NDk.*_ga_PPDYBESP2L*MTcyMDI0ODc0OS4xLjEuMTcyMDI0ODkxMS41OS4wLjA.)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m16ma2jg8fbbeiqb7ylg.png)

- **Machine Learning Models:** The references mention relevance processors using machine learning models stored in RocksDB. While the focus seems to be on channel selection (email vs. push), these models also consider factors like unseen notification status and user data to influence decisions about routing unseen notifications to your email.

#### 4. Keeping Your Email Inbox Clean & Manageable

To prevent being bombarded with emails, unseen notifications are grouped together and delivered in a digest format at an optimal time based on your preferences (if available) or historical activity. Here's how ATC achieves this:
- **Frequency Capping:** Samza's functionalities are used to implement frequency caps, also known as throttling, limiting the number of emails sent per user within a specific timeframe.
- Throttling is something, we have implemented at our end too. How it works is that it limits the number of workflow executions per user in a given time frame. You can set up two properties in this node.
- Max Executions - Maximum number of workflow executions permitted in the time window.
- Window - The duration for which the throttle count is maintained. For example, if max executions is set to 3 and the window is 15 minutes, a maximum of 3 workflows can execute within a 15-minute window. You can define the window as **d **h **m **s.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u0c47b2ph6x3s46pmld3.png)

- **Scheduling and Aggregation:** Samza's scheduler schedules emails for optimal delivery times. ATC also leverages RocksDB to aggregate unseen notifications for efficient delivery in a single email.

### Implementing Smart Notifications Channel Routing

Building a notification channel routing system like LinkedIn's ATC can involve significant engineering complexity. However, the benefits it offers – less confusion, fewer complaints

 about bombarding notifications, improved user experience, reduced costs, and potentially higher engagement – are highly valuable for any company that values customer experience highly.

The good news? You might not need to start from scratch.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lfmhsnx0j5kvjpxyfbtj.png)

At SuprSend, we've addressed this challenge by integrating an intelligent channel routing algorithm internally with the notification workflows, sparing you the need for extensive development. Simply activate the channel routing node within SuprSend's workflow builder to harness this functionality.

### How Does SuprSend’s Smart Channel Routing Node Work in the Notification Workflow?

While LinkedIn's ATC system's inner workings are somewhat shrouded in mystery, we can gain valuable insights by exploring a practical example of smart channel routing in action. Here's a breakdown of how SuprSend's routing node executes:

#### 1. Identifying Your Delivery Channels

Similar to how ATC considers user preferences, SuprSend starts by assembling the most suitable channels for notification delivery. This involves a two-factor approach:
- **Active Channels on the Template:** This consists of content for notification channels (email, push notifications, inbox, Slack, etc.) that are currently operational and ready for rendering on the trigger.
- **Active Channels in the User Profile:** These are the channels the user has provided for receiving notifications. However, any inactive channels (perhaps due to bounced emails or unregistered phone numbers) are excluded from the delivery. SuprSend even keeps a watchful eye, marking channels inactive if there are issues like spam complaints.

#### 2. User Preferences: Opting In/Out

Taking inspiration from the potential role of user preferences in LinkedIn's ATC, SuprSend goes a step further. Here, we check if the user has explicitly volunteered (opted-in) to receive notifications via a specific channel (e.g., email or SMS) for a particular notification category. This ensures the message reaches the user on their preferred channel, potentially leading to higher engagement.

#### 3. Prioritizing the Channel: Cost vs. Engagement

Now it's time to decide the order in which the channels will attempt delivery. SuprSend's routing logic echoes the concept of optimizing delivery based on factors like cost, user’s online status, and user engagement:
- **Current Focus: Cost-Effectiveness:** By default, SuprSend prioritizes cost-efficiency. This means the "cheapest" channels (e.g., push notifications) are sent first, similar to the potential cost-saving benefits mentioned about LinkedIn's ATC.
- **Future Innovation: Engagement Focus:** This involves sending notifications through channels with a higher likelihood of user interaction (opens and clicks) first. This aligns with the concept of user engagement highlighted throughout the article. For example, if a user is online on the application, send an in-app notification first, then wait and route it via email.
- **Custom Routing:** You can control which channels to send notifications on and in what order.

#### 4. Sequential Delivery with Time Intervals

Once the order of channels is determined, SuprSend sends the notification on each channel sequentially instead of bombarding the user all at once. We space out attempts with specific time intervals between each channel.

#### 5. Mission Accomplished (or Not!)

The notification delivery process continues through the prioritized channels until a pre-defined "success metric" is achieved. This metric could involve factors like user interaction or confirmation of receipt. Once successful delivery is confirmed, SuprSend stands down the remaining channels – the mission is a success!

### Closing Remarks

In essence, LinkedIn's ATC acts like a control tower, strategically managing unseen notifications. By considering factors like inactivity, priority, and potentially your preferences, ATC ensures you don't miss important updates even when you're away from the app. The technical underpinnings of ATC, including Apache Kafka, Apache Samza, and RocksDB, work together to achieve this seamless notification routing.

Though you can build the smart channel routing mechanisms in-house, it is a complex and resource-intensive activity. Being a notifications infrastructure platform, SuprSend provides you with an inbuilt channel routing node, which you can simply use in your notifications workflows.

---

You can check the documentation here: [SuprSend Documentation](https://docs.suprsend.com/docs/getting-started)