Redis as a Database — Live Data Updates with PubSub and GraphQL Subscriptions
Tags: redis, pubsub, graphql, react
Published: 2022-11-13 19:37:50

---
title: Redis as a Database — Live Data Updates with PubSub and GraphQL Subscriptions
published: true
description: 
tags: Redis, PubSub, GraphQL, React
series: Redis as a Database
cover_image: https://cdn-images-1.medium.com/max/2560/1*02RJMWpH7UtlZaJPEtGiiQ.gif
# published_at: 2022-11-13 19:35 +0000
---

My pet project, which I named **Visualizer** and that I use to learn more about Redis, can start ingesting tweets from Twitter’s [volume stream](https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/introduction) or from their [filtered stream](https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/introduction).

While it is doing so, **Visualizer** can produce a live stream of all the hashtags that are currently encountered in the ingested tweets and it ranks them too.

In this post I want to go over those parts of the system that handle new tweets, extract and publish the ranked hashtags and finally push them to the frontend to create that nice word cloud.

If you just want to see the code, here’s the GitHub [link for the backend](https://github.com/mariusmuntean/Visualizer) and here’s the [link to the frontend](https://github.com/mariusmuntean/VisualizerFrontend). Btw. this is my third post on using Redis, so make sure to read the others too.

## Visualizer Partial Architecture

**Visualizer** is made up of three components.

![Visualizer — Partial Architecture](https://cdn-images-1.medium.com/max/2000/1*X-r8qcz_rGe9_ZKEQlNckA.png)

**Visualizer Ingestion** — An ASP.NET Core 6 service that handles the stream of tweets and stores them in Redis as indexed documents (in JSON format). It also publishes each encountered hashtag together with its rank, i.e. a count of how many times it was ever encountered by **Visualizer**.

**Visualizer API** — Another ASP.NET Core 6 service that offers access to the stored tweet data and to the live stream of ranked hashtags. [GraphQL](https://graphql.org/) is another technology that I love and its [subscriptions](https://graphql.org/blog/subscriptions-in-graphql-and-relay/) are ideal to push the stream of ranked hashtags to any consumer.

**Visualizer Frontend** — A React Typescript + [ViteJs](https://vitejs.dev/) SPA to show of **Visualizer**’s features.

In a future post I’ll talk more about my reasons for this architecture, so follow me no to miss it.

## Publishing Ranked Hashtags

As I mentioned, **Visualizer Ingestion** gets tweets from Twitter’s API v2 and besides persisting them it also publishes any encountered hashtag together with its rank.

The tweet data retrieval isn’t very interesting. Suffice it to say that I’m using [Tweetinvi](https://linvi.github.io/tweetinvi/dist/streams-v1.1/streams-introduction.html) for this.

For every extracted hashtag a few things happen:

* **Store and Rank**: I’m storing every hashtag in a [Redis Sorted Set](https://redis.io/docs/data-types/sorted-sets/#:~:text=A%20Redis%20sorted%20set%20is,Leaderboards.). This Redis data structure has the expected time and space characteristics of a set, with the addition that adding an element multiple times will increase its rank each time. This is excellent for my use case as I don’t have to do much besides pumping all the hashtags in my sorted set.

Here’s the code to get an instance of *IDatabase.* It is probably best to add it to your DI container.
{% gist https://gist.github.com/mariusmuntean/9cb4533bf6c92b9de4cc85864a975283.js %}
Afterwards I’m adding the hashtag to a sorted set and I’m telling Redis to increment the rank of this hashtag by 1. Notice that I get the new rank as a response.
{% gist https://gist.github.com/mariusmuntean/dbd34ba67393c71b2231ed79bc8114cf.js %}
* **Publish Hashtag with Rank**: After I got the new rank of the hashtag I’m publishing it. Here I’m making use of [Redis’ Pub/Sub](https://redis.com/redis-best-practices/communication-patterns/pub-sub/) which allows me to publish string messages to a channel. This effectively decouples **Visualizer Ingestion **from **Visualizer API**.

First, you’ll need an *ISubscriber* instance. Like most reusable objects, I’m adding it to my DI container.
{% gist https://gist.github.com/mariusmuntean/318428002c7c872d219b21497b129151.js %}
With that subscriber I can then publish my ranked hashtag to a channel
{% gist https://gist.github.com/mariusmuntean/b66704f0a98ec21d311d91764904d385.js %}
Notice that I’m using a *RankedHashtag* DTO to represent the hashtag and its rank and that I’m serialising it to a JSON string before publishing. Also notice that I’m using a constant for the channel name. That’s because the constants live in a shared project in the same solution and are reused by **Visualizer API**.

## Consuming Ranked Hashtags

The stream of ranked hashtags is consumed by **Visualizer API**. It does so by subscribing to the same channel that **Visualizer Ingestion** uses to publish each hashtag with its current rank.

In a singleton service, I’m registering a handler for new hashtags in that stream
{% gist https://gist.github.com/mariusmuntean/813caedbcd576365b00817bc9a39901e.js %}
In the handler, each channel message is deserialised to an instance of *RankedHashtag *and passed on to any GraphQL subscriber
{% gist https://gist.github.com/mariusmuntean/f36876fae9ef7f2d417718b459ef0b42.js %}
The *_rankedHashtagStream* that you see there is a plain Reactive Extensions ReplaySubject with a buffer of 1
{% gist https://gist.github.com/mariusmuntean/c1ec618e0d0b73439f3ab8df427a12d6.js %}
It immediately broadcasts any notification to its subscribers. But who are its subscribers, I hear you asking? Its the GraphQL API users that subscribed to the ranked hashtags; they are the subscribers.

The way that’s done in **Visualizer** is quite specific to my needs and to the library that I’m using, called [GraphQl-Dotnet](https://github.com/graphql-dotnet/graphql-dotnet).
{% gist https://gist.github.com/mariusmuntean/e609c3a06b24449073b2b6bd40083358.js %}
Please notice that the GraphQL *rankedHashtag* subscription takes an argument called *sampleIntervalSec* with a default value of 0. That value is used to determine if a subscriber should receive all the ranked hashtags as soon as they come (commonly referred to as real-time, but not in its strict sense), or if the subscriber should receive another stream of ranked hashtags which is obtained by sampling the “real-time” stream every *sampleIntervalSec* seconds.

## Visualizer Frontend

This might be the least interesting part of the system because it only deals with data retrieval and rendering. It uses [Apollo Client for React](https://www.apollographql.com/docs/react/) to connect to **Visualizer API **and [react-wordcloud](https://github.com/chrisrzhou/react-wordcloud) to render the data.

At a high level, the frontend retrieves the top 50 hashtags, sorted descendingly by their rank (not shown in this post, but it is a simple sorted set operation). This is in order to have something to display in the wordcloud.

Then it subscribes to the stream of ranked hashtags and whenever a ranked hashtag arrives, it either overwrites one of the existing 50 hashtags or it is added to the same set.

**Subscribe to Ranked hashtags**

I’m using the codegen feature of Apollo Client to create a nice React hook for subscribing to the ranked hashtags
{% gist https://gist.github.com/mariusmuntean/a3dadfd62151c035b88c212ed88df997.js %}
Then my component is re-rendered every time a new ranked hashtag is available. I’m merging the new ranked hashtag with the initial 50
{% gist https://gist.github.com/mariusmuntean/49306d6c79de6cb99d0de3b8a79b2966.js %}
**Ranked Hashtags Wordcloud**
{% gist https://gist.github.com/mariusmuntean/b1ff1909548c4237682f30befb596fcf.js %}
## Conclusion

Displaying a nice little word cloud involves a few system components and multiple concepts, but each is easy to understand.

Let me know if you like these posts, with more code than usual. Give the post a ❤️ to let me know and follow me not to miss any future posts on Redis.