Real-time visualization with React and D3.js
Tags: react, tutorial, memgraph, d3js
Published: 2022-01-07 08:10:54

## Introduction

In this blog post, you will learn how to create a **React** app and **WebSocket** that connects to the **Flask** server, and visualize streaming data in real-time using **D3.js**. I decided to write this blog post as a part of my learning path, and I hope to help anyone struggling with the same problems while trying to implement something similar. I decided to tackle on the frontend implementation by visualizing **Twitter** users who retweeted something with the hashtag *#christmas*. Each community of users is presented with a different color which allowed me to notice the important communities in the Twitter network. 

The frontend service is a part of a web application that you can find in the GitHub [repository](https://github.com/memgraph/twitter-network-analysis/tree/add-backend-server). Besides that, the repository also holds a visualization that discovers [the most Christmassy person on Twitter](https://memgraph.com/blog/twitter-analysis-with-dynamic-pagerank) using the dynamic PageRank algorithm. 

## Prerequisites and app architecture 

If you are using Windows, you need to install [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) (WSL) and then [Docker Desktop](https://docs.docker.com/desktop/windows/install/). On the other hand, if you are a Linux user, just install Docker and [Docker Compose](https://docs.docker.com/compose/install/).

The app is dockerized and it consists of five services:
1. **`stream`**: A  Python script collects new retweets with the hashtag *#christmas* and sends them to a Kafka cluster.
2. **`kafka`**:  A Kafka cluster consisting of one topic named `retweets`. 
3. **`memgraph-mage`**:  The graph analytics platform where we store the incoming Twitter data from Kafka and perform [**dynamic PageRank**](https://memgraph.com/docs/mage/algorithms/dynamic-graph-analytics/pagerank-online-algorithm) and [**dynamic community detection**](https://memgraph.com/docs/mage/algorithms/dynamic-graph-analytics/community-detection-online-algorithm) on all Twitter users.
4. **`backend-app`**: A **Flask server** that sends all the data we query from `memgraph-mage` to the `frontend-app`. It also consumes the Kafka stream and sends it to the `frontend-app`.
5. **`frontend-app`**:  A **React app** that visualizes the **Twitter** network using the **D3.js** library.


![real-time-visualization-with-react-and-d3-js](https://public-assets.memgraph.com/real-time-visualization-with-react-and-d3-js/memgraph-tutorial-twitter-app-architecture.png)


## Project structure

You can see the whole project structure in the GitHub [**repository**](https://github.com/memgraph/twitter-network-analysis/tree/add-backend-server). The blog post focuses on the frontend service and explains how the visualization was created.
```
|   docker-compose.yml
|   
+---backend
|       Dockerfile
|   +---server
|   +---tests
| 
+---frontend
|   |   .dockerignore
|   |   Dockerfile
|   |   package.json
|   |   package-lock.json
|   +---node_modules
|   +---public
|   +---src
|
+---memgraph
|   |   Dockerfile
|   |   requirements.txt
|   +---procedures
|   +---transformations
|
+---stream
|   |   Dockerfile
|   |   kafka_utils.py
|   |   produce.py
|   |   requirements.txt
|   +---data
```

The `frontend` folder was created using the [`create-react-app`](https://www.npmjs.com/package/create-react-app) `npm` package. If you are starting from scratch and want to **create a React app** follow these steps:

1. Place yourself in the root folder of your project.
2. Run `npm install -g create-react-app` (if you don't want to install the latest version, you can specify the version of the `create-react-app`, for example, `create-react-app@5.0.0`).
3. Next, run `npm init react-app frontend --use-npm`, which will initialize the `react-app` package in the `frontend` folder.
4. In the end, place yourself in the `frontend` folder by running `cd frontend` and start the app with `npm start`.

Even a simpler way of creating a React app is by using `npx` - a package runner tool that comes with `npm 5.2+`. Then you just have to run:
```
npx create-react-app frontend
cd frontend
npm start
```

## Socket.IO library

I have used `socket.io-client@2.3.1` since I had issues with the latest version. I am going to explain the process on the [`CommunityDetection`](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js) component, since it's very similar to the [`PageRank`](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/PageRank.js) component. If you are running the frontend application locally, and not using the provided dockerized application, make sure to install the library by running: 
```
npm install socket.io-client@2.3.1
```
> Don't forget that [Node.js](https://nodejs.org/en/) is a prerequisite for using `npm`.

First, we are going to import the socket we're using on the client side. [Backend](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/backend/server/app.py) is implemented with Flask. 
```javascript=
import io from "socket.io-client"
```

After that, we are initializing the socket.

```javascript=
var socket = io("http://localhost:5000/", { 
                transports: ["websocket", "polling"] 
            })
```

We set the socket to listen to the server running at `http://localhost:5000/`. Then we established the connection to the server with `websocket` first. If `websocket` is not available, connection to the server will be established with `HTTP` `long-polling` - successive `HTTP` requests (`POST` for writing, `GET` for reading). Next, we need to handle different events with our socket are handled. When the connection is established, the socket is emitting the `consumer` signal. This signal is also emitted on the server side, whenever a new message is sent. This configuration allows the socket to receive all messages related to `consumer` signal.

```javascript=
socket.on("connect", () => {
    socket.emit('consumer')
    console.log("Connected to socket ", socket.id)
});

socket.on("connect_error", (err) => {
    console.log(err)
    // try reconnecting
    socket.connect()
});

socket.on("disconnect", () => {
    console.log("Disconnected from socket.")
});

socket.on("consumer", (msg) => {
    console.log('Received a message from the WebSocket service: ', msg.data);
    }
```

## `React.Component` lifecycle 

![memgraph-tutorial-react-component-lifecycle](https://public-assets.memgraph.com/real-time-visualization-with-react-and-d3-js/memgraph-tutorial-react-component-lifecycle)
<center>Image by <a href="https://github.com/wojtekmaj/react-lifecycle-methods-diagram">Wojciech Maj</a> from <a href="https://projects.wojtekmaj.pl/react-lifecycle-methods-diagram/">React Lifecycle Methods diagram</a></center>
<br></br>

You may ask yourself where to place all this `socket.io` code within a React component. First, I initialized the socket in the component's constructor. After that I have placed the socket events in [`componentDidMount()` lifecycle method](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L65). This part of the `React.Component` lifecyle is invoked once, immediately after a component is mounted. If you need to load data from a remote endpoint, this is a good place to instantiate the network request. This method is also a good place to set up any subscriptions. That's why I have decided to place all socket events there. On each `consumer` emit, the state of the component will be updated and that will trigger an extra rendering which will happen before the browser updates the screen, so the user won’t see the intermediate state. Before I set up the socket, at the beginning of the `componentDidMount()`, I have made [a simple `HTTP` request](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L37) that will trigger the backend to start producing the needed data.

```javascript=
firstRequest() {
    fetch("http://localhost:5000/api/graph")
        .then((res) => res.json())
        .then((result) => console.log(result))
}
```
After that, I initialized everything that was necessary for drawing using `D3.js` in the [`initializeGraph()` method](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L176). By setting a new state of nodes and links with `setState()` on each `consumer` emit, `componentDidUpdate()` lifecycle method will be called. In that method we are updating the graph by drawing new incoming nodes and links. This lifecycle method is not called for the initial render, and that's the reason why we initialized everything in the `initializeGraph()` method.

In the end, when the component unmounts (for example, when we click on the button to switch to the PageRank), the [`componentWillUnmount()` lifecycle method](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L121) is called and the client disconnects from the server.

```javascript=
componentWillUnmount() {
    this.socket.emit('disconnect');
    this.socket.disconnect();
}
```

>If you want to learn more about `React.Component` lifecycle methods, check out the React official [docs](https://reactjs.org/docs/react-component.html#the-component-lifecycle).


## Visualizing with D3.js

We want to draw the graph on `svg` using `D3.js` within the **class component**. We are going to do that by creating a reference in the component constructor which will be attached to the `svg` via the `ref` attribute. In [constructor](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L18) we have to use `createRef()` method.

```javascript=
constructor(props) {
    super(props);
    this.myReference = React.createRef();
    this.state = {
        nodes: [],
        links: []
    }
    this.socket = io("http://localhost:5000/", { transports: ["websocket", "polling"] })
}
```
Then, in the component's [`render()` method](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L334) we are adding the `ref` attribute with value `this.myReference` to the `svg`.

```javascript=
render() {
    return (<div>
        <h1>Community Detection</h1>
        <p>Number of users that retweeted so far: {this.state.nodes.length}</p>
        <svg ref={this.myReference}
            style={{
                height: 500,    //width: "100%"
                width: 900,
                marginRight: "0px",
                marginLeft: "0px",
                background: "white"
            }}></svg></div>
    );
}
```

Now, by [selecting the current attribute](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L177) of the reference, it's easy to get the `svg` on which we are going to draw our graph.
```javascript=
var svg = d3.select(this.myReference.current);
```

> If you want to know how to use `D3.js` within **function component**, check out one of my previous blog posts - [Twitch Streaming Graph Analysis - Part 2](https://dev.to/katelatte/twitch-streaming-graph-analysis-part-2-3d3d).

In the [`updateGraph()` method](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/CommunityDetection.js#L252) we have to draw the nodes and relationships using `D3.js`, where nodes will be colored depending on the community they belong to. We are receiving the community information through the `cluster` property of each node.

```javascript=
// Remove old nodes
node.exit().remove();
// Update existing nodes
node = node.data(nodes, (d) => d.id);
node = node
    .enter()
    .append('circle')
    .attr("r", function (d) {
        return 7;
    })
    .attr('fill', function (d) {
        if (!clusterColors.hasOwnProperty(d.cluster)) {
            clusterColors[d.cluster] = "#" + Math.floor(Math.random() * 16777215).toString(16)
        }
        return clusterColors[d.cluster]
    })
    .on("mouseover", function (d) {
        tooltip.text(d.srcElement["__data__"]["username"])
        tooltip.style("visibility", "visible")
    })
    .on("mousemove", function (event, d) { 
        return tooltip.style("top", (event.y - 10) + "px").style("left", (event.x + 10) + "px"); })
    .on("mouseout", function (event, d) { return tooltip.style("visibility", "hidden"); })
    .call(this.drag())
    .merge(node);
```
First we are removing the old nodes and setting the `node` value to the new nodes data. Next, we want each node to be a circle with radius *7* (that's just a random value which seemed quite okay to me). After that, we want each node to be colored depending on the cluster it belongs to. We have previously created a map of colors called `clusterColors`. When new cluster appears, a new key value pair is created in the map, where key is the cluster number and value is a randomly generated color. If the cluster of the node already exists, then the color of the node will be the value of that cluster key in the `clusterColors` map. Then if we want to see usernames on hover, we need `mouseover`, `mousemove` and `mouseout` events. In the next line, we are calling the `drag()` method which allows us to drag the nodes. At the end, new nodes are being merged with the old ones with the `merge()` method. We will add the links between the nodes in a similar manner. All that's left to do is to create the simulation on updated nodes and links.

```javascript=
try {
    simulation
        .nodes(nodes)
        .force('link', d3.forceLink(links).id(function (n) { return n.id; }))
        .force(
            'collide',
            d3
                .forceCollide()
                .radius(function (d) {
                    return 20;
                })
        )
        .force('charge', d3.forceManyBody())
        .force('center', d3.forceCenter(width / 2, height / 2));
} catch (err) {
    console.log('err', err);
}
```
Here we are creating the force between the nodes and links, where each link has an unique id which we created by adding the attribute id `.attr('id', (d) => d.source.id + '-' + d.target.id)` to each link. That id is created from the id's of the nodes the certain link is connecting. **Collide** force is there so that the nodes are not overlapping, considering the size of their radius. Here we have set the radius to size *20*, which is larger than *7* - the radius of the nodes. **Charge** force is causing the nodes in the graph to repel each other, that is, it prevents the nodes from overlapping each other in the visualization. In the end, we have a **center** force, which is forcing the nodes and links to appear at the middle of the `svg`.

And how this actually looks? Check out the `GIF` below for the preview, and if you want to start the app all by yourself, follow the instructions at the [README](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/README.md) in the repository.
![memgraph-tutorial-community-detection-stream](https://public-assets.memgraph.com/real-time-visualization-with-react-and-d3-js/memgraph-tutorial-community-detection-stream.gif)

The **PageRank** visualization [code](https://github.com/memgraph/twitter-network-analysis/blob/add-backend-server/frontend/src/components/PageRank.js) is similar, the notable difference is in the radius of each node and the color of the nodes.

```javascript=
 node = node
        .enter()
        .append('circle')
        .attr("r", function (d) {
            return d.rank * 1000;
        })
        .attr('fill', 'url(#gradient)')
        .on("mouseover", function (d) {
            tooltip.text(d.srcElement["__data__"]["username"])
            tooltip.style("visibility", "visible")
        })
        .on("mousemove", function (event, d) { return tooltip.style("top", (event.y - 15) + "px").style("left", (event.x + 15) + "px"); })
        .on("mouseout", function (event, d) { return tooltip.style("visibility", "hidden"); })
        .call(this.drag())
        .merge(node);
```
You can see that the attribute `r` is proportional to rank (calculated PageRank of each node). Also, the `fill` attribute is determined by the `gradient` created in the `defineGradient()` method.

![memgraph-tutorial-pagerank-stream](https://public-assets.memgraph.com/real-time-visualization-with-react-and-d3-js/memgraph-tutorial-pagerank-stream.gif)

## Conclusion

There is still a lot left to learn regarding the `React`, `D3.js` and `WebSocket`, but creating this demo application gave me a pretty good insight into the **real-time visualization**. It was fun playing with it, and I'm looking forward to learning more in the future. Also, I would like to emphasize that [Reddit network explorer](https://github.com/memgraph/reddit-network-explorer) application, developed by my colleagues Ivan, David and Antonio, helped me a lot. There, you can find real-time visualization with frontend in **Angular**. For any feedback or questions ping me or Memgraph team at our [**Discord**](https://discord.gg/memgraph) server.