Load Balance and Data Migration at Storage Layer in NebulaGraph Database
Tags: storage, loadbalance, graphdatabase, nebulagraph
Published: 2020-02-12 07:20:37

---
title: Load Balance and Data Migration at Storage Layer in NebulaGraph Database
published: true
date: 2020-02-11 01:41:09 UTC
tags: storage,loadbalance,graphdatabase,nebulagraph
canonical_url: https://nebula-graph.io/en/posts/nebula-graph-storage-banlancing-data-migration/
---

![](https://cdn-images-1.medium.com/max/1024/1*-7WxZDtA_hEz-htVxAyOSA.png)<figcaption>Load Balancing and Data Migration at Storage Layer in NebulaGraph</figcaption>

In our previous post, [_Introduction to NebulaGraph Storage Engine_](https://medium.com/nebula-graph/introduction-to-nebula-graph-storage-engine-3bad16952da2), we mentioned the distributed kv store is managed by the meta service. Both the partition distribution and machine status can be found in the meta service. Users can use commands in the console to add or remove machines to execute a balance plan for the storage service.

**NebulaGraph Database** ’s service is composed of three parts: graph, storage and meta. In this post, we will introduce how to implement data (partition) and work-load balance in the storage service.

The storage service can be scaled in or out horizontally by the BALANCE commands below:

- BALANCE DATA is used to migrate data from old machines to new machines;
- BALANCE LEADER only changes the distribution of leader partition to balance the work load without moving data.

### Intro to the balance mechanism

In **NebulaGraph** balance means to balance both the raft leader and partition data. But the balance **does not change the numbers of leaders or partitions**.

When you add a new machine with NebulaGraph service, the (new) storage will automatically register to the Meta service. Meta calculates an equally partition distribution, and then uses **remove partition** and **add partition** to make those partitions distributed evenly. The corresponding command is BALANCE DATA. Usually, the data migration is a time-consuming process.

However, BALANCE DATA only changes the replica distribution among the machines. But the leaders (corresponding work load) will not be changed. Next, you need to use the BALANCE LEADER command to achieve load balance. This process is also implemented through the meta service.

## Cluster data migration

The following example will show how to expand the cluster from three instances to eight instances.

### Step 1 Prerequisites

Suppose you’ve already start a cluster with three replicas.

**Step 1.1 Show the current cluster status**

Show the current status with command SHOW HOSTS:

{% gist https://gist.github.com/Jamie-VESoft/9cd89197688cfa8995a59f4034260ab8 %}

**_Explanations on the returned results:_**

- **_IP_** and **_Port_** are the storage instance. The cluster has three storaged instances (192.168.8.210:34600, 192.168.8.210:34700, 192.168.8.210:34500) without any data.
- **_Status_** shows the state of each instance. There are two kinds of states, i.e. online/offline. When a host crashed, metad will turn it to offline after its heart beat timed out. The default heart beat threshold is 10 minutes (You can find parameter expired\_threshold\_sec meta's config file).
- **_Leader count_** shows the number of raft leader of the instance served.
- **_Leader distribution_** shows how the leader distributed in each graph space. For now there are no spaces created. (You can regard space as an independent name space — similar to the Database in MySQL.)
- **_Partition distribution_** shows the partition number of different spaces.

![](https://cdn-images-1.medium.com/max/881/1*FGsfhqZy4vG1LfmZPQwghQ.png)<figcaption>Show the Status of the Current Cluster</figcaption>

We can see there is no data in the _Leader distribution_ and _Partition distribution_ for the time.

**Step 1.2 Create a graph space**

Create a graph space named ` **test`** with 100 partition and 3 replicas.

{% gist https://gist.github.com/Jamie-VESoft/d989092d3cd8b25d925e8da71d6993da %}

After a few seconds, run the command SHOW HOSTS again:

{% gist https://gist.github.com/Jamie-VESoft/b5d5a17a887a6161a304418a2d86ff01 %}

![](https://cdn-images-1.medium.com/max/882/1*mQzfssD3eNBKNHHQm2E9Sw.png)<figcaption>Show Hosts Again</figcaption>

After we created the space test with 100 _partitio_ns and 3 replicas, the host192.168.8.210:34600 serves NO leader, while 192.168.8.210:34700 serves 52 leaders and 192.168.8.210 serves 48 leaders。The leaders are not equally distributed.

### Step 2 Add five new instances

Now, let’s add five new instances into the cluster.

Again, show the new status using statement SHOW HOSTS. You can see there are already eight instances in serving. But no partition is running on the new instances.

{% gist https://gist.github.com/Jamie-VESoft/63e6b4fa482690794115f22dc1d11f33 %}

![](https://cdn-images-1.medium.com/max/947/1*gV-B9_of9TAOx1FA49Vayw.png)<figcaption>Cluster Status Comparison Before and After Adding Five New Instances</figcaption>

In the above picture, the five blue icons are the newly added ones. However, since we just add them, they serve no partitions.

### Step 3 Data migration

Run command BALANCE DATA:

{% gist https://gist.github.com/Jamie-VESoft/83c36cfd869d3f40fe055f668efcd473 %}

This command will generate a new plan and start a migration process if the partitions are not equally distributed. For a balanced cluster, re-run BALANCE DATA will not cause any new operations.

You can check the running progress of the plan by command BALANCE DATA $id.

{% gist https://gist.github.com/Jamie-VESoft/45d07ef6b55abb292b5cfd841cd4c856 %}

**_Explanations on the returned results:_**

1. The first column is a specific task.

Take 1570761786, 1:88, 192.168.8.210:34700->192.168.8.210:35940 for example

- **1570761786** is the balance ID
- **1:88** , 1 is the spaceId (i.e., space `test`), 88 is the partition id which is now being moved
- **192.168.8.210:34700->192.168.8.210:3594** , moving data from the source instance to the destination instance. The useless data on the source instance will be garbage collected after the migration is finished.

2. The second column shows the state (result) of the task, there are four states:

- Succeeded
- Failed
- In progress
- Invalid

The last row is the summary of the tasks. Some partitions are yet to be migrated.

### Step 4 If stop data balance halfway, …

BALANCE DATA STOP command will stop the running plan and return this plan ID. If there is no running balance plan, an error is thrown.

> Since a balance plan includes several balance (sub)tasks, BALANCE DATA STOP doesn't stop the running tasks, but rather cancel the subsequent tasks. The running tasks will continue until the executions are completed.

You can run BALANCE DATA $id to show the status of a stopped balance plan.

After all the running (sub)tasks are completed, you can re-run the BALANCE DATA command again to resume the previous balance plan (if applicable). If there are failed tasks in the stopped plan, the plan will retry. Otherwise, if all the tasks are succeed (and e.g., a new machine is added the cluster), a new balance plan will be created and executed.

### Step 5 Data Migration is Done

In some cases, the data migration will take hours or even days. During the migration, **NebulaGraph** online services are not affected. Once migration is done, the progress will show 100%. You can retry BALANCE DATA to fix those failed tasks. If it can't be fixed after several attempts, please contact us at [GitHub](https://github.com/vesoft-inc/nebula/issues).

Finally, use the SHOW HOSTS to check the final partition distribution.

{% gist https://gist.github.com/Jamie-VESoft/36c3bf872a1e1de8370ef4bc97e25480 %}

![](https://cdn-images-1.medium.com/max/1024/1*9kjGuNJ9qL3L3axgzA09nA.png)<figcaption>Status of the Cluster after Data Migration</figcaption>

#### Step 6 Balance leader

Statement BALANCE DATA only migrates partitions (with the data). But the leader distribution remains unbalanced, which means old hosts are overloaded-working, while the new ones are not fully used. We can re-distribute raft leader using the command BALANCE LEADER.

{% gist https://gist.github.com/Jamie-VESoft/6c06ad55feb0d6cf9eb159a20a07e75f %}

Seconds later, show the results using the statement SHOW HOSTS.

{% gist https://gist.github.com/Jamie-VESoft/cdf6b52b3b4fc4ca9de23e8e2b6c67e3 %}

According to the Leader distribution column, the RAFT leaders are distributed evenly over all the hosts in the cluster.

![](https://cdn-images-1.medium.com/max/1024/1*yziElnLE2yp3Y1Nxj8PqcA.png)<figcaption>Cluster Status after load balancing</figcaption>

As the above picture indicates, when BALANCE LEADER runs successfully, the number of _Leader distribution_ on the newly added (the blue icon) and the original instances (the black icon) are close to each other (12 or 13 for an instance). Besides, as there are no change to the Partition distribution number, it indicates that `balance leader` only involves the re-distribution of leaders from instances.

## Batch Scale in

**NebulaGraph** also supports to go offline a host (and scale in the cluster) during service. The command is

BALANCE DATA REMOVE $host\_list.

For example, command

BALANCE DATA REMOVE 192.168.0.1:50000,192.168.0.2:50000

removes two hosts, i.e. 192.168.0.1:50000，192.168.0.2:50000, from the cluster.

> If replica number cannot meet the quorum requirement after the remove (e.g., remote two machines from a three machine cluster), **NebulaGraph** will reject the request and return an error code.

## Conclusion

In this post, we showed how to balance data and balance work load on a raft-cluster. If you have any questions, please leave your comment. Finally let’s take a glance of the data migration process of instance _192.168.8.210:34600_.

![](https://cdn-images-1.medium.com/max/1024/1*eRvjrG-6j8qbmYq02geeWw.png)<figcaption>Data Migration Overview</figcaption>

## Appendix

This is the [GitHub Repo](https://github.com/vesoft-inc/nebula) for **NebulaGraph**. Welcome to try nebula. IF you have any problems please send us an [issue](https://github.com/vesoft-inc/nebula/issues).


* * *