Paper: HashOrder: Accelerating Graph Processing Through Hashing-based Reordering (ICLR-2024)

Reviewer comment (Writing):
The authors may be more detailed when discussing how hashing is used to handle graph problems in the related work, e.g., by describing how a graph problem is mapped to hashing. This does not hurt the novelty of the paper and allows readers to learn more.

Author response:
**1. Discussion on how hashing is used to handle graph problems in the related work.**

Some notable related works include [1, 2, 3]. Specifically, [1] develops a highly efficient graph community detection method based on MinHash, [2] derives meaningful features from MinHash and HyperLogLog for learning link prediction models on graphs, [3] develops a scalable set-based graph summarization algorithm based on LSH. We will add the discussion to the final paper.

Reviewer comment (Experiments):
A common trick in hashing is to hash multiple times to enlarge the collision probability gap between similar and dissimilar object pairs. Any idea on how HashOrder can benefit from multiple hash signatures? I think that may be an interesting problem.

Author response:
**2. Any ideas on how HashOrder can benefit from multiple hash signatures?**

Our final hash function is indeed composed of the concatenation of multiple hash functions. By increasing the number of hash functions, we can ensure that the hash function can distinguish whether a pair of nodes lie inside a cluster or in different clusters. The exact characterization of this property and required number of hash functions to obtain theoretical guarantees is provided in Theorem 1. Empirically, we found l=2 improves the ordering quality over l=1 (0.27% and 0.19% higher relative speedup), where l is the number of hash signatures. l=4 negatively impacts the ordering quality (7.89% and 1.46% lower relative speedup than 2 hashes), likely due to diminishing probability of collision for pairs inside a cluster.

| PageRank run time in seconds (relative speedup) |                 |                 |                 |
|-------------------------------------------------|-----------------|-----------------|-----------------|
| Dataset                                         | l=1             | l=2             | l=4             |
| web-Google                                      | 2.706 (57.68\%) | 2.689 (57.95\%) | 3.197 (50.06\%) |
| web-Stanford                                    | 0.830 (39.36\%) | 0.827 (39.55\%) | 0.847 (38.09\%) |

Reviewer comment (Novelty):
Throughout the paper, it isn't easy to see the difference between the existing and proposed methods. It appears as if the proposed method incorporates randomness into the existing hash-based graph sorting algorithm. It should be clarified what method is used for the existing method that is the basis of the proposed method and how it differs from the proposed method. We believe this will further clarify the contributions of the proposed method and make the paper a good one.

Author response:
Sorry for the confusion--there is no existing hash-based graph sorting algorithm. Our proposed method is a first of its kind probabilistic approach to the reordering problem, which is completely different from the existing methods. The existing methods can be categorized as follows 1) greedy optimization-based methods, such as GOrder 2) graph traversal-based methods, such as RCM 3) degree sorting-based methods, such as SortIn, SortOut, HubCluster, HubSort, and DBG. In contrast to these methods, HashOrder optimizes the fitness of all nodes simultaneously (unlike greedy and graph traversal-based methods) by leveraging randomized hashing and takes the higher order neighborhood into account (unlike degree sorting-based methods).

Reviewer comment (Experiments):
Only small-world and scale-free graphs such as web graphs, social networks, and citation networks were used in the experiments. Although I understand that these graphs are commonly used in machine learning, the proposed HashOrder may be effective only for these highly central graphs. Therefore, it would be better to experiment with a broader range of graph data with different properties.

Author response:
Most real-world graphs exhibit small-world and power-law properties. Hence we study random graphs generated from Erdős-Rényi (ER) and Barabási-Albert (BA) models. ER is a well-studied scale-free random network model, while a BA network exhibits preferential attachment. We compare the runtime of one iteration of PageRank averaged over 5 runs on random graphs with different configurations, with n as the number of nodes and m as the number of edges. HashOrder achieves higher speedups than GOrder across different random graphs.

|                                 | Original | GOrder (Speedup) | HashOrder (Speedup) |
|---------------------------------|----------|------------------|---------------------|
| Erdős-Rényi (n=100k, m=1M)      | 0.0866   | 0.0800 (7.68%)   | 0.0775 (10.54%)     |
| Erdős-Rényi (n=100k, m=10M)     | 0.2613   | 0.2396 (8.29%)   | 0.2376 (9.06%)      |
| Barabási-Albert (n=100k, m=1M)  | 0.1128   | 0.1021 (9.47%)   | 0.1014 (10.05%)     |
| Barabási-Albert (n=100k, m=10M) | 0.3361   | 0.3038 (9.63%)   | 0.3028 (9.91%)      |

Reviewer comment (Novelty):
Can you briefly describe what kind of graph reordering algorithms are used in the existing methods based on the proposed method? Also, can you explain the difference between the proposed and existing methods?

Author response:
We apologize for not understanding this question. Could you please further elaborate on the question?

Reviewer comment (Experiments):
Can you provide experimental results using other than small-world and scale-free graph datasets such as web graphs or social networks? Or can you discuss the performance of the proposed method on them?

Author response:
Most real-world graphs exhibit small-world and power-law properties. Hence we study random graphs generated from Erdős-Rényi (ER) and Barabási-Albert (BA) models. ER is a well-studied scale-free random network model, while a BA network exhibits preferential attachment. We compare the runtime of one iteration of PageRank averaged over 5 runs on random graphs with different configurations, with n as the number of nodes and m as the number of edges. HashOrder achieves higher speedups than GOrder across different random graphs.

|                                 | Original | GOrder (Speedup) | HashOrder (Speedup) |
|---------------------------------|----------|------------------|---------------------|
| Erdős-Rényi (n=100k, m=1M)      | 0.0866   | 0.0800 (7.68%)   | 0.0775 (10.54%)     |
| Erdős-Rényi (n=100k, m=10M)     | 0.2613   | 0.2396 (8.29%)   | 0.2376 (9.06%)      |
| Barabási-Albert (n=100k, m=1M)  | 0.1128   | 0.1021 (9.47%)   | 0.1014 (10.05%)     |
| Barabási-Albert (n=100k, m=10M) | 0.3361   | 0.3038 (9.63%)   | 0.3028 (9.91%)      |

Reviewer comment (Novelty):
novelty of this paper is limited. As admitted by the authors, LSH has been used in similar problems for graph reordering.

Author response:
To the best of our knowledge, our proposed method is the first probabilistic approach to the reordering problem, which is completely different from existing methods. The existing methods can be categorized as follows 1) greedy optimization-based methods, such as GOrder 2) graph traversal-based methods, such as RCM 3) degree sorting-based methods, such as SortIn, SortOut, HubCluster, HubSort, and DBG. In contrast to these methods, HashOrder optimizes the fitness of all nodes simultaneously (unlike greedy and graph traversal-based methods) by leveraging randomized hashing and takes the higher order neighborhood into account (unlike degree sorting-based methods). LSH has been used for problems such as sparse-dense matrix multiplication, task scheduling, and load balancing, which are distinct problems from graph reordering and require dedicated solutions.

Reviewer comment (Reproducibility):
The proposed HashOrder algorithm seems to be a CPU algorithm, since it is parallelizable, can it run on GPU?

Author response:
Yes, HashOrder can run on GPU since it conforms to the SIMD paradigm. The building blocks of the HashOrder algorithm are message passing with minimum aggregation and sorting, which are parallelizable operations on the GPU.

Reviewer comment (Evaluation):
My main concern lies in the experimental results. It does not seem to have significant improvements compared to Gorder in Figure 3.

Author response:
Our proposed method outperforms or is competitive with GOrder wrt downstream application runtime, while having significantly lower computational overhead. As shown in Figure 2, HashOrder surpasses GOrder in terms of ordering quality on average across all tasks, while being 592x (on average) more efficient in reordering.

Reviewer comment (Evaluation):
It is better to report median speedup, instead of "Upto xx". The median speedup seems to be limited, and some are experiencing degradation, e.g., in Figure 4.

Author response:
We see significant average speedups across tasks in downstream applications. We updated the paper to report the average speedup across datasets in the abstract. HashOrder speeds up PageRank by 1.44x and GNN data loaders by 1.93x on average.

Reviewer comment (Evaluation):
It would be good to show end-to-end time for GNN training, instead of only reporting GNN data loader time in Figure 4.

Author response:
We present additional experimental results on end-to-end GNN training time. We train a 3-layer GCN on ogbn-papers100M using the MultiLayerFullNeighborSampler with different reordering algorithms, and measure the data loading and training time per batch. HashOrder outperforms RCM and achieves up to 36.55% speedup on end-to-end GNN training. We will add the results to the final paper.

|                | Data Loading Time (Speedup) | Training Time (Speedup) | Total Time (Speedup) |
|----------------|-----------------------------|-------------------------|----------------------|
| Original Order | 4.971s                      | 1.245s                  | 6.216s               |
| RCM            | 3.284s (33.94%)             | 1.053s (15.42%)         | 4.337s (30.23%)      |
| HashOrder      | 2.952s (40.62%)             | 0.992s (20.32%)         | 3.944s (36.55%)      |

Reviewer comment (Evaluation):
It would be good if you can measure cache hit in the experiments.

Author response:
We have already included cache hit rates in Table 3 and 4 in the appendix.

Reviewer comment (Experiments):
Will reordering lead to other side effects? E.g., GNN training time and accuracy?

Author response:
We do not expect any negative side effects from reordering. Reordering improves the locality of the data layout so it accelerates training and inference. Accuracy is not affected as the graph remains the same with better memory layout.

Reviewer comment (Presentation):
Why there exist blanks in Figure 4, e.g., graphsaint + speedup panel?

Author response:
Blanks in Figure 4 means the experiments are not feasible for those settings due to the inherent limitations of the GNN library. GraphSAINT sampler fails on ogbn-papers100M since torch.multinomial does not support a number of categories beyond 2^24. Full neighbor sampling using UVA fails on ogbn-products and ogbn-papers100M due to GPU memory limitations. We will improve the figure to clarify this.

Reviewer comment (Experiments):
Why choose the number of hash functions l = 2? Will this affect the result?

Author response:
Empirically, we found l=2 marginally improves the ordering quality over l=1 (0.27% and 0.19% higher relative speedup). l=4 negatively impacts the ordering quality (7.89% and 1.46% lower relative speedup than 2 hashes), likely due to diminishing probability of collision.

| PageRank run time in seconds (relative speedup) |                |                |                |
|-------------------------------------------------|----------------|----------------|----------------|
| Dataset                                         | l=1            | l=2          | l=4          |
| web-Google                                      | 2.706 (57.68%) | 2.689 (57.95%) | 3.197 (50.06%) |
| web-Stanford                                    | 0.830 (39.36%) | 0.827 (39.55%) | 0.847 (38.09%) |

Reviewer comment (Theory):
I don't follow the structure of the theoretical results. Lemma 1 assumes existence of a hash family that is never proven, then Theorem 1 builds on Lemma 1, but drops the assumption that Lemma 1 had. This seems incorrect.

Author response:
Sorry for the confusion. For Jaccard similarity which we use in HashOrder, the hash family assumed in Lemma 1 always exists since for any given $\alpha$ and $R$ such that $R > \alpha$, MinHash is a $(R, \alpha, R, \alpha)$-sensitive family. We have improved the presentation of the definitions and theorems (marked in blue) in the paper, but the theoretical result remains unchanged.

Reviewer comment (Experiments):
The experimental results are based only on the authors' implementation. Some comparisons to existing libraries would be helpful in gauging whether the timings are competitive.

Author response:
In fact, we do use official implementations as much as possible. Our implementation extends the C implementation used in [2], which contains the official implementation of methods such as DBG, HubSort, HubCluster, SortIn and SortOut. As we have pointed out in the paper, we use the Ligra framework for running graph processing workloads and DGL for GNN workloads. We use the official implementation of RCM from the scipy package for the GNN experiments. Also, our experimental findings on reordering time and speedup of different methods are consistent with previous works.

Reviewer comment (Theory):
Please provide clarification or planned revision regarding concerns of in the theoretical results.

Author response:
Sorry for the confusion. For Jaccard similarity which we use in HashOrder, the hash family assumed in Lemma 1 always exists since for any given $\alpha$ and $R$ such that $R > \alpha$, MinHash is a $(R, \alpha, R, \alpha)$-sensitive family. We have improved the presentation of the definitions and theorems (marked in blue) in the paper, but the theoretical result remains unchanged.

Reviewer comment (Experiments):
Insufficient comparison with related techniques. The proposal performs node clustering using Minhash and parallelization. In fact, Rabbit order (Arai et al., 2016) shares a similar design concept, as it conducts node clustering using a modularity-based method and also incorporates parallelization. Hence, the authors should offer a comprehensive comparison between the proposal and Rabbit order, including performance experiments.

Author response:
**2. Comparison with Rabbit Order.**

We compare HashOrder with Rabbit Order on the efficiency of solving the SSSP (single-source shortest path) problem. For Rabbit Order, we use the official implementation from GitHub. The table below shows the relative speedup of different algorithms. HashOrder achieves greater speedups than Rabbit Order and GOrder. Rabbit Order generally performs worse than GOrder, which is consistent with the findings in [1].

|             | Rabbit Order | GOrder | HashOrder |
|-------------|--------------|--------|-----------|
| web-Google  | 33.62%       | 38.14% | 42.27%    |
| cit-Patents | 25.43%       | 27.92% | 36.75%    |

Reviewer comment (Novelty):
Insufficient comparison with related techniques. The definition of fitness is slightly extended from the one introduced in GO algorithm (Wei et al., 2016), so novelty is relatively weak.

Author response:
**3. The definition of fitness is extended from the GO algorithm.**

The definition of fitness in GO has drawbacks. It considers a sliding window and evaluates pairs of nodes inside the sliding window, and thus includes pairs which might lie in different cache lines/pages. However, in reality, evaluating pairs of nodes exclusively within a cache line/page better reflects the hardware. We fix this issue in our fitness definition. Additionally, we also generalize the inner product (neighbourhood intersection) used by GO to measure pairwise fitness. We find that our formulation allows for other metrics such as Jaccard similarity which can lead to efficient algorithms such as HashOrder.

Reviewer comment (Theory):
Theorem 1 is founded on the cluster quality (Definition 1). However, it does not provide the size of \eplsion for Minhash, which is used in the proposal.

Author response:
**4. The size of epsilon in the proposal.**

We have reformulated the theorems for ease of reading introducing the definition of $(S,\alpha, \beta)$ clusterable graphs and theorem 1 for such graphs. The value of $\epsilon$ (reworded as $\beta$ in new formulation) is a property of the graph and not the hash function. It affects the quality of ordering, as mentioned in theorem 1, for all similarities and their corresponding hash families $(S, \mathcal{H})$ including Jaccard similarity and Minhash family of hash functions.

Reviewer comment (Experiments):
It would be beneficial to include an end-to-end evaluation that encompasses both reordering and graph analysis.

Author response:
**5. End-to-end evaluation that encompasses both reordering and graph analysis.**

In real-world industry applications, graph analysis can be prohibitively expensive and [3] shows that even reordering with high overhead can benefit the end-to-end application run time. Even in our experiments with simple graph analysis, we found that HashOrder can consistently speedup the end-to-end run time of PageRank, which we present the results in the following table.

|          |            | Wiki-links   |            |             | LiveJournal  |            |             | Pokec        |           |            | web-Google   |           |            | web-Stanford |          |            | cit-Patents  |          |             |
|----------|------------|--------------|-----------:|------------:|--------------|-----------:|------------:|--------------|----------:|-----------:|--------------|----------:|-----------:|--------------|---------:|-----------:|--------------|---------:|------------:|
|          |            | Reorder Time | Run Time   | Total Time  | Reorder Time | Run Time   | Total Time  | Reorder Time | Run Time  | Total Time | Reorder Time | Run Time  | Total Time | Reorder Time | Run Time | Total Time | Reorder Time | Run Time | Total Time  |
| PageRank | DBG        |     0.160721 | 349.840633 |  350.001354 |     0.045862 |  200.65444 |  200.700302 |     0.015944 |  86.62113 |  86.637074 |     0.009877 | 15.960602 |  15.970479 |      0.00243 |   3.2405 |    3.24293 |     0.020491 | 5.334885 |    5.355376 |
|          | GOrder     |  3245.607901 | 323.327933 | 3568.935834 |  1055.639553 | 171.952428 | 1227.591981 |   361.788567 | 79.999555 | 441.788122 |     5.653868 |  8.809337 |  14.463205 |     3.662504 | 2.463918 |   6.126422 |  3467.710272 | 5.793895 | 3473.504167 |
|          | HubCluster |     0.148035 | 361.118788 |  361.266823 |     0.042016 | 194.890394 |   194.93241 |     0.015516 | 87.443333 |  87.458849 |     0.008445 | 17.573982 |  17.582427 |     0.002023 |  3.48898 |   3.491003 |     0.018954 | 5.577104 |    5.596058 |
|          | HubSort    |     1.555896 | 378.913859 |  380.469755 |     0.513904 |  224.32543 |  224.839334 |     0.158907 | 90.378812 |  90.537719 |      0.08431 | 15.786593 |  15.870903 |     0.023579 | 3.223704 |   3.247283 |     0.178334 | 5.583217 |    5.761551 |
|          | HashOrder  |    13.522054 | 304.888094 |  318.410148 |     4.715484 | 178.290475 |  183.005959 |     1.259555 | 81.726037 |  82.985592 |     0.337742 |   7.89198 |   8.229722 |     0.115728 | 2.340298 |   2.456026 |     0.945633 | 4.863815 |    5.809448 |
|          | Original   |            0 | 382.557465 |  382.557465 |            0 | 191.497494 |  191.497494 |            0 | 88.278854 |  88.278854 |            0 | 19.677888 |  19.677888 |            0 |   3.6446 |     3.6446 |            0 | 5.916659 |    5.916659 |
|          | RCM        |     9.455407 | 322.334798 |  331.790205 |     4.281879 | 193.203711 |   197.48559 |     1.691509 | 82.190626 |  83.882135 |     0.459709 |  9.762702 |  10.222411 |     0.158757 | 2.331306 |   2.490063 |      0.23346 | 5.315313 |    5.548773 |
|          | SortIn     |      1.57715 | 396.897457 |  398.474607 |     0.510017 | 227.540957 |  228.050974 |     0.152713 | 91.759981 |  91.912694 |     0.081956 | 14.885591 |  14.967547 |     0.023141 | 2.963184 |   2.986325 |     0.175311 | 5.151845 |    5.327156 |
|          | SortOut    |     1.544873 | 347.899902 |  349.444775 |     0.494126 | 226.379185 |  226.873311 |      0.15189 | 90.022815 |  90.174705 |     0.084075 | 14.943749 |  15.027824 |     0.023768 | 3.145041 |   3.168809 |     0.108394 | 5.213656 |     5.32205 |