Stars: Tera-Scale Graph Building for Clustering and Graph Learning
Abstract
A fundamental procedure in the analysis of massive datasets is the construction of similarity graphs.
Such graphs play a key role for many downstream tasks, including clustering, classiﬁcation, graph learning, and nearest neighbor search.
For these tasks, it is critical to build graphs which are sparse yet still representative of the underlying data.
The beneﬁts of sparsity are twofold: ﬁrstly, constructing dense graphs is infeasible in practice for large datasets, and secondly, the runtime of downstream tasks is directly inﬂuenced by the sparsity of the similarity graph.
In this work, we present Stars : a highly scalable method for building extremely sparse graphs via two-hop spanners, which are graphs where similar points are connected by a path of length at most two.
Stars can construct two-hop spanners with signiﬁcantly fewer similarity comparisons, which are a major bottleneck for learning based models where comparisons are expensive to evaluate.
Theoretically, we demonstrate that Stars builds a graph in nearly-linear time, where approximate nearest neighbors are contained within two-hop neighborhoods.
In practice, we have deployed Stars for multiple data sets allowing for graph building at the TeraScale , i.e., for graphs with tens of trillions of edges.
We evaluate the performance of Stars for clustering and graph learning, and demonstrate 10~1000-fold improvements in pairwise similarity comparisons compared to different baselines, and 2~10-fold improvement in running time without quality loss.
Introduction
Given a collection of unlabeled data, a critical technique in many data mining and machine learning pipelines is the construction of a similarity graph over the data.
Similarity graphs provide a sparse representation of the underlying structure of a dataset by creating edges between the most similar data points.
This allows downstream analytic procedures to run more efﬁciently over the data by restricting comparisons to edges in the graph.
In particular, many such procedures, such as
Hierarchical Agglomerative Clustering (HAC) and variants [17, 5, 1], can be run in time nearly-linear in the number of edges.
In order to be useful as part of a larger pipeline, graph building itself must be an extremely efﬁcient procedure.
Namely, if the time required to construct the graph exceeds the size of the graph itself, then graph building may become the primary bottleneck of the pipeline.
To avoid this scenario, graph building algorithms must circumvent all-pairs comparisons by employing selective ﬁltering methods, such as locality sensitive hashing (LSH).
However, with the extraordinary scale of modern datasets,
LSH alone is often no longer sufﬁcient to ensure efﬁciency.
Such methods still require an all-pairs 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
comparison within each hash bucket, which is infeasible for the bucket sizes produced by LSH on large datasets.
This is not deﬁcit of LSH speciﬁcally, but is rather an intrinsic dilemma of scale: even ground truth similarity graphs, such as threshold graphs and k -nearest neighbor graphs, would have too many edges to construct efﬁciently.
Instead, graph building at scale requires expanding our conceptions of similarity graphs.
In this work we go beyond the standard model of a similarity graph, introducing Stars , a novel graph building framework for constructing extremely sparse yet high-quality similarity graphs, based on the construction of two-hop spanners .
Unlike a classical similarity graph, two-hop spanners relax the constraint that similar points should be connected by an edge, instead requiring that such points be connected by a path of length at most two.
This allows for considerable improvements in sparsity, scaling up to tens of trillions of edges, with negligible loss in quality for downstream applications.
Stars employs a specialized locality sensitive hashing technique to bucket points, and then constructs a collection of star graphs within each bucket.
By creating star graphs, we signiﬁcantly reduce the number of pairwise comparisons required to process a bucket from quadratic to nearly-linear in the bucket size.
In practice, this results in signiﬁcant (e.g. 10 - 20 fold) reduction in the number of comparisons made when compared to LSH alone.
This reduction yields tremendous speedups for graph construction, especially for similarity measures derived from learned models, such as pre-trained deep neural networks, which are expensive to evaluate.
For such learning-based models, similarity computations constitute a majority of the overall runtime, therefore reducing the number of comparisons has enormous performance beneﬁts.
We formally introduce Stars in Section 3, and prove theoretical guarantees on its performance for similarity measures such as cosine and Jaccard similarity.
For these measures, we prove that Stars can construct two-hop spanners which capture the 1 /(cid:15)
-approximate k -nearest neighbors, for any value of k , in time at most ˜ O ( n
+ O ( (cid:15) ) ) , for any approximation factor (1 /(cid:15) ) ≥ 1 .
Additionally, we demonstrate a variant of Stars which, given a threshold r , constructs a two-hop spanner such that all points with similarity larger than r are connected by a path of length at most 2 , and no edge connects two points with similarity less than (cid:15)r .
We demonstrate that such two-hop spanners also approximately preserve connected components, allowing for a 2 -approximation of single-linkage clustering (Theorem 2.5).
We discuss implementation and design details in Section 4.
We then provide an empirical evaluation of Stars in Section 5, where we demonstrate that Stars performs signiﬁcantly better in practice than even the theoretical bounds suggest.
Speciﬁcally, in our experiments we set (cid:15) = .
99 , and demonstrate that Stars recovers (1 /(cid:15) )
-approximate nearest neighbors, as well as yields (1 /(cid:15) )
-approximations to threshold similarity graphs, all while constructing signiﬁcantly fewer than the O ( n 1 . 99 ) -edges (especially for large datasets) suggested by our theoretical guarantees.
In particular, Stars yields 10fold and 2 - 10 fold improvements in number of comparisons and runtime respectively, e.g., fromtrillion to 6 trillion comparisons, when compared to baseline LSH.
The improvement is certainly much larger (at least 1000 -fold) compared to the bruteforce (all-pairs) algorithm.
Importantly, these runtime improvements do not negatively impact on the quality of downstream tasks, such as hierarchical clustering.
The speedups become even more signiﬁcant as we employ more sophisticated similarity learning models; moreover, better similarity learning models further improve the quality of the graph.
1.1 Related Work
Graph Building and Nearest Neighbor Search.
The construction of similarity graphs, such as the k -nearest neighbor graph, is a key step for many established machine learning and data mining methods [10, 40, 11].
To efﬁciently compute approximate nearest neighbors, Locality Sensitive
Hashing (LSH) based methods [22] and their data-dependent counterparts [4, 29, 3] have proven tremendously successful.
In addition to LSH, other techniques such tree-based data structures [32, 31, 8], quantization approaches [24], and learning based methods [19], have also been utilized.
However, the majority of these methods were designed for quickly answering nearest neighbor search queries with a large dataset and a (generally) smaller number of queries, whereas in graph building the query set and the dataset are the same.
This scenario allows for an additional set of techniques and optimizations to be employed, such as local search [18], and the construction of two-hop spanners (as in this work).
Perhaps the most similar work to ours is [25], which builds large-scale graphs over learned similarity models.
In fact, our current results demonstrate that graphs with quality comparable to those in [25] can be constructed with substantially fewer edges via two-hop spanners.
Large Scale Graph Clustering Other than nearest neighbor search, one of the primary downstream analytic tasks run on similarity graphs is clustering.
One of the best studied class of such algorithms is hierarchical agglomerative clustering (HAC), which has been analyzed extensively from both a theoretical and practical perspective [16, 33, 23, 38].
For graph-based HAC, it was recently shown that an approximate clustering can be obtained for average linkage in nearly linear time [17].
Previously, this was also shown for Euclidean space using Ward’s linkage [1].
Several works also give sub-quadratic HAC algorithms, but sacriﬁce theoretical guarantees (such as approximation ratio) [15].
Finally, a related hierarchical clustering algorithm is the MST-based clustering algorithm Afﬁnity [5], a highly-scalable method which we also use for evaluating our two-hop spanners.
Two-Hop Spanners.
The concept of a two-hop spanner stems from the literature on metric spanners (see the surveys [21, 41, 2]), which are sparsiﬁcations of a graph with the property that distances in speciﬁer well approximate distances in the original graph.
Note that a two-hop spanner itself satisﬁes this property.
The ﬁrst usage of two-hop spanners can be attributed to [27], who demonstrated the existence of sparse two-hop spanners for Euclidean space with the (cid:96) 2 metric.
Two-hop spanners have also been implicitly studied in the context of min-size clustering in the MPC model [20].
Preliminaries
Let P be any set of points, and let µ : P 2 → R be either a similarity or a distance measure over
P .
For the remainder, we choose to focus only on similarity measures for clarity of presentation.
However, we remark that the techniques we present generalize naturally to distance measures by employing the appropriate locality sensitive hash functions for those measures (see deﬁnitions below).
We consider several types of similarity measures in the paper, including the dot-product similarity µ ( x, y )
= (cid:104) x, y (cid:105) between points x, y ∈ R d , and the cosine similarity µ ( x, y )
=
cos( θ x,y ) , where θ x,y is the angle between the vectors x, y .
Additionally, we consider the Jaccard similarity between sets: µ ( A, B ) =
| A ∪ B | / | A ∩ B | , where A, B ⊂ U for some universe U , and the weighted Jaccard similarity between non-negative vectors x, y , given by by µ ( x, y ) =
(cid:80) i min( x i ,y i ) (cid:80) i max( x i ,y i ) .
Finally, we use learned similarity measures µ which are computed by a pre-trained model, such as a deep neural network, and may be expensive to evaluate.
µ -Nearest Neighbors.
Given ( P, µ ) where n = | P | , for any p ∈ P we deﬁne the nearest neighbor ordering π p : [ n ] → P as any bijection which satisﬁes µ ( p, π p (1)) ≥ µ ( p, π p (2)) ≥ . . .
≥ µ ( p, π p ( n )) .
We write τ i ( p ) = µ ( p, π p ( i )) to denote the similarity to the i -th nearest neighbor, and we denote the set of i -nearest neighbors via N i ( p ) =
{ π p (1) , . . . , π p ( i ) } .
Locality Sensitive Hash Families.
We now introduce the notion of a locality sensitive hash (LSH) family H for similarity measures.
Given a similarity measure µ over a point set P , an LSH family H is a family of hash functions h : P → U , for some universe U , with the property that
Pr h ∼H [ h ( p ) =
h ( q )] is larger when the points p, q are more similar, namely when µ ( p, q ) is large.
For ease of presentation, we use a simpliﬁed parameterization of LSH families, as opposed to the more standard parameterization of ( r 1 , r 2 , p 1 , p 2 ) -sensitive families (see, e.g. [26]).
Deﬁnition 2.1 ( ( r 1 , r 2 , ρ ) -sensitive Family)
Fix any parameters r 1 < r 2 ∈ R , and ρ ∈ [0 , 1] .
Let P be any point set with similarity measure µ : P 2 → R , and ﬁx a universe of buckets U .
Then a family
H of hash functions h : P → U is said to be a ( r 1 , r 2 , ρ ) -sensitive locality sensitive hash family for ( P, µ ) if the following holds: for any p, q ∈ P , if µ ( p, q ) >
r 2 , then Pr h ∼H
[ h ( p ) =
h ( q )] ≥ n − ρ , and if µ ( p, q ) <
r 1 then
Pr h ∼H
[ h ( p ) =
h ( q )] < n − 4 .
Graph Building and Similarity Graphs.
In this work, we consider two distinct notions of similarity graphs, which are appropriate in different contexts based on the application.
The ﬁrst notion is that of an r -threshold graph, which uniformly connects points with similarity above a given threshold.
Deﬁnition 2.2 ( r -threshold graph)
Given a point set P , similarity measure µ , and a threshold r , the r -threshold graph for ( P, µ ) is the graph G =
( P, E ) where E = { ( x, y ) ∈ P | µ ( x, y ) ≥ r } .
The r -threshold graph for ( P, µ ) is a useful similarity graph when downstream tasks impose a uniform threshold on when points should be connected.
For instance, clustering with the constraint that pairs of points within a cluster have similarity above a threshold r .
On the other hand, from the perspective of nearest neighbor search, one would like edges to be created non-uniformly , where a point p is connected to its k closest neighbors.
Formally:
Deﬁnition 2.3 ( k -near neighbor graph ( k -NN graph))
Given a set of n points P and similarity measure µ , let π p : P → [ n ] be a nearest neighbor ordering over P .
Then the k -nearest neighbor graph for ( P, µ ) is the graph G =
( P, E ) where E = ∪ p ∈ P { ( p, q ) | q ∈ N k
( p ) } .
Two-Hop Spanners.
A two-hop spanner is a similarity graph which relaxes the guarantees of the prior two notions of similarity graphs, where similar points are only required to be connected by a path of length 2 , rather than by a direct edge.
The advantage of such spanners is that they can be considerably sparser than the associated similarity graph, and still produce high-quality results for downstream tasks (such as clustering).
For any graph G =
( V, E ) , v ∈ V , and integer k ≥ 1 , we write N k ( p ) to denote the k -hop neighborhood of p : namely, the set of vertices u ∈ V which are at (unweighted) shortest-path distance at most k from v in G .
The following deﬁnition captures the generalization of r -threshold graphs from one-hop to two-hop neighborhoods.
Deﬁnition 2.
(Threshold Two-Hop Spanners)
Given a point set P and similarity measure µ , a ( r 1 , r 2 ) -two-hop spanner is a graph G =
( P, E ) , such that: (1) for every edge ( p, q )
∈ E , we have µ ( p, q ) ≥ r 1 , and (2) for every pair p, q ∈ P with µ ( p, q ) ≥ r 2 , we have p ∈ N 2 ( q ) .
The above notion of two hop-spanners provide a good (two-hop) approximation of the r -threshold graph for a point set.
The approximation is controlled by the gap between r 1 , r 2 .
In Section 3.1, we give provable guarantees for Stars, by demonstrating that it constructs a two-hop spanners where this gap is small.
In Section 3.2, we deﬁne an analogous notion of two-hop spanners for approximation the k -nearest neighbor graph, and prove that Stars efﬁciently constructs such an approximation for k -nearest neighbors as well.
Two-hop spanners can be used as a powerful approximation of the underlying similarity structure of dataset.
For instance, we demonstrate that two-hop spanners can be utilized to obtain an approximation to the optimal k -single linkage clustering solution, which is to partition P into k clusters C 1 , . . .
, C k
so as to minimize max i (cid:54)
=
j max p ∈ C i ,q ∈ C j µ ( p, q ) .
Theorem 2.5 Let c ≥ 1 and r < OPT k /c , where OPT k is optimal cost of k -single-linkage clustering on ( P, µ ) .
Then any ( r/c, r ) -2-hop spanner G has at least k connected components.
Furthermore, for any two connected components C, C (cid:48) of G , min x ∈ C,y ∈ C (cid:48) µ ( x, y ) ≥ r .
Thus, by constructing log (cid:16) max x,y ∈ p µ ( x,y ) min x,y ∈ P µ ( x,y ) (cid:17) distinct ( r/c, r ) -2-hop spanner for geometrically increasing r , one obtains a 2 -approximation to k -single-linkage clustering.
Graph Building via Two-Hop Spanners
In this section, we theoretically analyze the performance of the Stars algorithm, by demonstrating that it efﬁciently constructs highly-sparse two-hop spanners.
We describe two variants of Stars, in
Sections 3.1 and 3.2 respectively.
The ﬁrst can be used to approximate the r -threshold similarity graph, and the second can be used to approximate the k -nearest neighbor graph.
3.1 Two-Hop Spanners via Locality Sensitive Hashing
We ﬁrst describe the Stars algorithm for constructing a two-hop spanner for approximating r -similarity threshold graph for a point set P and similarity measure µ .
Speciﬁcally, given a locality sensitive hash family H for ( P, µ ) , Stars proceeds by bucketing points via a randomly drawn hash function h ∼ H .
Within each bucket, Stars samples a random “leader” p , and then connects together all points in the bucket to p which are sufﬁciently similar to p , effectively creating a star graph centered at p .
The process is repeated with R independent sketches, to ensure that all sufﬁciently similar points land in the same bucket at least once.
The formal algorithm is given in the algorithm Stars 1.
Theorem 3.1 Let P be a point set equipped with a similarity measure µ , and ﬁx any r 1 < r 2 ∈ R .
Let H : P → U be a ( r 1 , r 2 , ρ ) family of hash functions, such that each h ∈ H can be evaluated
Evaluate
for
For
• Sample a uniformly random leader x ∼ B u .
• For all y ∈ B u \ { x } , if µ ( x, y ) >
r 1 , then create an edge ( x, y ) with weight µ ( x, y ) .
in time at most Time ( H ) .
Let G =
( P, E ) be the graph produced by the algorithm Stars 1, using the LSH family H .
Then with probability 1 − 1 /poly ( n ) , G is a ( r 1 , r 2 ) -two-hop spanner for ( P, µ ) .
Moreover, the number of edges produced is at most | E ( G ) | =
˜ O ( n 1+ O ( ρ ) ) , and the runtime of the algorithm is bounded by ˜ O ( n 1+ O ( ρ ) · Time ( H )) .
In the Appendix, we demonstrate that for parameters α, (cid:15) > 0 , there exist (1 − (cid:15) − 1 α, 1 − α, O ( (cid:15) )) - sensitive hash families for both cosine and Jaccard similarity measures.
3.2 Two-Hop Spanners via SortingLSH
We now describe a variant of the Stars algorithm for the construction of two-hop spanners which approximate that k -nearest neighbor graph ( k -NN graph, Deﬁnition 2.3).
While the k -NN Graph itself is sparse for small k , for large k this is no longer the case, and directly approximating the k -NN graph becomes infeasible.
Instead, by employing two-hop spanners, we show how one can construct extremely sparse graphs, with a almost-linear number of edges, such that the approximate k -nearest neighbors of every point p are contained in the two-hop neighborhood of p .
To do so, we will need to utilize a technique known as SortingLSH.
SortingLSH.
Since the similarities τ k ( p ) between points p and their k nearest neighbors may vary signiﬁcantly across the dataset, we cannot apply a single level of bucketing based on a hash family chosen with a ﬁxed set of parameters, as doing so would result in points being split around a uniform threshold.
Instead, we utilize a technique known as SortingLSH , which originates from the work [6].
SortingLSH evaluates a sequence of hash functions H ( p ) =
( h 1 ( p ) , . . .
, h (cid:96) ( p )) for each point p ∈ P , and interprets the resulting string of buckets as an key for p .
One then sorts the keys H ( p ) lexicographically, and breaks the sorted sequence into contiguous chunks of a given window size
W .
We then apply the two-hop spanner technique on each chunk, by sampling s leaders within that bucket, and comparing each leader to the rest of the chunk.
The key advantage of SortingLSH is that points p living in dense regions of the similarity space ( P, µ ) (i.e., the similarity τ k ( p ) is large) will share a longer preﬁx of hashes with their k -nearest neighbors, and therefore be more likely to be placed in the same window as them.
At the same time, points p whose k -nearest neighbors are not as similar will still be equally as likely to share a (shorter) preﬁx with their neighbors, and therefore also placed in the same window.
In order to describe our results, we ﬁrst introduce a notion of k -approximate nearest neighbors ( k -ANN), deﬁned with respect to a family of hash functions H .
In what follows, recall that given any p ∈ P and i ∈ [ n ] we write π p ( i ) to denote the i -th nearest neighbor to p with respects to µ .
Deﬁnition 3.2 (ANN with respects to an LSH Family)
Fix a point set P equipped with a similarity measure µ , and let
H = { h : P → U } be a family of hash functions.
Fix any ρ ∈ [0 , 1] .
We say that a collection of sets A =
{A p } p ∈ P is an ( k, ρ ) -ANN family with respect to H if for every p ∈ P : •
A p is a preﬁx of the sequence π p (1) , π p (2) , . . .
, π p ( n ) with size |A p
| ≥ k .
• There exists an integer (cid:96) =
(cid:96) ( p ) such that for all i ∈ [ k ] , we have: Pr
[Equation] [Equation]
[Equation] [Equation]
i ))
[Equation]
≥ n
[Equation]
ρ
and such that for all x / ∈
A p we have
[Equation]
If A has the property that (cid:96) ( p ) ≤ M for all p ∈ P , then we say that A is M -bounded ( ρ, M ) -ANN family (hereafter a ( k, ρ, M ) -ANN family).
Deﬁnition 3.2, while at ﬁrst somewhat unwieldy, is fairly straightforward to unpack.
Speciﬁcally, given a base hash family H , and let H (cid:96) denote the family of (cid:96) -wise concatenations of h ∈ H .
Then if, for every p ∈ P , there exists a smaller threshold r p ≤ τ k ( p ) and a sketch length (cid:96) ( p ) such that the family H (cid:96) ( p ) is a ( r p , τ k ( p ) , ρ ) -sensitive hash family (Deﬁnition 2.1), then the family A
= {A p } p ∈ P , where A p = { q ∈ P | µ ( p, q ) ≥ r p } , is a ( k, ρ ) -ANN family.
Note that the gap between r p and τ k ( p ) will depend on the properties of the family H .
Lastly, the condition of being M -bounded simply ensures that one can reach the appropriate splitting point (cid:96) ( p ) with a limited number of hash functions.
Thus, for points p with extremely large τ k ( p ) (e.g.., near-duplicate neighbors), in order to ensure that A is M -bounded, one may need to limit (cid:96) ( p ) =
M which would result in a value of r p that is smaller than what would otherwise be possible with a larger sketch length (cid:96) ( p ) .
We now prove explicit bounds for the above notion of k -ANNs for the case of angular (cosine) similarity, and the Jaccard similarity measure.
The bounds utilize the well-known SimHash family [14] for cosine similarity, and MinHash [12] for Jaccard similarity.
For the case of weighted Jaccard similarity, we remark that there is a straightforward reduction (i.e. similarity preserving mapping) from the (integer) weighted to unweighted case by simply duplicating coordinates, therefore the following bounds also apply to the weighted variant.
For weighted Jaccard distance over general non-integer vectors, one can instead use the variant of min-hash for probability distributions of [34].
Proposition 3.3 (Approximate Nearest Neighbors for Angular and Jaccard Similarity) Let P be a subset of either R d or 2 U for some universe U .
In the ﬁrst case, let µ be the Angular Similarity µ ( x, y )
= 1 − θ x,y where θ x,y is the angle between x, y ∈ P (normalized so that θ x,y ∈ [0 , 1] ), and in the second case we set µ to be the Jaccard similarity µ ( A, B ) =
| A ∩ B | | A ∪ B | between sets A, B ⊂ U .
For any p ∈ P , let s k ( p ) = 1 − τ k ( p ) .
Then for any (cid:15) ∈ (0 , 1) and M ≥ 1 , there exists a hash family
H for ( P, µ ) such the family A
=
{A p } p ∈ P , where
[Equation]
is a ( k, O ( (cid:15) ) , 4 M log n ) -ANN family with respect to H .
Since both the angular and Jaccard similarity are bounded by 1 , the value s k ( p ) in the Proposition 3.
can be interpreted as a natural dissimilarity measure d ( p, q ) =
1 − µ ( p, q ) .
Therefore, Proposition 3.3 demonstrates that we can take A p to be the set of points q which are at most a 1 /(cid:15) factor farther away than the k -th nearest neighbor to p , namely µ ( p, q ) ≤ (cid:15) − 1 d ( p, π k ( p )) .
We now state the main result of this section, which demonstrates that Stars provably recovers nearly k distinct (1 /(cid:15) )
-approximate k -nearest neighbors in the two-hop neighborhood of p .
Formally, for each point p ∈ P , we guarantee that there is a two-hop path from p to nearly k points q ∈
A p .
Moreover, we guarantee that this path utilizes only edges between other approximate nearest neighbors u ∈ A p .
Theorem 3.4 Let P be a point set equipped with a similarity measure µ , and ﬁx any δ ∈ (0 , 1) .
Let
H : P → U be a family of hash functions, such that each h ∈ H can be evaluated in time at most
Time ( H ) .
Let G =
( P, E ) be the graph produced by the algorithm Stars 2.
Let A =
{A p } p ∈ P be a ( k, ρ, M ) -ANN family with respects to H .
For any p ∈ P , let ˆ G p denote the subgraph of G induced by the set A p ∪ { p } .
Then with probability 1 − 1 /poly ( n ) , for every p ∈ P , we have: |N 2ˆ G p ( p ) | ≥ (1 − δ ) k .
Moreover, we have | E ( G ) |
=
˜ O ( δ − 1 n 1+ O ( ρ ) ) , and the runtime of the algorithm is ˜ O
( δ − 1 n 1+ O ( ρ ) M
· Time ( H )) .
Theorem 3.4 demonstrates that, even for extremely large values of k , for instance k = √ n , one can construct a two-hop spanner with a nearly-linear number of edges, such that each point p is connected with at least (1 − δ ) k of its k -approximate nearest neighbors, via a path containing only k -approximate nearest neighbors of p .
Notably, the runtime and the size of the graph produced are independent of k .
Stars 2: Constructing Approximate Nearest Neighbor Graphs
Input: Point set P , a hash H , ANN , M 0 , recall
Repeat: the following procedure R = c
1 n ρ log n times, for a sufﬁciently large constant c 1 . 1.
Let ( h 1 , h 2 , · · · , h M ) be a sequence of independent draws from H . 2.
Sort the points x lexicographically according to the hash values ( h 1 ( x ) , . .
, h M ( x )) .
Let x 1 , . . .
, x n be this ordering.
Set the window size to be W = 16 k 3.
Pick a random shift r ∼ [ W/ 2 , . .
, W ] , and split x 1 , . . .
, x n into consecutive blocks B 1 , B 2 , . . .
, B (cid:96) , each of size at most W , such that B 1 = { x 1 , . . .
, x r } . 4.
If: k > n 2 ρ (a) For each block B j , sample s = c 2 δ − 1 n ρ log 2 n uniformly random leaders cy j 1 , . .
, y js ∼ B j , where c 2 is a sufﬁciently large constant depending only on c 1 .
(b) Create an edge ( x, y ji ) with weight µ ( x, y ji ) for every x ∈ B j and leader y ji ∈ B
j 5.
Else if: k ≤ n 2 ρ (a) For each block B j , create an edge ( x, y ) with weight µ ( x, y ) for every pair x, y ∈ B j .
System Implementation and Design
We have implemented Stars as part of the Grale [25] graph building system using Flume - a C++ counterpart to FlumeJava [13].
which is based on the Adaptive Massively Parallel Computation (AMPC) model [7].
Each logical unit of computation is automatically distributed across a number of worker machines, with the experiments in this paper scaling to thousands of individual workers.
Our implementation has two primary phases: generating LSH tables using LSH or SortingLSH, then scoring pairs of points that share a sketch using all-pairs or Stars.
For efﬁciency we generate LSH tables containing only the identiﬁer of each point, excluding the associated point features, as the same point is sketched R > 1 times to aid in edge recall.
To compute pairwise similarity we thus require an additional round of communication join point features with the LSH tables.
We implement this in one of two ways: a MapReduce-style distributed shufﬂe sort, or via lookups in a distributed hash table (DHT).
These two options trade off CPU time and disk usage for memory.
In the shufﬂe case, we require O ( Rn ) additional disk storage and O ( Rn log( Rn )) time to materialize the joined table.
For problem settings with billions of points and large feature vectors, this extra storage requirement can be prohibitive.
The DHT caches the entire input dataset in memory across multiple machines, requiring O ( n ) RAM but no additional on-disk storage.
This enables online feature lookup as we process each bucket, eliminating the need for a shufﬂe and costly disk I/O.
An additional important implementation detail is the limit we impose on LSH bucket sizes.
In the worst case, a naive LSH implementation could have the same or worse running time than a brute-force all pairs comparison, because a poorly chosen LSH function could hash the entire dataset to a single value, up to R times.
To ensure robustness to sub-optimal LSH settings, we randomly partition large buckets into size-constrained sub-buckets prior to pairwise scoring.
This reduces the overall number of edges eligible for comparison, potentially impacting edge recall, but also capping the worst case running time for the scoring phase of graph construction.
Due to its nearly-linear runtime complexity, the Stars algorithm enables us to relax the sub-bucket size limitation signiﬁcantly.
See Appendix C for further implementation details.
Empirical Study
We evaluate the performance and quality of Stars on both real and synthetic datasets of varying scales.
We compare the LSH+Stars, SortingLSH+Stars, SortingLSH, and bruteforce (AllPair) algorithms.
[Figure] [Figure]
Datasets.
We run experiments on three real public datasets: MNIST [30], Amazon2m [9] (also known as OGBN-Products [28])
[Figure]
and Wikipedia [39]
[Figure]
and two synthetic datasets: random1B and random10B. MNIST contains 60k data points, each of which has a feature of a 784-dimensional ﬂoat vector.
Wikipedia contains 3,650,339 data points, where each point is represented by a set of strings with positive weights.
Amazon2m contains 2,449,029 data points, each of which has a feature of a 100-dimensional ﬂoat vector and a set of strings.
Random1B and random10B are generated from a
Gaussian mixture model with 100 modes, where each data point has 100 dimensions.
Random1B contains 10 9 data points and Random10B contains 10 10 data points.
We refer readers to Appendix D for further details regarding the datasets.
Sketching parameters.
For each similarity measure studied, we use a corresponding LSH function to build the graph.
In particular, for MNIST, Random1B and Random10B datasets, we study the cosine similarity between ﬂoat vector features, and thus we employ SimHash for them.
For the
Wikipedia dataset, we study the weighted Jaccard similarity between two sets of strings with weights, and therefore we use the weighted Minhash LSH to build the graph.
For the Amazon2m dataset, we study two different similarity measures: (1) a mixture of cosine similarity and Jaccard similarity, and (2) a neural network where the training set of candidate pairs are generated by SimHash over ﬂoat vector features and MinHash over sets of strings [25].
In summary, in both cases, we use a mixture of
SimHash and MinHash for graph building.
See Appendix D for more detailed sketching setups.
Number of comparisons.
In Figure 1, we illustrate the number of pairwise similarity comparisons of each algorithm on each dataset.
In all cases, Stars yields at least a ∼ 10 -fold improvement in number of comparisons over the other algorithms.
The number of comparisons used by Stars can be further reduced by choosing a smaller number of leaders (see Appendix D).
Coverage of Near(est) Neighbors.
We evaluate the number of (approximate) near(est) neighbors which can be found for each point in the dataset, and by each algorithm.
We run the bruteforce (AllPair) algorithm for MNIST, Wikipedia and Amazon2m (using mixture similarity) datasets to ﬁnd all ground truth 100 -nearest neighbors and all ground truth near neighbor points with similarity above 0 .
5 for each point.
For the non-Stars LSH based algorithm, we compute the fraction of points with similarity at least 0 .
5 that are direct neighbors for each point.
For LSH+Stars, we compute two ratios, the ﬁrst is the fraction of points with similarity at least 0 .
5 that can be found in two hops where each edge has similarity also at least 0 .
5 , and the second is the same except that the two hop edges
[Figure] [Figure]
can have similarity at least 0 .
495 (this aligns with the 1 . 01 -approximation mentioned in Section 3.2).
For SortingLSH based algorithms, we consider the fraction of exact 100 -nearest neighbors can be found in one hop and two hops for non-Stars and Stars respectively in the graph introduced by the 100 -nearest neighbors.
We also consider the relaxation of ﬁnding 1 .
01 -approximate 100 -nearest neighbors (i.e. 1 /(cid:15)
= 1 . 01 ).
Note that if we can ﬁnd more than 100 approximate 100 -nearest neighbors, we regard the ratio as 1 .
In Figure 2, we report the average of each ratio over all data points.
The graphs built using Stars are able to ﬁnd more near(est) neighbors in 2 hops, with fewer edges overall.
Note that the graphs built by SortingLSH-based algorithms have the same sparsity since we only keep the 250 closest points for each node (even for SortingLSH+Stars).
The sparsity of graphs built by LSH-based algorithms is presented in Figure 3.
In Figure 2, we show a good recall of ﬁnding 1 . 01 -approximate near(est) neighbors in 2 hops by our Stars algorithms.
In the meanwhile, the number of edges shown in Figure 3 is much less than n 1 . 99 which is suggested by our theoretical guarantees.
This phenomenon is observed in all 3 datasets used in Figure 2,3.
Clustering.
The points from MNIST are drawn from 10 classes, and the points from Amazon2m are drawn from 47 classes.
To cluster the graphs, we run average Afﬁnity clustering [5] on the graphs built by each algorithm.
In particular, for graphs built by LSH-based algorithms, we only keep the edges with similarity at least 0 .
5 , and for graphs built by SortingLSH-based algorithms, we only keep the closest direct 100 nodes for each point.
For Amazon2m, we consider both mixture similarity and learning similarity, i.e., we built graphs for each similarity and apply clustering on these graphs.
For all algorithms, we use the number of sketches R = 400 .
We measure the clustering quality via
VMeasure score [37] which is the harmonic mean between homogeneity score and completeness score of a clustering.
The VMeasure score is in (0 , 1) .
The higher the VMeasure score, the higher the quality of the clusters.
We report VMeasure scores in Figure 4.
Effect of the similarity function.
As shown in Figure 4, using a similarity function learned by a neural network in the graph building stage indeed helps other downstream tasks such as clustering on the built graphs.
However, a sophisticated similarity function increases the running time of computing the similarity between two points.
In fact, all of our algorithms become 5 × ∼ 10 × (in terms of the total running time over all workers) slower when using neural network similarity instead of using the mixture of cosine and Jaccard similarity to build the graph for Amazon2m dataset.
Fortunately, the Stars graph building algorithms use signiﬁcantly fewer comparisons, and are 10 × faster than the non-Stars versions (in terms of total running time over all workers).
In addition, they lose only negligible quality for downstream tasks.
Thus, given the same computational resources, switching to a Stars-based graph building strategy enables us to employ a wider range of similarity functions, and affords us the potential to signiﬁcantly improve the quality of downstream tasks.
Experiments on Random10B dataset.
Since we set the degree threshold to be 250 , each algorithm outputs exactly 2 .
5 trillion edges for the Random10B dataset.
However, since the non-Stars-based algorithms make ∼ 10 14 comparisons, more than 95% of these comparisons are redundant.
In contrast, the number of comparisons made by Stars-based algorithms is only 2 - 3 × the number of edges obtained.
Overall, the total running time of non-Stars LSH based algorithm is 100 × the total running time of LSH+Stars.
Similarly, the total running time of non-Stars SortingLSH algorithm is 10 × the total running time of SortingLSH+Stars.
The actual runtime of all Stars-based algorithms ﬁnish in 2 hours, whereas the non-Stars SortingLSH algorithm ﬁnishes in 4 hours and the non-Stars
LSH algorithm ﬁnishes in 23 hours.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] As discussed in Sections 2 and 3, for our theoretical bounds to hold we require a locality sensitive hash family for the similarity measure in question.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs are given in the supplementary of all theoertical statements.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Most our datasets are very large, and there the variance of the runtime and number of comparisons will be quite low due to concentration bounds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Connected Components and Single-linkage Clustering
We now demonstrate that given a ( r 1 , r 2 ) -two-hop spanner for ( P, µ ) , one can obtain an approximate solutions to single-linkage clustering by varing ( r 1 , r 2 ) .
Observation A.1 For c ≥ 1 and r > 0 , if two points are in the same connected components of r -threshold graph, they must be in the same connected components of ( r/c, r ) -two-hop spanner.
Furthermore, if two points are in the same connected components of ( r/c, r ) -two-hop spanner, they must be in the same connected components of r/c -threshold graph.
Corollary A.2 For c ≥ 1 and r > 0 , the number of connected components of ( r/c, r ) -two-hop spanner is at least the number of connected components of r/c -threshold graph, and is at most the number of connected components of r -threshold graph.
Given a parameter k ≥ 1 , the goal of the single-linkage clustering is to partition the input data P into k clusters C 1 , C 2 , · · · , C k such that the maximum similarity between points in separate clusters, namely the quantity minimized max i (cid:54) =
j max p ∈
[Equation]
i
[Equation]
j µ ( p,
[Equation]
) is minimized Let
[Equation]
i.e., OPT k denotes the optimal cost of k -single-linkage clustering.
Let r be in the range [ OPT k , OPT k +1 ) .
Then by the deﬁnition of OPT , it is easy to verify that the connected components of r -threshold graph yield the optimal k -single-linkage clustering.
In the following, we show that we can obtain an approximate k -single-linkage clustering via a two-hop spanner with the appropriate parameters.
Theorem A.3 Let c ≥ 1 and r <
OPT k /c , where OPT k is optimal cost of k -single-linkage clustering on ( P, µ ) .
Then any ( r/c, r ) -2-hop spanner G has at least k connected components.
Furthermore, for any two connected components C, C (cid:48) of G , min x ∈ C,y ∈ C (cid:48) µ ( x, y ) ≥ r , and the number of connected components of G is at least k .
Proof: By the deﬁnition of ( r/c, r ) - 2 -hop spanner, if µ ( p, q ) >
r , there is no edge between p and q .
Let us show that the connected components of G is at least k .
We prove it by contradiction.
Let
C 1 , C 2 , · · · , C k be the optimal k -single-linkage clusters.
Suppose G has at most k − 1 clusters.
By pigeonhole principle, we can ﬁnd p, q which are in the same connected components but
p ∈ C
i , q ∈
C j for some i (cid:54) =
j .
Since p, q are in the same connected components, there must be a path between p and q in G .
We can ﬁnd an edge ( p (cid:48) , q (cid:48) ) on the path such that p (cid:48) ∈ C (cid:48) i (cid:48) , q (cid:48) ∈ C (cid:48) j (cid:48) for some i (cid:48) (cid:54) =
j (cid:48) .
Since C 1 , C 2 , · · · , C k is the optimal k -single-linkage clustering, we have µ ( p (cid:48) , q (cid:48) ) ≥ OPT k > c ˙ r which leads to a contradiction.
Next let us show that for any two connected components C, C (cid:48) of G , min x ∈ C,y ∈ C (cid:48) µ ( x, y )
> r .
Suppose there are two points p, q satisfying µ ( p, q ) ≤ r .
By the deﬁnition of 2 -hop spanner, there is a path between p and q with at most 2 hops.
Therefore, for any two points that are in different connected components, their distance is at least r .
(cid:3)
A simple corollary of the above theorem is that we can easily obtain a k -single-linkage clustering solution with cost at least r by arbitrarily merging connected components to reduce the number of clusters to k .
B Proofs Omitted from Section
In this section, we ﬁll in missing details and proofs from statements and theorems made iun Section 3.
Firstly, we prove Theorem 3.1 below.
Afterwards, we ﬁrst formalize the discussion on the existence of certain ( r 1 , r 2 , ρ ) -sensitive families for the Jaccard and Cosine similarity measures.
B.1 Proof of Theorem 3.
Proof: First, note that by construction, the algorithm Stars 1 never creates an edge between two points x, y ∈ P with µ ( x, y ) <
r 1 .
Thus, the ﬁrst condition for a ( r 1 , r 2 ) -two hop spanner (Deﬁnition
2.4) is satisﬁed deterministically.
To show the second condition, ﬁx any point p ∈ P , and ﬁx any q ∈ P with µ ( p, q ) >
r 2 .
We show that q ∈ N 2 ( p ) with high probability, which will complete the proof.
Firstly, by deﬁnition of a ( r 1 , r 2 , ρ ) -sensitive family, the probability that p, z collide in a hash bucket B u when µ ( p, z ) < r 1 is at most 1 /n 4 .
Thus, the probability that any pair p, z collide with µ ( p, z ) < r 1 in a single repetition is at most 1 /n 2 , and by a union bound at most R/n 2 < 1 / (2 n ) over all R =
c
1 n ρ log n repetitions, where c 1 is a sufﬁciently large constant.
Call this event E , and condition on it now.
Then, again by deﬁnition of a ( r 1 , r 2 , ρ ) -sensitive family, we have that p, q collide in a hash bucket B u with probability at least n − ρ .
Thus, by repeating the hashing R = c 1 n ρ log n times, it follows p, q collide in at least one repetition with probability at least 1 − 1 /n 4 .
In this repetition, by event E , we have that all pairs of points which are contained in B u have pairwise similarity at least r 1 .
In particular, if x ∼ B u is the uniformly sampled leader in algorithm Stars 1, we have that µ ( x, p ) >
r 1 and µ ( x, q ) > r 1 , thus the edges ( x, p ) , ( x, q ) will be added to the graph with probability at least 1 − Pr
[ E ] − 1 /n − 4 > 1 − 1 /n , which completes the proof.
(cid:3)
We now formalize the claim made after the statement of Theorem 3.1 about the existence of ( r 1 , r 2 , ρ ) - sensitive families for the Jaccard and cosine (angular) similarity measures, where the latter is given by µ ( x, y )
= 1 − θ x,y .
The proof of the following proposition follows immedietly from the proof of Proposition 3.3, by simply replacing the threshold θ k ( p ) with the ﬁxed threshold α , and setting
M = ∞ (to avoid the second case in Proposition 3.3, which is not needed for the following claim).
Proposition B.1 Let µ be either the angular or Jaccard similarity measure on a dataset P .
Fix any (cid:15), α ∈ (0 , 1) .
Then there exists a (1 − (cid:15) − 1 α, 1 − α, O ( (cid:15) ))
-sensitive hash family for µ .
B.2 Proof of Propositions 3.
We now provide the proof of Proposition 3.3.
Proposition B.2 (Approximate Nearest Neighbors for Angular Similarity) Let P
⊂ R d be a subset, and let µ ( x, y )
= 1 − θ x,y where θ x,y is the angle between x, y ∈ P (normalized so that θ x,y ∈ [0 , 1] ).
Let H be the SimHash family, where h ∼ H is selected by ﬁrst drawing z ∈ R n uniformly from the unit sphere, and setting h ( x ) =
SIGN ( (cid:104) x, z (cid:105) ) .
Then for any (cid:15) ∈ (0 , 1) , integer
M ≥ 1 , and p ∈ P , let θ k ( p ) = 1 − τ k
( p ) be the normalized k -th closest angle to the point p .
Then the family A
= {A p } p ∈ P , where
[Equation]
is a ( k, O ( (cid:15) ) , 4 M log n ) -ANN family with respect to H .
Proof: Fix any p ∈ P , and set (cid:96) =
(cid:96) p = min { 4 (cid:15) log n θ k ( p ) , 4 M log n } .
In the ﬁrst case, suppose (cid:96) =
[Equation]
log n
[Equation]
p ) ≤ 4 M log n .
Then note that, for any x ∈ N k
( p ) , the probability that ( h 1 ( p ) , . . .
, h (cid:96) ( p )) =
( h 1 ( x ) , . . .
, h (cid:96) ( x )) is at least
[Equation]
Similarly, for x / ∈ A p , we have µ ( p, x ) < 1 − θ k
( p ) (cid:15) , so the probability that ( h 1 ( p ) , . . .
, h (cid:96) ( p )) =
( h 1 ( x ) , . . .
, h (cid:96) ( x )) is at most
[Equation]
where we used the inequality that (1 − x ) n/x ≤ (cid:0) 12 (cid:1) n for any x ∈ (0 , 1] and n ≥ 1 .
Next, suppose that (cid:96) p = 4 M log n < 4 (cid:15) log n θ k ( p ) .
In this case, since (cid:96) p is only smaller than the threshold used above, we still have that h 1 ( p ) , . . .
, h (cid:96) ( p ) =
h 1 ( x ) , . . .
, h (cid:96) ( x ) with probability at least n − O ( (cid:15) ) for any x ∈ N k
( p ) .
Next, for any x / ∈ A p , the probability that ( h 1 ( p ) , . . .
, h (cid:96) ( p )) =
( h 1 ( x ) , . . .
, h (cid:96) ( x )) is at most
[Equation]
4 M log n ≤ 1 /n which completes the proof.
[Equation]
Proposition B.3 (Approximate Nearest Neighbors for Jaccard Similarity) Let P ⊂ 2 U be a set of subsets of a universe U , and let µ ( x, y ) =
| x ∩ y | | x ∪ y | be the Jaccard Similarity between x, y ⊂ U .
Let
H be the MinHash family, where h ∼ H is selected by ﬁrst drawing a random number n u ∼ [0 , 1] for each u ∈ H , and setting h ( x ) = min u ∈ x n u .
Then for any (cid:15) ∈ (0 , 1) , integer M ≥ 1 , and p ∈ P , let s k ( p ) =
1 − τ k
( p ) be the k -th smallest Jaccard distance to the point p .
Then the family
A = {A p } p ∈ P , where
[Equation]
is a ( k, O ( (cid:15) ) , 4 M log n ) -ANN family with respect to H .
Proof: The proof of Proposition B.3 is identical to that of Proposition B.2, by simply using that fact that Pr[ h ( x ) = h
( y )]
= | x ∩ y | | x ∪ y | for the case of MinHash.
[Equation]
B.3 Proof of Theorem 3.
Proof: We begin by demonstrating that the ﬁrst condition holds, by demonstrating that our algorithm ﬁnds enough edges so that (cid:12)(cid:12)(cid:12) N 2ˆ G p ( p ) (cid:12)(cid:12)(cid:12) ≥ (1 − δ ) k .
In what follows, ﬁx any vertex v ∈ P .
We run Stars 2 with number of iterations R =
O (log n · n ρ ) .
For each i ∈ [ R ] , let H i
=
( h i, 1 , h i, 2 , . . .
, h i,M ) be the hash functions drawn on repetition i , and for t ∈
[ M ] , let H ti
= ( h i, 1 , h i, 2 , . . .
, h i,t ) be the ﬁrst t hash functions drawn in repetition i .
For each repetition i ∈ [ R ] , call i balanced for the point p if p is at distance W/ 4 from either boundary of the block B j which contains it if that block is not the ﬁrst or last block, and if B j is the ﬁrst or last block then i is well balanced if it is distance at least W/ 4 from the only block adjacent to B j .
It is easy to see that, in either case, a repetition i is balanced for p with probability at least 1 / 2 , taken only over the random choice of r ∈ { W/ 2 , . . .
, W } .
By Chernoff bounds, it follows that at least R/ 3 repetitions will be balanced for i with probability at least 1 − 1 /n 3 , which we condition on now.
In what follows, we restrict our attention only to the balanced repetitions, and show that
[Equation]
holds even restricted to just the edges found in those repetitions.
By deﬁnition of the set A p and a union bound over n points and at most R ≤ n repetitions, for each (balanced) repetition i there exists an (cid:96) =
(cid:96) p ≤ M such that simultaneously for all x / ∈
A p , we have H (cid:96)i ( p ) (cid:54) =
H (cid:96)i ( x ) with probability at least 1 − 1 /n 2 . Call the event that this holds E p , and condition on it for the remainder of the proof.
Next, for each x ∈ N k ( p ) , note that H (cid:96)i ( p ) =
H (cid:96)i ( x ) with probability at least n − ρ .
Moreover, the above analysis on the probability that a repetition i only took randomness over the choice of the random r ∈ { W/ 2 , . .
, W } , it follows that the event that both i is balanced and H (cid:96)i ( p ) =
H (cid:96)i ( x ) is at least n − ρ / 2 .
Thus, since we take R =
c
1 n ρ log n repetitions, taking c 1 > 50 with probability at least 1 − n − 10 we will have H (cid:96)i ( p ) =
H (cid:96)i ( x ) for at least one balanced repetition i .
Call this event E 2 and coniditon on it for the remainder of the proof.
We now split into cases based on the two branches of the if-statement in Stars 2.
Case 1: k > R/δ .
For any balanced repetition i , we say that i is bad if
[Equation]
We call a repetition good if it is not bad.
Now for any point x ∈ N k ( p ) , we say that x is good if there exists a balanced repetition
i such that H (cid:96) p
i ( p ) =
H (cid:96) p
i ( x ) and i is good.
For such a good repetition i , we say that x is good at i .
In what follows, we ﬁx any point x ∈ N k
( p ) .
We now prove the following claim.
Claim B.4 If x ∈ N k ( p ) is good at i , then with probability at least 1 − n − 10 , either x is added to
N 2ˆ G p ( p ) on repetition i , or we add at least k unique points from A p to N 2ˆ G p ( p ) on repetition i .
Proof: To see this, for any point y , deﬁne σ y ∈ { 1 , 2 , . . . , n } be the ranking of y in the lexicographical ordering induced by the set of hash functions H i corresponding to the i -th repetition.
Let t − ≤ t p be the smallest possible index such that the point y with σ y = t − satisﬁes H (cid:96)
p
i ( p ) =
H (cid:96) p
i ( y ) , and similarly deﬁne the point t + ≥ t p as the largest such index.
Note that, by conditioning on E p , for every y such that σ y ∈ [ t − , t + ] , we have y ∈
A p .
First suppose that either t − < t p − W/ 4 or t + > t p + W/ 4 .
Then since repetition i is balanced, it follows that there are at least W/ 4 > k points
y ∈
A p which are contained in the same block as p .
Since we sample s = c 2 δ − 1 n ρ log 2 n random leaders in this block, taking c 2 > 100 , with probability larger than 1 − 1 /n 10 we will sample at least one y ∈
A p which in that window.
Then for any other z ∈ A p which also fell in the same block, we create a path ( z, y ) , ( y, p ) of length two, each of which consists of only edges between points in A p ∪ { p } .
Thus, in this case we add at least W/ 4 > k unique points from A p to N 2ˆ G p ( p ) .
Otherwise, we have that t p − W/ 4 < t − < t p <
t + < t p + W/ 4 .
Again, since repetition i is balanced, it follows that every y with σ y ∈ [ t − , t + ] falls in the same window as p .
Since we know that repetition i is good, it follows that t + − t − > δk c 1 n ρ log n .
Thus, again by sampling s = c 2 δ − 1 n ρ log 2 n leaders for c 2 larger than some constant times c 1 , again with probability larger than 1 − 1 /n 10 we will sample at least one y such that σ y ∈ [ t − , t + ] as a leader, implying that y ∈
A p .
Conditioned on this, since x must also fall in the same window, we add the edges ( x, y ) , ( y, p ) to ˆ G p , thereby adding x is added to N 2ˆ G p ( p ) , which completes the proof of the claim.
[Equation]
By the above claim and a union bound, with probability at least 1 − n − 9 it holds that for every x ∈ N k
( p ) , if x is good then either we add x to N 2ˆ G p ( p ) or we already have that (cid:12)(cid:12)(cid:12) N 2ˆ G p ( p ) (cid:12)(cid:12)(cid:12) ≥ k .
To complete the proof for Case 1, it sufﬁces to show that the number of good points x ∈ N k ( p ) is at least (1 − δ ) · k .
To see this, note that if point x is not good, then for every balanced repetition i such that H (cid:96) p i ( p )
=
H (cid:96) p
i ( x ) (for which there is at least one due to conditioning on E 2 ), we have (cid:12)(cid:12)(cid:12) { x ∈ N k
( p ) |
H (cid:96) p i ( p ) =
H (cid:96) p
i ( x ) } (cid:12)(cid:12)(cid:12) ≤ δk/R .
For each bad point, we charge it’s “badness” to the ﬁrst balanced repetition i for which H (cid:96) p i ( p ) =
H (cid:96) p
i ( x ) and such that i was bad.
Since each bad repetition can be charged for at most δk/R bad points, it follows that there can be at most R · δk/R <
δk bad points in N k ( p ) , and therefore the number of points good points x ∈ N k
( p ) is at least (1 − δ ) · k , which completes the proof of Case 1.
Case 2: k ≤ R/δ .
In this case, we will demonstrate a stronger claim: for any x ∈ N k
( p ) , and any balanced repetition i such H (cid:96) p
i ( p ) =
H (cid:96) p
i ( x ) (for which there is at least one due to conditioning on
E 2 ), either either the edge ( x, p ) is added ˆ G p on repetition i , or for at least k unique points
y ∈
A p we add the edge ( y, p ) points to ˆ G p on repetition i .
Note that, given this claim, the stronger fact that (cid:12) (cid:12) (cid:12) N 1 ˆ G p ( p ) (cid:12) (cid:12) (cid:12) ≥ k in Case 2 follows immediately.
To see this, ﬁx any x ∈ N k
( p ) , and any balanced repetition i such H (cid:96) p
i ( p ) =
H (cid:96) p
i ( x ) .
First suppose that | σ x − σ p | ≥ W/ 4 , WLOG we have σ x ≥ σ p + W/ 4 (note that for the edge cases where p is in the ﬁrst or last block, there may be only one choice between σ x < σ p or σ x > σ p in this setting).
Since H (cid:96) p i ( p ) =
H (cid:96) p
i ( x ) , it follows that every y with σ x < σ y < σ x + W/ 4 also satisﬁes
H (cid:96) p i ( p ) =
H (cid:96) p
i ( y ) , and therefore y ∈
A p .
Since i is balanced, it follows that all such y are in the same block as p , and therefore we created at least W/ 4 > k such edges ( y, p ) for points
y ∈
A p .
In the second case, assume | σ x − σ p | ≥ W/ 4 .
In this case, by the balancedness of repetition i , it follows that x, p are in the same block, and therefore we connect ( x, p ) on this repetition, which completes the proof for case 2 .
Bounding The Number of Edges.
In case 1 , for each of the R repetitions, we create at most 2 ns edges, thus the total number of edges is O ( Rns ) =
˜ O ( δ − 1 n 1+2 ρ ) .
In Case 2, each vertex has degree at most W , so we create at most Wn
= O ( kn )
=
O ( nR/δ ) edges, thus the total number of edges is
RWn =
˜ O ( δ − 1 n 1+2 ρ ) as desired.
Bounding The Runtime.
Note that evaluating the M hash functions and sorting points lexicographically based on those hash functions requires O ( nM log n + nM · Time ( H )) time.
Thus, the overall runtime to construct all blocks used by the algorithm over R repetitions is ˜
O ( RnM · Time ( H )) =
˜ O ( RnM · Time ( H )) .
Once the blocks are complete, the remaining runtime is within a constant of | E ( G ) | , which is ˜ O
( δ − 1 n 1+2 ρ ) as above, which completes the proof.
(cid:3)
C Additional Implementation Details C.1 SortingLSH at Scale
The SortingLSH algorithm involves computing R sketches per point, then sorting the nR total sketches before subdividing into contiguous buckets with window size W .
To achieve this at billionpoint scales, we leverage the TeraSort algorithm [36].
C.2 Training a Pairwise Similarity Model
Following [25], we train a similarity model on the task of distinguishing pairs of nodes which are in the same category from pairs of nodes which are in different categories.
The model architecture uses shared-weight embedding towers to learn a symmetric representation of the node-level features, which are converted into a pairwise embedding using the Hadamard product.
This embedding is concatenated with additional pairwise features, then used as input for a feed-forward neural net that generates the ﬁnal binary prediction.
The unthresholded scalar output of this ﬁnal network provides a similarity score for any pair of nodes.
D Additional Experiments D.1 Additional Information of Datasets
• Each data item of MNIST is a 28 × 28 image of a hand written digit.
There are totaldifferent digits and thus MNIST has 10 ground truth clusters.
• Each data item of Wikipedia is an article.
The feature of a data item is the set of words appeared in the article together with the frequency of appearance of each word.
• Amazon2m consists of products available on the Amazon web site.
Each product has a 100dimensional real valued feature vector and co-purchase relationships with other products.
The products are in 47 categories and thus it has 47 ground truth clusters.
• Random1B and Random10B are generated from mixture of Gaussians.
Each data point has 100 dimensions.
The number of modes is 100 where the i -th mode is a Gassian distribution with mean (0 , 0 , . . .
, 0 , 1 , 0 , · · · , 0) (the i -th entry is 1 ), and standard deviation 0 .
1 of each entry.
Each data point is drawn uniformly at random from a mode.
D.2 Detailed Sketching Parameters
We set number of sketches R = 25 , 100 , 400 for all LSH and SortingLSH based algorithms.
For
Stars, if not speciﬁed otherwise, we set number of leaders s = 25 .
For SortingLSH-based algorithms: we set window size W = 250 , sketching dimension M = 30 , and the maximum allowed bucket size to be 20000 for all experiments.
For LSH-based algorithms:
• For non-Stars version of LSH based algorithm, we set the maximum allowed bucket size to be 1000 .
For Stars version of LSH based algorithm, we set the maximum allowed bucket size to be 10000 .
• We employ SimHash of sketching dimension M = 12 for MNIST and M = 16 for Random1B and Random10B. • For the Wikipedia dataset, we use the weighted Minhash with sketching dimension M = 3 .
• For the Amazon2m dataset, we use a mixture of SimHash and MinHash with sketching dimension M = 12 , i
e., randomly select each bit of hash value generated from SimHash or MinHash (it is easy to verify that the mixture of SimHash and MinHash satisﬁes Deﬁnition 2.1 for the similarity which is a mixture of cosine similarity and Jaccard similarity).
D.3 Similarity for Amazon2m Learned by Neural Network
The embedding tower for each node combines the product embedding and a 1-hot encoded vector of co-purchased products, hashed to 1000 buckets.
The Hadamard product of these embeddings is concatenated with: the cosine similarity of the product embeddings, an indicator variable conditioned on two products being copurchased, and the Jaccard similarity of the two products’ copurchase sets.
The embedding towers use two hidden layers of size 100 with ReLU activations [35].
The ﬁnal prediction is made using the concatenated pairwise features and embedding Hadamard product with another MLP having two hidden layers of size 100 and ReLU activations again.
The model is trained on all pairs which fall into an LSH bucket from a sample of 500,000 nodes from the original dataset and achieved an AUC of 0.92 on a holdout set from a distinct sample of nodes for the same-category task.
D.4 Experiments of Different Number of Leaders
In this section, we study the impact of number of leaders.
In particular, we ﬁx the number of sketches
R = 400 and evaluate the number of leaders s = 1 , 5 , 10 , 25 .
Number of comparisons.
In Figure 5, we show the number of pairwise similarity comparisons of each algorithm on MNIST, Wikipedia and Amazon2m datasets.
As we can observe, Stars yields ∼ 100 -fold improvement in number of comparisons over the other algorithms when the number of leaders s is chosen to be 1 or 5 .
[Figure]
Coverage of Near(est) Neighbors.
We evaluate the number of (approximate) near(est) neighbors which can be found for each point in Wikipedia, Amazon2m and MNIST, and by each algorithm with different number of leaders.
For all algorithms, the number of sketches R = 400 .
The evaluation metric is the same as Section 5.
In Figure 6, we report the average of each ratio over all data points.
When we choose more leaders, we obtain better recall.
We obtain sparser graphs when we use smaller number of leaders.
Again, note that the graphs built by SortingLSH-based algorithms have the same sparsity since we only keep the 250 closest points for each node (even for SortingLSH+Stars).
The sparsity of graphs built by LSH-based algorithms is presented in Figure 7.
D.5 Running Time
In this section, we investigate the running time of several experiments.
We use total running time to refer to the summation of running time of building edges over all machines.
We use real running time
[Figure] [Figure]
to refer to the actual running time of the entire job
[Figure]
which includes I/O, shufﬂe and scheduling time etc.
Since real running time depends on the number of machines assigned to the job, I/O latency
[Figure]
and network condition etc., the total running time is usually less noisy.
For all experiments, the worker pool contains 1000 number of machines (i
[Figure]
e., the maximum parallelism uses 1000 machines, though a job might not get all of them due to scheduling).
Effect of similarity function.
We report the relative total running time of Stars versions of algorithms and non-Stars versions of algorithms for Amazon2m for both mixture of similarities and the similarity learned by neural networks.
See Table 1 for LSH-based algorithms, and see Table 2 for
SortingLSH-based algorithms.
Experiments on 1B and 10B datasets.
For Random1B and Random10B datasets, we only run algorithms with number of sketches R = 25 .
We report the relative total running time of Stars versions of algorithms and non-Stars versions of algorithms for both random1B and random10B datasets in Table 3.
We also saw the speed-up of real running time on Random1B and Random10B datasets:
Mixture of similarities
[Table]
[Table] [Table]
For Random1B: All jobs of Stars-versions of algorithms ﬁnished in 1 hour.
SortingLSH+nonStars ﬁnished in 1 . 5 hours and LSH+non-Stars ﬁnished in 2 hours.
For Random10B: All jobs of Stars-versions of algorithms ﬁnished in 2 hours.
SortingLSH+non-Stars ﬁnished in 2 hours and LSH+non-Stars ﬁnished in 23 hours.
For large datasets
[Table]
the running time of ﬁnding edges dominate other overheads and thus we can observe a large speed-up as described above
[Table]