Sublinear Algorithms for Hierarchical Clustering
Abstract
Hierarchical clustering over graphs is a fundamental task in data mining and machine learning with applications in many domains including phylogenetics, social network analysis, and information retrieval.
Speciﬁcally, we consider the recently popularized objective function for hierarchical clustering due to Dasgupta [18], namely, minimum cost hierarchical partitioning.
Previous algorithms for (approximately) minimizing this objective function require linear time/space complexity.
In many applications the underlying graph can be massive in size making it computationally challenging to process the graph even using a linear time/space algorithm.
As a result, there is a strong interest in designing algorithms that can perform global computation using only sublinear resources (space, time, and communication).
The focus of this work is to study hierarchical clustering for massive graphs under three well-studied models of sublinear computation which focus on space, time, and communication, respectively, as the primary resources to optimize: (1) (dynamic) streaming model where edges are presented as a stream, (2) query model where the graph is queried using neighbor and degree queries, (3) massively parallel computation (MPC) model where the edges of the graph are partitioned over several machines connected via a communication channel.
We design sublinear algorithms for hierarchical clustering in all three models above.
At the heart of our algorithmic results is a view of the objective in terms of cuts in the graph, which allows us to use a relaxed notion of cut sparsiﬁers to do hierarchical clustering while introducing only a small distortion in the objective function.
Our main algorithmic contributions are then to show how cut sparsiﬁers of the desired form can be efﬁciently constructed in the query model and the MPC model.
We complement our algorithmic results by establishing nearly matching lower bounds that rule out the possibility of designing algorithms with better performance guarantees in each of these models.
Introduction
Hierarchical clustering (HC) is a popular unsupervised learning method for organizing data into a dendrogram (rooted tree).
It can be viewed as clustering datapoints at multiple levels of granularity simultaneously, with each leaf of the tree corresponding to a datapoint and each internal node of the tree corresponding to a cluster consisting of its descendent leaves.
Much of the technical development of HC originated in the ﬁeld of phylogenetics , where the motivation was to organize the different species into an evolutionary tree based on genomic similarities [20].
Since then, this tool has seen widespread use in data analysis for a variety of domains ranging from social networks , information retrieval , ﬁnancial markets [25, 8, 39] amongst many others.
Due to its popularity, HC has been extensively studied and several algorithms have been proposed.
The most prominent amongst these are bottom-up agglomerative algorithms such as average linkage, single linkage, complete linkage etc.
(see Chapter 14 in [30]).
However, despite these advances on the algorithmic front, very few formal guarantees were known for their performance, primarily owing to a historic lack of a well deﬁned objective function.
Therefore, the study of HC was largely empirical in nature for a long time.
A part of this issue was recently resolved, when [18] proposed an objective function for similaritybased HC.
This has since sparked interest in both the theoretical computer science as well as machine learning communities, for designing algorithms with provable guarantees for this objective [9, 12, 10, 17, 2].
The formal description is as follows: given as input a weighted undirected graph
G =
( V, E, w ) with n vertices (datapoints) and m edges with positive edge weights corresponding to pairwise similarities between its endpoints, the objective is to ﬁnd a hierarchy T over leaf nodes corresponding to the vertices V that minimizes the cost function
[Equation]
where T ij is the subtree rooted at the least-common ancestor of i, j in T and |T ij | is the number of descendent leaves in T ij .
Intuitively, cost G ( T ) incentivizes cutting heavy edges at lower levels in
T , thereby placing more similar datapoints closer together.
This objective has been shown to have several desirable properties, including one that guarantees an optimal tree which is binary [18].
This minimization objective however, turns out to be NP-hard.
Consequently, [18] and other subsequent work explored this objective from an approximation algorithms perspective [38, 9, 12, 10, 17, 2].
The best known polynomial time approximation is O ( √ log n ) which is achieved by the recursive sparsest cut (RSC) algorithm [18, 9, 17].
It is also known that no constant factor polynomial time approximation is possible for this objective under the small-set expansion (SSE) hypothesis [9].
In this paper, we study the above minimization objective for HC in the context of massive graphs.
While the currently known best algorithm can be considered “efﬁcient” in the classical sense, i.e. requires polynomial space and time 1 , this complexity can be prohibitive in many modern applications of HC that deal with staggering volumes of data.
For example, current social networks contain billions of edges which imposes serious limits on their storage and processing.
Therefore, alternative models of computation need to be considered in the context of such massive graphs.
In this work, we consider three widely-studied models, each aimed at optimizing a different fundamental resource: ( i ) the (dynamic) streaming model [22] for space efﬁciency, where the edges are presented in a stream, ( ii ) the general graph (query) model [28] for time efﬁciency, where the edges can be accessed via degree and neighbour queries, and ( iii ) the massively parallel computation model (MPC) for communication efﬁciency, where the edges are partitioned across multiple machines connected together through a communication channel.
The focus of our work is the following fundamental question:
Can we design sublinear (in the number of edges) algorithms for hierarchical clustering in each of these massive-graph computation models?
We provide an almost complete resolution to this question by providing matching upper and lower bounds for sublinear algorithms in all three canonical models of computation discussed above.
Remark 1.
When studying graph problems in the sublinear setting, one can consider an even more constrained setting where the available resource is o ( n ) , i.e. sublinear in the number of vertices.
However, we are interested in actually ﬁnding a hierarchical clustering of the data, the writing of which takes Ω( n ) time and space (and in MPC, Ω( n ) machine memory).
Since in most practical settings, the bottleneck is often the edges in the graph rather than the vertices, we believe it makes more sense for us to consider sublinearity only in the edge parameter, i.e. m , in all three models.
Due to the space needed to present full technical details of our main results, we focus here on giving a overview; the complete paper is deferred to the Appendix.
Our results are summarized in Table 1.
Notation.
In the following section, we use n and m to refer to the number of vertices and edges in the input graph, respectively.
We use (cid:101) O ( · ) to suppress multiplicative O (log c n ) factors for constant c ,
[Table]
and the term “w
[Table]
h
[Table]
p
[Table]
” implies with probability 1 − 1 / poly ( n ) .
Lastly, φ will denote the approximation ratio of any desired ofﬂine algorithm for hierarchical clustering.
For example, if allowed unbounded computation time, we have φ = 1 ; given polynomial time, the current best algorithm [9] gives φ = O ( √ log n ) .
We assume this abstraction as any improvement in the approximation ratio here automatically implies an identical improvement in our upper bounds
[Table]
Overview of Algorithmic Results
We begin by presenting our algorithmic results for the three models of computation, which at their core, are all based on the same meta-algorithm which follows from a new structural view of the minimization objective deﬁned in Eq.
(1) in terms of global cuts in the input graph.
2.1 A Meta-Algorithm for Sublinear-Resource Hierarchical Clustering
In their paper, [18] showed that cost G ( T ) can be viewed in two equivalent ways, the ﬁrst being the one deﬁned earlier in Eq.
(1), and the other in terms of the splits induced by the internal nodes in the hierarchy: given a hierarchy T with each internal node corresponding to a binary split 2 where some subset of vertices S ⊆ V of the input graph is partitioned into two pieces ( S (cid:96) , S r ) , then
[Equation]
where for any disjoint subsets S, T ⊂ V , w G ( S, T ) is the total weight of the edges in G going between S and T .
At this point, one might be tempted to think that if we could somehow construct a sparse representation of G such that the weights w G ( S, T ) are approximately preserved for any disjoint S, T ⊂ V , then the cost of every hierarchy would also be approximately preserved.
Following this, we could run any desired ofﬂine algorithm on this representation with improved efﬁciency due to its sparsity without much loss in the quality of its solution.
Unfortunately, this is just wishful thinking as such a representation can easily be shown to require Ω( m ) time and space 3 .
Our ﬁrst contribution is to show there is in fact a third equivalent view of this same objective function in terms of global cuts in G , and the above alternate formulation serves as our starting point.
This result follows from two critical observations, the ﬁrst of which is given any two disjoint
S, T ⊂ V , we can compute w G ( S, T ) exactly as w G ( S, T ) =
(1 / 2) · ( w G ( S, S )
+ w G ( T, T ) − w G
( S ∪ T, S ∪ T )) .
We could stop here as the quantities on the right are all graph cuts, and it is well known [7] that one can construct a (cid:101) O ( n ) sized sparsiﬁer that approximately preserves all graph cuts.
Unfortunately, the distortion in w G ( S, T ) can be very large depending on the quantities on the right, and the cumulative error in cost G ( T ) blows up with the depth of the tree which is even worse.
Here is the second observation: the negative term w G ( S ∪ T, S ∪ T ) that internal node S contributes to the cost also appears as a positive term in its parent’s contribution to the cost.
We can pass this term as a discount in its parent’s contribution to the cost, which after cascading gives a third view of Eq.
cost G ( T ) :
[Equation]

[Equation]
a linear combination of graph cuts.
This gives a strong blackbox reduction to cut-sparsiﬁers; preserving graph cuts to a (1 ± (cid:15) ) factor also preserves the cost of all hierarchies to a (1 ± (cid:15) ) factor.
However, we are not done yet, as cut-sparsiﬁers cannot be computed efﬁciently in certain models of computation; for instance, they necessarily require Ω( m ) queries to the underlying graph.
We therefore introduce a weaker notion of sparsiﬁcation that, for any cut ( S, S ) , allows for an additive error of δ min {| S | , | S |} in addition to the usual multiplicative error of (1 ± (cid:15) )
(Appendix B Defn. 1).
We term this generalization an ( (cid:15), δ ) -cut sparsiﬁer.
A similar notion was also proposed in an earlier work by [32], which unfortunately does not work here (see Appendix D.1 for details).
Our next result then shows that the distortion in the cost of any hierarchy under this weaker sparsiﬁer is also bounded.
Theorem 1 (informal) .
Given any weighted graph G , an ( (cid:15), δ ) -cut sparsiﬁer of G preserves the cost of any tree T up to a multiplicative
(1 ± (cid:15) ) factor
and an additive O ( δn 2 ) factor
Therefore, if we could lower bound the cost of the optimal hierarchical clustering by some quantity C , we could set δ =
(cid:15)C/n 2 .
The above result would then imply morally the same result as that achieved by traditional cut-sparsiﬁers: preserving graph cuts in this (cid:15), δ sense for a sufﬁciently small δ also preserves the cost of all hierarchies upto a (1 ± (cid:15) ) factor.
The last key result exactly establishes such a general purpose lower bound on the cost of any hierarchical clustering in a graph, which can be efﬁciently estimated in all models of computation we consider.
This chain of ideas results in the following meta-algorithm for sublinear HC given any parameter (cid:15) > 0 and model of computation: Compute the lower bound on the cost of an optimal HC which establishes the tolerable additive error in our ( (cid:15), δ ) sparsiﬁer, following which we efﬁciently (in the resources to be optimized) compute the said sparsiﬁer.
We ﬁnally run any φ -approximate HC algorithm, which is guaranteed to ﬁnd a (1 + (cid:15) )
φ -approximate HC tree.
Our subsequent results give sublinear constructions of these ( (cid:15), δ ) -cut
sparsiﬁers in each of the three models of computation.
2.2 Sublinear Space Algorithms in the (Dynamic) Streaming Model
We ﬁrst consider the dynamic streaming model for sublinear space algorithms, where the edges in the input graph are presented in an arbitrarily ordered stream of edge insertions and deletions.
Our upper bound here is a direct consequence of Theorem 1 used in conjunction with [1], a seminal result that showed an ( (cid:15), 0) -cut
sparsiﬁer can be constructed in (cid:101) O
( (cid:15) − 2 n ) space and a single pass in this setting.
Theorem 2 (informal) .
There exists a single-pass, (cid:101) O ( n ) space, streaming algorithm that given any weighted graph G presented in a dynamic stream, w.h.p.
ﬁnds a (1 + o (1)) · φ -approximate HC of G .
As outlined in our meta-algorithm, instantiating the ofﬂine algorithm with RSC with the input graph being the sparsiﬁer gives us a polynomial time, (cid:101) O ( n ) space, single-pass dynamic streaming algorithm with approximation ratio O ( √ log n ) as a corollary.
See Appendix C for more details.
We note that coupled with a recent result [15], our Theorem 1 also implies an (cid:101) O ( n ) -space algorithm for ﬁnding a (1 + o (1)) · φ -approximate HC in the more general turnstile streams , where arbitrary edge weight updates can appear.
2.3 Sublinear Time Algorithms in the Query Model
We next consider the general graph model [28] for sublinear time algorithms, where the input graph can be accessed via two 4 types of queries: ( i ) degree queries: given u ∈ V , returns degree d u , and ( ii ) neighbour queries: given u ∈ V , i ≤ d u , returns the i -th neighbour of u .
Note that this model can be easily implemented using an adjacency array representation of the graph (See Appendix D for a more detailed discussion).
We ﬁrst present the result for unweighted graphs, where it is easier to see the key intuition.
Our main result in this model is a sublinear time construction of an ( (cid:15), δ ) -sparsiﬁer.
Theorem 3 (informal) .
There exists an algorithm that given query access to any unweighted graph
G , and any parameters (cid:15), δ ∈ (0 , 1] , can ﬁnd an ( (cid:15), δ ) -cut sparsiﬁer of G w.h.p.
in (cid:101) O
( n/ ( (cid:15) 2 δ )) time.
Our algorithm is based on a simple yet elegant idea (which builds upon a slightly different idea proposed in [32]; see Appendix D.1 for a detailed discussion): if we embed a constant-degree expander with edge weights δ in an unweighted graph (with unit edge weights), then the effective resistance of every edge in the resulting composite graph is tightly bound in terms of the effective degrees of its incident vertices; the effective degree of a vertex is a weighted sum of its degree in the input graph and its degree in the expander.
We can then leverage the effective resistance sampling scheme of [41] to construct an ( (cid:15), 0) -cut sparsiﬁer of this composite graph, which then is deterministically an ( (cid:15), δ ) -cut sparsiﬁer of the input graph with the sources of error being the usual multiplicative (cid:15) term due to sparsiﬁcation itself, and the (small) additive δ term due to the (few) extra edges introduced by the expander.
We can construct constant degree graphs that are expanders with high probability in sublinear time, and we show that there is an efﬁcient rejection sampling scheme for sampling edges according to their effective resistances, giving the above result: an ( (cid:15), δ ) -cut sparsiﬁer with (cid:101) O
( n/ ( (cid:15) 2 δ )) edges in the same amount of time and queries.
Moreover, the queries are completely non-adaptive assuming prior knowledge of vertex degrees.
This construction of ( (cid:15), δ ) -cut sparsiﬁers in conjunction with Theorem 1 then gives our sublinear time upper bound in this model.
Theorem 4 (informal) .
There exists an algorithm that given query access to any unweighted graph
G with m =
Θ( n ζ ) for ζ ∈ [0 , 2] , can ﬁnd a (1 + o (1)) · φ -approximate HC of G w.h.p.
using (cid:101) O
( g ( n, ζ )) queries, where g ( n, ζ ) ≤ n 4 / 3 is given by g ( n, ζ )
= max { n, n ζ } when ζ ∈ [0 , 4 / 3] , g ( n, ζ )
= max { n, n 4 − 2 ζ } when ζ ∈ (4 / 3 , 2] .
Moreover, given any (arbitrarily small) constant τ > 0 , the algorithm can ﬁnd an O ( √ log n ) -approximate HC of G w.h.p.
in (cid:101) O
( g ( n, ζ ) + n 1+ τ ) time.
It is interesting to observe that the query complexity g ( n, ζ ) reduces as the graph becomes denser.
This is because the cost of the optimal HC increases with the density of the input graph, which allows us to tolerate a larger additive error in our cut sparsiﬁer, thereby making it sparser.
In Sectionwe also discuss lower bounds showing that this query complexity is also the best possible for any (cid:101)
O
(1) -approximate algorithm.
The sublinear time claim in Theorem 4 is implied by results from [40] and [14] .
Speciﬁcally, [40] showed that, for any constant τ ∈ (0 , 1 / 2) , sparsest cuts and balanced separators can be approximated to within O ( √ log n ) in (cid:101)
O ( m ) time plus (cid:101) O
( n τ ) maximum ﬂow computations on graphs with (cid:101) O
( n ) edges, and [14] showed that a maximum ﬂow on a graph of m edges can be computed in O ( m 1+ o (1) ) time.
These two combined with the fact that our sparsiﬁer contains just (cid:101)
O ( g ( n, ζ )) edges give us the desired running time bound.
We generalize the above result in Appendix D.2 to weighted graphs by grouping edges according to geometrically increasing weights and constructing ( (cid:15), δ ) -cut sparsiﬁers for each weight class, and get an algorithm with essentially the same worst case performance and O (log n ) rounds of adaptivity.
2.4 Sublinear Communication Algorithms in the MPC model
Lastly, we consider the MPC model [6, 3] for sublinear communication, which is a common abstraction of many MapReduce-style computational frameworks.
Here, the edges in the input graph are partitioned across several machines that communicate with each other in synchronous rounds.
Each machine has memory sublinear in m , with its total communication bounded by its memory.
A more detailed description of this model is given in Appendix E.
Our ﬁrst result is a 2 -round MPC algorithm that uses (cid:101)
O
( n ) memory per machine.
In our algorithm, we leverage a construction of ( (cid:15), 0) -cut
sparsiﬁers due to [1] using (cid:101)
O (1) random linear sketches per vertex in the graph.
We show that 2 rounds are sufﬁcient to construct these linear sketches for each vertex– the ﬁrst round is used to construct partial sketches using local edges and the second round is used to aggregate these partial sketches into complete sketches for each vertex.
This construction of ( (cid:15), 0) -cut sparsiﬁers in conjunction with Theorem 1 gives us the following result.
Theorem 5 (informal) .
There exists an MPC algorithm that given any weighted graph G with edges partitioned across machines with (cid:101) O ( n ) memory and access to public randomness, can ﬁnd a (1 + o (1)) · φ -approximate HC of G w.h.p.
on a designated machine in 2 rounds of MPC.
Our second result is a 1 -round MPC algorithm that solves this problem for unweighted graphs using machines with (cid:101) Θ( n 4 / 3 ) memory.
The execution of our algorithm depends on the density of the underlying graph.
If m ≤ n 5 / 3 , then we can again use the result from [1] by constructing local linear sketches on each machine and sending them to a coordinator who can aggregate them.
Note that we only require 1 round in this case as the number of machines is ≤ n 1 / 3 , and hence, we only need to communicate (cid:101) O ( n 1 / 3 ) sketches per vertex which is within our memory budget.
If m > n 5 / 3 , we show that the cost of the optimal hierarchy is sufﬁciently large such that a coarse ( (cid:15), δ ) -cut sparsiﬁer of size (cid:101) O
( n 4 / 3 ) obtained by randomly subsampling edges will sufﬁce.
As a consequence, each machine can subsample its edges independently and send these (cid:101) O ( n 4 / 3 ) edges to the coordinator.
We now summarize this 1 -round result below.
Theorem 6.
(informal)
There exists an MPC algorithm that given any unweighted graph G with edges partitioned across machines with (cid:101) O
( n 4 / 3 ) memory and access to public randomness, can ﬁnd a (1 + o (1)) · φ -approximate HC of G w.h.p.
on a designated machine in 1 round of MPC.
In the next section, we discuss a 1 -round lower bound for MPC which shows that n 4 / 3 − o (1) memory per machine is needed by any (cid:101) O (1) -approximate algorithm on unweighted graphs.
Overview of Lower Bounds
First note that in the (dynamic) streaming model, since our goal is for the algorithm to output a hierarchical clustering tree, we necessarily need Ω( n ) space.
Thus our (cid:101) O ( n ) space dynamic streaming algorithm that obtains a (1 + o (1))
-approximation is nearly optimal.
We then show that in the other two models of computation, our algorithms are also essentially the best possible.
3.1 Lower bounds in the Query Model
Note that the query complexity bound of our sublinear time algorithm is always at most (cid:101) O ( n 4 / 3 ) , where the worst-case input is an unweighted graph with about m ≈ n 4 / 3 edges.
We note that our algorithm obtains an O ( √ log n ) -approximation
and (i) is completely non-adaptive on unweighted graphs assuming prior knowledge of vertex degrees, and has O (log n ) rounds of adaptivity on weighted graphs; (ii) only uses degree queries and neighbor queries (no pair queries needed, see
Footnote 4).
We then show that n 4 / 3 − o (1) queries are indeed necessary for obtaining any (cid:101) O (1) - approximation even in unweighted graphs and given unlimited adaptivity and access to pair queries .
Theorem 7 (informal) .
Let A be a randomized algorithm that, on any input unweighted graph with Θ( n 4 / 3 ) edges, outputs with high probability a polylog ( n ) -approximate hierarchical clustering tree.
Then A necessarily uses at least n 4 / 3 − o (1) queries.
We brieﬂy describe the family of hard graph instances that we use to prove this result.
Roughly, a graph from such a family is generated by ﬁrst taking a union of n 2 / 3 vertex-disjoint cliques of size n 1 / 3 each, and then connecting them by a random “perfect matching”.
More speciﬁcally, we treat each clique as a supernode , and generate a perfect matching between these n 2 / 3 supernodes uniformly at random.
Then if the i th clique is matched to the j th clique in the perfect matching, we will add about n o (1) edges between these two cliques, which are also chosen in a random manner.
We show that, in order to output a good hierarchical clustering solution, it is necessary to discover a non-trivial portion of the edges that we add between the cliques, even though their number is relatively tiny compared to those within, and the latter task provably requires n 4 / 3 − o (1) queries.
While this plan looks intuitive, one has to be careful about not leaking information about the “perfect matching” between the cliques from the vertex degrees, which an algorithm knows a priori (or can otherwise acquire using O ( n ) non-adaptive degree queries).
In particular, once the inter-clique edges are added, one could tell that the vertices with degree higher than n 1 / 3 − 1 are those participating in the perfect matching.
Note that there are only n 2 / 3+ o (1) such vertices in total, and each of them has degree at most O ( n 1 / 3 ) .
As a result, by probing all neighbors of these vertices, one can easily ﬁnd all the inter-clique edges using n 1+ o (1) neighbor queries.
Our way around this issue is to also delete certain edges within the cliques based on what inter-clique edges we have added, so as to ensure that each vertex has the exact same degree of n 1 / 3 − 1 .
This of course further complicates things as it increases the correlation between the edge slots — for instance, whenever an edge between a matched pair of cliques is revealed to the algorithm, missing edges within each clique are no longer independent.
Consequently, our proof for this lower bound turns out to be considerably involved; we refer the reader to Appendix F.2 for more details.
Theorem 7 shows that the worst-case query complexity of our algorithm is nearly optimal.
Note that, however, for unweighted graphs, our algorithm also obtains improved query/time complexity when m is far from n 4 / 3 .
It is then natural to ask — are these improvements also the best possible?
We answer this question in the afﬁrmative.
In particular, we show that one can push further the ideas we discussed above to get a tight query lower bound for every graph density .
We summarize these lower bounds below.
Note in particular that, for m =
Θ( n 2 ) , as we will show later, any hierarchical clustering achieves an O (1) -approximation, thus trivially 0 queries are sufﬁcient.
Theorem 8 (informal) .
Let ζ ∈
[0 , 2] be any constant.
Let A be a randomized algorithm that, on any input unweighted graph with Θ( n ζ ) edges, outputs with high probability a polylog ( n ) -approximate
HC.
Then A necessarily uses at least Ω( g ( n, ζ )) queries, where g ( n, ζ ) = max { n, n ζ − o (1) } when ζ ∈ [0 , 4 / 3] , g ( n, ζ )
= max { n, n 4 − 2 ζ − o (1) } when ζ ∈ (4 / 3 , 2) , and g ( n, ζ ) = 0 when ζ = 2 .
3.2 Lower bounds in the MPC Model
As our goal is for some machine to output a good hierarchical clustering tree, Ω( n ) memory per machine is necessary.
Indeed, our 2-round MPC algorithm obtains a (1 + o (1))
-approximation for weighted graphs using a nearly optimal memory of (cid:101) O ( n ) per machine (Theorem 5).
To show that the number of rounds of our algorithm is also optimal, we prove that a superlinear (in particular, n 4 / 3 − o (1) ) memory per machine is necessary for any 1 -round MPC algorithm in which some machine outputs with high probability a polylog ( n ) -approximate hierarchical clustering tree.
Moreover, in our lower bound instance, the total memory of all machines is ≈ m , which means that the input is split across fewest possible machines.
We speciﬁcally prove the following result:
Theorem 9 (informal) .
Let P be any 1-round protocol in the MPC model where each machine has memory O ( n 4 / 3 − ε ) for any constant ε > 0 .
Then at the end of the protocol P , no machine can output a polylog ( n ) -hierarchical clustering tree with probability better than o (1) .
Note that this lower bound matches our upper bound result in Theorem 6.
Our family of hard instances is roughly deﬁned as follows.
Let α ≈ 2 / 3 , β ≈ 1 / 3 be certain constants.
A graph G of 2 n vertices from such a family consists of two vertex-disjoint parts, each supported on n vertices.
The ﬁrst part is supported on vertices V 1 and is a union of vertex-disjoint bi-cliques of size n α ; the second part is supported on vertices V 2 and is in turn a union of vertex-disjoint bi-cliques of size n β , where we have | V 1 | = | V 2 | = n .
We will also permute the vertex labels of G uniformly at random.
See Figure 1 for an illustrative example of such a graph G .
We ﬁrst show that in order to output a good hierarchical clustering solution, it is necessary to discover (almost) the exact clique structures of the vertex-induced subgraphs G [ V 1 ] , G [ V 2 ] , for otherwise a balanced cut has non-trivial probability of cutting too many edges within the cliques.
Then as an adversary, our strategy is to hide G [ V 2 ] , which has signiﬁcantly fewer edges than G [ V 1 ] , by splitting
G across multiple machines.
To this end, we observe that G [ V 1 ] can be tiled using edge-disjoint subgraphs G 1 , . . .
, G t that are each isomorphic to G [ V 2 ] (see Figure 1).
This suggests that we could give G [ V 2 ] to a uniformly random machine, and then give each G i to one of the other machines.
Note that, crucially, each machine’s input follows the exact same distribution, namely a union of bicliques of size n β with vertex labels permuted uniformly at random, although the input distributions of different machines are correlated.
As a result, each machine individually has no information whether its input graph is G [ V 2 ] or just a subgraph G i of G [ V 1 ] .
Therefore, each machine has to send a message to the coordinator such that, if its input graph happens to be G [ V 2 ] , the coordinator will be able to recover the clique structures with high probability.
G [ V
G [ V 2 ] :
Since the coordinator can only receive a total message size bounded by its machine memory, by choosing suitable values of α, β , each machine on average can only send a message of size o ( n ) .
Now the problem effectively becomes a one-way, two party communication problem, where Alice is given G [ V 2 ] and needs to send Bob a single message so that Bob can recover the clique structures with high probability.
We then conclude the proof by showing that this two-party problem requires Ω( n ) communication.
We refer the reader to Appendix G for more details.
Related Work
The work of Dasgupta [18] is the starting point of our work.
[18] deﬁned the objective function for hierarchical clustering, namely minimum cost hierarchical partitioning, that we study in this paper.
They showed that the resulting problem is NP-hard and the recursive sparsest-cut algorithm achieves an O ( φ log n ) approximation, where φ = O ( √ log n ) is the current best poly-time approximation for sparsest-cut.
[38] improved this approximation factor to O (log n ) using an LP-based algorithm.
[17, 9] showed that the recursive sparsest cut algorithm of [18] in-fact achieves an O ( φ ) approximation.
[38] and [9] also showed that no polynomial time algorithm can achieve constant factor approximation under the small set expansion (SSE) hypothesis.
[17, 33] showed that by imposing certain probabilistic or structural assumptions on the graph, one can circumvent this constant factor inapproximability.
There has also been work on maximization objectives for hierarchical clustering.
[35] considered a “dual” version of the Dasgupta objective: where the goal is to maximize the revenue n (cid:80) e ∈ E w e − cost G ( T ) .
While the optimal values for both objectives are achieved by the same solution, this objective behaves very differently from an approximation perspective.
[17] considered a setting where the edge weights correspond to dissimilarities rather than similarities and the goal is to maximize the dissimilarity-based objective cost G ( T ) .
[35] and [17] both study the average-linkage algorithm and show that it achieves approximation factors of 1 / 3 and 2 / 3 , respectively.
[10] provided algorithms with slightly better approximation factors of 1 / 3 + δ and 2 / 3 + δ , respectively.
[13] improved the approximation factor to 0 .
4246 for the dual objective in [35], which was later improved to 0 .
585 by [2].
Very recently, [36] improved the approximation to 0 .
71604 for the dissimilarity objective of [17].
Several other variations of this basic setup have been considered.
For example, [12] have considered this problem in the presence of structural constraints.
[11, 34, 37] considered a setting where vertices are embedded in a metric space and the similarity/dissimilarity between two vertices is given by their distances.
The most relevant to our work amongst these is [37] which considered this metric embedded hierarchical clustering problem in a streaming setting.
However, the stream in their setting is composed of vertices while edge weights can be directly inferred using distances between vertices; whereas the stream in our streaming setting is composed of edges while vertices are already known.
Moreover, their study is only limited to the streaming setting.
There has also been work on designing faster/parallel agglomerative algorithms such as single-linkage, average-linkage etc.
While these works share the same motivation as ours, namely, scaling HC algorithms to massive datasets, these results are largely orthogonal to ours.
The primary philosophical difference is that these aforementioned works are aimed at speeding up/parallelizing very speciﬁc kinds of linkage based algorithms, while recovering the same or similar cluster trees (under very different notions of similarity) that would have been computed by the slower/sequential algorithm.
Moreover, the speciﬁc algorithms considered in these works have no known approximation guarantees for Dasgupta’s objective.
Our work on the other hand approaches this problem from an optimization perspective.
Through data sparsiﬁcation, we aim to recover a cluster tree with marginal loss in objective function value as compared to one computed over the entire (dense) input data by any given HC algorithm as a blackbox, achieving a speedup in runtime or reducing its memory requirement due to sparsity.
[29] studied the hierarchical clustering problem in an MPC setting.
However, their work only considered the maximization objectives [35, 17], while our work is primarily focussed on the minimization objective of [18].
Recent Independent work: Very recently and independent of our work, [5] considered the problem of hierarchical clustering under Dasgupta’s objective in the (dynamic) streaming setting.
The primary focus of their work is on estimating the cost of an optimal hierarchy in o ( n ) space whereas the focus of our work is to actually output a near-optimal hierarchical clustering.
[5] also gives a single-pass, (cid:101) O ( n ) memory streaming algorithm for ﬁnding a clustering that also uses cut-sparsiﬁers.
However, their algorithm needs to restrict the solution space to only balanced trees, and hence, is only able to achieve an O ( φ ) approximation guarantee instead of the stronger (1 + o (1)) · φ approximation guarantee that we achieve for the streaming setting.
They also obtain a 2 -round algorithm for MPC that achieves an O ( φ ) approximation guarantee using (cid:101) O ( n ) memory per machine, which is again slightly weaker than the (1 + o (1))
φ approximation guarantee that we achieve.
But [5] does not show any lower bound results for the MPC model.
Moreover, their work does not consider sublinear time setting and their algorithms cannot be easily adapted to this setting.
Implications for Other Hierarchical Clustering Objectives
As noted in the previous section, two maximization objectives for hierarchical clustering were proposed subsequent to the work of [18]: (1) the revenue objective [35] for similarity-based HC which is a “dual” of cost G ( T ) , (2) the dissimilarity objective [17] where the edges correspond to pairwise dissimilarities and the objective is the same as cost G ( T ) .
In this section, we discuss the implications of our work for both objectives in the sublinear-resource regime.
We begin by noting a sharp contrast in the difﬁculty of achieving a “good” solution for the minimization objective [18], and the two maximization objectives described above.
In fact, it is possible to achieve a O (1) approximation to both maximization objectives non-adaptively ; a random binary hierarchy, in expectation, is a 1 / 3 approximation of the optimal revenue [35], and is a 2 / 3 approximation of the optimal dissimilarity objective [17], constructing which requires no knowledge of the input graph.
On the other hand, it is not hard to see that one would achieve an arbitrarily bad approximation of the minimization objective unless something non-trivial was learned about the input graph.
That said, one can further question whether it is possible to match the solution quality of any given ψ -approximate ofﬂine algorithm for the maximization objectives in the models of computation we consider.
We answer this in the afﬁrmative; we can in fact achieve even stronger performance guarantees for both objectives in the sublinear resource regime by exploiting the fact that their corresponding optimal hierarchies have large objective function values 5 , allowing us to tolerate even larger additive errors in our cut-sparsiﬁers.
A straightforward application of our structural decomposition of the cost function along with its downstream implications in each of the three models of computation directly gives us ( ψ − o (1))
-approximate algorithms for both HC maximization objectives in weighted graphs that use ( i ) a single-pass and (cid:101) O ( n ) space in the dynamic streaming model, ( ii ) (cid:101) O
( n ) queries 6 in the general graph (query) in the MPC model, which can be further improved to use just ( iv ) 1-round, and (cid:101) O ( n ) communication for unweighted graphs.
Conclusions and Future Directions
In this paper, we studied hierarchical clustering problem under Dasgupta’s objective [18] in the regime of sublinear computational resources.
We gave sublinear space, query, and communication algorithms for ﬁnding a (1 + o (1))
φ -approximate hierarchical clustering, where φ is the approximation ratio of any ofﬂine algorithm for this problem, and a sublinear time algorithm for ﬁnding an O ( √ log n ) - approximate hierarchical clustering.
At the core of our sublinear algorithms is a novel meta-algorithm which ﬁrst obtains an ( ε, δ ) -cut sparsiﬁer of the graph and then runs hierarchical clustering algorithm on the sparsiﬁer.
We also proved sharp information-theoretic lower bounds showing that the performance of all our sublinear algorithms is essentially optimal for any polylog ( n ) -approximation.
Note that all our algorithms and lower bounds are aimed at ﬁnding an explicit hierarchical clustering tree.
Therefore a natural direction for future work is to understand whether we can get even more efﬁcient sublinear algorithms if we only want to estimate the cost of the optimal hierarchical clustering to within some small error.
We note that two recent and independent works have already considered this question.
In the streaming model, [5] have proved extensive lower bounds for optimal cost estimation under various regimes of space and number of passes over the input stream.
In the query model, [31] have proved both upper and lower bounds for sublinear time cost estimation under certain “well-clusterability” assumptions and given auxiliary information about the underlying input graph instance.
However, this question remains open in the query model (sublinear time) for more general input graph instances, and completely unexplored in the MPC model (sublinear communication).
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF awards CCF-1763514, CCF-1934876, and CCF-
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text
instructions
[Table]
participants
screenshots,
applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Notation and Preliminaries
We use the notation G =
( V, E ) to represent unweighted graphs, and G =
( V, E, w ) for weighted graphs.
We use lowercase letters u, v to refer to vertices in V , and given a vertex v , we use d G ( v ) to refer to its degree in graph G .
We use capital letters S, T to represent subsets of vertices, and given a vertex set S ⊂ V , we use | S | to refer to its cardinality, S :
=
V \ S to refer to its complement, and G [ S ] to refer to the subgraph of G induced by vertex set S .
Furthermore, given two disjoint vertex sets S, T , we use w G ( S, T ) :
= (cid:80) ( u,v ) ∈
E : u ∈ S,v ∈ T w ( u, v ) to represent the total weight of the edges in graph G with one endpoint in S and the other in T .
In the case of an unweighted graph, this is equivalent to the number of edges going from S to T .
For ease of notation, we use w G ( S ) :
= w G ( S, S ) , and when the implied graph is clear from context, w e to refer to the weight of an edge e ∈ E in that graph.
Given a graph G =
( V, E ) , we use T to refer to a hierarchical clustering (tree) of the vertex set V , and cost G ( T ) to refer to the cost of this clustering in graph G .
Without loss of generality, we restrict our attention to just full binary hierarchical clustering trees, since the optimal tree is binary [18].
Any internal node S of a hierarchical clustering tree corresponds to a binary split ( S (cid:96) , S r ) (the left and right children of S in T ) of the set of leaves in the subtree rooted at S .
With some overload of notation, we let S represent both, the internal node of the clustering tree as well as the set of leaves S ⊆ V in the subtree rooted at internal node S .
Furthermore, since (the leaves in the subtree rooted at) an internal node S can correspond to an arbitrary subset of V , we use the term split to refer to a partition ( S (cid:96) , S r ) of S to disambiguate it from cuts , which are a partition of the entire vertex set V .
We conclude the preliminaries by presenting two useful facts from [18]; the ﬁrst is an equivalent reformulation of the similarity based hierarchical clustering cost function deﬁned earlier in the introduction, and the second is the cost of any hierarchical clustering in an unweighted clique.
Fact 1.
The hierarchical clustering cost of any tree T with each internal node S corresponding to a binary split ( S (cid:96) , S r ) of the subset S ⊆ V of vertices is equivalent to the sum
[Equation]
Fact 2.
The cost of any hierarchical clustering in an unweighted n -vertex clique is ( n 3 − n ) / 3 .
B Hierarchical Clustering using ( (cid:15), δ ) -Cut Sparsiﬁcation
In this section, we shall present the key insight behind all our results: the hierarchical clustering cost function can equivalently be viewed as a linear combination of global cuts in the graph .
As a consequence, approximately preserving cuts in the graph also approximately preserves the cost of hierarchies in the graph, effectively reducing the sublinear-resource hierarchical clustering problem to a cut-sparsiﬁcation problem.
However, there are some hard lower bounds that refute an efﬁcient (sublinear) computation of traditional cut-sparsiﬁers in certain models of interest.
Therefore, we begin by introducing a weaker notion of cut sparsiﬁcation, which we call ( (cid:15), δ ) -cut sparsiﬁcation.
Deﬁnition 1 ( ( (cid:15), δ ) -cut sparsiﬁer) .
Given a weighted graph G =
( V, E, w ) and parameters (cid:15), δ ≥ 0 , we say that a weighted graph (cid:101) G =
( V, (cid:101) E, (cid:101) w ) is an ( (cid:15), δ ) -cut sparsiﬁer of G if for all cuts S ⊂ V ,
[Equation]
The above is a generalization of the usual notion of cut-sparsiﬁers (which are ( (cid:15), 0) -cut
sparsiﬁers as per the above deﬁnition) that allows for an additive error in addition to the usual multiplicative error in any cut of the graph.
A variant of this idea has been proposed before under the term probabilistic ( (cid:15), δ ) -spectral
sparsiﬁers in [32] which was similarly motivated by designing sublinear time algorithms for (single) cut problems on unweighted graphs.
However, as the name might suggest, the key difference between the prior work and ours is that the above bounds on the cut-values hold only in expectation (or any given constant probability) for any ﬁxed cut in the former.
Due to this limitation, we cannot use this previous work in a blackbox, and new ideas are needed.
We now show that for any two graphs that are close in this (cid:15), δ sense, the cost of any hierarchy in these two graphs is also close as a function of these parameters, effectively allowing the use of ( (cid:15), δ ) -cut sparsiﬁers in a blackbox .
Lemma 1.
Given any input weighted graph G =
( V, E, w ) on n vertices, and an ( (cid:15), δ ) -cut sparsiﬁer (cid:101) G of G , then for any hierarchy T over the vertex
set V , we have
[Equation]
Therefore, running a φ -approximate hierarchical clustering oracle A with input as the sparsiﬁer (cid:101) G with (cid:15) ≤ 1 / 2 produces a hierarchical clustering T
A whose cost in G is at most
[Equation]
where T ∗ is an optimal hierarchical clustering of G .
Proof.
Consider any graph H (not necessarily G or (cid:101) G ) over vertex set V .
Given any hierarchy T over the vertex set V , let S 0 be the root node with left and right children S 0 (cid:96) , S 0 r , respectively.
Then we have the the cost of this hierarchy in H is given by
[Equation]
Now observe that since the split at the root S 0 is a partition of the entire vertex set V into S 0 (cid:96) , S 0 r , we have w
H ( S 0 (cid:96) , S 0 r ) =
w H ( S 0 (cid:96) ) =
w H ( S 0 r ) .
Furthermore, observe that for any split of S , S (cid:96) ∪ S r = S , and therefore, we can represent the total weight of the edges crossing the split w H ( S (cid:96) , S r ) =
( 1 / 2 ) · ( w H ( S (cid:96) )
+ w H ( S r ) − w H ( S )) .
Therefore,
[Equation]
Therefore, the hierarchical clustering cost function can equivalently be represented as a non-negative weighted sum of cuts in a graph.
We shall now use this reformulation of the clustering cost function to bound the error in the cost of any hierarchy T over a graph G and its ( (cid:15), δ ) -sparsiﬁer (cid:101) G as a function of the error in the cuts in these two graphs, which is parameterized by (cid:15), δ .
Our claimed lower bound is easy to see since for every cut ( S, S ) , w (cid:101) G ( S ) ≥ (1 − (cid:15) )
w G ( S ) , and therefore,
[Equation]
To show the upper bound, we have cost
[Equation]
Finally, we claim that for any binary hierarchical clustering tree T over n vertices (leaves), (cid:88)
[Equation]
We shall prove this claim by induction on the number of leaves of T .
The base case is easy to see, which is a binary tree on 2 leaves.
Assuming this claim holds for all binary trees on n (cid:48) < n leaves, consider any binary tree
T with n leaves.
Suppose the split at the root partitions the set of n leaves S 0 into sets S 0 (cid:96) and S 0 r .
Let T (cid:96) , T r be the subtrees of T rooted at S 0 (cid:96) , S 0 r , respectively.
Then we have
[Equation]
Since both | S
[Equation]
| , | S 0 r | < n , applying our induction hypothesis on the subtrees T
[Equation]
, T r gives us that
[Equation]
Substituting these bounds on the above sum proves our claim as
[Equation]
Finally
[Equation]
observe that the φ -approximate hierarchical clustering oracle on input (cid:101) G ﬁnds a tree T A such that
[Equation]
Applying the above bound with T
=
T ∗ , an optimal hierarchical clustering of G gives us that
[Equation]
Therefore
[Equation]
for (cid:15) ≤ 1 /
[Equation]
we have that
[Equation]
The above result shows that these weaker cut sparsiﬁers also approximately preserve the cost of any hierarchical clustering, but only up to an additive O ( δn 2 ) factor.
Therefore, supposing we could efﬁciently estimate a lower bound OPT on the cost of an optimal hierarchical clustering in a graph G , we could then set the additive error δ = (cid:15) OPT /n 2 , giving us that any φ -approximate hierarchical clustering for (cid:101) G is a (1 + 5 (cid:15) )
φ -approximate hierarchical clustering for G .
This implies that hierarchical clustering is effectively equivalent to efﬁciently computing an ( (cid:15), δ ) -cut sparsiﬁer with a sufﬁciently small additive error δ .
The following result ﬁlls in the ﬁnal missing link in our chain of ideas by establishing a general-purpose lower bound on the cost of any hierarchical clustering in an unweighted graph as a function of the number of vertices and edges in the graph.
Lemma 2.
Let G be any unweighted graph on n vertices and m edges.
Then the cost of any hierarchical clustering in G is at least 4 m 2 / (3 n ) .
Proof.
Given any unweighted graph G =
( V, E ) over n vertices and m edges, ﬁx any hierarchy T of the vertices V .
In order to lower bound the cost of T , we shall iteratively modify the “base graph” graph G by moving edges, strictly reducing the cost of T with each modiﬁcation such that the ﬁnal graph has a structure that makes the hierarchical clustering cost of T easy to analyze.
In particular, the ﬁnal graph would be such that each connected component is either a clique or two cliques connected together by some number of edges.
This is done as follows: given any hierarchy T of V , we perform a level order traversal over the internal nodes of T , and at each node S , we modify the graph by pushing edges crossing the split ( S (cid:96) , S r ) down to lower level splits.
Formally, let S 1 , · · · , S n − 1 be a level-order traversal over internal nodes of T .
We denote by
G t =
( V, E t ) the modiﬁed graph after visiting internal node S t , with G 0 = G .
Given G t , we visit S t + and modify the graph as follows: if the subgraphs G t [ S t +1 (cid:96) ] , G t [ S t +1 (cid:96) ] induced by vertex sets S t +1 (cid:96) , S t +1 r respectively, are both cliques, then G t +1 =
G t ; else move a maximal number of (arbitrary) edges crossing the split ( S t +1 (cid:96) , S t +1 r ) to any (arbitrary) edge slots that are available in subgraphs G t [ S t +1 (cid:96) ] , G t [ S t +1 r ] until either (a) the split ( S t +1 (cid:96) , S t +1 r ) has no more edges going across in which case the two subgraphs become disconnected, or (b) both of the subgraphs become cliques with the edges remaining going across these cliques.
We call the resulting graph G t +1 .
Observe that the cost of T in G t +1 is at most the cost of T in G t .
Let the ﬁnal graph obtained after this traversal be G n − 1 .
It is easy to see that G n − 1 is a collection of connected components, with each connected component being either a clique or two cliques with edges going across them, and that cost
G n − 1 ( T ) ≤ cost G ( T ) .
In this graph G n − 1 , (1) let k 1 , . . .
, k r be the cliques, with k j being the number of vertices in clique j , and (2) let t 1 , . . .
t s be the connected components that are two cliques connecting by edges, where each t i =
{ k i, 1 , k i, 2 , c i } with k i, 1 , k i, 2 being the number of vertices in the two cliques of component t i , and c i < k
i, 1 · k i, 2 being the number of edges going across the two cliques.
Then the cost of T on G n − 1 is given by
[Equation]
which follows by construction of G n − 1 and Fact 2.
We also observe that n = r (cid:88)
[Equation]
= k j +
[Equation]
(cid:88)
[Equation]
=
[Equation]
( k
[Equation]
k
[Equation] [Equation]
and m =
[Equation]
(cid:33)
[Equation]
k
[Equation] [Equation]
(cid:33)
[Equation]
With these three observations
[Equation]
we shall now prove our claimed lower bound.
We have that
[Equation]
=
[Equation] [Equation]
n
[Equation]
where ( a ) follows by Cauchy-Schwarz inequality, ( b ) follows by Eq.
(4), ( c ) follows by observing
c i < k i, 1 · k i,due to which c 2 i ( k − 1 i, 1 + k − 1 i, 2 ) <
c i ( k i, 1 + k i, 2 ) , and ( d ) follows from the cost of hierarchical clustering T in
G n − 1 established in Eq.
Therefore, we have that
[Equation]
for any hierarchical clustering T in any graph G on n vertices and m edges.
We conclude this section with a remark about one particular instantiation of a φ -approximation oracle for hierarchical clustering.
Speciﬁcally, [17] showed that the recursive sparsest cut algorithm, i.e. recursively splitting the vertices using either the uniform sparsest cut or the balanced cut (sparsest cut that breaks the graph into two roughly equal parts) in the subgraph induced by the vertices, is a 6 .
75 γ -approximation to hierarchical clustering given access to a γ -approximation algorithm for sparsest cut or balanced cut.
The best known polynomial time approximation for either is O ( √ log n ) , a celebrated result due to [4].
These results in combination give us the following corollary.
Corollary 1.
Given any input weighted graph G =
( V, E ) on n vertices, and an ( (cid:15), δ ) -cut sparsiﬁer (cid:101) G of G for any constant 0 ≤ (cid:15) ≤ 1 / 2 and a sufﬁciently small 0 ≤ δ , there exists a polynomial time algorithm that given sparsiﬁer (cid:101) G as the input, ﬁnds a hierarchical clustering T whose cost in G is at most O ( √ log n ) · cost G ( T ∗ ) , where T ∗ is the optimal hierarchical clustering in G .
In the following sections, we use this idea of constructing ( (cid:15), δ ) -cut
sparsiﬁers in three well-studied models for sublinear computation: the streaming model for sublinear space, the query model for sublinear time, and the MPC model for sublinear communication.
C Sublinear Space Algorithms in the Streaming Model
We ﬁrst consider the space bounded setting in the dynamic streaming model, where the input graph is presented as an arbitrary sequence of edge insertions and deletions.
The objective is to compute a good hierarchical clustering of the input graph given O ( n polylog ( n )) memory, which is sublinear in the number of edges in the graph (referred to as a semi-streaming setting).
The following theorem describes the main result of this setting.
Theorem 10.
Given any weighted graph G =
( V, E, w ) with n vertices and the edges of the graph presented in a dynamic stream, a parameter 0 < (cid:15) ≤ 1 / 2 , and a φ -approximation oracle for hierarchical clustering
, there exists a single-pass semi-streaming algorithm that ﬁnds a (1 + (cid:15) )
φ -approximate hierarchical clustering of G with high probability using (cid:101) O
( (cid:15) − 2 n ) space.
This result is a direct consequence of Lemma 1 and known results [26, 1] for constructing an ( (cid:15), 0) -cut
sparsiﬁers in single-pass dynamic streams using polynomial time and (cid:101) O
( (cid:15) − 2 n ) space.
Lastly, Corollary 1 gives us a complete polynomial time single-pass semi-streaming algorithm that ﬁnds an (cid:101) O
(1) -approximate hierarchical clustering of the input graph in (cid:101) O ( n ) space in a dynamic stream.
D Sublinear Time Algorithms in the Query Model
We now move our attention to the bounded time setting in the general graph (query) model [28], where the input graph is accessible via the following two 7 queries: ( a ) Degree queries: given v ∈ V , returns the degree d G ( v ) , and ( b ) Neighbour queries: given v ∈ V , i ≤ d G ( v ) , returns the i th neighbour of v (neighbours are ordered arbitrarily).
The objective is to compute a good hierarchical clustering of the input graph in time and queries sublinear in the number of edges in the graph.
This problem becomes substantially more interesting in this setting, as ﬁnding an ( (cid:15), 0) -cut
sparsiﬁer necessarily takes linear Ω( n + m ) queries.
Therefore, the key to achieving such a result crucially depends upon being able to efﬁciently construct these weaker ( (cid:15), δ ) -cut
sparsiﬁers with a small additive error δ , which is the backbone of our sublinear time result.
For simplicity, we begin by presenting our result for unweighted graphs, and then extend it to weighted graphs in subsection D.2.
Theorem 11.
Given any unweighted graph G =
( V, E ) with n vertices and m =
αn 4 / 3 edges accessible via queries in the general graph model, and any parameter 0 < (cid:15) ≤ 1 / 2 , there exists an algorithm that
(a) given a
φ
-approximate
hierarchical
clustering
oracle, ﬁnds
a (1 + (cid:15) )
φ -
approximate hierarchical clustering of G with high probability using f ( n, α, (cid:15) ) queries,
and (b given any arbitrarily small parameter 0 < τ <
/ 2 , ﬁnds an O ( (cid:112)
τ −
1 log n ) -approximate hierarchical clustering of G with high probability using (cid:101)
O ( f ( n, α, (cid:15) )
+ n 1+ τ ) time and queries, where
[Equation]
Note that unlike the sublinear space and communication settings, we cannot directly give a sublinear time (1 + (cid:15) ) φ -approximation guarantee here; even though the rest of our algorithm (that constructs the ( (cid:15), δ ) -cut sparsiﬁer) has a sublinear time and query complexity, the running time of the φ -approximate hierarchical clustering oracle to which we are given access can be arbitrarily large 8 .
Therefore in this setting, we give a two-part result - the ﬁrst is a sublinear query , (1 + (cid:15) )
φ -approximation result, and the second is a sublinear time and query , O ( √ log n ) -approximation result, which follows from a speciﬁc sublinear time implementation of a φ -approximate hierarchical clustering oracle with φ
=
O ( √ log n ) .
The query (and time) complexity in the above result is linear in the number of edges for sparse graphs with fewer than n 3 √ n edges, decays as (cid:101)
O
( α − 2 n 4 / 3 ) for moderately dense graphs when the number of edges is in the range n 3 √ n and n √ n , and is (cid:101) O ( n ) for dense graphs with more than n √ n edges.
As we will see in our lower bounds, this complexity is essentially optimal for achieving a (cid:101) O (1) -approximation in each of these three regimes.
Proof of Theorem 11.
The proof of both parts of Theorem 11 relies on ( (cid:15), δ ) -cut
sparsiﬁers, which we show in Theorem 12, can be constructed with high probability in (cid:101) O
( (cid:15) − 2 δ − 1 n ) time and queries.
Assuming this construction, the sublinear query, (1 + (cid:15) ) φ -approximation claim (Theorem 11 ( a )) is relatively straightforward to see: we ﬁrst determine the number of edges m = αn 4 / 3 in the input graph by performing n degree queries.
If the graph is sufﬁciently sparse ( m ≤ n 4 / 3 ), then we simply read the entire graph, which takes O ( m ) neighbour queries.
If not, then the lower bound established in Lemma 2 implies that the cost OPT of any hierarchical
, v ∈ V , clustering in the input graph is at least α 2 n 5 / 3 .
As a consequence, the additive error δ = (cid:15) OPT /n 2 ≥ (cid:15)α 2 n − 1 /we can tolerate in our ( (cid:15), δ ) -sparsiﬁer is also relatively large.
Such a sparsiﬁer can then be constructed with high probability in (cid:101) O
( (cid:15) − 3 max { α − 2 n 4 / 3 , n } ) time and queries.
The rest of the proof follows directly by Lemma 1, since the φ -approximate hierarchical clustering oracle uses only the ( (cid:15), δ ) -cut sparsiﬁer as input, and therefore, makes no additional queries to the input graph.
To prove the sublinear time, O ( (cid:112) τ − 1 log n ) -approximation claim (Theorem 11 ( b )) where τ ∈ (0 , 1 / 2) is any arbitrarily small parameter, we complement the above proof with an instantiation of a sublinear time, φ -approximate hierarchical clustering oracle with φ
=
O ( (cid:112)
τ − 1 log n ) .
As discussed in Corollary 1, this essentially reduces to a sublinear time, O ( (cid:112)
τ − 1 log n ) -approximation algorithm for balanced cuts (also called balanced separators in the literature).
However, the algorithm of [4] cannot be used here due to its quadratic running time.
Therefore, we instead refer to another result of [40] that achieves O ( (cid:112)
τ − 1 log n ) -approximation for balanced separators by reducing this problem to (cid:101) O
( n τ ) single-commodity max-ﬂow computations for any given τ ∈ (0 , 1 / 2) .
While [40] could only achieve this in (cid:101) O ( m + n 3 / 2+ τ ) time, the bottleneck being the (cid:101) O ( m 3 / 2 ) time ﬂow-computation algorithm of [27] (with the sparsiﬁcation result of [7] being used to improve this complexity to (cid:101) O
( n 3 / 2 ) )
, we can leverage a very recent breakthrough [14] that gives an (cid:101) O ( m 1+ o (1) )
algorithm for exact single-commodity max-ﬂows.
This improves the running time of the algorithm of [40] from (cid:101)
O ( m + n 3 / 2+ τ ) to (cid:101)
O ( m + n 1+ τ ) without any loss in the approximation factor.
Since our ( (cid:15), δ ) -cut sparsiﬁer (cid:101) G is very sparse with f ( n, α, (cid:15) ) edges, we can ﬁnd a O ( (cid:112)
τ − 1 log n ) -approximate balanced separator in (cid:101) G in sublinear (cid:101) O
( f ( n, α, (cid:15) ) + n 1+ τ ) -time, for any given τ ∈ (0 , 1 / 2) .
Although we use this subroutine repeatedly (at each split of the graph until we are left with singleton vertices), observe that at any level of the hierarchical clustering tree, the splits at that level together form a disjoint partition of (cid:101) G .
Now let the set of all internal nodes (splits) of the resultant hierarchical clustering tree at level i ∈
[ d ] be S i , where d is the depth of the tree.
Furthermore, for any internal node S in this tree, let m S be the number of edges in the subgraph (cid:101) G
[ S ] induced by the set of vertices S .
Therefore, the running time of the recursive sparsest cut algorithm on (cid:101) G with the aforementioned O ( (cid:112) τ − 1 log n ) -approximate oracle for balanced separators is given by
[Equation]
Finally, observe that since the splits in the tree are balanced, i.e. a split S → ( S (cid:96) , S r ) is such that min {| S (cid:96) | , | S r |} ≥ | S | / 3 , the depth of this hierarchical clustering tree produced d = O (log n ) , which gives the total running time of the recursive sparsest cut algorithm on (cid:101) G as (cid:101)
O
( f ( n, α, (cid:15) ) + n 1+ τ ) , proving our sublinear time claim.
We shall now present our sublinear time construction of ( (cid:15), δ ) -cut sparsiﬁers for unweighted graphs.
D.1 A Sublinear Time ( (cid:15), δ ) -Cut Sparsiﬁcation Algorithm for Unweighted Graphs
Theorem 12.
There exists an algorithm that given a query access to an unweighted graph G =
( V, E ) and parameters 0 < δ ≤ 1 , 0 < (cid:15) ≤ 1 / 2 , can ﬁnd a ( (cid:15), δ ) -cut
sparsiﬁer of G with high probability in (cid:101) O ( nδ − 1 (cid:15) − 2 ) time and queries.
Our ( (cid:15), δ ) -cut
sparsiﬁer construction broadly builds upon ideas developed in [32] for probabilistic spectral sparsiﬁers.
At a high level, to achieve an additive error of δ , we embed a constant-degree expander G x =
( V, E x ) with edge weights δ ≤ 1 on all n vertices in the input graph G =
( V, E ) .
This trick gives a tight (and very friendly) bound on the effective resistance of every edge in the resultant composite graph H
=
( V, E ∪ E x , w ) , which is a weighted graph with edge set consisting of the union of edges E in the input graph G , each having weight 1 , and edges E x in the constant-degree expander G x , each with weight δ (edges in E ∩ E
x are assumed to be two parallel edges, one with weight 1 and the other with weight δ ).
This is useful as it allows for efﬁcient sparsiﬁcation of this composite graph using effective resistance sampling of [41], with the sources of error being the usual multiplicative error due to sparsiﬁcation itself, and a small additive error due to the few extra edges introduced by the expander.
There are several well known (cid:101) O ( n ) time constructions of constant degree expanders, for example, a random d -regular graph is an expander with high probability [23].
This roughly outlines the sparsiﬁcation algorithm and proof of the above theorem.
A similar idea was used in [32], with the key difference being that they embed an unweighted constant degree expander in a random δn subset of vertices.
Since the set of vertices where the expander is embedded is random, it is easy to see why this gives a small additive error in expectation for any ﬁxed cut, but could be very large for some cuts in the graph.
Our construction on the other hand provides a sparsiﬁer with stronger guarantees that hold for every cut.
As outlined above, we start by showing that the effective resistance of any edge ( u, v )
∈ E ∪ E x is tightly bounded.
Lemma 3.
Given parameter δ ∈ (0 , 1) , and a composite graph H =
( V, E ∪ E x , w ) where G =
( V, E ) is an arbitrary input graph with edges of weight 1 , and G x = ( V, E x ) is a constant-degree expander graph with edges of weight δ , then for any edge ( u, v )
∈ E ∪ E x , we have
[Equation]
where R H ( u, v ) is the effective resistance of edge ( u, v ) in graph H , and for any vertex u ∈ V , d G ( u ) , d G x ( u ) are the degrees of vertex u in graphs G and G x , respectively.
Proof.
For any edge ( u, v ) ∈ E ∪ E x , let us assume without loss of generality that k :=
d H ( u ) ≤
d H ( v ) .
The proof of our upper bound on the effective resistance R H ( u, v ) relies on a basic property of expander graphs: there are Ω( k ) edge-disjoint paths, each of length at most O (log n ) connecting u to v .
Since each edge on these paths has weight at least δ , by Rayleigh’s monotonicity principle, the effective resistance between ( u, v ) can be no more than a graph containing exactly k edge-disjoint paths, each of length O (log n ) with each edge on this path having resistance 1 /δ , which gives us that
[Equation]
To show this many edge-disjoint, short paths property of expanders, we consider two possibilities: either k < n/ log n , in which case let { u i } ki =1 be the neighbours of u , and let { v i } ki =1 be a set of k neighbours of v , chosen and ordered arbitrarily.
Then the well known multicommodity ﬂow result of [24] already guarantees the existence of these short edge-disjoint paths connecting every u i to v i .
In the case that k ≥ n/ log n , consider the (unweighted) subgraph H x = ( V, E x ∪ E u ∪ E v ) induced by the expander edges E x and edges E u , E v ⊆
E incident on vertices u, v in G , respectively.
We ﬁrst claim that the min u - v cut in H x contains at least k/ 2 edges; let ( S, S ) be the min u - v cut, with | S | ≤ n/ 2 and s ∈ { u, v } being the vertex in S .
Furthermore, let k s ≥ k be the number of neighbours of s , with k (cid:48)
s ≤ k s of them being contained in S .
Now there are two possibilities, either ( a ) k (cid:48)
s <
k s / 2 in which case the cut ( S, S ) already contains the k s − k (cid:48)
s ≥ k s / 2 edges connecting s to its remaining neighbours in S , or ( b ) k (cid:48)
s ≥ k s / 2 in which case ( S, S ) must necessarily cut at least | S | ≥ k s / 2 edges in E x due to expansion.
Therefore, by the (integral) min-cut max-ﬂow theorem, there are at least k/ 2 edge-disjoint paths from u to v .
Moreover, we claim that at least half of these paths must be short, speciﬁcally, of length O (log n ) .
To see this, observe that graph H x contains just Cn edges for some constant C , which follows from that fact that u , v each can have at most n neighbours and G x is a constant degree expander.
Now let the integral ﬂow which is of size f ≥ k/ 2 ≥ n/ (2 log n ) be routed along arbitrary edge-disjoint paths P 1 , . .
P f .
It is easy to see why at least f/ 2 of these paths must be of length at most 2 C log n , because otherwise, the number of edges contained in just the long paths alone would exceed ( f/ 2) · (2 C log n ) >
Cn , the total number of edges in H x which is a contradiction.
Therefore, there are at least k/ 4 edge disjoint paths between u, v in H x ⊆ H , each of length
O (log n ) .
Now to prove the lower bound on the effective resistance of any edge ( u, v )
∈ E ∪ E x , we add an extra vertex w and replace the edge ( u, v ) with two edges ( u, w ) and ( w, v ) , each with weight/capacitance 2 w uv (doing this twice if edge (
u, v ) ∈
E ∩ E x , once for the edge ( u, v ) with w uv = 1 and again for the edge with w uv = δ ).
We then apply Rayleigh’s monotonicity principle by shorting all vertices other than u, v in the graph, which gives us that
[Equation]
where the ﬁnal inequality follows from the fact that w uv < min s ∈{ u,v } { d G ( s ) + δd G x ( s ) } , which proves our claimed lower bound.
This tight bound of the order (
[Equation]
)) − 1 on the effective resistances directly allows us to apply the effective resistance sampling scheme of [41] outlined in Algorithm 1 to construct a spectral sparsiﬁer of H , which is even stronger than the simple cut sparsiﬁer we require.
The following theorem then establishes the properties of the resulting sparsiﬁer (cid:101) G .
Theorem 13 (Theorem 1 + Corollary 6 of [41]) .
Given any weighted graph G =
( V, E, w ) on n vertices with Laplacian L , let Z e be numbers satisfying Z e ≥ R e /α for some α ≥ 1 and (cid:80)
e ∈ E w e Z e ≤ (cid:80) e ∈
E w e R e .
Then given any parameter 0 ≤ (cid:15) ≤ 1 , the subroutine Sparsify ( G, p, q ) with sampling probabilities p e =
Laplacian
L high probability, satisﬁes
[Equation]
From the effective resistance bound established in Lemma 3, it is easy to see that sampling edges with parameter Z
[Equation]
x
[Equation] [Equation]
2 satisﬁes the condition in Theorem 13 with α = O (log n/δ ) for the graph H =
( V, E ∪ E x , w ) .
Given this choice of parameters Z e , it is easy to see that (cid:80) e ∈ E ∪ E x w e Z e = n/ 2 , which gives us that the sampling probability for any edge e ∈ E ∪ E x is given by
[Equation]
for which there is a very simple sublinear time rejection sampling scheme given query access to G : sample a uniformly at random vertex u ∈ V , and toss a coin with bias d G ( u ) /
( d G ( u ) + δd G x ( u ))
(degree query).
If heads, sample a uniformly at random edge incident on u in G (neighbour query).
Otherwise, sample a uniformly at random edge incident on u in G x .
The complete algorithm is given below.
Algorithm 2 ( (cid:15), δ ) -Sparsify
Construct a constant degree expander G x = ( V, E x ) .
Let H =
( V, E ∪ E x , w ) be the composite weighted graph with edge weights w e = 1 for e ∈ E and w e = δ for e ∈ E x .
Set (cid:15) (cid:48)
= (cid:15) (cid:112) δ/ ( C 1 log n ) for a sufﬁciently large constant C 1 , repetitions q = C 2 n log n/ ( (cid:15) (cid:48) )
2 for a sufﬁciently large constant C 2 .
Set sampling probabilities p , where for each edge e ∈ E ∪ E x , p e is as deﬁned in Eq.
Sparsiﬁer (cid:101) G =
Sparsify ( H, p, q )
It is easy to see that the above algorithm produces a graph (cid:101) G with O ( n log n/ ( (cid:15) (cid:48) ) 2 )
=
O ( n log 2 n/ ( δ(cid:15) 2 )) edges, and runs in time O ( n log n/ ( (cid:15) (cid:48) ) 2 )
=
O ( n log 2 n/ ( δ(cid:15) 2 )) .
We shall now prove that (cid:101) G is an ( (cid:15), δ ) -sparsiﬁer of G as claimed in Theorem 12.
Proof of Theorem 12.
We start by observing that Theorem 13, by restricting to vectors x ∈ { 0 , 1 } n (corresponding to partitions of V ) and choice of (cid:15) (cid:48) =
(cid:15) (cid:112) δ/ ( C 1 log n ) with a sufﬁciently large constant C 1 , implies that with high probability, the sparsiﬁer (cid:101) G produced by Algorithm 2 is such that
[Equation]
Now observe that for any cut ( S, S ) in the composite graph H ,
[Equation]
where the ﬁnal inequality follows by observing that the weight of any edge e ∈ E x is δ , and since the degree of any vertex in G x is Θ(1) , the number of edges in G x that cross any cut ( S, S ) is Θ(min {| S | , | S |} ) .
Combining the above bounds with Eq.
(6) gives us the ( (cid:15), δ ) -cut sparsiﬁcation guarantees for (cid:101) G as
[Equation]
D.2 Extension to Weighted Graphs
In this section, we extend our sublinear time results to weighted graphs G =
( V, E, w ) , where edges e ∈ E take weights 1 ≤ w e ≤ W , where W is an upper bound on the maximum edge weight.
Since there is no universally accepted query model for weighted graphs, we propose the following generalization where the algorithm can make ( a ) Degree queries: given v ∈ V , returns the degree d G ( v ) , and ( b ) Neighbour queries: given v ∈ V , i ≤ d G ( v ) , returns both the i th neighbour of v and the connecting edge weight, with the additional constraint that the neighbours are ordered by increasing edge weights (neighbours connected by equal weight edges are ordered arbitrarily).
Note that this generalization reduces to the general graph model when all edge weights are equal.
The following theorem describes our upper bound in this more general setting.
Theorem 14.
Let G =
( V, E, w ) be any weighted graph with n vertices and edge weights taking values in a bounded range [1 , W ] .
Given any parameter 0 < (cid:15) ≤ 1 / 3 , let m i
= α i n 4 / 3 be the number of edges in G with weights in the interval [(1 + (cid:15) )
i − 1 , (1 + (cid:15) ) i ) .
Then given query access to G , there exists an algorithm that (a) given a φ -approximate hierarchical oracle, ﬁnds a (1 + (cid:15) ) φ - clustering of G with high probability using (cid:101) O (cid:0) (
− 1 log W
· ( n + max i f n, α i ,
)) (cid:1) queries, and (b) given any arbitrarily parameter
<
<
O cal
G probability using n
[Table]
+
[Table]
+ (
[Table] [Table]
W ) · ( n + max i
[Table] [Table]
time and queries, where
[Equation]
Before discussing this result, one might naturally ask whether this stricter requirement of ordering neighbours by weight is really necessary or whether it is possible to achieve a similar result for arbitrary or even random orderings.
Towards the end of this section, we will show that this is unfortunately necessary; without the ordering constraint, no non-trivial approximation for hierarchical clustering is possible unless a constant fraction of the edges in the graph are queried, and this holds even if we were to additionally allow pair queries: given u, v ∈ V , return whether ( u, v ) ∈ E (and edge weight w uv if afﬁrmative).
At a high level, our sublinear time upper bound for weighted graphs is morally the same as that achieved in the unweighted case, with a O ( (cid:15) − 1 log W ) hit to query and time complexity.
Algorithmically, we build upon the ideas developed for the unweighted case.
We begin by partitioning the edge set E of the input graph G =
( V, E, w ) into weight classes, where the i th weight class consists of all edges E i with weights in the interval [(1+ (cid:15) )
i − 1 , (1+ (cid:15) ) i ) .
By construction, there are log (1+ (cid:15) )
W ≤ 2 (cid:15) − 1 log W weight classes in total, with the edge sets { E i } log (1+ (cid:15) )
W i = being a disjoint partition of E .
We then approximately sparsify each unweighted subgraph G (cid:48) i
=
( V, E i ) using our sublinear time ( (cid:15), δ ) -Sparsify routine outlined in the previous section, and scale up all the edge weights of the resultant sparsiﬁer (cid:101) G (cid:48)
i by the maximum edge weight W i
=
(1 + (cid:15) )
i of that class.
Since for every weight class i , the weights of all the edges E i in that class are within a (1 + (cid:15) ) factor of each other, the resultant scaled sparsiﬁer (cid:101) G i is a good approximate sparsiﬁer for the weighted subgraph G i
= ( V, E i , w ) .
Finally, since the input graph G = ( V, E, w ) is partitioned into subgraphs G i =
( V, E i , w ) , the sum of the scaled sparsiﬁers (cid:101) G i is a good sparsiﬁer for the input graph.
Given this sparsiﬁer, the proof of (both claims of) Theorem 14 then follows identically as that of Theorem 11.
An important point to note here is that we do not need to explicitly construct the subgraphs G (cid:48)
i corresponding to each of the weight classes i ∈
[log 1+ (cid:15) W ] (which would naively take O ( m ) time) as our ( (cid:15), δ ) -sparsiﬁcation subroutine only requires query access to G (cid:48)
i .
This is easy to do in (cid:101) O ( n ) time for any weight class assuming the edges incident on vertices are sorted by weights.
For any weight class i and any vertex v , the set of edges incident on v in subgraph G i lie in the range of indices [ x i − 1 ( v ) , x i ( v ) − 1] where for any weight class j ∈ [log 1+ (cid:15) W ] , vertex u ∈ V , x j ( u ) is the ﬁrst occurrence of an edge incident on u with weight at least (1+ (cid:15) )
j .
Both indices x i ( v ) , x i − 1 ( v ) can be found in O (log n ) time and queries using binary search; the degree d G (cid:48)
i ( v ) =
x i ( v ) − x i − 1 ( v ) , and the j th neighbour of v in G (cid:48)
i is simply the ( x i − 1 ( v ) + j − 1) th neighbour of v in G .
Therefore, the total time and query complexity of setting up query access to G (cid:48)
i is O ( n log n ) .
We now present a formal proof of Theorem 14, which is achieved by Algorithm 3.
Proof of Theorem 14.
As with the analysis for unweighted graphs, we begin by establishing a lower bound on the cost of any hierarchical clustering for weighted graphs.
Given any weighted graph G =
( V, E, w ) and a parameter 0 < (cid:15) ≤ 1 / 3 , we begin by partitioning the edge set into weight classes, where the i th weight class consists of all edges E i with weights in the interval [(1+ (cid:15) )
i − 1 , (1+ (cid:15) ) i ) .
Therefore, we have that the clustering
Algorithm
Input .
Weighted graph G =
( V, E, w ) , parameter 0 < (cid:15) ≤ 1 / 3 .
Output .
Sparsiﬁer (cid:101) G =
( V, (cid:101) E, (cid:101) w ) .
For every vertex v ∈ V , x 0 ( v ) =
for i = 1 , . . .
, log (1+ (cid:15) )
W do
For every vertex v ∈ V , binary search for x i ( v ) , the ﬁrst occurrence of an edge incident on v with weight at least (1 + (cid:15) )
i .
Establish query access to G (cid:48)
i ← ( V, E i ) , where E i :=
{ e ∈ E : (1 + (cid:15) )
i − 1 ≤ w e <
(1 + (cid:15) )
i } using { ( x i − 1 ( v ) , x i ( v )) } v ∈ V .
Let | E i |
= m i = α
i n 4 / 3 .
if α i ≤ 1 then
Read G i
=
( V, E i , w ) entirely, and let this graph be (cid:101) G i .
else
Set additive error δ i ← (cid:15) · min { α 2 i /n 1 / 3 , 1 } , and W i =
(1 + (cid:15) ) i .
(cid:101) G (cid:48)
i ← ( (cid:15), δ i ) -Sparsify
( G (cid:48) i ) , where (cid:101) G (cid:48) i
=
( V, (cid:101) E
i , (cid:101) w (cid:48) i )
Construct sparsiﬁer (cid:101) G i =
( V, (cid:101) E
i , (cid:101) w i =
W i · (cid:101) w (cid:48) i ) with edge weights scaled by W i .
end if end for (cid:101)
G = (cid:101) G 1 + . . .
+ (cid:101) G log (1+ (cid:15) )
W cost of
[Table]
is cost G
[Equation]
where the ﬁrst inequality follows from the fact that the clustering cost function is monotone in edge weights, and every edge in G i
= ( V, E i , w ) has weight at least (1 + (cid:15) )
i − 1 .
We now claim that for every weight class i , the scaled sparsiﬁer (cid:101) G i is a ( O ( (cid:15) ) ,
O ( W i δ i )) -sparsiﬁer of the subgraph G i
=
( V, E i , w ) .
To see the lower bound, observe that for any cut ( S, S )
[Equation]
where the ﬁnal inequality follows from the fact that every edge in G i has weight at most W i .
To see the upper bound, observe that for any cut ( S, S ) ,
[Equation]
where the second inequality follows from the fact that every edge in G i has weight at least W i /
(1 + (cid:15) ) .
Since we have that the edge set E = E 1 + . . .
+ E log (1+ (cid:15) )
W , this directly gives us that the scaled sparsiﬁer returned (cid:101) G = (cid:101) G 1 + . . .
+ (cid:101) G log (1+ (cid:15) )
W is a (cid:0) O ( (cid:15) ) , O ( (cid:80)
i W i δ i ) (cid:1) -cut sparsiﬁer of G , where for any cut ( S, S ) ,
[Equation]
S
[Equation]
By choice of each δ i ≤ (cid:15) | E i | 2 /n 3 , we further have that
[Equation]
∀ hierarchies T .
Given this guarantee, the bound on the hierarchical clustering cost claimed in Theorem 14 ( a ) follows by a straightforward application of Lemma 1.
To complete this proof, the last thing we need to verify is the time and query complexity of Algorithm 3.
We shall break down the complexity of this algorithm across the weight classes i ∈ [log 1+ (cid:15) W ] .
As described earlier, for any weight class i ,
Let | E i |
=
α i n 4 / 3 be the number of edges in this subgraph.
In the case α i ≤ 1 , this subgraph is sufﬁciently sparse and G i is read entirely which takes O
( α i n 4 / 3 ) time and queries.
Otherwise ( α i > 1) , in which case it is sparsiﬁed in (cid:101) O ( (cid:15) − 3 max { α − 2 i n 4 / 3 , n } ) time and queries as established in Theorem 12.
Therefore, the total complexity of processing a weight class i is (cid:101) O
( n + f ( n, α i , (cid:15) )) , where f ( n, α, (cid:15) )
= (cid:101)
O ( αn 4 / 3 ) if α ≤ 1 , and (cid:101) O
( (cid:15) − 3 max { α − 2 n 4 / 3 , n } ) otherwise.
Since there are O ( (cid:15) − 1 log W ) weight classes in total, Algorithm 3 runs in time (cid:101) O
( (cid:15) − 1 n log W + (cid:80) i f ( n, α i , (cid:15) )) ≤ (cid:101) O
(( (cid:15) − 1 log W ) · ( n + max i f ( n, α i , (cid:15) )) .
Lastly, for any given parameter τ ∈ (0 , 1 / 2) , the sublinear time, O ( (cid:112) τ − 1 log n ) -approximation claim (Theorem 14 ( b )) follows by the same φ -approximate hierarchical clustering oracle construction described in the proof of Theorem 11 combined with the fact that our ( (cid:15), δ ) -cut sparsiﬁer for the weighted graph G now contains (cid:101)
O (( (cid:15) − 1 log W ) · max i f ( n, α i , (cid:15) )) edges.
D.2.1 Necessity of Ordering Neighbours by Weight
We conclude this section by showing that the assumption that the adjacency list of each vertex u orders the neighbours of u by weight, is in fact necessary.
Otherwise, no non-trivial approximation for hierarchical clustering is possible even when one is allowed to query a constant fraction of edges in the graph.
We shall naturally consider only sufﬁciently dense graphs with Ω( n 4 / 3 ) edges.
While this isn’t strictly necessary for our example, our upper (and lower) bounds allow us to simply read the entire graph otherwise, rendering the sparse regime uninteresting.
While this is straightforward to see when the upper limit on edge weights W = poly ( n ) is large, we can even show this for a relatively small W = n 1+ (cid:15) for any constant (cid:15) > 0 .
The example is as follows: consider an input graph G =
( V, E 1 ∪ E 2 , w ) with n vertices, and an edge set of size m consisting of the union of two Erd˝os-Renyi random graphs, where E 1 ∼ G n,p for any p > n − 2 / 3 with all edges having weight 1 and E 2 ∼ G n, 1 / 3 n with all edges having weight W = n 1+ (cid:15) for some constant (cid:15) > 0 .
We can assume that edges in both E 1 and E 2 are given the larger weight.
We shall ﬁrst establish an upper bound on the cost of the optimal hierarchical clustering in G , which we claim is at most nm + O ( nW log n ) .
To prove this, we shall use the fact that with probability at least 1 − 1 /n , ( a ) the subgraph G 2 = ( V, E 2 ) is a union of connected components, each either a tree or a unicyclic component, and ( b ) the degree of every vertex in G 2 is at most 3 .
The former is well known in the random graph literature, [21] and the latter follows from Bernstein’s concentration inequality.
Therefore, hierarchical clustering that ﬁrst separates the different connected components of G 2 , following which each connected component is partitioned recursively using a balanced sparsest cut, i.e. the sparsest cut with a constant fraction of the remaining vertices on either side of the cut, will achieve a cost of at most O ( nW log n ) .
The remaining edges in E 1 , regardless of how they are arranged can cumulatively add no more than n | E 1 | to the cost of this hierarchical clustering.
Now consider any (randomized) algorithm that performs at most 2 m/ 9 neighbour and pair queries in total, and let T be the hierarchical clustering returned by this algorithm.
Consider a balanced cut ( S, S ) in this tree, i.e. an internal node S with min {| S | , | S |} ≥ n/ 3 .
Since the number of queries made is bounded by 2 m/ 9 , there necessarily are at least 2 n 2 / 9 − 2 m/ 9 ≥ n 2 / 9 unqueried edge pairs from the cut ( S, S ) .
Furthermore, there are at least m − 2 m/ 9 = 7 m/ 9 unqueried edges in G .
For every unqueried edge, there is at least a constant ( n 2 / 9) / (cid:0) n 2 (cid:1) ≥ 2 / 9 probability that it realized into an edge slot from the cut ( S, ¯ S ) , and then at least a 1 / (3 n ) marginal probability that it came from E 2 .
Therefore in expectation, there are at least (7 m/ 9) · (2 / 9) · (1 / 3 n ) ≥ m/ (18 n ) edges from G 2 , each having weight W that go across the cut ( S, ¯ S ) .
Since | S | ≥ n/ 3 , the contribution of each heavy edge to the cost of T is at least n/ 3 · W , and therefore, the expected cost of T is at least ( m/ 18 n ) · ( n/ 3) · W = mW/ 54 due to these heavy edges alone.
Note that this argument holds even if the neighbours of every vertex are ordered randomly.
Now by comparing the cost of the optimal clustering, which is at most nm + O ( nW log n ) , to the expected cost of the hierarchical clustering produced by an algorithm that makes at most 2 m/ 9 queries, which is Ω( mW ) , it is easy to see that the approximation ratio in expectation is Ω( n (cid:15) ) when W = n 1+ (cid:15) and m ≥ n 1+ (cid:15) log n .
E Sublinear Communication Algorithms under MPC Model
Finally, we consider the bounded communication setting in the massively parallel computation (MPC) model of [6], where the edge set of the input graph is partitioned across several machines which are inter-connected via a communication network.
The communication proceeds in synchronous rounds.
During each round of communication, any machine can send any information to an arbitrary subset of other machines.
However, the total number of bits a machine is allowed to send or receive is limited by the memory of the machine.
Between two successive rounds, each machine is allowed to perform an arbitrary computation over their inputs and any other bits received in the previous rounds.
At the end, a machine designated as the coordinator is required to output a solution based on its initial input and the communication it receives.
The objective is to study the trade-off between the number of rounds and communication required by each machine, or as alternatively stated, minimize the number of rounds given a ﬁxed communication budget for each machine.
Note that the communication budget of each machine is same as the memory given to the machine.
E.1 A 2 -Round (cid:101) O ( n ) Communication Algorithm
We ﬁrst give a 2 round algorithm that requires (cid:101) O
( n ) communication per machine.
The following is the main result of this section.
Theorem 15.
There exists a randomized MPC algorithm that, given a weighted graph G =
( V, E, w ) over n vertices where edge weights are O ( poly ( n )) , and a φ -approximate hierarchical clustering oracle, can compute with high probability a (1 + (cid:15) )
φ -approximate hierarchical of G in 2 rounds using (cid:101) O
( (cid:15) − 2 n ) communication per machine and access to public randomness.
In order to prove this theorem we will utilize a result from [1] for constructing ( (cid:15), 0) -cut
sparsiﬁers using linear graph sketches .
Given L : R d → R d (cid:48) and x ∈ R d , we say that L ( x ) is a sketch of x .
In order to sketch a graph, we represent each vertex in the graph using a (cid:0) n 2 (cid:1) -dimensional vector and then compute a sketch for each vertex.
Let the vertices in the graph be indexed as 1 , · · · , n .
For each i ∈ [ n ] , we will deﬁne a vector x ( i ) ∈ {− 1 , 0 , 1 } ( n 2 ) as follows: we ﬁrst compute a matrix M of size n × n with
[Equation]
i
[Equation]
otherwise .
The vector x i is then obtained by ﬂattening M after removing all the diagonal entries.
The following theorem summarizes the result of [1] for computing cut-sparsiﬁers using linear sketches.
Theorem 16 ([1]) .
For any (cid:15) > 0 , there exists a (random) linear function L : R ( n 2 ) → R O ( (cid:15) − 2 poly log n ) such that, given any graph G =
( V, E, w ) over n vertices with edge weights that are O ( poly ( n )) , a ( (cid:15), 0) -cut
sparsiﬁer can be constructed with high probability using the sketches L ( x (1) ) , · · · , L ( x ( n ) ) of each vertex.
Moreover, each of these sketches can by computed using O ( (cid:15) − 2 poly log n ) space
given
access to fully independent random hash functions.
Note that an important property of this sketch is that it is linear, which means that (partial) independently computed sketches L ( x ( i, 1) ) , · · · , L ( x ( i,t ) ) for a vertex i can be added together to get a sketch L ( x ( i ) ) =
L ( x ( i, 1) ) + · · · + L ( x ( i,t ) ) .
We will now use this result for computing a cut-sparsiﬁer using 2 rounds of MPC computation.
We will use the same construction of linear sketches for each vertex as in this result.
2 -round MPC Algorithm:
Input: Parameter (cid:15) ∈ (0 , 1 / 2] , graph G
=
( V, E, w ) such that edges are partitioned over k machines.
Let each machine be responsible for constructing the sketch for n/k (arbitrarily chosen) vertices.
Divide the weights into O (log n ) weight classes similar to [1].
Each machine locally constructs a (random) linear sketch of size O ( (cid:15) − 2 poly log n ) for each vertex and weight class.
Each machine computes the sketches according to the same function L using Theorem 16 by computing the same random hash functions through public randomness.
Round 1: Each machine sends its local linear sketches of a vertex to the machine that is responsible for this vertex.
For each weight class, each machine constructs the linear sketches for each of its responsible vertices by adding the corresponding partial linear sketches.
Round 2: Each machine sends its n/k linear sketches to the coordinator.
The coordinator
constructs ψapproximate
The above pseudo-code outlines the 2 -round algorithm for hierarchical clustering in the MPC model.
Here, we arbitrarily partition vertices into k sets of size ( n/k ) each, and designate each machine to be responsible for constructing the sketch for n/k (arbitrarily chosen) vertices.
Since, the edges are partitioned over k machines, a given machine might not have all the edges incident on a vertex.
Hence, in the ﬁrst round each machine will locally construct a linear sketch for each vertex based on its edges.
Note that each machine will construct linear sketches using the same function L by sampling the same random hash functions.
Then each machine sends their local linear sketches for each vertex to the responsible machines.
Each machine can send (cid:101) O ( (cid:15) − 2 n ) bits in total as the sketch of each vertex is of size O ( (cid:15) − 2 poly log n ) .
Moreover, each machine can receive (cid:101) O
( (cid:15) − 2 n ) bits as it can receive at most (cid:101)
O ( (cid:15) − 2 n/k ) bits from k − 1 other machines.
Each machine then constructs the linear sketches for its vertices by adding the corresponding sketches.
Note that these sketches are valid due to linearity and the fact that the random hash functions are shared across all machines.
Finally, each machine will send its sketches to the coordinator.
The coordinator will then compute a cut sparsiﬁer using these sketches and run a φ -approximate hierarchical clustering algorithm over the sparsiﬁed graph.
Using Theorem 16 and
Lemma 1, we can easily argue that the coordinator’s hierarchical clustering will be (1 + O ( (cid:15) )) · φ -approximate with high probability.
E.2 A 1 -Round (cid:101) O ( n 4 / 3 ) Communication Algorithm
We next consider the possibility of computing a good hierarchical clustering in just a single round in the MPC model.
However, as we will show in Section G, computing in one round requires Ω( n 4 / 3 ) communication (and hence, machine memory) requirement, even for unweighted graphs.
In this setting, give a 1 -round (cid:101) O ( n 4 / 3 ) communication MPC algorithm for hierarchical clustering of unweighted graphs assuming knowledge of the number of edges in the input graph, and the number of machines being bounded by m/n 4 / 3 .
Theorem 17.
There exists a randomized MPC algorithm that, given an unweighted graph G =
( V, E ) over n vertices and m edges, a parameter 0 < (cid:15) ≤ 1 / 2 , and a φ -approximate hierarchical clustering oracle, can compute with high probability a (1+ (cid:15) )
φ -approximate hierarchical clustering of G in 1 round using (cid:101) O
( (cid:15) − 2 n 4 / 3 ) communication per machine and k ≤ m/n 4 / 3 machines with access to public randomness.
The following pseudo-code outlines the 1 -round algorithm.
Input: Parameter (cid:15) ∈ (0 , 1 / 2] , graph G
=
( V, E ) with m edges such that edges are partitioned over k ≤ m/n 4 / 3 machines each with memory (cid:101) Ω( (cid:15) − 2 n 4 / 3 ) .
If m = βn 4 / 3 for β ≥ n 1 / 3 then (a) Each machine samples its each of its local edges independently with probability p =
C ( (cid:15) 2 β ) − 1 log n for some sufﬁciently large constant C and sends all the sampled edges with weight 1 /p to the coordinator.
(b) Let δ := m 2 /n 3 = β 2 /n 1 / 3 , and let
H =
( V, E h , w h ) be the weighted graph induced by the sampled edges received from all machines.
The coordinator constructs a constant degree expander G x = ( V, E x , w x ) with all edges having weight (cid:15)δ , and embeds this weighted expander in H .
Let (cid:101) G =
( V, E h ∪ E x , w h + w x ) be the resultant composite graph.
(c)
The coordinator runs a φ -approximate hierarchical clustering on (cid:101) G and returns the answer.
Else if m = αn for α < n 2 / 3 then (a) Each machine computes a (random) linear sketch of size O ( (cid:15)
− 2 poly log n ) for all vertices using the local edges.
Each machine computes the sketches according to the same function L using Theorem 16 by computing the same random hash functions through public randomness.
(b) Each machine sends its local linear sketches to the coordinator.
(c) The coordinator adds the partial linear sketches corresponding to each vertex to get one linear sketch per vertex.
It then runs the algorithm of [1] for computing a ( (cid:15), 0) -cut sparsiﬁer.
Finally, it runs a φ -
approximate hierarchical clustering over the cut sparsiﬁer and returns the answer.
The execution of the above algorithm is divided into two cases based on the number of edges in the graph.
We analyze these two cases separately below.
Analysis for Case 1:
We ﬁrst observe that in this case, since the total number of edges in the graph G that is distributed across all machines is βn 4 / 3 , and each machine samples its local edges with probability p =
C ( (cid:15) 2 β ) − 1 log n , the total number of edges sampled across all machine, and therefore, the total communication to the coordinator is (cid:101) O
( (cid:15) − 2 n 4 / 3 ) .
We shall now bound the cost of the hierarchical clustering returned by our scheme.
We begin by lower bounding the cost of any hierarchical clustering T of G , which by Lemma 2 is at least β 2 n 5 / 3 , which implies that δ :=
β 2 /n 1 / 3 ≤ cost G ( T ) /n 2 for any hierarchy T .
We shall argue that in the weighted sampled graph
H =
( V, E h , w h ) received by the coordinator, with probability at least 1 − 1 / poly ( n ) , the weight of any cut ( S, S ) is such that
[Equation]
Assuming this bound, it is relatively straightforward to prove that running the φ -approximate hierarchical clustering algorithm on the composite graph (cid:101) G =
( V, E h ∪ E x , w h + w x ) would produce a (1 + O ( (cid:15) ))
φapproximate clustering.
This follows by observing that the composite graph (cid:101) G is an ( (cid:15), Θ( (cid:15)δ ))
-sparsiﬁer of the input graph G , as the weight of any cut ( S, S ) in (cid:101) G is
[Equation]
where both inequalities follow by substituting the bounds in Eq.
(11), and observing that for any cut ( S, S ) , the weight w G x ( S ) =
(cid:15)δ · Θ(min {| S | , | S |} ) due to expansion and choice of edge weights in G x .
This guarantee together with Lemma 1 proves our claimed bound on the cost of the hierarchical clustering computed by our algorithm.
We shall now prove the bounds claimed in Eq.
Consider any cut ( S, S ) , and let us assume that | S | = k ≤ n/ 2 .
Let E S be the edges that cross the cut ( S, S ) in graph G .
For every edge e ∈ E S , we deﬁne a random variable X e
that is Bernoulli with parameter p =
C ( (cid:15) 2 β ) − 1 log n , taking value 1 if edge e is sampled.
Therefore, the weight of this cut in H is the random variable w H ( S ) =
p − 1 (cid:80) e ∈ E S X e .
We shall now bound the probability of the bad event where the value of this cut w H ( S ) >
(1 + (cid:15) )
w G ( S ) + (cid:15)δk as
Pr
[Equation]
H ( S ) >
[Equation] [Equation]
where for any edge e ∈ E S , random variable Y e
=
X e − p is such that E [ Y e ] = 0 , | Y e | ≤ 1 − p , and E [ Ye ] = p (1 − p ) .
Therefore, by Bernstein’s inequality,
[Equation]
Now there are two cases, either the cut ( S, S ) is such that ( a ) E [ w G ( S )] ≥ δpk or ( b ) E [ w G ( S )] < δpk .
In the ﬁrst case, we have the upper bound in Eq.
(12) is at most
Pr
[Equation]
where C (cid:48) is a constant, with ( a ) following by choice of p and δ , and ( b ) following by observing that β ≥ n 1 / 3 .
In the second case, we have the upper bound in Eq.
(12) is at most
Pr
[Equation]
where ( a ) follows by the same calculation as the previous case.
Therefore, by taking a union bound over all cuts ( S, S ) with | S | = k ≤ n/ 2 , we have that
Pr ( ∃ S : | S | = k, and w H ( S ) > (1+ (cid:15) )
w G ( S )+ (cid:15)δk
[Equation]
Following an identical analysis for Y e = p − X e gives us that with probability at least 1 − 1 / poly ( n ) , we have for all cuts ( S, S ) w H ( S ) ≥ (1 − (cid:15) )
w G ( S ) − (cid:15)δ min {| S | , | S |} , proving the bound claimed in Eq.
(11), completing the analysis for this case where β ≥ n 1 / 3 .
Analysis for Case 2:
In this case the number of edges in the graph is at most n 5 / 3 .
Since the memory of each machine is (cid:101) Ω( (cid:15) − 2 n 4 / 3 ) , the number of machines can be at most n 1 / 3 .
Each machine constructs linear sketches over its input and sends these to the coordinator similar to the 2 -round algorithm in Section E.1.
Note that the total communication is at most n 1 / 3 × (cid:101) O ( (cid:15) − 2 n ) = (cid:101) O
( (cid:15) − 2 n 4 / 3 ) as each machine can only send (cid:101) O ( (cid:15) − 2 n ) bits to the coordinator.
The coordinator then adds all the sketches corresponding to each vertex and computes a cut sparsiﬁer using the algorithm of [1].
Using Theorem 16, we can again argue that these linear sketches are such that one can recover a ( (cid:15), 0) -cut sparsiﬁer with high probability.
The claimed bound on the cost of the hierarchical clustering recovered then follows by a direct application of Lemma 1.
F Tight Query Lower Bounds for (cid:101) O (1) -approximation
We note that, for unweighted graphs, our sublinear time algorithm requires only 2 rounds of adaptive queries, where the ﬁrst round only needs to query vertex degrees.
Thus if one assumes prior knowledge of vertex degrees, our algorithm is in fact non-adaptive .
For weighted graphs, our algorithm requires at most O (log n ) rounds of adaptive queries due to the binary searches.
In any case, our algorithm makes at most (cid:101) O
( n 4 / 3 ) queries, where the worst-case input is an unweighted graph of about ≈ n 4 / 3 edges.
We now show that, in a sharp contrast, even with unlimited adaptivity, our algorithm’s query complexity is essentially the best possible for any randomized algorithm that computes a polylog ( n ) -approximate hierarchical clustering tree with high probability.
In particular, we establish below tight query lower bounds when the input is an unweighted graph with m =
Θ( n ζ ) edges for any constant ζ ∈ [0 , 2] .
By plugging in ζ = 4 / 3 in Case 4 , we get a matching lower bound for the worst-case input graph.
Case 1: ζ = 2 .
Any binary hierarchical clustering tree has cost O ( n 3 ) (Fact 2), and by Lemma 2, the optimal cost is at least Ω( n 3 ) .
Thus trivially 0 queries are sufﬁcient for O (1) -approximation.
Case 2: ζ ∈ [0 , 1] .
It is not hard to show an Ω( n ) query lower bound even for o ( n ) -approximation.
Speciﬁcally, consider using a random matching of size Θ( n ζ ) as a hard distribution, whose optimal hierarchical clustering cost is Θ( n ζ ) .
However, any o ( n ) -query algorithm can only discover an o (1) -fraction of the matching edges, and with an Ω(1) fraction of the matching edges having high entropy, any balanced cut of the graph has nontrivial probability of cutting Ω( n ζ ) matching edges, incurring a cost of Ω( n 1+ ζ ) .
On the algorithmic side, one can simply probe all edges with O ( n ) queries and then run any hierarchical clustering algorithm on the entire graph.
Thus the query complexity for (cid:101) O
(1) - approximation is settled at Θ( n ) .
Case 3: ζ ∈ [3 / 2 , 2) .
One can show an Ω( n ) query lower bound for (cid:101)
O
(1) -approximation, by considering an input graph obtained by randomly permuting the vertices of a union of vertex-disjoint cliques.
We include a proof of this lower bound in Section F.1.
On the algorithmic side, our sublinear time algorithm obtains an O ( √ log n ) -approximation using (cid:101) O
( n ) queries in this case, which is nearly optimal.
Case 4: ζ ∈ (1 , 3 / 2) .
Let γ :=
ζ − 1 ∈ (0 , 1 / 2) .
Our sublinear time algorithm obtains an O ( √ log n ) - approximation using (cid:101) O
( n min { 1+ γ, 2 − 2 γ } ) queries.
We show in Section F.2 that this is nearly optimal even for (cid:101) O (1) -approximation.
F.1 Lower bound for m between n 3 / 2 and n
Theorem 18 (Lower bound for m between n 3 / 2 and n 2 ) .
Let γ ∈ [1 / 2 , 1) be an arbitrary constant.
Let A be a randomized algorithm that, on an input unweighted graph G =
( V, E ) with | V | = n and | E | =
Θ( n 1+ γ ) , outputs a polylog ( n ) -approximate hierarchical clustering tree with probability Ω(1) .
Then A necessarily uses Ω( n ) queries.
We will show that there exists a distribution D over graphs with n vertices and Θ( n 1+ γ ) edges, on which no deterministic algorithm using o ( n ) queries can output a polylog ( n ) -approximate hierarchical clustering tree with probability ≥ .
This coupled with Yao’s minimax principle [43] will prove Theorem 18.
We deﬁne D such that a graph G ∼ D is generated by ﬁrst taking a union of n 1 − γ vertex-disjoint cliques of size n γ , and then permuting the n vertices uniformly at random.
More formally, we ﬁrst pick a uniformly random permutation π : [ n ] → [ n ] , and then let G be a union of vertex-disjoint cliques C 1 , . . .
, C n 1 − γ each of size n γ such that C
i is supported on vertices
[Equation]
By Fact 2, we know that the optimal hierarchical clustering cost of each clique is O ( n 3 γ ) .
Therefore, summing this cost over all cliques, we have:
Proposition 1.
The optimal hierarchical clustering tree of G has cost O ( n 1+2 γ ) .
We now describe a process that interacts with any given deterministic algorithm
A using o ( n ) queries while generating a uniformly random permutation π :
[ n ] → [ n ] along with its inverse function π − 1 : [ n ] → [ n ] .
Speciﬁcally, we will generate π, π − 1 by realizing them entry by entry adaptively based on the queries made be the algorithm.
Thus, when realizing an entry of π or π − 1 , we will always do so by conditioning on their already realized entries.
Also note that since the degree of each vertex is the same (namely n γ − 1 ), we will give the degree information to A for free at the start.
The process then proceeds by the following two principles:
Principle 1: Upon a pair query between i, j , realize π − 1 ( i ) , π − 1 ( j ) and then answer the query accordingly.
Principle 2: Upon a neighbor query about the (cid:96) th neighbor of i , ﬁrst realize π − 1 ( i ) .
Let k be such that the (cid:96) th neighbor of i is π ( k ) .
Then realize π ( k ) and answer the query accordingly.
Clearly, each query triggers the realization of O (1) entries of π and π − 1 .
Thus, after A terminates, the number of realized entries of π and π − 1 is at most o ( n ) .
Let U ⊂ [ n ] with | U | ≥ (1 − o (1))
n be the set of indices whose π values are not realized, and similarly let W ⊂ [ n ] with | W | =
| U | ≥ (1 − o (1))
n be the set of indices whose π − 1 values are not realized.
Let T be the hierarchical clustering tree output by A , which we suppose for the sake of contradiction is polylog ( n ) -approximate.
We ﬁrst make T a full binary tree such that the bi-partition of each internal node is [1 / 3 , 2 / 3] -balanced, during which we increase the cost of the tree by at most an O (1) factor.
We next consider the bi-partition of the root, which is a cut ( S, ¯ S ) with | S | ∈ [ n/ 3 , 2 n/ 3] .
Let S (cid:48) :=
S ∩ W and T (cid:48) :
= ¯ S ∩ W , and thus ( S (cid:48) , T (cid:48) ) is a bi-partition of W .
Since | W | ≥ (1 − o (1)) n , we have | S (cid:48) | ∈
[ | W | / 6 , 5 | W | / 6] .
Since also | U | ≥ (1 − o (1)) n , we have that for at least Ω(1) fraction of the cliques
C i ’s (which are supported on S i ’s), we have
[Equation]
For each such clique C i , the number of edges within C i that are across ( S (cid:48) , T (cid:48) ) is Ω( n 2 γ ) with high probability.
Therefore, the size of the cut ( S, ¯ S ) is at least Ω( n 1+ γ ) with high probability.
This means that the cost of T is at least Ω( n 2+ γ ) , which together with γ < 1 contradicts T being polylog ( n ) -approximate.
F.2 Lower bound for m between n and n 3 /
Theorem 19 (Lower bound for m between n and n 3 / 2 ) .
Let γ ∈ (0 , 1 / 2) be an arbitrary constant.
Let A be a randomized algorithm that, on an input unweighted graph G =
( V, E ) with | V | = n and | E | =
Θ( n 1+ γ ) , outputs with Ω(1) probability a polylog ( n ) -approximate hierarchical clustering tree.
Then A necessarily uses at least n min { 1+ γ, 2 − 2 γ }− o (1) queries.
By Yao’s minimax principle [43], to prove Theorem 19, it sufﬁces to exhibit a hard input distribution on which every deterministic algorithm using a small number of queries fails with nontrivial probability.
Speciﬁcally, we will show that there exists a distribution D over graphs with n vertices and Θ( n 1+ γ ) edges such that, on an input graph drawn from D , any deterministic algorithm using n min { 1+ γ, 2 − 2 γ }− δ queries for any constant δ >can only output a polylog ( n ) -approximate hierarchical clustering tree with o (1) probability.
The hard distribution.
We start by deﬁning the hard distribution D over graphs with n vertices and Θ( n 1+ γ ) edges.
Roughly speaking, we will generate an input graph G by ﬁrst taking the union of a certain number of cliques C 1 , . .
, C k of equal size n/k , and then adding some artiﬁcially structured edges between them.
We will then show that even the edges between the cliques are relatively tiny compared to those within, it is necessary to discover them in order to output a good hierarchical clustering solution.
More speciﬁcally, we will decide what edges to add between cliques based on the structure of a randomly generated “meta graph” H on k supernodes, with supernode i in H representing the clique C i .
We generate the meta graph H by picking a uniformly random perfect matching between the k supernodes (assuming for simplicity k is even).
Then for each matched pair of supernodes i, j in the meta graph H , we will add between
C i and C j a random bipartite matching of certain size (note that this matching is in the actual graph G rather than the meta graph H ).
Moreover, when adding the latter matching edges in G , we will also delete some edges inside C i , C j to ensure that every vertex has the exact same degree, so that an algorithm cannot tell which vertices participate in the meta graph’s perfect matching by only looking at the vertex degrees.
We will then show:
Any deterministic algorithm using n min { 1+ γ, 2 − 2 γ }− δ queries for any δ > 0 can only discover an o (1) fraction of the matching edges in the meta graph H .
If Ω(1) fraction of the matching edges have high entropy, an algorithm cannot output a polylog ( n ) - approximate hierarchical clustering tree with Ω(1) probability.
We now formally describe how we generate a graph G from D .
Let the vertices of G be numbered 1 through n .
We divide the vertices into n 1 − γ groups S 1 , . . .
, S n 1 − γ each of size n γ , where
S i :=
{ ( i − 1) n γ + 1 , . .
, in γ } .
We then generate the edges of G by the process in Figure 2.
Generate a meta graph H on supernodes numbered 1 , . .
. , n 1 − γ by picking a uniformly random perfect matching (of size n 1 − γ / 2 ) between them.
Initially, add a clique C i of size n γ to each vertex group S i , and insert the clique edges into the adjacency list of G in an arbitrary order.
Let t ← n max { 0 , 3 γ − 1 } + 1 √ log n .
In what follows, we will add a matching of size 2 t between each matched clique pair.
For each matched pair of supernodes i, j in the meta graph H : (a) Add a uniformly random bipartite matching M i,j of size 2 t between S i and S j , and let T i,j denote the vertices matched by M i,j (thus | T i,j ∩ S i | =
| T i,j ∩ S j | = 2 t ).
(b) Inside S i (resp.
S j ), pick a uniformly random perfect matching of size t between vertices T i,j ∩ S i (resp.
T i,j ∩ S j ), and delete its edges from clique C i (resp.
C j ).
(c) Modify the adjacency list of the vertices in G by replacing the edges deleted at Step 4b with the edges added at Step 4a.
This modiﬁcation is valid because the degree of each vertex is preserved.
Proposition 2.
All vertices in G have degree exactly n γ − 1 .
Proposition 3.
The optimal hierarchical clustering tree of G has cost O ( n 1+2 γ ) .
Proof.
We will construct a hierarchical clustering tree as follows.
At the ﬁrst level, we divide the entire vertex set into n 1 − γ / 2 clusters where each cluster is a connected component.
This step incurs zero cost.
We then construct a binary hierarchical clustering tree of each cluster arbitrarily.
Since each cluster has 2 n γ vertices, the hierarchical clustering tree we construct for it has cost bounded by O ( n 3 γ )
(Fact 2).
Summing this upper bound over all n 1 − γ / 2 clusters ﬁnishes the proof.
Analysis of deterministic algorithms on D .
Let A be a deterministic algorithm that makes n min { 1+ γ, 2 − 2 γ }− δ queries for some constant δ > 0 .
Since all vertices have the same degree n γ − 1 in
G , we will give the degree information to A for free at the start.
We shall then describe a process that interacts with the algorithm A while generating a G ∼ D .
To that end, we ﬁrst deﬁne the notion of revealed vertex groups.
Deﬁnition 2 (Revealed vertex groups) .
At any given point of the algorithm, we say a vertex group S
i is revealed by A if at least one of the following is true:
Condition 1: At least n 2 γ 10000 t pair queries involving vertices in S i are made by A .
Condition 2: At least n 2 γ 10000 t neighbor queries on vertices in S i are made by A .
Condition 3: A pair query by A ﬁnds a pair u, v ∈ S
i not connected by an edge.
Condition 4: A pair query or a neighbor query by A ﬁnds a pair u ∈ S i , w / ∈ S
i connected by an edge.
We now describe a process that answers queries made by A while adaptively realizing the edge slots and the adjacency list of G , as well as the perfect matching in the meta graph H .
Whenever realizing a part, we will always do so following the distribution D conditioned on the already realized parts.
This means that if a part is already realized or determined by other realized parts, realizing it again will not change it.
The process proceeds according to the following three principles:
Principle 1: Upon a pair query, realize the corresponding edge slot and answer accordingly.
Principle 2: Upon a neighbor query, realize the corresponding entry of the adjacency list and answer accordingly.
Principle 3: As soon as a group S i becomes revealed after a query, due to either large query count or what we have answered by Principle 1 and Principle 2 , right away do:
• Realize the supernode j that is matched to i in the meta graph H .
• Realize all edge slots incident on (and hence also all neighbors of) vertices in S i , S j .
At any given point of this process, we say a vertex group S
i is realized if all edge slots incident on S i are realized.
That is, the realized vertex groups are exactly those revealed by A and the ones matched to them.
This in particular implies that a perfect matching has been realized between the realized vertex groups in the meta graph H , while none of the unrealized vertex group is matched.
As a result, one can show that the queries made so far that involve unrealized vertex groups must have deterministic answers:
Proposition 4.
At any point of the algorithm A , for the queries already made, we have:
• Every pair query between an unrealized vertex group and a realized one discovered no edge.
• Every pair query between two unrealized vertex groups discovered no edge.
• Every pair query within a same unrealized vertex group discovered an edge.
• Every neighbor query on a vertex in an unrealized vertex group found a neighbor within the same group.
In what follows, we will consider the conditional distribution of D on all edge slots incident on realized vertex groups, which we denote by D rz .
Note that G (cid:48) ∼ D rz is not necessarily consistent with the answers we gave to the queries that involve unrealized vertex groups, though these answers are themselves deterministic by Proposition 4.
By deﬁnition, a graph G (cid:48) ∼ D rz can be generated by the process in Figure 3.
Add the edges incident on the realized vertex groups to G (cid:48) .
Add the perfect matching between the realized vertex groups to the meta graph H . 3.
Add a clique C i of size n γ to each unrealized vertex group S i . 4.
For each unrealized vertex group S i : (a) If supernode i is not matched in the meta graph H , then match i to another uniformly random unmatched j , and change the edges within S
i ∪ S j using Steps 4a-4c in Figure 2.
Proposition 5.
Consider generating G (cid:48) ∼ D rz conditioned on that an unrealized S
i is matched to another unrealized S j in the meta graph H .
Then G (cid:48)
[ S i ∪ S j ] is consistent with previous answers with probability at least .
Proof.
First note that, when changing the edges within S i ∪ S j at Step 4a in Figure 3, the edges we delete from C i (resp.
C j ) distribute as a uniformly random matching of size t in C i (resp.
C j ), and the edges we add between S i , S j distribute as a uniformly random bipartite matching of size 2 t between S i , S j , though these distributions are correlated.
Then note that G (cid:48)
[ S i ∪ S j ] is not consistent with previous answers only if (i) the slot of an edge we delete within S i or S j was queried by A , or (ii) an edge we add between S i , S j was queried by A .
Since S i , S j are both unrevealed, they do not satisfy Condition 1 or Condition 2 .
As a result, we can bound the probability of
G (cid:48)
[ S i ∪ S j ] being inconsistent with previous answers via a union bound by
[Equation]
which proves the proposition.
We show that the number of realized vertex groups can be at most a o (1) fraction of the total.
Proposition 6.
Upon termination of the algorithm A , the total number of realized vertex groups S i ’s is bounded by o ( n 1 − γ ) with probability at least 1 − 1 /n 4 .
Proof.
The number of vertex groups that satisfy Condition 1 or Condition 2 can be at most 2 · #queries
[Equation]
This means that the total number of realized vertex groups that satisfy Condition 1 or Condition 2 and those matched to them is at most o ( n 1 − γ ) .
We then bound the number of realized vertex groups that do not satisfy Condition 1 or Condition 2 and are not matched to those who satisfy Condition 1 or Condition 2 .
Each such vertex group must be (matched to) a revealed one that satisﬁes Condition 3 or Condition 4 .
We thus consider the probability that a query makes an unrealized vertex group satisfy Condition 3 or Condition
• Pair query: If a pair query involves a vertex in an already realized vertex group, then its answer is already determined and it does not reveal any unrealized groups.
Otherwise, if a pair query only involves unrealized vertex groups, we show that the probability it reveals any unrealized groups is at
most 8 t
n 2 γ .
First consider the case that the query is within a single unrealized group S i .
For a G (cid:48) ∼ D rz , the probability that this query discovers a non-edge is at most t ( nγ 2 ) .
By Proposition 5, conditioned on S i being matched to another S j in the meta graph H , the probability that G (cid:48)
[ S i ∪ S j ] is consistent with previous answers is ≥ .
Therefore, this query discovers a non-edge with probability ≤ 2 t ( nγ
Then consider the case that the query is between two unrealized groups
S i , S j .
If S i is not matched to S j in the meta graph H , then the pair query does not discover an edge, since there is no edge between S i , S j .
Otherwise, for a G (cid:48) ∼ D rz , conditioned on S i being matched to S j , the pair query discovers an edge with probability 2 t n 2 γ .
By Proposition 5, the probability that G (cid:48)
[ S i ∪ S j ] is consistent with previous answers with probability ≥ .
Therefore, this query discovers an edge with probability ≤ 4 t n 2 γ .
• Neighbor query: Consider a neighbor query on a vertex u in an unrealized vertex group S i .
For a G (cid:48) ∼ D rz , the query ﬁnds an edge going out of S i with probability t ( nγ 2 ) .
By Proposition 5, conditioned on S i being matched to another S j in the meta graph H , the probability that G (cid:48)
[ S i ∪ S j ] is consistent with previous answers with probability ≥ .
Therefore, this query discovers an outgoing edge with probability ≤ 2 t ( nγ 2 ) .
Combining the above, a query makes an unrealized vertex group satisfy Condition 3 or Condition 4 with probability at most 8 t n 2 γ .
Also, by doing so, a query can increase the number of realized vertex groups by at most 4 .
As a result, the expected increase in the number of realized groups that do not satisfy Conditionor Condition 2 over all queries made by A is at most
[Equation]
(plugging in the value of t )
[Equation]
Then the proposition follows by an application of Chernoff bounds.
Suppose we are now at the end of the algorithm A .
Let D rz be D conditioned on all edge slots incident on realized vertex groups, as deﬁned above.
Similarly, G (cid:48) ∼ D rz is not necessarily consistent with the answers we gave to A ’s queries that involve unrealized vertex groups, albeit these answers are deterministic by Proposition 4.
Also, let aaa denote the answers we gave to all queries made by A , and let D rz , aaa denote the conditional distribution of D rz on aaa .
Lemma 4.
Let ( S, ¯ S ) be any ﬁxed cut with | S | ∈
[ n/ 3 , 2 n/ 3] .
With probability at least 1 − 1 /n , the size of the cut ( S, ¯ S ) in G (cid:48)(cid:48) ∼ D rz , aaa is at least n 2 γ + 1 √ log n / 10 7 .
Proof.
Suppose after A terminates, the number of realized vertex groups is bounded by o ( n 1 − γ ) , which by
Proposition 6 happens with high probability.
Suppose we generate a G (cid:48) ∼ D rz using the process in Figure 3.
Consider an S i that is among the ﬁrst unmatched n 1 − γ / 13 unrealized vertex groups that we iterate over at
Step 4a in Figure 3.
We claim that, with probability at least .
1 over the choice of S j matched to S i and the edges we add between S i , S j , t/ 100 of the latter edges are across the cut ( S, ¯ S ) .
To prove the claim, note that the number of choices of S j to be matched to S i is at least 5 n 1 − γ / 6 .
Let U denote the vertices in these S j ’s, and thus we have | U | ≥ 5 n/ 6 .
Deﬁne T := S ∩ U and T (cid:48) :
= ¯ S ∩ U , which satisfy | T | + | T | (cid:48) =
| U | and | T | ∈ [ | U | / 6 , 5 | U | / 6] .
Then the expected number of edge slots between S i , S j that are across the cut ( S, ¯ S ) is given by
[Equation]
by |
[Equation]
>.
13 n
[Equation]
Then the expected number of edges that we add between S i , S j that fall in these slots is at least
[Equation]
Since the number of edges between S i , S j is 2 t , by Markov’s inequality, the number such edges across the cut ( S, ¯ S ) is at least t/ 100 with probability ≥ .
1 , as desired.
Thus, for a G (cid:48) ∈ D rz , in expectation, at least n 1 − γ / 130 of the S
i ’s satisfy that between S i and the matched
S j , t/ 100 edges are across the cut ( S, ¯ S ) .
By a Chernoff bound, with probability at least 1 − e − n 1 − γ / 500 , the number of such S
i ’s is at least n 1 − γ / 1300 , in which case the cut size of ( S, ¯ S ) in G (cid:48) is at least
[Equation]
[Figure]
[Equation]
On the other hand, by Proposition 5, G (cid:48) is consistent with all answers aaa that we gave to A with probability at least
[Equation]
As a result, the cut ( S, ¯ S ) in G (cid:48)(cid:48) ∼ D rz , aaa has size at least 10 − 7 n 2 γ + 1 √ log n with probability at least 1 − e − 0 .
0005 n 1 − γ , which sufﬁces for proving the lemma.
We now conclude this section by proving Theorem 19.
Proof of Theorem 19.
Let A be a deterministic algorithm that makes n min { 1+ γ, 2 − 2 γ }− δ queries for some constant δ > 0 .
Suppose for the sake of contradiction, on an input graph G ∼ D , A outputs with probability Ω(1) a polylog ( n ) -approximate hierarchical clustering tree.
First, we turn this tree into a full binary tree such that the bi-partition of each internal node is [1 / 3 , 2 / 3]
-balanced, while increasing the cost by at most an O (1) factor.
We then consider the bi-partition of the root, which is a cut ( S, ¯ S ) with | S | ∈ [ n/ 3 , 2 n/ 3] .
By Lemma 4, conditioned on the answers A got, this cut has size at least n 2 γ + 1 √ log n / 10 7 with high probability.
However, by
Proposition 3, the cost of the optimal hierarchical clustering tree of G is at most O ( n 1+2 γ ) .
This means that A only obtains an n o (1) -approximation with high probability, a contradiction.
G A One-Round MPC Lower Bound for (cid:101) O (
) -approximation
Theorem 20.
Let P be any one-round protocol in the MPC model where each machine has memory O ( n 4 / 3 − ε ) for any constant ε > 0 .
Then at the end of the protocol P , no machine can output a polylog ( n ) -approximate hierarchical clustering tree with probability better than o (1) .
To prove the theorem, we will (i) describe the graph distribution from which we generate an input graph, (ii) specify how we split the input graph across multiple machines, and (iii) analyze the performance of any one-round protocol on such input.
The hard graph instance.
Let ε ∈
(0 , 1 / 3) be an arbitrary constant.
We ﬁrst deﬁne a “base” graph G of 2 n vertices and Θ( n 5 / 3 − ε ) edges as follows.
G consists of two vertex-disjoint parts, each supported on n vertices:
Part 1: A union of n 1 / 3+ ε bipartite cliques, each of size n 2 / 3 − ε (with each side having n 2 / 3 − ε /vertices), supported on vertex set V 1 with |V 1 | = n .
Part 2: A union of n 2 / 3+ ε bipartite cliques, each of size n 1 / 3 − ε (with each side having n 1 / 3 − ε /vertices), supported on vertex set V 2 with |V 2 | = n that is disjoint from V 1 .
We show that the induced subgraph G [ V 1 ] can be tiled using edge-disjoint subgraphs that are isomorphic to G [ V 2 ] .
Proposition 7.
The vertex-induced subgraph G [ V 1 ] can be decomposed into n 1 / 3 edge-disjoint subgraphs
G 1 , . .
, G n 1 / 3 , each supported on V 1 and consisting of n 2 / 3+ ε vertex-disjoint bipartite cliques of size n 1 / 3 − ε .
Proof.
Consider ﬁrst arbitrarily partitioning vertices on each side of each bipartite clique in G [ V 1 ] into vertex subsets of size n 1 / 3 − ε / 2 , and then collapsing each vertex subset into a supernode.
By further treating the parallel edges between a same pair of supernodes as a single edge, we have made G [ V 1 ] a union of n 1 / 3+ ε bipartite cliques each supported on 2 n 1 / 3 supernodes (thus we have 2 n 2 / 3+ ε supernodes in total).
Note that in this contracted version of G [ V 1 ] , each perfect matching between the 2 n 2 / 3+ ε supernodes correspond to an edge-induced subgraph that is isomorphic to G [ V 2 ] .
It is now not hard to show that this contracted version of
G [ V 1 ] can be decomposed into n 1 / 3 edge-disjoint perfect matchings, which proves the proposition.
In what follows, we will ﬁx an arbitrary such tiling G 1 , . .
, G n 1 / 3 .
We next deﬁne a distribution D such that a graph G ∼ D is generated by permuting the vertices of G uniformly at random.
Let π : [2 n ] → [2 n ] be the permutation we use to generate G .
We will then let V 1 , V 2 be, respectively, V 1 , V 2 under the vertex permutation π .
We also use G 1 , . .
, G n 1 / 3 to denote G 1 , . .
, G n 1 / 3 under the vertex permutation π , where the former form an edge-disjoint tiling of G [ V 1 ] , and each G i is isomorphic to G [ V 2 ] .
The next proposition bounds the optimal hierarchical clustering cost for any input graph G generated as above.
Proposition 8.
The optimal hierarchical clustering tree of G has cost at most O ( n 7 / 3 − 2 ε ) .
Proof.
We construct a hierarchical clustering tree by the following steps.
At the root of the tree, we divide the entire vertex set into n 1 / 3+ ε + n 2 / 3+ ε clusters with each cluster being a connected component.
This incurs zero cost of the tree.
We next construct a binary hierarchical clustering tree of each cluster arbitrarily.
If a cluster is a bipartite clique of size n 2 / 3 − ε , then we incur a cost of at most O ( n 2 − 3 ε )
(Fact 2).
If a cluster is a bipartite clique of size n 1 / 3 − ε , then we incur a cost of O ( n 1 − 3 ε ) .
Thus the total cost is O ( n 2 − 3 ε ) · n 1 / 3+ ε + O ( n 1 − 3 ε ) · n 2 / 3+ ε ≤ O ( n 7 / 3 − 2 ε ) .
The MPC input distribution.
Consider that in the MPC model each machine has Θ( n 4 / 3 − ε log n ) bits of memory, and there are in total Θ( n 1 / 3 ) machines.
We will give G [ V 2 ] to a uniformly random machine.
We then give each of G 1 , . . .
, G n 1 / 3 to a uniformly random remaining machine, while ensuring that each machine gets at most one subgraph G i .
Note that, each machine’s input has exactly the same distribution.
Namely, each machine has the same probability of having a non-empty graph.
Moreover, for each machine, conditioned on that it gets at least one edge, the graph it gets is a union of n 2 / 3+ ε bipartite cliques of size n 1 / 3 − ε plus n isolated vertices, with all vertices permuted uniformly at random.
However, the input distributions of different machines are correlated.
Analysis of
one-round protocols
on the
input
distribution.
We show that for any one-round protocol P , no machine can output a polylog ( n ) -approximate hierarchical clustering tree of G with probability Ω(1) .
We will do so by a reduction from a two-party one-way communication problem, which we deﬁne next.
We speciﬁcally consider the following one-way communication problem in the two-party model, with players Alice and Bob who have shared randomness.
Alice is given as input a graph H on n vertices, which is obtained by ﬁrst taking a union of n 1 − γ bipartite cliques each of size n γ (with each side having n γ / 2 vertices), for some constant γ ∈ (0 , 1) , and then permuting the n vertices uniformly at random.
The goal of this communication problem is as follows:
For Alice to send Bob a single (possibly randomized) message such that, for some constant δ > 0 , Bob can then output with probability Ω(1) a cut ( S, ¯ S ) in H with | S | ∈ [ n/ 3 , 2 n/ 3] and size at most O ( n 1+ γ − δ ) .
( (cid:63)
We show that this problem requires Ω( n ) communication.
In particular, we will prove the following theorem in Section G.1.
Theorem 21.
For any constant γ ∈ (0 , 1) , Alice needs to send a message of size Ω( n ) to achieve goal ( (cid:63) ) .
To reduce this two-party communication problem to our MPC problem, we prove the following lemma.
Lemma 5.
Suppose for ε > 0 , there exists a one-round protocol P in the MPC model with Θ( n 4 / 3 − ε log n ) bits of memory per machine such that, at the end of P , some machine can output a polylog ( n ) -approximate hierarchical clustering tree with probability Ω(1) .
Then there exists a one-way protocol Q with message size o ( n ) in the two-party communication model that achieves goal ( (cid:63) ) for γ = 1 / 3 − ε .
Proof.
Suppose at the end of protocol P , some machine M ∗ can output with probability Ω(1) a polylog ( n ) - approximate hierarchical clustering tree.
We ﬁrst show that such a protocol P implies another protocol P (cid:48) in which M ∗ can output a balanced cut of G [ V 2 ] with small size.
Claim 1.
There is another one-round protocol P (cid:48) in the MPC model in which M ∗ can ﬁnd with probability ≥ Ω(1) a vertex set S ⊂ V 2 such that | S | ∈
[ n/ 3 , 2 n/ 3] and the number of edges between S and V 2 \ S in G is at most n 4 / 3 − 2 ε polylog ( n ) .
Proof.
P (cid:48) proceeds by ﬁrst simulating P , and then, in parallel, letting each machine M (cid:54) =
M ∗ sample its input edges with probability 100log n n 2 / 3 − ε and send the sampled edges to M ∗ .
With high probability, the total number of edges that M ∗ receives is O ( n log n ) , and each bipartite clique of size n 2 / 3 − ε in G [ V 1 ] is connected by the sampled edges.
Therefore, by looking at the connected components of size n 2 / 3 − ε in the subsampled graph,
M ∗ can recover V 1 , V 2 exactly.
M ∗ then uses the protocol P to output a hierarchical clustering tree T .
We ﬁrst make T a fully binary tree without increasing the cost.
We then look at an internal node of T corresponding to a vertex set T such that | T ∩ V 2 | ∈
[ n/ 3 , 2 n/ 3] .
One can show that such an internal node exists by starting at the root of the tree and keeping moving to the child that has a larger intersection with V 2 until ﬁnding a desired node.
Let S :
= T ∩ V 2 .
If we consider the edges between S and V 2 \ S in G , each of them incurs a cost of at least | T | ≥ n/ 3 in T .
Since the cost of T is at most n 7 / 3 − 2 ε polylog ( n ) with probability Ω(1) , the number of edges between S and
V 2 \ S must be bounded by n 4 / 3 − 2 ε polylog ( n ) with probability Ω(1) , as desired.
We now show how to use the protocol P (cid:48) to construct a one-way protocol Q in the two-party communication model that achieves goal ( (cid:63) ) with γ = 1 / 3 − ε .
First note that, since the memory per machine is Θ( n 4 / 3 − ε log n ) , the total message size received by M ∗ is at most Θ( n 4 / 3 − ε log n ) .
This means that, on average, another machine sends a message of size Θ( n 1 − ε log n ) to M ∗ .
For each machine M i , let p i be the success probability of P (cid:48) conditioned on that G
[ V 2 ] is given to machine M i .
Since P (cid:48) succeeds with probability Ω(1) , the average of p i must be Ω(1) .
Based on the above two observations, by applying Markov’s inequality twice and then a union bound, we have that there exists a machine M j (cid:54)
=
M ∗ that sends M ∗ a message of size O ( n 1 − ε log 2 n ) and has p j ≥ Ω(1) .
The protocol Q proceeds as follows.
Upon receiving the input graph, which is a union of n 2 / 3+ ε bipartite cliques of size n 1 / 3 − ε , Alice shall treat its vertices as V 2 and add n isolated vertices as V 1 .
Then she uses her shared randomness with Bob to permute the 2 n vertices uniformly at random.
Note that now the new graph has the exact same distribution as the input given to machine M j in the MPC model conditioned on G [ V 2 ] being given to M j .
Alice then uses the same message generation algorithm as M j to produce a message and sends it to Bob.
Upon receiving the message from Alice, Bob himself then simulates protocol P (cid:48) for other machines M i (cid:54) =
M j by generating their inputs conditioned on the realization of V 1 , V 2 and simulating their message generation algorithms.
Finally, Bob runs the recovery algorithm of M ∗ to recover a vertex set S ⊂ V 2 , which satisﬁes with probability =
p j ≥ Ω(1) that | S | ∈ [ n/ 3 , 2 n/ 3] and that the number of edges between S, V 2 \ S is at most n 4 / 3 − 2 ε polylog ( n ) ≤ O ( n 4 / 3 − 1 .
5 ε ) .
This means that the protocol Q achieves goal ( (cid:63) ) for γ = 1 / 3 − ε with message size O ( n 1 − ε log 2 n ) ≤ o ( n ) .
Lemma 5 and Theorem 21 together rule out any one-round protocol in the MPC model with O ( n 4 / 3 − ε ) memory per machine for any constant ε > 0 , and thus prove Theorem 20.
G.1 A Lower Bound in the Two-Party Communication Model
In this section we prove Theorem 21, which gives a lower bound on the communication needed to achieve goal ( (cid:63) ) .
We ﬁrst show that any cut in the input graph H given to Alice that has size ≤ O ( n 1+ γ − Ω(1) ) can be made into a cut of size 0 by changing the sides of an o (1) fraction of vertices.
Proposition 9.
For any cut ( S, ¯ S ) in H with size at most O ( n 1+ γ − δ ) for any constant δ > 0 , one can obtain from it another cut of size 0 by switching the sides of at most O ( n 1 − δ ) vertices.
Proof.
Let us call the bipartite cliques in H C 1 , . .
, C n 1 − γ with C i supported on vertices S i .
For each i , let s i :=
| S i ∩ S | and t i :=
| S i ∩ ¯ S | .
Then the number of edges of C i across the cut ( S, ¯ S ) is at least Ω(1) · min { s i , t i } · n γ .
This means that we have
[Equation]
which by rearranging gives
[Equation]
Therefore we can obtain a new cut of size 0 from ( S, ¯ S ) by switching the sides of the vertices that correspond to the summation on the LHS of the above inequality, whose total number is at most O ( n 1 − δ ) .
Using the above lemma, we then show that if one can achieve ( (cid:63) ) using o ( n ) communication, then one can also output a balanced cut with size 0 using o ( n ) communication.
Lemma 6.
If there is a protocol that achieves goal ( (cid:63) ) with message size o ( n ) , then there is another protocol in which Alice sends Bob a message of size o ( n ) , such that Bob can then output with probability Ω(1) a cut ( S (cid:48) , ¯ S (cid:48) ) in H with | S (cid:48) | ∈
[ n/ 6 , 5 n/ 6] and size 0 .
Proof.
We will design the second protocol by simulating the ﬁrst protocol.
To this end, let us ﬁx a protocol P that achieves goal ( (cid:63) ) for some δ > 0 , which consists of a message generation algorithm A for Alice and a cut recovery algorithm B for Bob.
At the start, Alice runs A to generate a message M of size o ( n ) .
Then,
Alice and Bob use their shared randomness to generate sufﬁciently many random bits for running B .
Alice ﬁrst runs B given the message M on her own and gets a cut ( S, ¯ S ) .
If the cut satisﬁes | S | ∈ [ n/ 3 , 2 n/ 3] and has size at most O ( n 1+ γ − δ ) , then Alice sends Bob the message M along with a subset U of O ( n 1 − δ ) vertices whose switching sides makes cut ( S, ¯ S ) have zero size (existence guaranteed by Proposition 9).
Then Bob, upon receiving the message M and the subset U of vertices, runs the recovery algorithm B using the shared random bits with Alice and gets the same cut ( S, ¯ S ) with
| S | ∈
[ n/ 3 , 2 n/ 3] as Alice.
By switching the sides of vertices in U , Bob then gets a cut ( S (cid:48) , ¯ S (cid:48) ) with | S (cid:48) | ∈ [ n/ 6 , 5 n/
] and
size
, as
desired
In light of the above lemma, we now consider another one-way two-party communication problem, where Alice gets a same input graph H obtained by ﬁrst taking a union of n 1 − γ bipartite cliques of size n γ
and then permuting all n vertices uniformly at random, and the goal is
For Alice to send Bob a single (possibly randomized) message such that Bob can output with probability Ω(1) a cut ( S, ¯ S ) in H that satisﬁes | S | ∈
[ n/ 6 , 5 n/ 6] and has size 0 .
( (cid:63)(cid:63) )
We show that it requires Ω( n ) communication to achieve ( (cid:63)(cid:63) ) .
Lemma 7.
For any constant γ ∈ (0 , 1) , Alice needs to send a message of size Ω( n ) to achieve goal ( (cid:63)(cid:63) ) .
Proof.
Let P be a protocol that achieves ( (cid:63)(cid:63) ) .
Then on an input graph H ∼ D , at the end of the protocol P ,
Bob outputs with probability Ω(1) a cut ( S, ¯ S ) in H with | S | ∈ [ n/ 6 , 5 n/ 6] and size 0 .
We now analyze the entropy of the distribution D and that of D conditioned on the message M Bob receives from Alice.
First, note that an input graph H can be determined by ﬁrst dividing the n vertices into n 1 − γ groups each of size n γ , and then picking a balanced bi-partition for each group.
Thus the total number of different H ’s can be calculated by
[Equation]
Thus the entropy of D is
[Equation]
On the other hand, by Fano’s inequality, we have
[Equation]
We then do a case analysis to calculate H ( D| ( S, ¯ S )) :
Case 1: The cut ( S, ¯ S ) satisﬁes | S | ∈ [ n/ 6 , 5 n/ 6] and has size 0 , which happens with probability Ω(1) .
Since the cut ( S, ¯ S ) does not cut through any cliques, it must be that S contains entirely | S | /n γ cliques and ¯ S contains entirely the remaining | ¯ S
| /n γ cliques.
Therefore, the total number of graphs
H that is consistent with this proﬁle is
[Equation]
Thus the entropy of D conditioned on the cut ( S, ¯ S ) is
H
[Equation]
We can then calculate the difference between the entropy H ( D ) and H ( D| ( S, ¯ S )) by
H ( D
[Equation]
H
[Equation]
S ))
[Equation]
= logn !
[Equation]
| !
[Equation]
S
[Equation]
n
[Equation]
γ
[Equation]
(plugging in the values of N 1 , N 2 ) = log
[Equation]
n
[Equation] [Equation]
(rewriting as binomials)
[Equation]
Case 2: The cut ( S, ¯ S ) does not satisfy | S | ∈
[ n/ 6 , 5 n/ 6] or has nonzero size, which happens with probability 1 − Ω(1) .
Since conditioning can only reduce entropy, we have
[Equation]
S ))
[Equation]
Combining the above two cases with (14), we have
[Equation]
On the other hand, by the chain rule, we have
[Equation]
As a result, H ( M ) ≥ Ω( n ) , and therefore M must contain Ω( n ) bits.
We now prove Theorem 21.
Proof of Theorem 21.
Suppose for the sake of contradiction there exists a protocol with message size o ( n ) that achieves goal ( (cid:63) ) .
Then by Lemma 6, there also exists a protocol with message size o ( n ) that achieves goal ( (cid:63)(cid:63) ) , contradicting Lemma 7.
Therefore Alice needs to send a message of size Ω( n ) in order to achieve goal ( (cid:63) ) , as desired.
H Experimental Results
We now present our experimental results.
The goal of our experiments is to test our data/graph sparsiﬁcation framework (Algorithm 3) which forms the basis of all our algorithms under the different models of computation.
We ask the following question: if we run an HC algorithm on the sparsiﬁed graph produced by Algorithm 3, then what will be the loss in the quality of solution compared to the same HC algorithm run on the original graph?
We would also like to understand the tradeoff between the sparsiﬁcation rate and the solution quality of certain HC algorithms.
We test our data sparsiﬁcation algorithm on two large real-world datasets Boston and Newsgroup, which are both part of the standard Scikit-learn library and were used in previous HC papers [16, 38].
We calculate the similarity between two data points using the Gaussian kernel as in [38, 33] and only insert an edge if its weight (i.e. similarity between its two endpoints) is bigger than 10 − 10 .
Since we are not aware of any existing implementation of the recursive sparsest cut algorithm, we choose to test two linkage algorithms, namely average linkage and complete linkage, which, as shown in [33], are among the best-performed algorithms on the two datasets we consider.
We run these two algorithms both on the entire graph and on the sparsiﬁed graphs obtained by our Algorithm 3, and compare the cost of the HC we get in each case, where the cost is measured using Dasgupta’s objective function (Eq.
When running our Algorithm 3, we also try different sampling rates to obtain subgraphs of various edge densities.
Note that, we always measure the cost of an HC in the original graph as opposed to in the subsampled graph.
We present our experimental results in the tables below.
Here we write absolute HC cost to denote the cost of the HC obtained by running the corresponding algorithm on the entire graph, and write relative HC cost to denote the cost of the HC obtained by running the algorithm on the subsampled graph divided by the absolute HC cost.
For the Boston dataset, one can see that even when we only sample <5% of the edges, we already get a very good relative HC cost of 1.15; for the Newsgroup dataset, sampling only <
[Table] [Table]
[Table] [Table]