E ND - TO -E ND L EARNING OF P ROBABILISTIC H IERARCHIES ON G RAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel probabilistic model over hierarchies on graphs obtained by continuous relaxation of tree-based hierarchies.
We draw connections to Markov chain theory, enabling us to perform hierarchical clustering by efﬁcient end-to-end optimization of quality metrics such as Dasgupta cost or Tree-Sampling Divergence (TSD).
We show that our model learns rich, high-quality hierarchies present inreal world graphs, including a large graph with 2.3M nodes.
Our model consistently outperforms recent as well as strong traditional baselines such as average linkage.
We demonstrate that our model also obtains competitive results on link prediction despite not being trained on this task, highlighting the quality of the hierarchies discovered by our model
I NTRODUCTION
Clustering is a fundamental problem in unsupervised learning, both in theory and practice.
In contrast to traditional ‘ﬂat’ clustering, hierarchical clustering brings several advantages.
It enables us to naturally analyze (Zhao & Karypis, 2002) and visualize (Himberg et al., 2004) a given dataset on different scales.
Further, different downstream tasks may require different granularities.
Hierarchical clustering lets us pick the desired granularity after learning.
It can also be used for personalized recommendation (Zhang et al., 2014) or to solve record linkage tasks (Wick et al., 2012).
In an inﬂuential paper, Eisen et al.
(1998) perform hierarchical clustering on gene expression data and argue that it could be used to discover the yet-unknown meaning of certain genes, which could lead to advances in medicine.
Besides traditional vector data, real-world graphs are often scale-free and hierarchically organized (Ravasz & Barabási, 2003; Barabási & Pósfai, 2016).
This makes them interesting candidates for uncovering the underlying hierarchy via hierarchical clustering.
Thus, the hierarchical clustering method we introduce is particularly useful for analyzing scale-free real-world networks such as Web graphs, citation networks, ﬂight networks, or biological networks.
Related work.
We can group hierarchical clustering algorithms into discrete and continuous optimization methods.
Discrete optimization methods can themselves be decomposed into agglomerative approaches and divisive approaches.
Agglomerative methods – like the famous linkage algorithms (Gower & Ross, 1969; Jardine & Sibson, 1968; Bonald et al., 2018) – follow a bottom-up approach and iteratively aggregate the two most similar clusters w.r.t.
a given similarity measure.
Divisive methods follow a top-down approach and recursively split larger clusters into smaller ones using, e.g., k-means (Steinbach et al., 2000) or spectral clustering (Charikar & Chatziafratis, 2016).
While Moseley & Wang (2017); Kamvar et al.
(2002) have shown connections with explicit objective functions, those methods are mainly based on heuristics to avoid exhaustive combinatorial optimization.
On the other hand, continuous optimization methods circumvent this issue by (i) relaxing the discrete tree structure, (ii) deﬁning an explicit quality metric for hierarchies, and (iii) optimizing it using gradient-based optimizers.
Chierchia & Perret (2019); Monath et al.
(2017; 2019); Chami et al.
(2020) focus on minimizing the Dasgupta cost (Dasgupta, 2016)
(see Eq.
(1)) to ﬁt an ultrametric, hyperbolic embeddings, or a probabilistic model over cluster assignments.
Monath et al.
(2017) learns a probabilistic assignment of the samples to leaves of a tree with a ﬁxed binary structure.
The method of Monath et al.
(2019) requires approximation for the computation of lowest common ancestors (LCA) probabilities and regularization losses.
Beyond the Dasgupta cost, Charpentier & Bonald (2019) propose the Tree Sampling Divergence (TSD), a quality metric for hierarchies on graphs.
Both metrics have the beneﬁt to be internal metrics, i.e., they do not require ground truth information to evaluate the quality of a hierarchy.
Most of the aforementioned methods focus on vector data.
Contributions.
In this work, we focus on hierarchical clustering in homophilic graphs, where the similarity of two nodes is made explicit by the presence or absence of an edge.
We (1) propose a probabilistic model over hierarchies via continuous relaxation of a tree’s parent assignment matrices.
We (2) theoretically analyze the model by drawing connections to absorbing Markov chains, which allows (3) efﬁcient and exact computation of relevant quantities (e.g., LCA probabilities).
This enables us to (4) learn hierarchies on graphs by efﬁcient, end-to-end optimization of quality metrics such as
Dasgupta cost and the TSD score.
Our extensive experimental evaluation on 12 real-world graphs, including a massive graph with more than 2M nodes and 60M edges, highlights the effectiveness of our Flexible Probabilistic Hierarchy model (FPH) .
It outperforms all baselines, traditional and recent, on the quality of the learned hierarchies measured both by TSD and Dasgupta cost.
Remarkably, FPH also performs competitively on link prediction despite not being trained on this task.
H IERARCHICAL GRAPH CLUSTERING PRELIMINARIES
[Figure]
(𝑧 𝑗 |𝑧 𝑖
[Figure]
(𝑧 𝑗 |𝑣 𝑖 )
[Figure]
[Figure]
𝑣 4 𝑣
[Figure]
𝑣 6 𝑣 [Figure] 𝑣 4 𝑣
[Figure]
𝑣𝑣𝑣𝑣𝑣𝑣
[Figure]
[Figure]
𝐵 𝐴
We deﬁne a graph G =
( E, V ) with n nodes and m undirected edges.
We let w ( v i , v j ) be equal to the weight of edge ( v i , v j ) if they are connected and to 0 otherwise, and w ( v i ) =
(cid:80) j w ( v i , v j ) to be equal to the weight of node v i .
This allows us to deﬁne the edge distribution P ( v i , v j ) ∝ w ( v i , v j ) normalized over all edges s.t.
(cid:80) ij P ( v i , v j ) =
and the node distribution
P ( v i ) ∝ w ( v i ) s.t.
(cid:80) i P ( v i )
= 1 , which are later needed to compute the quality metrics of learned hierarchies.
Discrete hierarchical clustering.
We deﬁne a hierarchical clustering T of a graph G as a tree structured partitioning of the nodes V .
We denote the nodes of the tree T with n (
cid:48) ∈
N internal nodes by { x 1 , ..., x n + n (cid:48) } .
The tree nodes can be decomposed into the leaf nodes { v 1 , ..., v n } corresponding to the nodes of the input graph, and the internal nodes { z 1 , ..., z n (cid:48) } representing clusters of leaf nodes; overall { x 1 , ..., x n + n (cid:48) }
= { v 1 , ..., v n } ∪ { z 1 , ..., z n (cid:48) } .
Since any tree is a directed acyclic graph (DAG), we impose w.l.o.g.
a topological ordering on the internal nodes of the tree in the remainder of this work.
That is, z i cannot be a parent of z j if i < j ; z n (cid:48) denotes the root node.
We can uniquely deﬁne a tree as T =
( ˆ A , ˆ B ) with two adjacency matrices, ˆ A ∈ { 0 , 1 } n × n (cid:48) and ˆ B ∈ { 0 , 1 } n (cid:48) × n (cid:48) .
ˆ A describes the connections from the leaves to the internal nodes, i.e. ˆ A i,k
=if internal node z k is the parent of leaf node v i .
ˆ B describes the connections between the internal nodes, i.e. B k,l = 1 if internal node z l is the parent of internal node z k .
Since in a tree any node has exactly one parent, ˆ A is row-stochastic and ˆ B is row-stochastic; except its last row ˆ B n (cid:48) corresponding to the root, whose sum is 0.
We denote by v i ∧ v j the internal node which is the lowest common ancestor (LCA) of v i , v j in the tree.
The maximum possible number of internal nodes is n (cid:48)
=
n − 1 .
Probabilistic hierarchical clustering.
The central idea of FPH is to deﬁne a probabilistic tree via continuous relaxation of the binary-valued adjacency matrices ˆ A , ˆ B .
I.e., we deﬁne A , B such that A ∈ [0 , 1] n × n (cid:48) and B ∈ [0 , 1] n (cid:48) × n (cid:48) while keeping the row-stochasticity constraints, i.e., (cid:80) n (cid:48)
j A i,j = 1 for 1 ≤ i ≤ n , (cid:80) n (cid:48)
l B k,l = 1 , 1 ≤ k < n (cid:48) , and for the root node we have (cid:80) n (cid:48)
l B n (cid:48) ,l = 0 .
This describes a probabilistic hierarchy; the probability of an internal node z j to be a parent of a leaf or an internal node is A ij :=
p ( z j | v i ) or B ij := p ( z j | z i ) , respectively.
See Fig.
1 for a visualization.
Our key technical contribution are efﬁcient closed-form expressions for the lowest common ancestor probabilities that arise when sampling discrete trees from the probabilistic one.
2. I NTERNAL METRICS FOR HIERARCHICAL CLUSTERING
Internal metrics are commonly used to evaluate clustering methods, and they have the important advantage of not requiring external labels.
For hierarchical clustering on graphs, two internal metrics have been proposed to assess the quality of a discrete hierarchy T given an input graph G : Dasgupta cost (Das.)
(Dasgupta, 2016)
TSD )
(Charpentier & Bonald, 2019)
While the Dasgupta cost can be used for graphs and vector data, TSD is tailored speciﬁcally to graphs.
[Equation]
For Das. , c ( z ) =
| z | denotes the number of leaves for which the internal node, i.e., z = v i ∧ v j , is an ancestor.
Intuitively, the cost is large if v i and v j are connected by an edge and their lowest common ancestor contains many leaves, i.e., is close to the root.
Thus, Dasgupta favors leaves connected by an edge to have LCAs low in the hierarchy.
For Das., lower scores indicate better hierarchies.
TSD deﬁnes two distributions over the internal nodes, p ( z ) =
(cid:80) v i ,
[Equation]
j : z = v i ∧ v j P ( v i , v j ) , induced by the edge distribution, and q ( z )
= (cid:80)
[Equation]
∧
[Equation]
P
[Equation]
v
[Equation]
) P ( v j ) , induced by the independent node distribution.
Intuitively, we expect the graph distribution p ( z ) (induced by edge sampling) and the null model distribution q ( z ) (induced by independent node sampling) to differ signiﬁcantly if the tree
T indeed represents the hierarchical structure of the graph, leading to high TSD scores.
Continuous relaxation.
To enable end-to-end gradient-based optimization of Dasgupta or TSD, we replace the discrete hierarchy T
=
( ˆ A , ˆ B ) with its continuous relaxation T =
( A , B ) .
This leads to probabilistic parent assignments of leaves to internal nodes.
By itself, this does not help us in hierarchical clustering as it is not clear how to compute the lowest common ancestor probabilities when sampling discrete hierarchies from the relaxed one, i.e., the probability that an internal node is the lowest common ancestor of two leaves.
Our main theoretical contribution is to derive closed-form expressions for the lowest common ancestor probabilities in Sec. 3.
This gives rise to continuous versions of TSD and Dasgupta (see equations in Appendix A.1), which we will refer to as Soft-TSD and Soft-Das., respectively.
For the ﬁrst time, this allows us to directly optimize for hierarchical clustering quality metrics in an end-to-end fashion instead of proxy losses or heuristic algorithms.
For optimization over hierarchies, we either minimize the Dasgupta cost or maximize TSD:
[Equation]
where A ∈ [0 , 1] n × n (cid:48) , B ∈ [0 , 1] n (cid:48) × n (cid:48) are the continuous relaxations of the parent assignment matrices as described in Sec. 2.
Note that for both metrics we recover the same score as their discrete versions in the case of a deterministic probabilistic model, i.e. when A , B are binary-valued.
2. S AMPLING DISCRETE HIERARCHIES
Given the probability matrices A and B we can easily recover a discrete hierarchical clustering by sampling discrete matrices ˆ A and ˆ B , which in turn describe a (discrete) tree T .
For each leaf and internal node we independently sample its parent from the categorical distribution described by the respective row in A or B .
As we show in Appendix A.2, this tree-sampling procedure (denoted
T =
( ˆ A , ˆ B ) ∼ P A , B ( T ) ) leads to valid tree hierarchies, and we can directly compute the probability of any discrete hierarchy given the probabilistic one.
We denote probabilities associated with the tree-sampling perspective as p ( T ) ( · ) .
Note that we can easily obtain a discrete tree given continuous
A , B in a deterministic way by selecting for each leaf and internal node its most likely parent.
E FFICIENT , DIFFERENTIABLE HIERARCHIES VIA M ARKOV CHAINS
Outline.
This section is organized as follows.
The goal is to derive efﬁcient, closed-form equations to compute (lowest common) ancestor probabilities which are consistent with the tree-sampling procedure explained in the previous section.
For this, we draw connections to absorbing Markov chains in Sec. 3 and show in Secs.
3.1, 3.2 how to compute the desired quantities efﬁciently under this simplifying Markov Chain perspective .
Then, we show that these equations under the Markov chain and the tree-sampling perspectives are equivalent, i.e., we do not introduce any error by the Markov chain assumptions.
Finally, we show in Sec.
3.3 how to exploit the independence assumptions of the Markov chain to compute the (lowest common) ancestor probabilities efﬁciently and in a vectorized way.
We start by showing that the probabilistic model described in the previous section indeed deﬁnes an absorbing Markov chain.
Intuitively, A and B can be interpreted as transition matrices from leaves to internal nodes and among internal nodes, respectively.
Deﬁnition 1 (Tree Markov Chain) .
Let A ∈ [0 , 1] n × n (cid:48) , (cid:80) n (cid:48)
j =1 A ij = 1 ∀ 1 ≤ i ≤ n and B ∈ [0 , 1] n (cid:48) × n (cid:48) , (cid:80) n (cid:48)
j =1 B ij = 1 ∀ 1 ≤ i < n (cid:48) , (cid:80) n (cid:48)
j =1 B n (cid:48)
j = 0 .
We deﬁne a Markov chain M =
( S ∪ { ω } , T ) with state set S ∪ { ω } and transition matrix T , where S = { v 1 , v 2 , . .
. , v n } ∪ { z 1 , z 2 , . . .
, z n (cid:48) } .
Further,
[Equation]
is the transition matrix of the Markov chain M in canonical form, where w ∈ R |S| = (0 , 0 , . . .
, 0 , 1) T is the vector of transition probabilities to state ω .
Note that we add here an auxiliary state ω , which acts as ﬁnal absorbing state from the root.
Theorem 1.
Let M be a tree Markov chain as deﬁned in Deﬁnition 1.
M is an acyclic absorbing
Markov chain with ω being its only absorbing state.
(See proof in App. A.3)
Thus, our probabilistic hierarchy deﬁned by A and B describes an absorbing and acyclic Markov chain.
In our hierarchical clustering interpretation, a random walk starts at one of the leaves and ﬁrst randomly transitions to the internal nodes based on A , followed by transitions among the internal nodes via B .
Once the root is reached, the random walk is absorbed after one last step to ω .
In the remainder, we denote the probabilities arising from the tree Markov-chain as p ( M ) ( · ) to distinguish them from their tree-sampling counterparts p ( T ) ( · ) .
In general, the two perspectives are not equal, e.g., when considering pairs of leaf nodes.
Here, the Markov chain introduces an independence assumption of pairs of random walks.
Under tree-sampling, two leaves’ paths to the root are dependent in general, i.e., two paths traversing some internal node z k implies that all subsequent transitions are identical.
While Markov-chain probabilities p ( M ) ( · ) often have efﬁcient and analytical solutions grounded in Markov chain theory, it is unclear a-priori how to leverage them for hierarchical clustering.
In the following, we relate central tree-sampling probabilities p ( T ) ( · ) , i.e., (lowest common) ancestor probabilities, to their Markov chain counterparts p ( M ) ( · ) .
3. A NCESTOR PROBABILITIES
First, we want to compute the probability p ( T ) anc ( z k | v j ) of an internal node z k being an ancestor of leaf v j under the row-wise tree-sampling procedure (see Sec. 2.2).
We call this an ancestor probability.
Theorem 2.
Let T =
( ˆ A , ˆ B ) ∼ P A , B ( T ) be a discrete hierarchy obtained by tree-sampling.
Further, let W ( v i ) be a random walk on M rooted in v i resulting in path ˆ r i .We have that
[Equation]
(3) (See proof in App.
A.4)
Note that any random walk on M must result in a path since, because M is acyclic, no transient state can be visited more than one time in a random walk.
Theorem 2 means that we can use ancestor probabilities arising from the tree-sampling and Markov chain views interchangeably.
3. L OWEST COMMON ANCESTOR PROBABILITIES
For the hierarchical clustering problem we are interested in the lowest common ancestor (LCA) of two leaf nodes v i and v j , which we denote by v i ∧ v j .
The LCAs are required to compute both
the
Dasgupta cost as well as the Tree-Sampling Divergence (TSD).
Therefore, when optimizing for good
Dasgupta or TSD scores, it is crucial to exactly and efﬁciently compute the LCA probabilities that arise.
We denote these LCA probabilities by p ( T ) ( z k = v i ∧ v j ) .
Previous work resorts to heuristics in approximating the LCA probabilities (Monath et al., 2019).
In contrast, our connection to absorbing
Markov chains admits efﬁcient closed-form computation of LCA probabilities, as we now show.
Theorem 3.
Let A and B be transition matrices describing a (soft or discrete) hierarchy as introduced in Section 2.
Let r i =
( r (1) i , . . .
, r ( T − 1) i , z n
[Equation]
) denote a path from leaf v i
=
r (1)
i to the root z n (cid:48) , where | r i | =
T .
Then, the probability of internal node z k being the lowest common ancestor v i ∧ v j of leaf nodes v i , v j under the tree-sampling procedure from Section 2.2 is
[Equation] [Equation]
where r i is the part of a path from v i up to (and including) internal node z k , i.e., r i =
( r (1) i , . . .
, z k ) .
(See proof in App. A.5)
From Eq.
(4) it seems that a straightforward way to compute the LCA probabilities is to enumerate the set { ( r i , r j ) :
z k = v i ∧ v j } of all pairs of potential paths from v i and v j to the internal node z k and sum their probabilities.
However, this is intractable in practice because the size of the set grows exponentially in the ‘depth’ of the internal node k (see App.
A.8, Theorem 7).
In Theorem 4 we show how to compute the LCA probabilities that arise when sampling pairs of random walks from the Markov chain M .
Theorem 4.
Let M be a Markov chain as deﬁned in Deﬁnition 1, z k be an internal node, and v i , v j be leaf nodes.
Let W ( v i ) , W ( v j ) be independent random walks on M rooted in v i and v j , respectively.
Then, the probability that internal node z k is the lowest internal node (as by the topological order) traversed by both random walks, i.e., the lowest common ancestor of v i and v j , is
[Equation]
(See proof in App. A.6)
Thus, due to the independence of pairs of random walks in the Markov chain M , we can compute LCA probabilities efﬁciently in closed form on the Markov chain.
However, the joint ancestor probability p ( M ) anc ( z k | v i , v j ) does not reﬂect the underlying process of sampling T =
( ˆ A , ˆ B ) ∼ P A , B ( T ) any more, i.e., in general p ( M ) anc ( z k | v i , v j ) =
p anc ( z k | v i ) · p anc ( z k | v j ) (cid:54) =
p ( T ) anc ( z k | v i , v j ) .
As our main theoretical result , we show next that, remarkably, the LCA probabilities obtained from the Markov chain are indeed equivalent to the tree-sampling LCA probabilities:
Theorem 5.
Let A and B be transition matrices describing a (soft or discrete) hierarchy as described in Sec. 2.
The LCA probabilities arising from tree-sampling and the Markov chain are equal, i.e.,
[Equation]
(See proof in App. A.7)
This result is surprising at ﬁrst, since we made the assumption that pairs of random walks are independent in the Markov chain sampling process.
However, recall that two paths in the treesampling process are disjoint and independent until they meet at their LCA z k .
Thus, for this subset of the pairs of paths leading to z k , the independence assumption in M does not lead to an error.
The result is also very useful in practice, since it means that we can use efﬁcient computations from the Markov chain view to exactly compute the tree-sampling LCA probabilities.
As we show in the following, we can jointly compute LCA probabilities for all pairs of leaves in a vectorized way.
3. E FFICIENT VECTORIZED COMPUTATION
Deﬁnition 2 (Fundamental Matrix) .
Let M be an absorbing Markov chain and Q as in Deﬁnition 1.
The fundamental matrix N of Markov chain M is
[Equation]
N ij equals the expected number of visits of state j when starting a random walk in state i .
Since M is acyclic, each transient state can be visited at most once, i.e., N ij is the probability of state j being on a random path to the root starting from i .
Observing the block structure of N in Eq.
(7), we deﬁne
[Equation]
P anc ij =
p anc ( z j | v i ) is the probability of internal node
z j ending up being an ancestor of v i under the tree-sampling procedure.
Analogously, ˜ P anc provides the ancestor probabilities among the internal nodes, i.e. ˜ P anc ij =
p anc ( z j | z i ) .
Note that we subtract the identity matrix in Eq.
(8) since the diagonal entries of the fundamental matrix block ( I − B ) − 1 are always trivially 1, as they correspond to the probability of traversing an internal node z k when also starting from z k .
By subtracting I we obtain the ancestor probabilities by enforcing that an internal node is not its own ancestor.
Since B is strictly upper triangular, the inverse ( I − B ) − 1 always exists and is efﬁcient to compute in O ( n (cid:48) 2 ) .
Theorem 6.
The vector of LCA probabilities of all internal nodes w.r.t.
leaf nodes v i and v j can be computed in a vectorized way via
[Equation]
where (cid:12) denotes the element-wise (Hadamard) product.
(See proof in App. A.9)
Complexity analysis.
As we show in App.
A.10, we can exploit sparsity in real-world graphs and thus do not have to construct the full R n × n × n (cid:48)
LCA tensor P LCA .
This leads to a complexity of
O ( m × n (cid:48) 2 ) for both Soft-Das.
and Soft-TSD, which is efﬁcient since typically we have n (cid:48)
(cid:28) n .
All equations are vectorized and thus beneﬁt from GPU acceleration.
More details in App. A.12.
3. I NTEGRAL SOLUTIONS
In App.
A.13 we analyze the properties of the relaxed problem Soft-TSD which our method FPH optimizes.
We prove in Theorem 10 that the optimization problem is integral , i.e., the global maximum is discrete.
This is remarkable, since we are actually optimizing over continuous hierarchies parameterized by A and B .
This implies that the global maximum of the relaxed problem is the same as for the combinatorial problem of optimizing over discrete hierarchies.
Soft-Das. is not obviously convex or concave, thus not obviously integral.
3. F URTHER CONSIDERATIONS
Choice of n (cid:48) .
The number of internal nodes n (cid:48) is an important hyperparameter of our method (as well as most baselines).
Similar to, e.g., the number of clusters k in k -means, large numbers of internal nodes n (cid:48) lead to more expressive hierarchies, which on the other hand are less interpretable by a human and require more memory and computation.
In Fig.
2, we show how the expressiveness (as measured by TSD/Dasgupta) improves for increasing values of n (cid:48) , and in App.
B.6 we visualize learned hierarchies with different n (cid:48) .
Since FPH typically trains within a few minutes, our general recommendation is to use the elbow method (Thorndike, 1953) to determine n (cid:48) .
Constrained vs. unconstrained optimization.
Since our probabilistic hierarchy model leads to fully differentiable metrics (i.e., Soft-Das.
and Soft-TSD)
, we can optimize the metrics in an end-to-end fashion via gradient descent.
Note that the matrices A and B are constrained to be row-stochastic; we therefore experiment with two optimization schemes: unconstrained optimization of A and B (e.g. using Adam optimizer and softmax to obtain row-stochastic matrices), or constrained optimization via projected gradient descent (PGD).
In our ablation study we found that the PGD optimization consistently leads to better results (see Fig. 4).
We attribute this performance difference to the severe gradient re-scaling in the softmax operation when the parameters become very large or small, leading to very small step sizes (Niculae, 2020); thus, unless otherwise stated, FPH uses PGD optimization.
Scaling to large graphs.
For fast training on large graphs on commodity GPUs, we propose a simple yet effective batching scheme.
We uniformly pick K random leaves from the graph at each iteration.
Then we select the induced subgraph of these K nodes and their neighbors while capping the total number of nodes by some constant C .
We then compute the loss and perform the update based on the selected subgraph.
By using our batching procedure we do not need to have all parameters on the
GPU, which enables scaling to very large graphs.
Node embeddings.
In our default setting, we directly parameterize the matrices A and B as learnable parameters via gradient descent.
That is, the direct parent probabilities p ( z k | v i ) and p ( z k | z k (cid:48) ) are the only trainable parameters.
As an alternative, we also experiment with learning a node embedding for each leaf and internal node.
We compute the parent probabilities A and B via softmax on the negative Euclidean distances of the embeddings.
Given a ﬁxed embedding size, this leads to a parameterization that scales linearly in the number of leaf nodes and internal nodes.
Initialization.
We have found that our model (as well as most of the baselines) can greatly beneﬁt from a “smart” initialization scheme.
For the direct parameterization, we have found initializing from the solution obtained from the average linkage algorithm
(Jardine & Sibson, 1968) to work well.
Unless stated otherwise, FPH uses initialization from average linkage.
In contrast to vector data, linkage algorithms are fast on graph data and can be performed in O ( m ) (Benzécri, 1982; Murtagh &
Contreras, 2012), thus not affecting the complexity of FPH.
In the embedding parameterization, we experiment with initializing the embeddings using DeepWalk (Perozzi et al., 2014).
[Table]
38.79 ogbl
[Figure] collab 13,835 91, OOM 34,934 21,057 15,714 13,661 45.33 27.90 OOM 24. 48.02 DBLP 31,138 148,439 OOM 94,384 44,424 36,463 31,406 38.26 20.21 OOM 15.
[Figure]
Avg.
[Figure]
RGHC
HGHC
HypHC UF [Figure]  E XPERIMENTS
Datasets.
We use 11 real world datasets (McCallum et al., 2000;
Sen et al., 2008; Aspert et al., 2019;
Amunts et al., 2013; Cho et al., 2014; Adamic & Glance, 2005; Patokallio; Wang et al., 2020; Yang & Leskovec, 2015), including the very large ogbn-products dataset with around 2.3M nodes and 62M edges (Hu et al., 2020).
We always select the largest connected component (LCC) as a preprocessing step and convert each graph to an undirected one.
We provide further information about the datasets in Table 7, including number of nodes, number of edges and mutual information (MI) between nodes which is an upper-bound on the TSD score (Charpentier & Bonald, 2019).
In our experiments we report normalized TSD in percent.
Baselines.
We compare our method with the following deep learning baselines:
Routing Gradientbased clustering (RGHC) (Monath et al., 2017), Hyperbolic Gradient-based Clustering (HGHC) (Monath et al., 2019), Ultrametric Fitting (UF) (Chierchia & Perret, 2019), and HypHC (Chami et al., 2020).
Importantly RGHC, HGHC and HypHC also optimize a relaxed version of the Dasgupta cost.
Moreover, we compare to the average linkage algorithm (Avg. link.).
as well as the Ward linkage algorithm (Ward) (Ward Jr., 1963).
Finally, we compare to the Louvain method (Blondel et al., 2008) (Louvain) .
For RGHC, HGHC, and HypHC, which require vector data as input (as opposed to graphs), we use DeepWalk embeddings as node features.
Note that linkage algorithms return a full hierarchy with n (cid:48)
=
n − 1 .
Therefore we use the compression scheme introduced in (Charpentier & Bonald, 2019) to reduce the size of the hierarchies to the desired number of internal nodes.
See App.
B.7 for our hyperparameter choices.
For the results on Dasgupta and TSD, we train each baseline 5 times and report results for the best run.
1 For all models, we apply a post-processing treatment consisting in pruning unused internal nodes.
For FPH, we select for each node its most likely parent to obtain a discrete tree.
4. R ESULTS
Hierarchical clustering.
We report results on hierarchical clustering with n (cid:48)
= 512 internal nodes in Table 1.
Results for Louvain are in Table 9.
FPH outperforms all baselines on all datasets on the Dasgupta cost function as well as TSD.
Remarkably, the average linkage algorithm outperforms most of the deep-learning based baselines.
We partially attribute this to the fact that the baselines use DeepWalk embeddings as node attributes.
[Table] [Table] Moreover, we noticed that HypHC requires an excessive amount of triplet samples to obtain good results.
Even n 2 , as suggested by the authors, performs poorly.
We use 50 M triplets for all datasets, which is more than 4.5 times the recommended number of n 2 on Wiki-Physics.
This took almosthours to compute, which is why we could not go higher.
We clearly see a dependence of HypHC’s performance on the number of triplets, as it performs competitively on PolBlogs, our smallest dataset, and poorly on Wiki-Physics, the largest dataset on which we were able to run HypHC.
In Tbls.
10,we show baseline results with DW d = 128 embeddings, which did not improve results consistently.
In Figure 2 we show how the models perform for different numbers of internal nodes n (cid:48) .
As expected, the models generally obtain better scores for higher capacity.
Again, we observe that FPH performs best across datasets and values of n (cid:48) , highlighting the effectiveness of our approach.
Speciﬁcally on the Dasgupta cost, FPH substantially outperforms the baselines.
This is remarkable since RGHC,
HGHC and HypHC also optimize a relaxed version of the Dasgupta cost.
External evaluation.
As we typically have no knowledge about ground-truth hierarchies in realworld data, it is difﬁcult to perform external evaluation.
To address this, we propose the following:
(i)
We compute the normalized mutual information (NMI) between the ground-truth class labels and the learned hierarchies when cutting them appropriately to divide the leaves into the same number of (ﬂat) clusters.
We set n (cid:48)
= 256 for all models (except RGHC, since it ran OOM; we use n (cid:48)
=instead)
and set FPH to optimize TSD.
Note, though, that the node labels are not necessarily solely based on connectivity of nodes, but e.g., also on node attributes, to which the models do not have access.
Thus, we cannot expect perfect correlation of “good” hierarchies in the sense of explaining edges in the graph and NMI w.r.t.
ground-truth node labels.
In Table 2 (more datasets in Tab. 12) we show the results; the hierarchies learned by FPH achieve strong scores, highlighting its effectiveness.
(ii) Similarly, we can use the ground-truth clustering from synthetic hierarchical stochastic blockmodel (HSBM)
(Lyzinski et al., 2016)
graphs to compute the NMI scores at each level of the hierarchy.
We use ﬁve HSBM graphs with 100 and 1000 nodes, respectively.
The graphs have three levels (see
Figs.
6,7 for example graphs).
On the small graphs we set n (cid:48)
= 64 ; on the large graphs, we use the same settings as in (i) .
In Table 3 we show the results.
FPH outperforms all baselines and is able to recover the ground-truth hierarchies to a very high accuracy.
Ablation study.
In our ablation study, we compare (i) FPH
(i.e., with constrained optimization using PGD) vs. unconstrained optimization with Adam (FPH-U) and using softmax; (ii) average linkage initialization vs. random initialization (FPH-R, FPH-UR); (iii) direct optimization of A , B vs. learning node embeddings (FPH Emb.).
Here, we experiment with random initialization (FPH-R
Emb.) as well as DeepWalk initialization (FPH Emb. DW).
The results, obtained on the WikiPhysics dataset, are displayed in Figure 4 (App.).
We observe that FPH, i.e. constrained PGD optimization of A and B with initialization from average linkage performs best.
In a similar way, “smart” initialization from DeepWalk tends to improve the results on FPH Emb., too, but the effect is less pronounced.
In general, FPH variants learning embeddings perform worse than our full version; this may be due to the gradient-rescaling due to softmax, which we also suspect is a reason why unconstrained optimization achieves weaker results than FPH.
Scalability.
We run experiments on the very large graph dataset ogbn-products , which has about 2.3M nodes and 62M edges.
We train FPH both with n (cid:48)
= 512 and 1024 internal nodes performing batching as explained in Sec.
We show the results in Table 4.
FPH outperforms all baselines both on TSD as well as Dasgupta, and with both n (cid:48)
= 512 and 1024 .
Remarkably, FPH achieves more than 50% better Dasgupta results compared to the best baseline Ward, and cuts the average linkage [Table] [Table] results by a factor of more than 3 for n (cid:48)
= 1024 , highlighting that FPH outperforms all baselines by a large margin.
Note that we could not compare with HypHC, since (a) the implementation constructs a dense n × n matrix, which would require more than 14TB memory assuming single precision; (b) sampling n 2 triplets, as recommended by the authors, would take weeks.
Further, we do not report results on UF since it did not converge.
RGHC ran out of memory for n (cid:48)
= 1024 .
Runtime.
Due to the efﬁcient, vectorized computations, a full epoch of a complete dataset to compute
Soft-TSD or Soft-Dasgupta
using our model is very fast.
A complete forward pass on Wiki-Physics with 512 internal nodes takes only about 130ms on GPU and about 2s on CPU.
On the other hand, a full evaluation on ogbn-products takes only about 13 minutes (758s) on CPU, and training the model on a GPU takes approximately 3.5 hours for 2K epochs (with batching).
Link prediction.
As the TSD score can be interpreted in terms of a reconstruction loss, we can use the scores of the reconstruction scheme in Eq.
(25) (App. B.1) for predicting links.
We compare with
DeepWalk (DW) and the variational graph autoencoder (VGAE) (Kipf & Welling, 2016), where the link prediction score of two nodes is the cosine of their respective embeddings.
Further, we compare with degree-corrected stochastic blockmodels (DCSBM) (Karrer & Newman, 2011) and Adamic
Adar (Ad./Ad.)
(Adamic & Adar, 2001), which are established strong baselines for link prediction in non-attributed graphs.
See App.
B.2 for details.
Importantly, none of the models use node features, including FPH.
As shown in Table 5, FPH achieves best or second-best performance on all datasets.
This is remarkable, since FPH is not trained speciﬁcally for this task, which highlights the generality and usefulness of the hierarchies discovered by FPH.
[Table]
Qualitative analysis.
In Table 6 we provide insight into our model trained on the Cora-ML dataset.
We show the ﬁrst three levels of internal nodes of the learned hierarchy.
Each cell corresponds to an internal node and contains the three most frequent words of the abstracts of all papers (i.e., leaf nodes) assigned to the respective internal nodes.
Words from a node are excluded from its children to avoid duplicates.
Terms tend from more general at the root (e.g., ‘problem’) towards more speciﬁc at the lower levels (e.g., ‘mcmc’).
Further, we can identify categories of machine learning approaches, e.g., reinforcement learning (L2, row 2), symbolic AI (L2, row 4), or variational inference (L2, row 6).
Again, FPH did not see any text information (node attributes) and performed the clustering based on citations (edges) alone.
Besides offering qualitative evidence that the learned hierarchies are reasonable, this also showcases a potential real-world application of FPH: a scholar can explore the hierarchy of topics in an academic ﬁeld to discover relevant papers.
C ONCLUSION
We propose a new probabilistic model over hierarchies on graphs which can be learned using end-to-end gradient-based optimization.
By drawing connections to absorbing Markov chains we can compute relevant quantities such as lowest common ancestor probabilities exactly and efﬁciently in a vectorized way.
For the ﬁrst time, this allows to directly optimize for quality metrics for hierarchical clustering such as Dasgupta cost or Tree-Sampling Divergence (TSD) in an end-to-end fashion.
Our Flexible Probabilistic Hierarchy model outperforms strong traditional as well as recent deep-learning-based baselines on nearly all datasets and tasks considered and easily scales to massive graphs with millions of nodes.
E THICS STATEMENT
Since our method does not directly focus on a speciﬁc real-world application, good or bad societal outcomes depend on how practitioners and researchers use it in practice.
This means that it could potentially be abused, e.g., by corporations or governments, to identify groups and hierarchies of dissidents via recorded metadata.
For instance, phone call metadata is routinely collected at scale by service providers and governments.
By using our method on the large-scale call graph it could be possible to identify groups of political dissidents and to repress them.
On the other hand, we argue that our method can also have positive impact, e.g., by making corpora of literature more accessible to users or by enabling scientists, e.g., to discover cliques and hierarchies in biological networks.
Moreover, our contribution is algorithmic in nature and does not consider the effects of potential biases or discrimination in the underlying network data.
R EPRODUCIBILITY STATEMENT
For reproducibility and veriﬁability of our theoretical results, we provide complete proofs of all ten theorems of our work in Appendices A.3-A.13.
We further make explicit all assumptions and deﬁnitions we use to derive our results.
For reproducibility of our experimental results, we ﬁrst highlight that our model’s core implementation is a straightforward PyTorch implementation of the matrix equations in this work.
Further, we detail our experimental approach in Sec. 4 and provide hyperparameter choices for our method as well as the baselines in Table 8 in the appendix.
For the baselines, we use the authors’ ofﬁcial implementations and use the suggested hyperparameters.
To compute the Dasgupta and TSD metrics (as well as to obtain the results for the Louvain algorithm), we use the sknetwork Python library.
2 We share our codebase with the reviewers in a private comment on OpenReview; upon acceptance, we will make our implementation available to the general public.
A P ROOFS AND T HEORETICAL A NALYSIS A. C ONTINUOUS VERSIONS OF TSD AND D ASGUPTA
Our probabilistic hierarchy model enables us to replace the discrete parent assignments in the
Dasgupta cost and TSD with the parent probabilities from the relaxed adjacency matrices A and B .
This, in turn, leads to LCA probabilities which are consistent under the tree-sampling procedure.
For the ﬁrst time, this allows us to directly and efﬁciently optimize for hierarchical clustering quality metrics in an end-to-end fashion instead of proxy losses or heuristic algorithms.
Hence, we propose the soft Dasgupta cost (Soft-Das) by replacing the cardinality of the lowest common ancestor | v i ∧ v j | by the expected cardinality under the probabilistic model, i.e. c ( z )
=
(cid:80) v p anc ( z | v ) .
[Equation]
In the same way, we propose a differentiable version of the soft Tree Sampling Divergence (SoftTSD) .
To this end, we replace the discrete assignments in the distributions p ( z ) and q ( z ) with probabilistic assignments, i.e.
[Equation]
Note that for both metrics we recover the same score as their discrete formulations in the case of a deterministic probabilistic model, i.e. when A and B are binary-valued.
A. T REE - SAMPLING PROCEDURE
Recall our assumption that the internal nodes are ordered, and that B ij = 0 if j ≤ i .
This implies that there are no possible cycles, or equivalently that B is a strictly upper-triangular matrix, i.e., it describes a directed acyclic graph (DAG).
Combined with the fact that each node in a tree (except the root) has exactly one parent, we see that the sampled discrete hierarchy is indeed a tree.
We denote this tree-sampling process by T =
( ˆ A , ˆ B ) ∼ P A , B ( T ) .
We can also compute the probability of any tree T under the sampling procedure described above:
[Equation] [Equation]
Note that A i,j and B i (cid:48) ,j (cid:48) are the probabilities of internal nodes z j and z j (cid:48) to be a parent of leaf and internal nodes v i and z i (cid:48) respectively while ˆ A i,j and ˆ B i (cid:48) ,j (cid:48) are equal to 1 if these connections exist in the tree T , else 0 .
A. P ROOF OF T HEOREM
Proof.
For M to be absorbing (i) it must have at least one absorbing state, and (ii) at least one absorbing state must be reachable from any state in a ﬁnite number of steps.
For (i), ω is an absorbing state since its self-transition probability T k,k = 1 , where k = |S| + 1 is its corresponding index in the transition matrix T .
Thus, once reached, a random walk cannot leave the state ω .
To show (ii), we note that since the transient state transition matrix Q is a strictly upper-triangular matrix, any random walk on M must lead to state z n (cid:48) .
From state z n (cid:48) , the random walk transits to ω with probability w n (cid:48)
= 1 .
Thus, since n (cid:48) is ﬁnite, state ω can be reached from any state in a ﬁnite number of steps.
Further, ω is the only absorbing state since all self-transition probabilities of states in S are zero as diag ( Q ) = 0 .
Since Q is strictly upper-triangular, none of the transient states can be visited more than once on a random walk, and therefore M is acyclic.
A. P ROOF OF T HEOREM
Proof.
We can arbitrarily deﬁne the order in which we sample from the categorical distributions in
A and B because of the independence of the sampling steps.
We choose to start by sampling ﬁrst from A i , i.e., the row corresponding to the leaf node v i under consideration: w (1) ∼ Cat ( A i ) .
Next, we sample from the row corresponding to w (1) , and repeat until we reach z k , i.e. w ( t ) ∼ Cat ( B w ( t − 1) ) for 1 < t ≤ T, where w ( T ) =
z k .
For the remaining entries, we continue in arbitrary order.
Observe that ( w (1) , . . .
, w ( T ) ) are the ancestors of leaf v i in T .
Further observe that this sampling procedure is identical to how the path ˆ r i is generated in the random walk W ( v i ) , completing the proof.
A. P ROOF OF T HEOREM
Proof.
First, recall that all paths end in the root node, such that r i necessarily ends in z n (cid:48)
=
r ( T ) i .
When reasoning about the lowest common ancestors, it is no longer sufﬁcient to consider the ancestors (or, equivalently, path to the root) of a single leaf node in isolation.
Instead, we need to consider pairs of dependent paths ( r i , r j ) , i (cid:54) =
j rooted in v i and v j , respectively.
Note that r i and r j necessarily converge at some internal node z k
= v i ∧ v j — the latest at the root node z n (cid:48) .
Thus, we denote with r i =
( r (1) i , . .
. , v i ∧ v j ) the ﬁrst part of the path r i until (and including) its lowest common ancestor with r j , i.e., v i ∧ v j .
Analogously, r i =
( r ( | r i | +1)
i , . .
, z n (cid:48) ) , such that r i =
( r i , r i ) .
Further, note that the paths r i and r j are on the same underlying hierarchy T , thus r i =
r j , as both paths have the same trajectory to the root once they have reached their lowest common ancestor, i.e., they are dependent.
The probability of observing the pair of paths ( r i , r j ) under the tree-sampling perspective is
[Equation]
More compactly,
[Equation]
Importantly, we can see from Eq.
(12) that, in general, p ( T ) (( r i , r j )) (cid:54) =
p ( r i ) · p ( r j ) i.e., the paths r i and r j are not independent.
We denote p ( T ) anc ( z k | v i , v j ) the probability of the internal node z k to be the ancestor of leaf nodes v i and v j under the tree-sampling perspective.
Hence, the probability of the internal node z k to be the ancestor of leaf nodes v i and v j under dependent and independent random walks are different i.e. p ( T ) anc ( z k | v i , v j ) (cid:54) =
p anc ( z k | v i ) · p anc ( z k | v j ) .
This makes intuitive sense because knowing that z k (cid:48) is an ancestor of v i and v j in a tree T , additional knowledge that z k , k > k (cid:48) is an ancestor of v i implies that z k is also an ancestor of v j .
However, for the parts of r i and r j before they converge, Eq.
(12) shows that p (( r i , r j ))
=
p ( r i ) · p ( r j ) .
This is because all transitions in r i and r j are disjoint thus independent.
This is an important insight because it means that the probability of observing two paths both converging at an internal node z k factorizes.
A. P ROOF OF T HEOREM
Proof.
We start by reorganizing Eq.
[Equation]
In words, we can split the event “z k is an ancestor of v i and v j ” (i) into two mutually exclusive events: (ii) z k is the lowest common ancestor of v i and v j ; or (iii) some internal node lower in the topological order is the LCA of v i and v j , and further, both random walks also traverse through z k .
(ii) and (iii) are mutually exclusive since exactly one internal node is the LCA for v i and v j on any two random walks.
Since the two random walks are independent, the probability of z k being traversed on both walks factorizes.
Thus, p ( M ) anc ( z k | v i , v j ) = p anc ( z k | v i ) p anc ( z k | v j ) and therefore (i) is the probability of z k being an ancestor of v i and v j in our Markov chain M .
The events in (iii) can indeed be expressed:
[Equation]
where in the last step we exploit that z k (cid:48)
= v i ∧ v j implies z k (cid:
) ∈ anc ( v i , v j ) as well as the Markov property of the random walks.
Further, note that
[Equation]
) where we ﬁrst exploit factorization due to independence and in the last step again the Markov property of the random walks.
A. P ROOF OF T HEOREM
Proof.
Let ˆ r i ∈ P ( v i ) , ˆ r j ∈ P ( v j ) be two independent random walks on M rooted in v i and v j , respectively.
Then,
[Equation]
where ˆ r i =
(ˆ r (1) i , . . .
, z k ) is the ﬁrst part of ˆ r i until it reaches z k .
Note that the second part of the paths, ˆ r i and ˆ r j , which are theoretically independent under our Markov chain model, are marginalized out in the LCA formula.
However, due to the independence of the ﬁrst part of the paths ˆ r i and ˆ r j under both models (see
Eq.
(4)), we can write:
[Equation]
A. N UMBER OF PAIRS OF LCA PATHS .
Theorem 7.
Let M be a Markov chain as deﬁned in Deﬁnition 1.
The number of pairs of paths from two leaves v i , v j for which an internal node z k is the lowest common ancestor is 3 k − 1 .
Proof.
Proof by induction over k .
For the base case k = 1 we have one pair of paths for which z k is the LCA, i.e. directly from v i to z k and v j to z k .
Assume that for internal node z k there are 3 k −unique pairs of paths for which z k is the LCA.
For each of these paths we can generate three unique paths for which z k +1 is the LCA.
(1) rewire the last transition of v i ’s path to go to z k +1 instead of z k .
(2) do the same but for v j .
(3) rewire both v i ’s and v j ’s last transition to go to z k +1 instead of z k .
Thus, the number of pairs of paths from v i and v j to z k +1 is 3 · 3 k − 1 = 3 k .
A. P ROOF OF T HEOREM
We provide here the proof for the fast vectorized computation of P LCAv i , v j in Theorem. 6.
Proof.
We start by reorganizing Eq.
[Equation]
Note that the inverse ( I + ˜ P anc (cid:12) ˜
P anc ) − 1 is guaranteed to exist and is efﬁcient to compute because ˜ P anc (cid:12) ˜
P anc is a strictly upper triangular and therefore nilpotent matrix.
The k -th entry is thus:
[Equation]
Plugging in the deﬁnitions of Eq.
(8), using Theorem 2, and observing that due to the upper triangular structure ˜ P
anc k (cid:48) ,k = 0 for k (cid:48) >
k we obtain
[Equation]
A. V ECTORIZED COMPUTATIONS
All quantities involved in Soft-Das and Soft-TSD can be computed in closed-form based on the
Markov chain M and its fundamental matrix.
However, their computation should not be done naively, as this involves unnecessary computations.
Constructing the full tensor of LCA probabilities is expensive since P LCA ∈ R n × n × n (cid:48) .
Note, however, that to compute the Soft Dasgupta loss or the distribution p
( z ) in TSD we only require the LCA probabilities p ( z = v i ∧ v j ) for pairs of leaves connected by an edge (i.e., P ( v i , v j ) > 0 ).
That is, we only need to construct an LCA probability matrix of shape R m × n (cid:48) .
Thus, we can exploit the sparsity of real world graphs, as typically m (cid:28) n 2 .
In a similar way, the computation of the distribution q ( z ) does also not require the expensive explicit computation of p ( z k = v i ∧ v j ) for all pairs of leaf nodes.
Instead, we can again exploit insights from the Markov chain M .
First, observe that the equation of q ( z ) in Soft-TSD describes an expectation:
[Equation]
Deﬁning ˆ p anc =
p T · P anc , the computation of this expectation can be vectorized similarly to
Theorem 6 (see derivation in App.
A.11): q = (ˆ p anc (cid:12) ˆ
[Equation]
anc )
[Equation] [Equation]
+
[Equation]
˜ P anc ) − 1 .
A. V ECTORIZED q COMPUTATION .
We provide here the proof for the fast vectorized computation of q in Eq. 19.
Proof.
We ﬁrst rewrite the expectation Eq.
19 in vectorized form:
[Equation]
where we denote P ( v i )
= p v i .
Subsequently, we can plug the P LCA formula Eq.. (9) and pull p v i into the Hadamard product which is done over the internal node dimension:
[Equation]
where ˆ p anc =
p T · P anc .
A. C OMPLEXITY ANALYSIS .
Both Dasgupta and TSD computations require to compute the ancestor probabilities (Eq. 8) which can be done in O ( n × n (cid:48) 2 ) (i.e. inverse of triangular matrix ( I − B ) ∈ R n (cid:48) × n (cid:48) , plus matrix multiplication with A ∈ R n × n (cid:48) ).
Then, Soft-Das.
or the distribution p ( z ) for the Soft-TSD loss require the LCA probabilities (Eq. 9) for all leaves connected by an edge only, amounting to O ( m × n (cid:48) 2 ) operations, where m is the number of edges in the graph.
Note that similarly to Eq.
(8), the inverse computation in
Eq.
(9) can be done in O ( n (cid:48) 2 ) .
Additionally, Soft-TSD requires the computation of q ( z ) ( complexity
O ( n × n (cid:48) 2 ) ).
Both Soft-Das.
and Soft-TSD computations are dominated by the O ( m × n (cid:48) 2 ) term.
This leads to an efﬁcient time complexity as long as we assume a small number of internal nodes n (cid:48) (cid:28) n , which is reasonable in practice.
A. C ONVEXITY OF S OFT -TSD
Theorem 8.
Let H ∈
[0 , 1] n (cid:48)
× n × n be a tensor whose elements H kij
= p ( z k
= v i ∧ v j ) are the
LCA probabilities of internal nodes w.r.t.
pairs of leaf nodes.
Soft-TSD ( H ) is convex in H .
Proof.
Let H (1) , H (2) be two LCA probability tensors deﬁned as above, and 0 ≤ α ≤ 1 .
We ﬁrst compute the distribution p induced by the edge distribution.
We have that
[Equation]
Analogously we compute the distribution q induced by the independent ndoe distribution.
We have that
[Equation]
We combine this with the well-known fact that KL-divergence is convex w.r
[Equation]
t. pairs of distributions,
i
[Equation]
e.,
[Equation]
to obtain the desired result:
[Equation]
Theorem 9.
Let H FPH
=
{ H : ∃ A , B ∈ Φ( n, n (cid:48) ) :
H = FPH ( A , B ) } denote the set of probabilistic hierarchies which can be represented by FPH.
Here, Φ( n, n (cid:48) ) are the constraints FPH places on
A , B (see Sec. 2)
[Equation]
and FPH ( A , B ) is shorthand the mapping from transition matrices to lowest common ancestor probability tensors deﬁned in Theorems 4 and 6.
Here, H kij = p ( z k
= v i ∧ v j ) are the LCA probabilities of internal nodes w.r.t.
pairs of leaf nodes.
This set H FPH is convex.
Proof.
We start by recalling form Theorems 4 and 5 that FPH computes lowest common ancestor probabilities p ( z k = v i ∧ v j ) from the continuous parent probability matrices A and B such that
the
LCA probabilities are consistent with the expected result from the tree-sampling procedure described in Sec.
A.2.
More formally,
[Equation]
where ˆ H ∈ { 0 , 1 } n (cid:48) × n × n is a discrete hierarchy obtained via tree-sampling from A and B .
By deﬁnition of the expectation we write
[Equation]
where H ( n, n (cid:48) ) is the set of all valid discrete hierarchies with n leafs and n (cid:48) internal nodes.
Thus, any continuous hierarchy H learned by FPH is a convex combination of discrete hierarchies ˆ H .
This completes the proof.
Theorem 10.
The Soft-TSD optimization problem solved by FPH is integral.
That is, the global maximum of the Soft-TSD optimization problem solved by FPH is the same as the global optimum of the discrete optimization problem of optimizing TSD over discrete hierarchies.
Proof.
This follows from Theorems 8 and 9.
Theorem 8 establishes that the Soft-TSD objective function is convex in the hierarchy tensors H ; Theorem 9 proves that the set of hierarchies FPH optimizes over is convex.
When maximizing a convex function over a convex set, we are guaranteed to ﬁnd the global optimum at a vertex of the constraint set, which are discrete hierarchies in the case of FPH.
Thus, the global maximizer is a discrete hierarchy; this discrete hierarchy must also be the maximizer of the discrete TSD optimization problem, since our relaxation optimizes over a superset of all discrete hierarchies.
Implications of Theorems 8, 9, and 10.
In the previous theorems, we have shown that we are maximizing a convex function over a convex set.
In general, maximizing a convex function over a convex set is NP-hard (Benson, 1995).
Thus, we cannot hope to efﬁciently recover the global optimum.
However, our continuous relaxation brings several practical beneﬁts for the optimization.
First, observe that directly optimizing over the convex set of continuous hierarchies described in
Theorem 9 is not practical.
This is because there are exponentially many corners of the set, and encoding the constraints of the set is very difﬁcult.
Our parameterization of (continuous) hierarchies via A , B and being able to efﬁciently compute the expected lowest common ancestor probabilty tensor enables us to optimize over a fairly low-dimensional and convex set.
The constraints on A , B , i.e., entries in [0 , 1] , unit row sums and upper-triangular structure of B , are easy to encode and enforce during optimization.
This comes at the cost that mapping from A and B to the LCA tensor
H is nonconvex (yet describes, as per Theorem 9, a convex set over hierarchies).
Thus, we can solve the optimization problem with off-the-shelf methods such as projected gradient descent and beneﬁt from the elaborate techniques from nonconvex optimization.
While it is possible that FPH gets stuck in a non-discrete local optima during optimization, we can easily obtain a discrete and valid hierarchy given the non-discrete local optimizer via tree-sampling or selecting the most likely parent for all leaves and internal nodes under A and B , as described in Sec. 2.2.
B E XPERIMENT I NFORMATION B. L INK P REDICTION WITH S OFT -TSD
The TSD can be interpreted in terms of retrieved information when reconstructing the original graph from the tree representation (Charpentier & Bonald, 2019).
In this case, the reconstruction scheme for the edge weights of the reconstructed graph ˆ G is:
[Equation]
B. L INK PREDICTION SETUP
For all datasets, we randomly select 10% of edges to hold out for testing while making sure that the graph remains connected.
Further, we set n (cid:48)
= 256 and minimize Soft-TSD via FPH.
For DC-SBM, we use the Python package ‘graph-tool’ and follow the documentation 3 with default parameters to learn the model.
For VGAE, we use the default hyperparameters by the authors (one hidden layer, latent dimensions [32 , 16] , learning rate 0 .
01 , training for 200 epochs).
We use the variant described in the paper which replaces the node attributes by the n × n identity matrix.
For DeepWalk, we set the embedding dimension to 10.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
B. D ATASET SUMMARY
See Table 7 for an overview of the datasets we used.
B. A DDITIONAL RESULT FIGURES
In Fig.
3 we show results for four more datasets.
B. A BLATION STUDY
See Fig 4 for the comparison of different FPH model variants, and our full discussion in Sec. 4.1.
B. H IERARCHY V ISUALIZATION
We conduct a qualitative study of the structure discovered by FPH.
In Figure 5 we compare the hierarchies learned by FPH when optimizing for TSD or Dasgupta, respectively.
We cut at different levels of the dendrogram to obtain a coarse hierarchy (10 clusters) and ﬁne-grained structure ( clusters).
Comparing Figure 5 (a) and (d)
[Figure]
we notice that the coarse structure learned by optimizing
Soft-Dasgupta looks more appealing
[Figure]
as TSD essentially splits only into the Americas and the rest
[Figure] [Figure]
of the world. At 50 clusters, however, we observe the opposite: TSD splits the airports across the world into meaningful, coherent geographical regions, whereas Dasgupta looks mostly unchanged from the coarse version, highlighting the complementarity of both quality metrics.
In addition, the dendrogram learned by TSD in (f) appears to be of higher quality and more balanced than the
Dasgupta dendrogram in (c).
B. H YPERPARAMETERS
We use the hyperparameters for models and baselines described in Tab. 8.
Note that we train FPH for 1,000 epochs and restore the best hierarchy after training.
For ogbn-products we train for 2,000 epochs.
Further, to ensure convergence we reduce the learning rate to 0.02 and introduce weight decay of 1e-5 for FPH (Das.) for n (cid:48) ≥ 128 on Polblogs, Cora-ML, and Brain, and train for 2,000 epochs.
Similarly, we use different learning rates for A and B for FPH (Das.) on ogbn-arxiv, ogbl-collab, and DBLP ( lr A = 1 e − 2 , lr B = 1 e − 5 ).
B. C OMPUTING INFRASTRUCTURE
We train all models on a single GPU (NVIDIA GTX 1080 Ti or NVIDIA GTX 2080 Ti, 11 GB memory) in our own in-house compute cluster.
The machines have 10-core Intel CPUs.
We use
Python 3 and PyTorch for all our experiments.
B. HSBM GRAPHS
We generated HSBMs with n = 100 leaf nodes and n = 1000 leaf nodes for our external evaluation.
The small HSBMs have 3 levels with edge probabilities in [ . 01 , .
6] , a branching factor ofand core community sizes in [10 , 15] .
The large HSBMs have 3 levels with edge probabilities in [ . 001 , .
4] , branching factor in [2 , 3 , 4] and core community sizes in [30 , 35] .
In Fig.
6 and
Fig.
7, we plot one of the ﬁve synthetic HSBM graphs used in our experiments (right), and their corresponding dendrograms (left).
The graphs have n = 100 and n = 1000 leaf nodes and three levels of hierarchy. C A DDITIONAL RESULTS [Figure] [Table]
[Table]
[Table]
[Table]
[Table]