L EARNING WEIGHTED PRODUCT SPACES REPRESENTATIONS FOR G RAPHS OF H ETEROGENEOUS S TRUCTURES
The-Anh Ta FPT Software, AIC
Hanoi, Vietnam
A BSTRACT
Graph-like data are heterogeneous with different types of nodes and edges.
Heterogeneous structure in a graph means it has various structures such as trees, circles, and grid-like.
In representation learning, it is important to learn embedding spaces whose geometry matches the underlying structure of the data.
In the literature, an active research direction aims at using product spaces, which consists of Euclidean and non-Euclidean manifolds to represent data of varying curvatures.
However, real-world data are usually heterogeneous and consist of a mixture of varying structures, requiring the representation learning process to flexibly select and combine the member spaces accordingly.
Since previous works only consider a combination of embedding spaces with equal weights, in this paper, we propose a data-driven method to learn the embeddings in a weighted product space for graph data.
Specifically, our model utilizes the topological information of the input graph to learn the weight of each component of the product spaces.
Experiments on synthetic and real-world datasets show that our models produce better representations in terms of distortion measures, and perform better on tasks such as word similarity learning.
I NTRODUCTION
Representation learning aims to find parameterizations of data distribution manifolds of low intrinsic dimension from data samples into high dimensional feature spaces (Bengio et al., 2013).
Euclidean embedding spaces have been traditionally used in most representation learning models (Balazevic et al., 2019; Mikolov et al., 2013b).
However, due to their uniform geometric structure, it has been realized that Euclidean spaces cannot faithfully represent various types of structure data such as treelike (Nickel & Kiela, 2017) or circle-like (Wilson et al., 2014) graphs.
Thus, increasing attention has been paid to feature embedding into non-Euclidean spaces (Wilson et al., 2014; Nickel & Kiela, 2017; Chami et al., 2019;
Meng et al., 2019; Ganea et al., 2018).
Recent works (Gu et al., 2018; Skopek et al., 2020) also show promising results when learning representation in product spaces of components with different geometries.
This is mainly motivated by the fact that real-world graphs usually have varying patterns and complicated geometry, rather than being uniformly structured, while models with embedding spaces of the single geometric type usually fail in capturing the underlying mixed structures leading to higher distortion in representation (Gu et al., 2018).
Since existing models (Gu et al., 2018; Skopek et al., 2020) use product spaces with equally weighted components, training data samples are used to fit the learnable parameters in all these spaces evenly after the component spaces are determined.
We argue that this approach limits the robustness of models when learning data of mixed geometric structures since all components in product spaces are updated equally, even if the input data have a dominating geometric type compared to others.
As a result, it is difficult to appropriately adjust the geometry of product spaces to that of the input graph.
To address this problem, (Zhang et al., 2021) recently proposes a method called Switch Spaces which learns to choose a combination of K components from the total of N spaces with input specification.
Switch Spaces starts with two embedding points, then learns to choose the top K spaces whose combination is best suited for a specific scoring function.
However, with such a gating mechanism (Zhang et al., 2021) could be seen as a similarity learning between two graph nodes, thus Switch
Spaces cannot capture the overall relation of points from input data.
To deal with this problem, we propose a new data-driven approach with a new scoring mechanism for each component in the product spaces that enables automatic learning of the soft weights for each component space based on the overall structure of the entire input data.
More precisely, we introduce a new soft gating mechanism for automatic component spaces weighting and learning the weights of all component spaces from the input graph data.
Our method allows us to construct general product spaces with weighted signatures based on given input data, which is capable of learning better representation from data with varying geometric structures.
Our main contributions are summarized in the following.
Firstly, to our best knowledge, this work is the first one that discusses the problem of learning weighted product spaces from the entire input graph.
Secondly, we propose a new approach to learn embedding product spaces for representation learning via a soft gating mechanism in Section 2.
Thirdly, we perform experiments on synthetic and real-world datasets to validate our approach in Section 3.
P ROPOSED METHOD
In this section, we describe our new approach for adaptively learning weighted product spaces for representation learning.
Our goal is to construct product space models that make data with certain patterns be handled by suitable subsets of component spaces.
To this end, we introduce a new soft weight gating mechanism.
The main idea is to weigh the score for each component in embedding product spaces with specialization to enforce the curvature of the embedding to be approximate to the graph curvature.
Problem formulation.
Given three types of geometry Euclidean ( E ), Hyperbolic ( H ), and Spherical ( S ).
Let M 1 , M 2 , . .
, M N be N component spaces where M i ∈ { E , H , S } and dim M i =
b i .
The goal is to learn from the input data the weighting scores w = ( w 1 , . . .
, w N ) ∈ R N for each component space such that the embedding of data into P = w 1 M 1 × w 2 M 2 × · · · × w N M N will have lowest geometric distortion.
Weighting scores and graph coarsening.
To find the soft weight for each component space, we use the DiffPool architecture from (Ying et al., 2018) to distill topology information from the graph.
In particular, let S ( l ) ∈ R n l × n l +1 denotes the cluster assigment matrix at pooling layer l . Then
[Table] S ( l ) gives a soft assignment of each node at layer l to a cluster at the next layer l + 1 .
We treat the number of cluster as a hyperparameter and n l +1 = N .
Intuitively, each row of S ( l ) shows a soft margin of each node to each component space in P .
After the final pooling layer, we apply a multi-head attention block to get w ∈ R N which is the average weight of N spaces in P .
We refer to (Ying et al., 2018) for more detailed information on DiffPool.
Objective function and optimization.
Let w ∈ R N is the weight vector of N spaces are captured from A.3.
Our total average distortion objective function is defined as:
[Equation]
where d 2 P ( x i , x j ) =
N (cid:80) k
=1 w k dist 2 ( x ki , x kj ) , d G is the graph distance and L aux is a combinations of the link prediction ( L LP ) loss and entropy regularization ( L e ).
More precisely, L LP =
(cid:13)(cid:13)(cid:13)
A ( l ) − S ( l ) S ( l ) T (cid:13)(cid:13)(cid:13) F at each layer l , where ∥ · ∥ F denotes the Frobenius norm, and L e = 1 n n (cid:80) i =
1 H ( S i ) , where H ( S i ) is the entropy of the row i th in matrix S .
Minimizing L e makes the output cluster assignment for each node close to a one-hot vector so that the membership for each cluster is clearly defined.
Our objective function L ( x ) in (1) is optimized with the Riemannian gradient descent algorithm from (Gu et al., 2018, Algorithm 1).
E XPERIMENTS 3. E MBEDDINGS OF SYNTHETIC AND REAL - WORLD DATASETS .
Metrics and baselines.
We use the average distortion D avg and mean average precision mAP metrics whose formulas can be found in A.4 to measure the quality of graph embeddings.
Pure product spaces (Gu et al., 2018) is used as the state-of-the-art baselines for comparison with our models.
Results on synthetic datasets.
We first perform experiments of finding the best matching embedding spaces in total dimensions d = 3 and d = 5 for the three synthetic datasets Cycle, Tree, and Ring of trees introduced in Gu et al.
(2018), each of them is a graph of 40 nodes with canonical geometry.
The average distortion results are reported in Table 1.
Overall, our best models significantly outperform best models from Gu et al.
(2018) in terms of D avg .
For d = 3 , our method improves upon the pure product model by 18 .
6% and 16 .
3% on Ring of trees and Tree datasets, respectively.
When d = 5 , our models perform better than the models of Gu et al.
(2018) on Cycle and Ring of trees by 17 .
3% and 11 .
9% , respectively.
Results on real-world datasets.
We evaluate the quality of embeddings of our models on two realworld datasets with embeddings into spaces of 10 and 50 total dimensions.
The first dataset is Cs
PhDs , a graph of computer science Ph.D advisor-advisee relationships (de Nooy et al., 2011).
The second dataset is Power , a power distribution network (Watts & Strogatz, 1998).
[Table]
Results on the Cs PhDs and Power datasets are reported in Tables 2.
On both dimensions and both datasets, our best models learn better embeddings than the product spaces models of (Gu et al., 2018).
On the Cs PhDs dataset, our best models improve upon the best models of the pure product space approach (Gu et al., 2018) by 15 .
6% for d = 10 , and 19 .
3% for d = 50 in terms of average distortion, respectively.
Similarly, on the Power dataset, our models outperform best models of (Gu et al., 2018) by 28 .
4% and 13 .
9% on D avg for d = 10 , 50 , respectively.
We provide more experimental results on the quality of embeddings in the Appendix.
For the total embedding dimension d = 10 , our method finds that the optimal weighted product spaces to embed the Power dataset is 0 .
83 H 5 × 0 .
16 S 5 .
The rate between hyperbolic and spherical components is 0 .
16 ≈ 5 : 1 which is only possible to capture by our approach.
An heuristics way to estimate the component in the product of model space. The component in the pool of the product spaces is chosen based on the distortion value of a single space on the synthetic and benchmark dataset. For example, Table 1 witness the best distortion of S 3 on the Cycle dataset.
Thus, we can choose the product of model spaces that has spherical properties as a component in the product of model spaces.
3. P ERFORMANCE ON THE WORD SIMILARITY TASK
We evaluate our models on applications that require the understanding of the underlying manifold structure with a downstream task of learning word embeddings on the Word Similarity (WS-353) benchmark dataset as in previous works (Gu et al., 2018; Leimeister & Wilson, 2018).
Our implementation is based on the hyperbolic skip-gram embeddings from (Leimeister & Wilson, 2018).
Setup.
We use the standard skip-gram model (Mikolov et al., 2013a) and extend the loss function to a generic objective suitable for arbitrary manifolds, which is a variant of the objective used in (Leimeister & Wilson, 2018).
More precisely, given a word u and a target w with label y
=
if w is a context word for u , and y =
if it is a negative sample, the model is
P [Equation]
Word similarity.
We measure the Spearman rank correlation ρ between our scores and annotated ratings on the word similarity datasets WS-353 ((Finkelstein et al., 2001)) 1 .
The results are reported in Table 3.
In general, our models outperform both the hyperbolic word embeddings of (Leimeister & Wilson, 2018) and the pure product space models of (Gu et al., 2018) on both total dimension settings. [Table]
C ONCLUSION
Real-world data usually have complicated geometric structures which are difficult to capture by embedding into spaces of uniform curvature.
We introduce a data-driven method of weighted product spaces for learning better representation.
Our models improve both the embedding quality and downstream task performance compared to previous works
A A PPENDIX A. R ELATED WORK
In recent years, learning representation in non-Euclidean spaces has gained increasing interest.
For example, hyperbolic representation learning has shown state of the art performance on a number of tasks such as taxonomic entities modeling (Nickel & Kiela, 2017), network embeddings (Nickel &
Kiela, 2017), recommender systems (Vinh Tran et al., 2020), question answering (Tay et al., 2018), knowledge graphs completion (Balazevic et al., 2019; Chami et al., 2020b), graph-related classification (Chami et al., 2019;
Liu et al., 2019), similarity-based hierarchical Clustering (Chami et al., 2020a).
Hyperbolic space is better for the embedding of trees and excels in modeling hierarchical structures (Chami et al., 2020a), while spherical space is a more suitable choice for directional similarity modeling and data with cyclical structure (Meng et al., 2019; Wilson et al., 2014).
Recent works have applied spherical spaces in text embeddings (Meng et al., 2019), texture mapping (Wilson et al., 2014), time-warping functions embedding (Wilson et al., 2014).
As discussed above, different spaces have their own special geometric feature, and the choice of embedding spaces depends on the characteristics of the data.
The quality of such feature embeddings crucially depends on whether the geometry of the space matches that of the graph.
Euclidean spaces are often a poor choice for many types of real-world graph data, where one encounters hierarchical structure and power-law degree distribution due to negative curvature are linked to negative curvature (Chami et al., 2020a).
In particular, recent works (Chami et al., 2020b; Gu et al., 2018;
Zhang et al., 2021) show that hyperbolic spaces and more general manifolds, such as products of constant-curvature spaces and matrix manifolds, are advan- tageous to approximate better capture from graphs.
A product space is a direct product of embedding spaces with heterogeneous curvature (e.g., Euclidean space, hyperbolic space, spherical space, etc.)
(Zhang et al., 2021;
Gu et al., 2018).
Each component of product space has a constant curvature while the curvature of product space is the sum of curvatures of components which is still constant (Di Giovanni et al., 2022), which makes it possible to capture a wider range of curvatures with lower distortion than a single space.
Theoretically, product space is well-suited for modeling real-world data with a mixture of structures.
Empirically, product spaces have demonstrated their efficacy in graph reconstruction (Gu et al., 2018), wording embedding with low dimensions (Gu et al., 2018).
A. B ACKGROUND
Embeddings.
For metric spaces U, V equipped with distances d U , d V , an embedding is an injective continuous map f : U → V .
The quality of an embedding is measured by various fidelity measures.
A standard measure is the average distortion D avg ( f ) .
The distortion of a pair of points a, b is (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) d
V ( f ( a ) ,f ( b ))
d U ( a,b ) (cid:17)
2 − 1 (cid:12)(cid:12)(cid:12)(cid:12) , and D avg ( f ) is the average distortion over all pairs of points.
Distortion is a global metric; it considers the explicit value of all distances.
At the other end of the global-local spectrum of fidelity measures are the mean average precision (mAP), which applies to unweighted graphs.
Let G =
( V, E ) be a graph and node a ∈ V have neighborhood
N a = { b 1 , · · · , b deg(a) } , where deg ( a ) is the degree of a .
In the embedding f , define R a,b i to be the smallest ball around f ( a ) that contains b i which means R a,
b
i is the smallest set of nearest points required to retrieve the i th neighbor of a in f .
Thus, define mAP( f ) = 1 | V | (cid:80) a ∈ V 1 deg( a ) |N a | (cid:80) i
= |N a ∩ R a,b | | R a,bi | .
mAP ( f ) is a ranking-based measure for local neighborhoods; it does not track explicit distances like a distortion measure.
We see that, mAP ( f ) ≤ 1 (higher is better) while D avg ≥ 0 (lower is better) (Gu et al., 2018).
Riemannian manifolds.
Let M be a smooth manifold,
p ∈ M be a point, and T p M be the tangent space to the point p .
If M is equipped with a Riemannian metric g , then the pair ( M, g ) is called a
Riemannian manifold.
The shortest-distance paths on manifolds are called geodesics.
To compute distance functions on a Riemannian manifold, the metric tensor g is integrated along the geodesic.
This is a smoothly varying function (in p ) g p ( ., . )
T p M × T p M → R that induces geometric notions such as length and angle by defining an inner product on the tangent space.
For example, the norm of v ∈ T p M is defined as ||
v ||
g :=
g p ( v, v ) 12 .
In Euclidean space R d , each tangent space
T p R d is canonically identified with R d , and the metric tensor g E is simply the usual inner product.
We refer to (Willmore, 2013; Petersen, 2006) for a formal introduction of Riemannian geometry.
Product manifolds.
Consider a sequence of smooth manifolds M 1 , M 2 , · · · , M k .
The product manifold is defined as the Cartesian product M = M 1 × M 2 × · · · × M k .
We write points p ∈ M through their coordinates p
=
( p 1 , . . .
, p k ) , p i ∈ M i and similarly a tangent vector v ∈ T p M can be written ( v 1 , . .
, v k ) : v i ∈
T p i M i .
If the M i are equipped with metric tensor g i , then the product M is also Riemannian with metric tensor g p
[Equation]
Geodesics and distances.
Gradient descent algorithms on manifolds requires a notion of taking a step.
This step can be performed in the tangent space and transferred to the manifold via the exponential map Exp p : T p M
→ M (Willmore, 2013).
In a product manifold P , for tangent vectors v = ( v 1 , . . .
, v k ) at p
=
( p 1 , . . .
, p k ) ∈ P , the exponential map simply decomposes, as do squared distances:
[Equation]
In other words, the shortest path between points in the product travels along the shortest paths in each component simultaneously.
Note the analogy to Euclidean products
R d ≡ (cid:0) R 1 (cid:1) d .
Hyperbolic and spherical models.
We use the hyperboloid model of hyperbolic space with points in R d +1 .
Let J ∈ R
( d +1) ×
( d +1) be the diagonal matrix with J
= − 1 and J ii = 1 : i > 0 .
For p, q ∈ R d +1 , the Minkowski inner product is ⟨ p, q ⟩ ∗ :=
p T Jq = − p
q 0 + p 1 q 1 + . . .
+ p d q d , and the corresponding norm is || p
|| ∗ = ⟨ p, p ⟩ 12 ∗ .
For any K > 0 , the hyperboloid H dK is defined as the subset (cid:8) p ∈
R
d +1 : ∥
p ∥ ∗ = − K 1 / 2 , p 0 > 0 (cid:9) .
When the subscript K is omitted, it is taken to be 1.
The hyperbolic distance on H d is d H ( p, q ) =
acosh ( −⟨ p, q ⟩ ∗ ) where acosh () is the inverse hyperbolic cosine function (Willmore, 2013; Petersen, 2006).
Similarly, spherical space S dK is most easily defined when embedded in R d +1 .
The manifold S dK is defined as the subset
p ∈ R d
+1 :
|| p ||
= K 12 , with metric tensor g S is induced by inner product on the tangent space at each point only.
The spherical distance on S d is d S ⟨ p, q ⟩ = arccos ( ⟨ p, q ⟩ ) where arccos () is the inverse cosine function (Willmore, 2013; Petersen, 2006).
A. W EIGHTED PRODUCT MANIFOLDS AND GRAPH COARSENING .
We aim to obtain the weight for each model which reflects how much the component space contributes to the product of the model spaces.
For example, in the case of the Ring of tree dataset which has a lot of tree structures and one ring, we can think that it will be better if we evaluate the score for the hyperbolic model is bigger than the spherical model.
In this work, to get the soft weight information for all the spaces, we use the DiffPool (Ying et al., 2018) architecture to distill topology information from the graph.
Formally, let Z =
GNN ( A, X ) ∈ R n × d be the output of a
GNN module, where A ∈ R n × n is a graph adjacency matrix.
DiffPool defines a strategy to output a new coarsened graph with m < n nodes and a new weighted adjacency matrix A ′ ∈ R m × m and node embeddings Z
′ ∈
R m × d .
DiffPool has the interactive property which means that the new coarsen graph can turn to be used as input to another DiffPool layer, and this whole process can be repeated L times, generating a model with L GNN layers that operate on a series of coarser and coarser versions of the input graph.
For specific, S ( l ) ∈ R n l × n l +1 denotes the cluster assigment matrix at layer l .
S ( l ) gives a soft assignment of each node at layer l to a cluster at the next layer l + 1 .
In other words, each row of
S ( l ) corresponds to one of the n l nodes (or clusters) at layer l , and each column of S ( l ) corresponds to one of the n l +1 clusters at the next layer l + 1 .
In our work, we treat the number of cluster as a hyperparameter and n l +1 = N , where N is the number of components in the product space P .
Intuitively, each row of S ( l ) shows a soft margin of each node to each each component space in P .
At the end, we use a multihead attention pooling to achieve w ∈ R N which is presented for the average weight of N spaces in P over the n l clusters.
We refer to (Ying et al., 2018) for more information.
A. M ETRIC
We use two common metrics average distortion D avg , and mean average precision mAP to measure the quality of embeddings.
Distortion measure.
Distortion measure is used to quantify how far an embedding map is from an isometry.
Assume we have a set data points P = { p 1 , . .
, p N } ⊂ X .
An embedding map f : X → Y is called D -embedding, where D > 1 , and d X ( ., . ) is the distance in the input space (Euclidean or graph distance) and d Y ( ., . )
is the distance in the embedding spaces.
The total distortion is defined as:
[Equation]
Mean average precision.
Mean average precision ( mAP ) is used as a local measure of fidelity for embeddings of unweighted graphs.
Let G =
( V, E ) be a graph and node a ∈ V have neighborhood
N a = { b 1 , . . .
, b deg ( a ) } where deg ( a ) is the degree of a.
In the embedding f , define R a,b i to be smallest ball around f ( a ) that contains b i (that is, R a,
b i is the smallest set of nearest points required to retrieve the i -th neighbor of a in f ).
The formula of mAP is:
[Equation]
[Table] [Table] mAP ( f ) is a ranking-based measure for local neighborhoods; it does not track explicit distances like a distortion measure.
A. P ARAMETER SETTINGS
We choose the best parameters for each model.
Several hyper-parameters having a major impact on the models are lr (learning rate); λ (to balance the influence of auxiliary loss).
We run 500 epochs and 5000 epochs with the synthetic and benchmark datasets respectively.
We turn hyper-parameters with lr in { 0 . 001 , 0 . 003 , 0 .
01 } for Cities and three synthetic graphs.
lr is searched in the range { 1 , 3 , 5 , 10 } with the remaining dataset.
λ is searched in range { 0 . 5 , 1 , 1 . 5 , 2 , 3 } .
The tuning process gives us lr = 0 .
005 for Cities graph, lr = 0 .
01 for the three synthetic graphs, and lr = 5 for the two remaining benchmark datasets.
We choose λ = 1 . 5 for the best distortion.
In the weighted module, we use DiffPool with two GraphSAGE (Hamilton et al., 2017) pooling layers.
A. A DDITIONAL EXPERIMENTAL RESULTS
[Table] [Table]
[Table] [Table]