R OBUST G RAPH D ICTIONARY L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional Dictionary Learning (DL) aims to approximate data vectors as sparse linear combinations of basis elements (atoms) and is widely used in machine learning, computer vision, and signal processing.
To extend DL to graphs,
Vincent-Cuaz et al.
2021 propose a method, called GDL, which describes the topology of each graph with a pairwise relation matrix (PRM) and compares
PRMs via the Gromov-Wasserstein Discrepancy (GWD).
However, the lack of robustness often excludes GDL from a variety of real-world applications since GWD is sensitive to the structural noise in graphs.
This paper proposes an improved graph dictionary learning algorithm based on a robust Gromov-Wasserstein discrepancy (RGWD) which has theoretically sound properties and an efﬁcient numerical scheme.
Based on such a discrepancy, our dictionary learning algorithm can learn atoms from noisy graph data.
Experimental results demonstrate that our algorithm achieves good performance on both simulated and real-world datasets.
I NTRODUCTION
Dictionary learning (DL) seeks to learn a set of basis elements (atoms) from data and approximates data samples by sparse linear combinations of these basis elements (Mallat, 1999; Mairal et al., 2009;
To ˇ si ´ c and Frossard, 2011), which has numerous machine learning applications including dimensionality reduction (Feng et al., 2013; Wei et al., 2018), classiﬁcation (Raina et al., 2007; Mairal et al., 2008), and clustering (Ramirez et al., 2010; Sprechmann and Sapiro, 2010), to name a few.
Although DL has received signiﬁcant attention, it mostly focuses on vectorized data of the same dimension and is not amenable to graph data (Xu, 2020; Vincent-Cuaz et al., 2021; 2022).
Many exciting machine learning tasks use graphs to capture complex structures (Backstrom and Leskovec, 2011; Sadreazami et al., 2017; Naderializadeh et al., 2020; Jin et al., 2017; Agrawal et al., 2018).
DL for graphs is more challenging due to the lack of effective means to compare graphs.
Speciﬁcally, evaluating the similarity between one observed graph and its approximation is difﬁcult, since they are often with different numbers of nodes and the node correspondence across graphs is often unknown (Xu, 2020; Vincent-Cuaz et al., 2021).
The seminal work of Vincent-Cuaz et al.
(2021) proposes a DL method for graphs based on the Gromov-Wasserstein Discrepancy (GWD) that is a variant of the Gromov-Wasserstein distance.
Gromov-Wasserstein distance compares probability distributions supported on different metric spaces using pairwise distances (M´emoli, 2011).
By expressing each graph as a probability measure and capturing the graph topology with a pairwise relation matrix (PRM), comparing graphs can be naturally formulated as computing the GWD, since both the node correspondence and the discrepancy of the compared graphs are calculated (Peyr´e et al., 2016; Xu et al., 2019b; Chowdhury and M´emoli, 2019).
However, observed graphs often contain structural noise including spurious or missing edges, which leads to the differences between the obtained PRMs and the true ones (Donnat et al., 2018; Xu et al., 2019b).
Since GWD lacks robustness (S´ejourn´e et al., 2021; Vincent-Cuaz et al., 2022; Tran et al., 2022), the inaccuracies of PRMs may severely affect GWD and the effectiveness of DL in real-world applications.
Contributions.
To handle the inaccuracies of PRMs, this paper ﬁrst proposes a novel robust
Gromov-Wasserstein discrepancy (RGWD) which adopts a minimax formulation.
We prove that the inner maximization problem has a closed-form solution and derive an efﬁcient numerical scheme to approximate RGWD.
Under suitable assumptions, such a numerical scheme is guaranteed to ﬁnd a δ -stationary solution within O ( 1 δ 2 ) iterations.
We further prove that RGWD is lower bounded and the lower bound is achieved if and only if two graphs are isomorphic.
Therefore, RGWD can be employed to compare graphs.
RGWD also satisﬁes the triangle inequality which is of its own interest and allows numerous potential applications.
A robust graph dictionary learning (RGDL) algorithm is thereby developed to learn atoms from noisy graph data, which assesses the quality of approximated graphs via RGWD.
Numerical experiments on both synthetic and real-world datasets demonstrate that RGDL achieves good performance.
The rest of the paper is organized as follows.
In Sec. 2, a comprehensive review of the background is given.
Sec.
3 presents RGWD and the numerical approximation scheme for RGWD.
RGDL is delineated in Sec. 4.
Empirical results are demonstrated in Sec.
We ﬁnally discuss related work in Sec. 6.
P RELIMINARY 2. O PTIMAL T RANSPORT
We ﬁrst present the notation used throughout this paper and then review the deﬁnition of the
Gromov-Wasserstein distance that originates from the optimal transport theory (Villani, 2008; Peyre and Cuturi, 2018).
Notation.
We use bold lowercase symbols (e.g. x ), bold uppercase letters (e.g. A ), uppercase calligraphic fonts (e.g. X ), and Greek letters (e.g. α ) to denote vectors, matrices, spaces (sets), and measures, respectively.
1 d ∈ R
d is a d -dimensional all-ones vector.
∆ d is the probability simplex with d bins, namely the set of probability vectors ∆ d = (cid:8) a ∈ R d + | (cid:80) di
=1 a i = 1 (cid:9) .
A [ i, :] and A [: , j ] are the i th row and the j th column of matrix A respectively.
Given a matrix A , we denote by (cid:107) A (cid:107) F its Frobenius norm, by (cid:107) A (cid:107)
its number of non-zero elements, and by (cid:107) A (cid:107) ∞ its element-wise (cid:96) ∞ norm (i.e., (cid:107) A (cid:107) ∞ =
max ij | A ij | ).
The cardinality of set A is denoted by |A| .
The bracketed notation (cid:74) n (cid:75) is the shorthand for integer sets { 1 , 2 , . .
. , n } .
A discrete measure α can be denoted by α = (cid:80) mi =1 p i δ x i where δ x is the Dirac at position x , i.e., a unit of mass inﬁnitely concentrated at x .
Gromov-Wasserstein distance.
Optimal Transport addresses the problem of transporting one probability measure towards another probability measure with the minimum cost (Villani, 2008;
Peyre and Cuturi, 2018).
The induced cost deﬁnes a distance between the two probability measures.
Gromov-Wasserstein (GW) distance extends classic optimal transport to compare probability measures supported on different spaces (M ´ emoli, 2011).
Let X
and Y be two sample spaces.
Endowing the spaces X and Y with metrics (distances)
d X and d Y , the r-GW distance between α = (cid:80) mi =1 p
i
δ x i and β = (cid:80) ni (cid:48)
=1 q i
δ x i is deﬁned as
[Equation]
where the feasible domain of the transport plan T =
[ T ii (cid:48) ] is given by the set
[Equation]
and D ii (cid:48)
jj (cid:48) calculates the difference between pairwise distances, i.e., D ii (cid:48)
jj (cid:48)
=
| d X ( x i , x j ) − d Y
( y i (cid:48) , y j (cid:48) )
| with x 1 , x 2 , . . .
, x m ∈ X and y 1 , y 2 , . .
. , y n ∈ Y .
G RAPH R EPRESENTATION AND C OMPARISON
In this subsection, we formalize the idea of comparing graphs with GWD, which addresses the challenges that graphs are often with different numbers of nodes and the node correspondence is unknown (Xu et al., 2019b; Xu, 2020; Vincent-Cuaz et al., 2021).
Pairwise relation and graph representation.
Given a graph G with n nodes, assigning each node an index i ∈ (cid:74) n (cid:75) , G can be expressed as a tuple ( C , p ) , where C ∈ R n × n is a matrix encoding the pairwise relations (e.g. adjacency, shortest-path, Laplacian, or heat kernel) and p ∈ ∆ n is a probability vector modeling the relative importance of nodes within the graph (Peyr´e et al., 2016;
Xu et al., 2019b; Titouan et al., 2019; Vincent-Cuaz et al., 2022).
Gromov-Wasserstein Discrepancy GWD can be derived from the 2 -GW distance by replacing the metrics with pairwise relations (Xu et al., 2019b; Vincent-Cuaz et al., 2022).
More speciﬁcally, given the observed source graph G s and the target graph G t that can be expressed as ( C s , p s ) and ( C t , p t ) respectively, GWD is deﬁned as
[Equation]
where n s and n t are the numbers of nodes of G s and G t respectively.
GWD computes both a soft assignment matrix between the nodes of the two graphs and a notion of discrepancy between them.
For conciseness, we abbreviate GWD (cid:0) ( C s , p s ) , ( C t , p t ) (cid:1) to GWD( C s , C t ) in the sequel.
2. D ICTIONARY L EARNING
Traditional DL approximates data vectors as sparse linear combinations of basis elements (atoms)
(Mallat, 1999; Mairal et al., 2009; Toˇsi´c and Frossard, 2011; Jiang et al., 2015), and is usually formulated as
[Equation]
where X ∈ R d × K is the data matrix whose columns represent samples, the matrix D ∈ R d × M contains M atoms to learn and is constrained to the following set
[Equation]
W ∈ R M × K is the new representation of data whose k th -column w k
=
[ w km ] m ∈ (cid:74) M (cid:75) stores the embedding of the k th sample, and λ Ω( w k ) promotes the sparsity of w k .
Such a formulation only applies to vectorized data.
Recently, Xu 2020 proposes to approximate graphs via the highly non-linear GW barycenter.
Specifically, given a dataset of K graphs which has PRMs { C k } k ∈ (cid:74) K (cid:75) such that C k ∈ R n k × n k , the basis elements {
[Equation]
are learned by solving
[Equation]
where w k ∈ ∆ M is referred to as the embedding of the k th graph G k , and the GW barycenter
B (cid:0) w k , { ¯ C m } m ∈ (cid:74) M (cid:75) (cid:1) gives the approximation of G k and is deﬁned as
[Equation]
Therefore, a complex bi-level optimization problem is involved, which is computationally inefﬁcient (Vincent-Cuaz et al., 2021).
DL for graphs.
To overcome the above computational issues, Vincent-Cuaz et al.
2021 propose
GDL which approximates each graph as a weighted sum of PRMs and is formulated as
[Equation]
where each atom ¯ C m is a n a × n a matrix.
In contrast to the (cid:96) 2 loss in Eq.
(1), GWD is used to assess the quality of the linear representation (cid:80) M m =1 w km ¯ C m for k ∈ (cid:74) K (cid:75) .
However, the observed graphs often contain noisy edges or miss some edges in real-world applications (Clauset et al., 2008; Xu et al., 2019b; Shi et al., 2019; Piccioli et al., 2022), which leads to the inaccuracies of the PRMs C k , that is, the deviation between C k and the true PRM C k ∗ .
Since GWD lacks robustness (S´ejourn´e et al., 2021; Vincent-Cuaz et al., 2022; Tran et al., 2022), the quality of the learned dictionary may be severely affected.
R OBUST G ROMOV -W ASSERSTEIN D ISCREPANCY
To deal with the inaccuracies of PRMs, this section ﬁrst deﬁnes a robust variant of GWD, referred to as RGWD.
The properties of RGWD are then rigorously analyzed.
We then derive a theoretically guaranteed numerical scheme for calculating RGWD approximately.
Due to the limit of space, all proofs can be found in the appendix.
Deﬁnition
Given the observed source graph G s and the target graph G t that can be expressed as ( C s , p s ) and ( C t , p t ) respectively, RGWD is deﬁned by the solution to the following problem
[Equation]
where the objective f ( · ) is given by
[Equation]
and the perturbation E is in the bounded set U (cid:15) =
{ E | E
= E (cid:62) and (cid:107) E (cid:107) ∞ ≤ (cid:15) } .
RGWD requires the sought transport plan to have low transportation costs for all perturbation E in set U (cid:15) .
For succinctness, we omit C s and C t in f ( T , E ; C s , C t ) in the following.
3. P ROPERTIES OF RGWD
The properties of RGWD are presented as follows.
Firstly, although RGWD involves a non-convex non-concave minimax optimization problem, the inner maximization problem has a closed-form solution, which allows an efﬁcient numerical scheme for RGWD.
Secondly, RGWD has a lower bound that is achieved if and only if the expressions of compared graphs are identical up to a permutation, which implies RGWD can be employed to evaluate the similarity between one observed graph and its approximation in DL.
Thirdly, RGWD satisﬁes the triangle inequality, which allows numerous potential applications including clustering (Elkan, 2003; HajKacem et al., 2019), metric learning (Pitis et al., 2019), and Bayesian learning (Moore, 2000; Xiao et al., 2019). Finally, arbitrarily changing the node orders does not affect the value of RGWD.
More formally, we state the properties in the following theorem.
Theorem 1 Given two observed graphs G s =
( C s , p s ) and G t =
( C t , p t ) with n s and n t nodes respectively, RGWD satisﬁes
for all T ∈ Π( p s , p t ) , E ( T ) = [ E i
[Equation]
j (cid:48) ( T )]
where
[Equation]
−
[Equation]
otherwise.
solves the inner maximization problem
[Equation]
RGWD is lower bounded, that is,
[Equation]
where the equality holds if and only if there exists a bijective π ∗ :
[Equation] n s
[Equation] → (cid:74) n t (cid:75) such that p s [Equation] [Table]
The triangle inequality holds for RGWD,
i
[Equation]
e.,
[Equation]
RGWD is invariant to the permutation of node orders, i
e., for all permutation matrices Q s and
[Equation] As is implied by Theorem 1, RGWD does not deﬁne a distance between metric-measure spaces.
Firstly, the identity axiom is not satisﬁed.
Secondly, the symmetry generally does not hold either, which we exemplify below.
Example 1 (Asymmetry of RGWD) Consider the case p s =
p t =
5 ] , C s =
[ 0 1 1 0 ] , C t =
Then RGWD (cid:0) ( C s , p s ) , ( C t , p t ) , 1 (cid:1) = 11 .
5 with the solution given by T ∗ =
25 ] and E ∗ =
[ − 1 1 1 − 1 ] .
In contrast, RGWD (cid:0) ( C t , p t ) , ( C s , p s ) , 1 (cid:1) = 10 . 5 with T ∗ =
25 ] and
E ∗ =
[ − 1 − 1 − 1 − 1 ] .
One has RGWD (cid:0) ( C s , p s ) , ( C t , p t ) , 1 (cid:1) (cid:54) = RGWD (cid:0) ( C t , p t ) , ( C s , p s ) , 1 (cid:1)
Example 1 showcases that RGWD is asymmetric even if n s = n t .
3. N UMERICAL S CHEME OF RGWD
We derive a gradient based numerical scheme to solve RGWD by exploiting the property that the inner maximization problem has a closed-form solution, which is summarized in Algorithm 1.
In each iteration, E τ that solves the inner problem for current T τ is calculated.
Then, the transport plan is updated using the projected gradient descent. Update the transport plan via
[Equation]
where the partial gradient takes the form
[Equation]
denoting the -wise multiplication. To present the convergence guarantee of Algorithm 1, we introduce the notion of the Moreau envelope.
The stationarity of any function h ( x ) can be quantiﬁed by the norm of the gradient of its
Moreau envelope h λ ( x ) = min x (cid:48) h ( x (cid:48) ) + 12 λ (cid:107) x − x (cid:48)
(cid:107) 2 .
The following theorem gives the convergence rate of Algorithm 1 and the proof is deferred to the appendix.
Theorem 2 Deﬁne φ ( · ) = max E ∈U (cid:15) f
( · , E ) .
The output ˆ T of Algorithm 1 with step-size η = γ √ N + satisﬁes
[Equation]
where l = √ 2 max { 10 n 3 U 21 + 6 n 3 U 1 (cid:15) + 4 nU 1 U 2 + 4 n 3 (cid:15) 2 , 6 n 2 U 1 U 2 + 2 U 22 + 4 n 2 U 2 (cid:15) } and L = √ 2 max { (4 U 1 +2 (cid:15) )
U 22 n 3 , 2(2 U 1 + (cid:15) )
2 U 2 n 3 } with n = max { n s , n t } , U 1 = max {(cid:107) C s (cid:107) ∞ , (cid:107) C t (cid:107) ∞ } and U 2 = max {(cid:107) p t (cid:107) 2 , max T (cid:48)
∈ Π( p s , p t ) (cid:107) T (cid:48)
(cid:107) F } .
When U 1 and (cid:15) are of the order
O ( 1 n 3 ) , both l and L are of the order O (1) and Theorem 2 states that an δ -stationary solution can be obtained within O ( 1 δ 2 ) iterations.
Note that we can multiply C s , C t , and (cid:15) by the same number without affecting the resulted transport plan.
R OBUST G RAPH D ICTIONARY L EARNING
The problem of learning a robust dictionary for graph data is now formulated as follows.
Given a dataset of K graphs expressed by { ( C k , p k ) } k ∈ (cid:74) K (cid:75) , estimating the optimal dictionary is formalized by
[Equation]
where { ¯ C m } m ∈ (cid:74) M (cid:75) and { w k } k ∈ (cid:74) K (cid:75) are the dictionary and graph embeddings respectively, and ¯ p is obtained by sorting and averaging { p k } k ∈ (cid:74) K (cid:75) following Xu et al. (2019a).
To resolve (2), we propose a nested iterative optimization algorithm that is summarized in Algorithm 2.
The main idea is that the dictionary and embeddings are updated alternatingly.
We discuss some crucial details below. [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
2: Output: The learned dictionary { ¯ C m } m ∈ (cid:74) M (cid:75) .
3: for t = 0 , 1 , . . .
, T − 1 do 4:
Sample a mini-batch of graphs whose indices are denoted by B such that |B| = b .
5: for k ∈ B do 6: Initialize w k = M 1 M and T k
= ¯ pp k (cid:62) . 7: repeat 8: Calculate ( T k , E k ) via Algorithm 1 with ﬁxed w k . Compute w k solving (4) for the ﬁxed T k and E k with conditional gradient.
10: until Convergence 11: end for 12:
Update the atom ¯ C m for m ∈
[Equation]
M
[Equation]
with stochastic gradient ˆ ∇ ¯ C m which has the form
[Equation]
[Equation]
13: end for
Solving w k .
We now formulate the problem of obtaining the embedding of the k th graph G k when the dictionary is ﬁxed and the PRM is inaccurate.
Given dictionary { ¯ C m } m ∈ (cid:74) M (cid:75) where each ¯ C m ∈
R n a × n a , the embedding of G k expressed by ( C k , p k ) is calculated by solving
[Equation]
where λ ≥ 0 induces a negative quadratic regularization promoting sparsity on the simplex (Li et al., 2020; Vincent-Cuaz et al., 2021).
When w k is ﬁxed, updating T k and E k can be solved by
Algorithm 1 whose convergence is guaranteed by Theorem 2.
For ﬁxed T k and E k , the problem of updating w k is a non-convex problem that can be tackled by a conditional gradient algorithm.
Note that for non-convex problems, the conditional gradient algorithm is proved to converge to a local stationary point (Lacoste-Julien, 2016).
Such a procedure is described from Line 5 to Line 11 in
Algorithm 2, which we observe converges within tens of iterations empirically.
Stochastic updates.
To enhance computational efﬁciency, each atom is updated with stochastic estimates of the gradient.
At each stochastic update, b embedding learning problems are solved independently for the current dictionary using the procedure stated above, where b is the size of the sampled mini-batch.
Each atom is then updated using the stochastic gradient given in Eq.
Note that the symmetry of each atom is preserved as long as the initialized atom is symmetric, since the stochastic gradients are symmetric.
E XPERIMENTS
This section provides empirical evidence that RGDL performs well in the unsupervised graph clustering task on both synthetic and real-world datasets.
The heat kernel matrix is employed for the
PRM since it captures both global and local topology and achieves good performance in many tasks (Donnat et al., 2018; Tsitsulin et al., 2018; Chowdhury and Needham, 2021).
5. S IMULATED D ATASETS
We ﬁrst test RGDL in the graph clustering task on datasets simulated according to the well-studied
Stochastic Block Model (SBM) (Holland et al., 1983; Wang and Wong, 1987).
RGDL is compared against the following state-of-the-art graph clustering methods: (i) GDL (Vincent-Cuaz et al., 2021) learns graph dictionaries via GWD; (ii) Gromov-Wasserstein Factorization (GWF) (Xu, 2020) that approximates graphs via GW barycenters; (iii) Spectral Clustering (SC) of Shi and Malik (2000); Stella and Shi (2003) applied to the matrix with each entry storing the GWD between two graphs.
[Table] [Table]
Dataset generation.
We consider two scenarios of inaccuracies.
In the ﬁrst scenario (S1), Gaussian noise is added into the heat kernel matrix of each graph.
More speciﬁcally, denoting the heat kernel matrix of the k th graph as C k ∗ for k ∈ (cid:74) K (cid:75) , the PRM available to DL methods is C k =
C k ∗ + Z
+ Z (cid:62)
where each entry
Z ij of Z is sampled from the Gaussian distribution N (0 , σ ) .
In the second scenario (S2), we randomly add ρ |E| edges into the graph and then randomly remove ρ |E| edges while keeping the graph connected, where E is the edge set of the graph.
The heat kernel matrix is then constructed for the modiﬁed graph.
Such two scenarios allow us to study the performance of RGDL against different scales of inaccuracies.
In both S1 and S2, we generate two datasets, both of which involve three generative structures (also used to label graphs): dense (only one community), two communities, and three communities.
We ﬁx p = 0 .
1 as the probability of inter-community connectivity and 1 − p as the probability of intra-community connectivity.
The ﬁrst dataset includes 20 graphs for each generative structure and thus is referred to as the balanced dataset.
The second dataset consists of 12, 18, and 30 graphs for the three generative structures respectively, and is hence named as the unbalanced dataset.
The number of graph nodes is uniformly sampled from [30 , 50] .
The magnitude of the observed PRM C k satisﬁes (cid:107) C k (cid:107) ∞ ≤ 15 .
Evaluating the performance.
The learned embeddings of the graphs are used as input for a kmeans algorithm to cluster graphs.
We use the well-known Adjusted Rand Index (ARI) (Hubert and Arabie, 1985; Steinley, 2004), to evaluate the quality of clustering by comparing it with the graph labels.
RGDL with varied (cid:15) is compared against GDL, GWF, and SC.
RGDL, GDL and GWF use three atoms which are R 6 × 6 matrices.
We run each method for 5 times and report the averaged ARI scores and the standard deviations.
Experimental results reported in Table 1 and Table 2 demonstrate RGDL outperforms baselines signiﬁcantly.
Inﬂuence of (cid:15) .
RGDL with moderate (cid:15) values outperforms baseline methods by a large margin and is more robust to the noise.
Even when (cid:15) is relatively small ( (cid:15) =0.01), RGDL achieves better performance than them.
Increasing (cid:15) within a suitable range can boost ARI and RGDL is not sensitive to the choice of (cid:15) .
If (cid:15) becomes too large, the performance of RGDL slowly decreases.
In practice, when a small quantity of data labels are available, (cid:15) can be chosen according to the performance on this small subset of data. [Figure]
5. R EAL - WORLD D ATASETS
We further use RGDL to cluster real-world graphs.
We consider widely utilized benchmark datasets including MUTAG (Debnath et al., 1991), BZR (Sutherland et al., 2003), and Peking 1 (Pan et al., 2016).
The labels of the graphs are employed as the ground truth to evaluate the estimated clustering results.
For each dataset, the size of the atoms is set as the median of the numbers of graph nodes following Vincent-Cuaz et al.
The number of atoms M is set as M = β (# classes) where β is chosen from { 2 , 3 , 4 , 5 } .
RGDL is run with different values of (cid:15) . Speciﬁcally, (cid:15) is chosen from { U, 10 − 1 U, 10 − 2 U, 10 − 3 U } where U = max k ∈ (cid:74) K (cid:75) (cid:107) C k (cid:107) ∞ .
Results.
The experimental results on real-world graphs are reported in Figure 1.
RGDL with (cid:15) = 10 − 1 U or (cid:15) = 10 − 2 U outperforms baselines on all datasets, which implies that the observed graphs contain structural noise and (cid:2) 10 − 2 U, 10 − 1 U (cid:3) is often a suitable range for (cid:15) . The time required for RGDL to converge is comparable to that of state-of-the-art of methods.
Since GDL, GWF, and RGDL can output graph embeddings, we further illustrate the embeddings generated by them respectively based on PCA.
As is shown in Figure 2, the embeddings of the two types of the graphs are less likely to be mixed together, which explains why RGDL achieves higher ARI values.
R ELATED W ORK
Unbalanced OT.
Enhancing the robustness of the optimal transport plan has received wide attention recently (Balaji et al., 2020; Mukherjee et al., 2021; Le et al., 2021; Nietert et al., 2022; Chapel et al., 2020;
S´ejourn´e et al., 2021; Vincent-Cuaz et al., 2022).
Originally, robust variants of classical
OT are proposed to compare distributions supported on the same metric space (Balaji et al., 2020;
Mukherjee et al., 2021; Le et al., 2021; Nietert et al., 2022), which model the noise as outlier supports and reduce the inﬂuence of outlier supports by allowing mass destruction and creation.
Following the same spirit, variants of the GW distance that also relax the marginal constraints are proposed (Chapel et al., 2020; S ´ ejourn ´
e et al., 2021; Vincent-Cuaz et al., 2022).
However, these methods do not take the inaccuracies of the pairwise distances/similarities into account.
The proposed RGWD aims to handle such cases.
Graph representation learning and graph comparison.
Comparing graphs often requires learning meaningful graph representations.
Some methods manually design representations that are invariant under graph isomorphism (Bagrow and Bollt, 2019; Tam and Dunson, 2022). Such representations are often sophisticated and require domain knowledge. Graph neural network-based methods learn the representations of graphs in an end-to-end manner (Scarselli et al., 2008; Zhang et al., 2018; Lee et al., 2018; Errica et al., 2019), which however requires a large amount of labeled data.
Another family of methods that uses graph representations implicitly is referred to as graph kernels (Shervashidze et al., 2009; Vishwanathan et al., 2010).
GWD and its variants based methods can estimate the node correspondence and provide an interpretable discrepancy between compared graphs (Xu et al., 2019b; Titouan et al., 2019;
Barbe et al., 2020; Chapel et al., 2020).
In this paper, we propose a novel graph dictionary learning method based on a robust variant of GWD to learn representations of graphs which are useful in downstream tasks.
Non-linear combination of atoms.
Classic DL methods are linear in the sense that they attempt to approximate each vectorized datum by a linear combination of a few basis elements.
Recently, nonlinear operations are also considered.
In order to exploit the non-linear nature of data, Autoencoderbased methods encode them to low-dimensional vectors using neural networks, and decode data with another neural network (Hinton and Salakhutdinov, 2006; Hu and Tan, 2018).
Another family of methods replace the linear combinations by geodesic interpolations (Boissard et al., 2011; Bigot et al., 2013; Seguy and Cuturi, 2015; Schmitz et al., 2018).
More closely related to our work,
Xu 2020 proposes to approximate graphs via the GW barycenter of graph atoms, which however involves a complicated and computational demanding optimization problem.
C ONCLUSION
In this paper, we propose a novel graph dictionary learning algorithm that is robust to the structural noise of graphs.
We ﬁrst propose a robust variant of GWD, referred to as RGWD, which involves a minimax optimization problem.
Exploiting the fact that the inner maximization problem has a closed-form solution, an efﬁcient numerical scheme is derived.
Based on RGWD, a robust dictionary learning algorithm for graphs called RGDL is derived to learn atoms from noisy graph data.
Numerical results on both simulated and real-world datasets demonstrate that RGDL achieves good performance in the presence of structural noise.
Appendix
The appendix is organized as follows.
We ﬁrst provide omitted proofs in the main paper in Sec. A. Then, algorithmic details are presented in Sec. B. Finally, Sec. C gives additional experimental results.
A O MITTED P ROOFS
Theorem 1 Given two observed graphs G s =
( C s , p s ) and G t =
( C t , p t ) with n s and n t nodes respectively, RGWD satisﬁes
for all T ∈ Π( p s , p t ) , E ( T ) =
[ E i (cid:48) j (cid:48) ( T )] where
[Equation]
otherwise.
solves the inner maximization problem
[Equation]
RGWD is lower bounded, that is,
[Equation]
where the equality holds if and only if there exists a bijective π ∗ : (cid:74) n s
[Equation]
→ (cid:74) n t (cid:75) such that p s i = p t π ∗ ( i ) for all i ∈ (cid:74) n s (cid:75) and C s ij = C t π ∗ ( i ) π ∗ ( j ) for all i, j ∈ (cid:74) n s (cid:75) .
The triangle inequality holds for RGWD, i.e.,
[Equation]
RGWD is invariant to the permutation of node orders, i
[Equation]
e., for all permutation matrices Q s and Q t , [Equation]
Proof: ( i )
The objective can be rewritten as follows,
[Equation]
which by the property of quadratic functions yields the closed-form solution E ( T )
=
[ E i (cid:48) j (cid:48) ( T )] , where
[Equation]
It is easy to verify that the such a choice guarantees the symmetry of E ( T ) .
( ii ) We now prove the lower boundedness.
By Eq.
(5), we have
[Equation]
∈ Π( p s , p t ) n s (cid:88) i,j = n t (cid:88) i (cid:48) ,j (cid:48) = T ii (cid:48) T jj (cid:48) (cid:15) 2 = (cid:15) 2 . Note that when there exists a bijective π ∗ : (cid:74) n s (cid:75) → (cid:74) n t (cid:75) such that p s i
= p t π ∗ ( i ) for all i ∈ (cid:74) n s (cid:75) and
C s ij =
C t π ∗ ( i ) π ∗ ( j ) for all i, j ∈ (cid:74) n s (cid:75) , choosing the transport plan ˆ T =
[ ˆ T ii (cid:48) ] where [Equation]
=
[Equation] if [Equation] 0 , otherwise, we have for all i (cid:48) , j
[Equation]
∈
[Equation] [Equation]
i,
[Equation]
= 0 , which implies that E i (cid:48) j (cid:48)
( T ) =
(cid:15) for all i (cid:48) , j (
cid:48) ∈ (cid:74) n t (cid:75) .
We then have
[Equation]
Therefore, in such a case, RGWD (cid:0) ( C s , p s ) , ( C t , p t ) , (cid:15) (cid:1) = (cid:15) .
On the other hand, when such a bijective does not exist,
[Equation]
where the strict inequality is due to the fact that (cid:80)
[Equation]
( iii ) Thirdly, we prove the triangle inequality.
Given tuples ( C 1 , p 1 ) , ( C 2 , p 2 ) , and ( C 3 , p 3 ) which have the node numbers n 1 , n 2 , and n 3 respectively, let ( T ∗ 12 , E ∗ 12 ) , ( T ∗ 23 , E ∗ 23 ) , and ( T ∗ 13 , E ∗ 13 ) be the solutions of RGWD (cid:0) ( C 1 , p 1 ) , ( C 2 , p 2 ) , (cid:15) (cid:1) , RGWD (cid:0) ( C 2 , p
[Equation]
, ( C 3 , p 3 ) , (cid:15) (cid:1) , and RGWD (cid:0) ( C 1 , p 1 ) , ( C 3 , p 3 ) , (cid:15) (cid:1) .
Deﬁne T 13 =
[ T 13 i 1 i 3 ] where
[Equation]
Then we have
[Equation]
[Equation]
[Table]
23 i 2 i 3 T ∗ 23 j 2 j≤ RGWD (cid:0) ( C 1 , p 1 ) , ( C 2 , p 2 ) , (cid:15) (cid:1) + RGWD (cid:0) ( C 2 , p 2 ) , ( C 3 , p 3 ) , (cid:15) (cid:1) .
( iv ) Finally, we prove the invariance to the node order permutation.
Denote the solution to the objective of RGWD by T ∗ =
[ T ∗ ii (cid:48) ] and E ∗ =
[ E ∗ i (cid:48)
j (cid:48) ] , which implies
E ∗ i (cid:48) j (cid:48) = (cid:40) (cid:15), if (cid:80)
[Equation]
− (cid:15), otherwise, and n s (cid:88) i,j = n t (cid:88) i (cid:48)
[Equation]
=
[Equation]
n
[Equation]
for all T ∈ Π( p s , p t ) .
The two permutation operations can be equivalently denoted as two bijectives π s and π t .
Denote ˜ C s = [ ˜ C s ij ] where ˜ C s ij = C s π s − 1 ( i ) π s − 1 ( j ) , ˜ C t =
[ ˜ C t i (cid:48) j (cid:48) ] where ˜ C t i (cid:48) j (cid:48) =
C t π
[Equation]
( i (cid:48) )
[Equation]
( j (cid:48) ) , ˜ E ∗ =
[ ˜ E ∗ i
[Equation]
j
[Equation]
] where ˜ E ∗ i (cid:48)
j (cid:48)
[Equation]
E
[Equation]
, ˜ T ∗ =
[ ˜ T s ii
[Equation]
] where ˜ T s ii (cid:48)
=
T ∗ π s − 1 ( i
[Equation]
π t − 1 ( i
[Equation]
We ﬁrst prove ˜ E ∗ solves the inner maximization problem for ˜ T ∗ .
For all i (cid:48) , j (cid:48) ∈ (cid:74) n t (cid:75) , when (cid:80)
[Equation]
0 , we have [Equation] ≤ 0 , which is consistent with ˜ E ∗ i
[Equation]
j
[Equation]
=
[Equation]
The case when
[Equation]
) > 0 is similar.
Since n
[Equation]
and max
[Equation]
where
[Equation]
and ˜ E ∗ solve the optimization problem of
[Equation]
To prove Theorem 2, we require the following lemma.
Lemma 3 f ( · ) is l -smooth and L -Lipschitz, where l = √ 2 max { 10 n 3 U 21 + 6 n 3 U 1 (cid:15) + 4 nU 1 U 2 + 4 n 3 (cid:15) 2 , 6 n 2 U 1 U 
 + 2 U 22 + 4 n 2 U 2 (cid:15) } and L = √ 2 max { (4 U 1 + 2 (cid:15) )
U 22 n 3 , 2(2 U 1 + (cid:15) ) 2 U 2 n 3 } with n = max { n s , n t } , U 1 = max {(cid:107) C s (cid:107) ∞ , (cid:107) C t (cid:107) ∞ } and U 2 = max {(cid:107) p t (cid:107) 2 , max T (cid:48) ∈ Π( p s , p t ) (cid:107) T (cid:48) (cid:107) F } .
Proof: ( i )
We ﬁrst prove that f ( · ) is L -Lipschitz.
For all T , T (cid:48) ∈ Π( p s , p t ) and E , E (cid:48)
∈ U (cid:15) , [Equation] f [Equation]
T , E
[Equation]
− [Equation] [Equation] iji
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Table]
For the second term
[Table]
[Equation]
For the third term
[Equation]
Combining the three relations above
[Equation]
we have
[Equation]
( ii ) Now we prove that f ( · ) is l -smooth, which requires ﬁnding a constant l satisfying
[Equation]
where vec( X ) means the vectorization of matrix X and [ ab ] denotes the concatenation of vectors a and b .
Since the left hand side satisﬁes
[Equation]
and the right hand side satisﬁes
[Equation]
it sufﬁces to ﬁnd a constant l satisfying
[Equation]
We bound (cid:107)∇
[Equation]
as follows,
[Equation]
For the ﬁrst term,
[Equation]
For the second term,
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For the third term
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Since ∇
[Equation]
[Equation]
can be bounded as follows,
[Equation]
Combining the above four relations, we have
[Equation]
which yields the desired result.
(cid:4)
Theorem 2 Deﬁne φ ( · ) = max E ∈U (cid:15) f
( · , E ) .
The output ˆ T of Algorithm 1 with step-size η = γ √ N + satisﬁes
[Equation]
, where l = √ 2 max { 10 n 3 U 21 + 6 n 3 U 1 (cid:15) + 4 nU 1 U 2 + 4 n 3 (cid:15) 2 , 6 n 2 U 1 U 2 + 2 U 22 + 4 n 2 U 2 (cid:15) } and L = √ 2 max { (4 U 1 +2 (cid:15) )
U 22 n 3 , 2(2 U 1 + (cid:15) )
2 U 2 n 3 } with n = max { n s , n t } , U 1 = max {(cid:107) C s (cid:107) ∞
∞
Proof: By the smoothness of f ( · ) , for any ˜ T ∈ Π( p s , p t ) and T τ from Algorithm 1, we have
[Equation]
Let ˆ T τ = argmin T ∈ Π( p s , p t ) φ ( T ) + l (cid:107) T − T τ (cid:107) 2 F .
We have
[Equation]
where second line uses Lemma 3.1 of Bubeck et al.
and last line follows from (6).
Taking a telescopic sum over τ [Equation] we obtain
[Equation]
Rearranging this, we obtain
[Equation]
Since φ ( T ) + l (cid:107) T − T τ (cid:107) 2 F is l -strongly convex [Equation] we have
[Equation]
Plugging this in (7) and combining Lemma 3 proves the result. [Equation]
B A LGORITHMIC D ETAILS B. I MPLEMENTATION D ETAILS OF P ROJECTED G RADIENT D ESCENT
The Projected Gradient Descent (PGD) consists of the following three steps in each iteration τ .
Find E τ that maximizes f ( T τ , E ) .
By Theorem 1, we need to calculate an auxiliary matrix
[Equation]
where (cid:12) denotes the element-wise multiplication.
And we have
E
[Equation]
if G
[Equation]
−
[Equation]
otherwise.
Such a step involves computational cost O ( n 3 ) where n = max { n s , n t } .
Gradient descent.
Calculate H
τ
=
T τ − η ∇ T f ( T τ , E τ ) where ∇ T f ( T
[Equation]
, E
[Equation]
=
[Equation]
C
[Equation]
E τ (cid:1) , which also involves computational cost O ( n 3 ) .
Projection into the feasible domain.
This requires solving the following problem
[Equation]
This optimization problem has a strongly convex objective and linear constraints, and hence can be solved efﬁciently via Augmented Lagrangian Method with computational complexity n 2 | log ρ | ρ 1 /(Xu, 2021) where ρ measures the optimality, that is, the violation of the two linear constraints.
When ρ = O (cid:0) 1 n 2 (cid:1) , this step also has cubic costs if we ignore the log term.
Therefore, the overall complexity of PGD obtaining a δ -stationary solution is O (cid:16) n 3 δ 2 + n 2 | log ρ | ρ
[Equation]
δ 2 (cid:17) .
B. A N A LTERNATIVE TO P ROJECTED G RADIENT D ESCENT
Similar to GWD, RGWD can also be solved by linearzing the objective and applying the Sinkhorn algorithm (Titouan et al., 2019).
This approach is very similar to the conditional gradient in Titouan et al.
and the pseudocode is given in Algorithm 3.
Algorithm 3 Conditional Gradient (CG) for RGWD.
Solve optimal transport problem via Sinkhorn algorithm with ground loss ∇ T f
( T τ , E τ ) , i.e.
[Equation]
Find the step-size γ via line search.
[Equation]
8: end for
Such an approach, however, is more time-consuming than PGD.
The Sinkhorn algorithm in Lineinvolves complexity O (cid:16) n 2 log n ρ(cid:17) (Dvurechensky et al., 2018).
Since CG requires O ( 1 δ 2 ) iterations to ﬁnd a δ -stationary solution (Lacoste-Julien, 2016), the overall complexity is O (cid:16) n 3 δ 2 + n 2 log n ρ 2 δ 2 (cid:17) .
Since ρ is relatively small to guarantee each iteration has high precision, the complexity of CG is higher than PGD.
The empirical evidence is given in Sec.
C.1.
C A DDITIONAL E XPERIMENTS C. N UMERICAL A NALYSIS FOR RGWD
Comparison between the transport plan of RGWD and that of GWD.
We ﬁrst visualize and compare the the transport plans of RGWD and GWD.
In this example, we match discrete distributions p and q that are supported on R 2 and R 3 respectively.
The PRMs are given by the (cid:96) 2 distances of supports.
We sample n = 30 supports in R 2 from a mixture of two Gaussians 23 N (cid:0) [ 00 ] , [ 1 0 0 1 ] (cid:1) + 13 N (cid:0) [ 00 ] , [ 0 . 5 0 0 0 .
5 ] (cid:1) .
For each support ( x, y ) in R 2 , we obtain a support ( x, y, z ) in R 3 where the third coordinate z ∼ N (0 , 0 . 3) .
The transport plans of GWD and RGWD are visualized in Figure 3.
The third coordinate z can be considered as introducing inaccuracies to the PRM of z .
This example demonstrates RGWD is robust to such inaccuracies.
PGD vs. CG.
We then compare PGD and CG by matching distributions generated by the same procedure described above but with different numbers of supports.
Given the same initialization of the transport plan, PGD converges faster than CG within the same amount of time, as is shown in
Figure 4.
[Figure]
[Figure]
C. A DDITIONAL E XPERIMENTAL R ESULTS OF G RAPH C LUSTERING
Sensitivity analysis of λ .
As is discussed in Li et al.
Vincent-Cuaz et al.
(2021), the negative quadratic term can promote the sparsity of graph embeddings.
We further conduct sensitivity analysis of λ by varying the value in { 0 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 } .
As is shown in Table 3, λ ∈ (cid:2) 10 − 4 , 10 − 2 (cid:3) often yields good performance.
The experiments in the main paper are run with λ = 10 −
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Peking
[Table]
C. G RAPH C LASSIFICATION
The learned embeddings of graphs can also be used in the graph classiﬁcation task.
RGDL is thus compared against GDL (Vincent-Cuaz et al., 2021), GWF (Xu, 2020), and other state-of-the-art graph classiﬁcation methods including WGDL (Zhang et al., 2021) and GNTK (Du et al., 2019) on the benchmark datasets MUTAG (Debnath et al., 1991), IMDB-B, and IMDB-M (Yanardag and
Vishwanathan, 2015).
RGDL, GDL, and GWF use 3 -NN as the classiﬁer due to its simplicity.
We perform a 10-fold nested cross validation (using 9 folds for training, 1 for testing, and reporting the average accuracy of this experiment repeated 10 times) by keeping same folds across methods.
[Table]
The results are reported in Table 4 and RGDL outperforms or matches state-of-the-art methods.
RGDL outperforms GDL and GWF signiﬁcantly, which indicates the necessity of taking into account the structural noise of observed graphs.
Although WGDL and GNTK have similar performance, they are more computation and memory demanding due to the usage of graph neural networks.