Orthogonal Gromov-Wasserstein Discrepancy with Efﬁcient Lower Bound
Abstract
Comparing structured data from possibly different metric-measure spaces is a fundamental task in machine learning, with applications in, e.g., graph classiﬁcation.
The Gromov-Wasserstein (GW) discrepancy formulates a coupling between the structured data based on optimal transportation, tackling the incomparability between different structures by aligning the intra-relational geometries.
Although efﬁcient local solvers such as conditional gradient and Sinkhorn are available, the inherent nonconvexity still prevents a tractable evaluation, and the existing lower bounds are not tight enough for practical use.
To address this issue, we take inspirations from the connection with the quadratic assignment problem, and propose the orthogonal Gromov-Wasserstein (OGW) discrepancy as a surrogate of GW. It admits an efﬁcient and closedform lower bound with O ( n 3 ) complexity, and directly extends to the fused Gromov-Wasserstein (FGW) distance, incorporating node features into the coupling.
Extensive experiments on both the synthetic and real-world datasets show the tightness of our lower bounds, and both OGW and its lower bounds efﬁciently deliver accurate predictions and satisfactory barycenters for graph sets.
INTRODUCTION
Similarity based learning has been a popular approach in many machine learning applications.
Instead of directly modeling each individual object which may pose challenge in some areas, it resorts to models based on pairwise similarity, possibly across different domains.
The most common example is the kernels used in support vector machines and
Gaussian process, including RBF kernel and covariance matrices that measure similarity in the Euclidean space [Chang et al., 2010], and string or tree kernels that compare discrete objects [Lodhi et al., 2002].
More generally, featured graphs have been a useful tool for capturing similarities and relations in the structured data that are commonly not Euclidean.
Examples include social network [Fan et al., 2019], recommendation systems [Wu et al., 2020], fraud detection [Li et al., 2020], quantum chemistry [Coley et al., 2019] and topology-aware IoT applications [Abusnaina et al., 2019].
Despite the inherent challenge in comparing graphs from possibly different metric-measure spaces, there has been a wealth of reﬁned discrepancy measures between graphs, including kernels [Vishwanathan et al., 2010, Shervashidze et al., 2011] and GCNs based approaches [Bronstein et al., 2017, Defferrard et al., 2016].
Recently, the GromovWasserstein discrepancy [ GW , Peyre et al., 2016], which extends the Gromov-Wasserstein distance [Memoli, 2011], has emerged as an effective transportation distance between structured data, alleviating the incomparability issue between different structures by aligning the intra -relational geometries.
Thanks to its favorable properties such as efﬁciency and isometry-awareness, GW has been applied to domain adaptation [Yan et al., 2018], word embedding [Alvarez-Melis and Jaakkola, 2018], graph classiﬁcation [Vayer et al., 2019a], metric alignment [Ezuz et al., 2017], generative modeling [Cohen and Sejdinovic, 2019], and graph matching and node embedding [Xu et al., 2019b,a].
However, different from the standard Wasserstein distance which is a linear program, GW is unfortunately intractable to evaluate.
Despite the practical success of non-convex optimization techniques such as conditional gradient method and entropic regularization [Peyre et al., 2016, Gold and Rangarajan, 1996], it remains NP-hard to ﬁnd the global optimum.
Hence, existing practice settles with local solutions, lacking an analyzable guarantee.
This signiﬁcantly challenges the trustworthiness of the GW discrepancy.
Towards a tractable approximation, Memoli [2011] proposed three lower bounds of GW , which cost O ( n 2 ) , O ( n 4 ) and O ( n 5 ) .
They can be useful for branch-and-bound based global optimization, as well as recent algorithms for certifying the robustness of nonconvex models. However, these lower bounds are rarely used in practice, and as we will show later in Figure 3, even the most expensive lower bound can be quite loose, raising the concerns on their effectiveness. Instead of developing yet another lower bound or tight approximation of GW , our goal in this paper is to design a surrogate of it (namely orthogonal GW , or OGW ) such that: (a) It retains GW ’s desirable properties such as permutation invariance, non-negativity, triangle inequality, and good performance in machine learning tasks such as classiﬁcation and barycenter.
We stress that OGW does not need to be either an upper bound or a lower bound of GW . We are also not concerned about the gap between OGW and GW because what matters is the desirable mathematical properties and the performance in learning, instead of how close OGW is to GW . (b) It does not have to be tractable, but it must admit a tight and efﬁcient lower bound – upper bound is easier from local optimization as in GW . The tightness will potentially contribute to global optimization such as certiﬁcation, a topic that is beyond this paper.
Ideally, such a lower bound should also possess the aforementioned good properties of the surrogate itself.
Our inspiration, as unrolled in Section 2, stems from the connection of GW to quadratic assignment problems (QAPs), which was tapped into by sliced GW
[Vayer et al., 2019b] and Gromov-Monge problems [Memoli and Needham, 2021].
It paved the way for approximating the set of doubly stochastic matrices (used in GW ) by orthonormal matrices under marginal constraints [Rendl and Wolkowicz, 1992, Hadley et al., 1992, Anstreicher and Brixius, 2001].
The resulting problem is well known to admit tight approximate solutions, and accommodates fused GW to account for node features [ FGW , Vayer et al., 2019a].
Experiments on classiﬁcation and barycenter demonstrate the effectiveness of our proposed OGW and its lower bounds.
OGW -DISCREPANCY WITH TRACTABLE LOWER BOUNDS
We represent an undirected graph G with n nodes by an adjacency matrix A ∈ { 0 , 1 } n × n , where A ii = 0 , and
A ij = 1 if there is an edge between nodes i and j ( i (cid:54) = j ), and 0 otherwise.
To start with, we consider the discrepancy between two graphs with the same order ( i.e. , number of nodes).
A detailed generalization to any graph orders will be addressed in Section 2.3.
Associated with the nodes is a distribution, encoding some prior information about their importance, e.g., the normalized degree of each node [Xu et al., 2019b].
However, many applications lack such natural normalization [Vayer et al., 2019a,b, Peyre et al., 2016].
Therefore, we will stick with a uniform distribution over all nodes and include our extension of non-uniform distribution in Appendix A.5.
Letting
=
, 1) (cid:62) whose dimensionality can be implicitly induced from the context, the standard GW distance between graph G and H based on (cid:96) 2 distance can be formulated as (the square root of) GW ( G, H ) :=
[Equation]
where E := { P ∈ R n × n : P1 = P (cid:62) 1 = 1 } and N :=
R n × n + [Memoli, 2011].
Here c G ( i, j ) represents a distance measure between node i and j on G , and common choices include their shortest-path distance, or simply 1 − A ij (the complement of adjacency).
When both c G and c H are a metric, (1) is a squared metric on isomorphism classes of measurable metric spaces.
However, as pointed out by Peyre et al.
[2016], c G and c H do not have to be restricted to metrics and (cid:96) 2 can be extended to other asymmetric or nonsubadditive losses such as f -divergence.
They call it GW discrepancy, which broadens its applicability in machine learning.
We will refer to it as GW , without even taking the square root of (1) just like in Peyre et al. Remark 1. It is noteworthy that although the original GW requires c G ( i, j ) to be a distance metric
[Memoli, 2011], it can be relaxed in (1) where (cid:96) 2 loss is used. Indeed, c G ( i, j ) can also be served by similarities between nodes instead of distance, e.g. , by simply ﬂipping the sign of the distance measure. This also opens up the use of non-metric dissimilarity measures such as constrained shortest path [Lozano and Medaglia, 2013].
Deﬁne two n -by- n symmetric matrices C and D whose ( i, j ) -th elements are C ij =
c G ( i, j ) and D ij =
c H ( i, j ) , respectively.
For example, the complement of adjacency can be written as C = 11 (cid:62) − A .
The above GW can be compactly rewritten in the Koopmans-Beckmann form [Koopmans and Beckmann, 1957]:
[Equation]
Here (cid:107)·(cid:107) F is the Frobenius norm.
Obviously, GW is permutation invariant, nonnegative, and equals 0 when G and
H are isomorphic.
The major drawback is that the maximization over P is intractable, although efﬁcient local algorithms are available such as conditional gradient [Vayer et al., 2019a] and Sinkhorn [Peyre et al., 2016].
2.1 CONNECTING OGW WITH THE QUADRATIC ASSIGNMENT PROBLEM
Rewriting GW with quadratic optimization over E ∩N as in (2) reveals an innate connection to the quadratic assignment
[Figure]
problem (QAP).
Noting that by the Birkhoff–von Neumann theorem [Birkhoff, 1946], E ∩N is the convex hull of the set of n × n permutation matrices (denoted as Π ).
Indeed, the connection with QAP has been used to formulate GromovMonge distances
[Memoli and Needham, 2021], and to accelerate the evaluation of GW via projection (slicing) to 1-D [Vayer et al., 2019b].
Fortunately, a number of tractable relaxations of QAP are available, many of which are based on the following characterization of Π :
[Equation]
where O := { P ∈ R n × n : P (cid:62) P = PP (cid:62) = I } .
Here I is the identity matrix.
Whenever necessary, we will explicitize the dimensionality of O by writing O n .
Interestingly, tr( CPDP (cid:62) ) can be maximized exactly by a simple eigendecomposition if P is restricted to O [Umeyama, 1988].
Speciﬁcally, assume the eigen-decomposition of C and D are C
= P C diag( λ C ) P (cid:62) C and D = P D diag( λ D ) P (cid:62) D , respectively, and suppose the eigenvalues in λ C and λ D are both arranged in a descending order.
Then
[Equation]
and
[Equation]
Based on this result, Hadley et al.
[1992] proposed tightening the domain approximation from O to O ∩ E , which, despite the original intention of approximating inhomogeneous
QAPs, happens to be useful in our context too.
Compared with N ∩ E , O ∩ E offers more convenience in constructing upper and lower bounds that are tight and efﬁcient.
This substitution leads to our proposed new metric, named as orthogonal Gromov-Wasserstein ( OGW ) discrepancy:
[Equation]
Figure 1 illustrates how QAP is connected with GW and
OGW through the different convex outer approximations of the domain of the permutation matrices.
2.2 UPPER AND LOWER BOUNDS OF OGW
The evaluation of OGW is hindered by the nonconvex objective and the nonconvex domain in the optimization of P in (6).
So it is natural to resort to its lower and upper bounds.
Upper bound of OGW . Obviously, any locally optimal P in (6) yields an upper bound of OGW . To ease the local optimization, we ﬁrst leverage the characterization of O ∩ E [Hadley et al., 1992]:
[Equation]
= [Equation] [Equation] , (7) where V is any n × ( n − 1) matrix satisfying V (cid:62) 1 = 0 and
V (cid:62) V = I n − 1 . An example is given in Appendix A.1.
Plugging P = 1 n 11 (cid:62) + VQV (cid:62) into the optimization objective in (6) yields
[Equation]
where ˆX := V (cid:62) XV and s X := 1 (cid:62) X1 for any matrix
X , and E := 2 n C11 (cid:62) D .
Since Q ( ˆ C , ˆ D , ˆ E ) involves both linear and quadratic terms in Q , no closed-form solution remains available.
Clearly, any locally optimal Q yields a lower bound for Q (denoted as Q lb ), i.e. , an upper bound for OGW (denoted as OGW ub ).
Locally optimizing Q over O ( a.k.a.
Stiefel manifold) has been very well studied [Absil et al., 2009,
Wen and Yin, 2013, Arasu and Mohan, 2018], and we adopt a straightforward approach of projected quasi-Newton, noting that the projection of any matrix Q on O is simply U Q V (cid:62) Q , where the singular value decomposition (SVD) of Q is U Q Λ Q V (cid:62) Q .
With the locally optimal Q in hand, the locally optimal P for OGW can be recovered by plugging Q into the formula in (7). Similarly to the practice of GW which resorts to locally optimal solutions, we will use OGW ub as a practical “evaluation” of OGW . Whenever there is no confusion (especially in empirical investigation), we will simply refer to the performance of OGW ub as the performance of OGW .
Lower bounds of OGW .
The simplest way to lower bound OGW is by relaxing the domain of P into O in (6):
[Equation]
where the last step is by (4).
We note in passing that OGW o embodies a different design principle from heat kernel signature [Sun et al., 2009] and wave kernel signature [Aubry et al., 2011], in that neither of the kernel signatures sort the kernel spectrum.
In practice, we found that completely dropping the constraint E may lead to over relaxation.
To bring back E , we follow Hadley et al.
[1992] and decompose Q in (8) into quadratic and linear terms by decoupling their Q :
[Equation]
As a result, we obtain an upper bound of Q (denoted as Q ub ), which produces a lower bound of OGW :
[Equation]
Q ub can be evaluated analytically.
First, Q 1 can be solved by (4).
As for Q 2 , the von Neumann’s trace inequality implies that its optimal value is U E V (cid:62) E , where U E Λ E V (cid:62) E is the SVD of ˆE , and the maximum value of tr( ˆE (cid:62) Q 2 ) is || ˆE || ∗ , the trace norm of ˆE , which is the sum of the singular values of ˆE . Hadley et al.
[1992] showed that such an upper bound in (11) is often quite tight, which is also observed in our experiments.
Indeed, we noticed that the magnitude of ˆC and ˆD in (11) is signiﬁcantly larger than that of ˆE .
Therefore, although the optimal Q 1 and Q 2 are different, the resulting gap is small.
We next summarize the mathematical properties of OGW and its lower bounds as follows: Theorem 1.
OGW , OGW o , and OGW lb are all nonnegative and symmetric.
Their square root satisﬁes the triangle inequality. Their values are 0 if (but not only if) the two graphs are isomorphic. The proof is in Appendix A.2. Compared with the requirement of distance metric, OGW and its lower bounds only fall short of the “only if” part of the identity of indiscernibles.
To see why “only if”cannot hold, consider OGW o whose closed form in (9) shows that its value can be 0 as long as
C and D are similar, i.e. , share the same eigenvalues.
In general, however, C and D are derived from graphs with certain discrete properties, leaving permutation the most likely path to similarity.
Remark 2.
The coupling matrix P in GW provides a useful matching between two sets of nodes.
Although the P in
OGW and its lower bounds may contain negative entries, it optimizes over the orthonormal domain, which may still provide useful insights between the two groups of node.
For example, invariance to orthogonal transformation is a longstanding pursuit in learning [Kornblith et al., 2019].
Despite the hardness of exactly optimizing P for OGW , we can use Q 1 from (11) to recover P via the transformation in (7) .
This is reasonable because ˆ E is generally much smaller in magnitude than ˆ C and ˆ D .
Computational complexity.
The analytic solution for
OGW lb and OGW o is achieved by singular value decomposition and eigen decomposition, whose computational complexity is O ( n 3 ) .
ˆC , ˆD , and ˆE can be computed in O ( n 3 ) thanks to the structure of V (see Appendix A.1).
It is worth mentioning that Memoli [2011] also derived the lower bounds of GW by solving a set of linear assignment problems, named First Lower Bound (FLB), Second Lower
Bound (SLB), and Third Lower Bound (TLB).
The following inequalities provide the connections between different lower bounds of GW :
[Equation]
And the complexities of FLB, SLB and TLB are
O ( n 2 ) , O ( n 4 ) , O ( n 5 ) , respectively.
In general, TLB provides the tightest lower bound for GW , and SLB has resemblant performance compared with TLB.
This is also observed in our experiments.
2.3 GRAPHS WITH DIFFERENT SIZES
So far, we have been restricting the two graphs to have the same number of nodes, primarily because the orthonormal domain O only contains square matrices.
In order to deal with the non-square matrix, i.e. , graphs of different order, we introduce the semi-orthogonal domain
[Equation]
where m > n without loss of generality.
That is a domain of “tall” matrices, whose columns are orthonormal.
When m = n , ˜ O recovers O n because T (cid:62) T
= I n is equivalent to TT (cid:62) =
I n for a square matrix T .
For a non-square matrix P ∈ R m × n and m > n , it is no longer feasible to impose the constraint of E := { P ∈ R n × n : P1 = P (cid:62) 1 = 1 } because the dimensionality does not match.
Instead, it can be generalized into
[Equation]
To summarize, we can extend the OGW in (6) to graphs with different orders by replacing the domain of P , amounting to [Equation]
[Equation]
where G and H have the graph order of m and n , respectively.
We reuse the symbol OGW because the constraints ˜
O and ˜ E recover O and E respectively when m = n .
It is also easy to see that OGW is nonnegative because
[Equation]
In the similar spirit to (7), any P ∈ ˜ O ∩ ˜ E can be reparameterized as follows Theorem 2.
Let U ∈ R m × ( m − 1) and V ∈ R n × ( n − 1) be arbitrary projection matrices satisfying
[Equation]
Then
[Equation]
where Q ∈ ˜
O m − 1 ,n − 1 .
The proof is relegated to Appendix A.4.
Letting ˆC :=
U (cid:62) CU and ˆD := V (cid:62) DV , the optimization over P in (16) turns into a projected QAP (PQAP) with an additional linear term and some constant terms
[Equation]
where ˆE = 2 √ mn U (cid:62) C1 m 1 (cid:62) n DV . Finally, in order to leverage the favorable properties of orthonormal matrices, we right-pad the matrix Q by an ( m − 1) × ( n − m ) matrix J , such that [ Q , J ] ∈
O m − 1 .
Indeed
Q ∈
[Equation]
Such a J matrix only needs to be any basis of the kernel space of Q .
As a result, the quadratic term max Q ∈ ˜ O tr( ˆCQ ˆDQ (cid:62) ) is equivalent to
[Equation]
[Equation] and we can optimize [ Q J ] as a whole, utilizing the closedform solution to the squared matrix case.
Similarly, for the linear term, we have
[Equation]
where K serves the same role as J .
To conclude, by padding zero on the non-square matrices ˆD and ˆE to square matrices, we can enjoy the analytic solutions to the problems in (22) and (23) in the same way as in the square case. Remark 3.
We refrain from the interpretation of adding dummy nodes with 0 distance because, as pointed out in Remark 1, C and D can represent similarity measures. In such cases, padding with 0 is still justiﬁed with the above derivation, but not amenable to dummy node interpretations. Remark 4.
Disconnected graphs can be modeled by any existing heuristic that is also required by GW . In (1) , c G ( i, j ) cannot be ∞ because it would push GW to ∞ as long as all c H ( k, l ) < ∞ and all nodes have nonzero marginals. A simple heuristic is to employ a large distance value between two nodes that belong to two separate/disconnected subgraphs. Our experiments only involved connected graphs, because all the graphs from the real datasets are already connected – none was discarded.
We stress that the heuristic in Remark 3 is independent of padding 0 in (22) that tackles different graph sizes. The latter is on how to characterize the alignment of two matrices, which is orthogonal to the design of base measure itself within each graph.  BARYCENTER We next study the application of OGW to the Barycenter problem, where given a set of sampled graphs and their associated weights, we aim to ﬁnd their Fréchet mean by minimizing the weighted average discrepancy between the barycenter and the sampled graphs.
Here the discrepancy is measured by our proposed OGW , and the sampled graphs are represented by a set of cost matrices
D i , along with normalized weight λ i for i ∈ { 1 , ..., S } .
D i can also encode pairwise similarities instead of dissimilarities, and our method accommodates both cases naturally.
The barycenter problem can be formalized as minimizing the weighted average OGW discrepancy:
[Equation]
For simplicity, we specify the barycenter with a ﬁxed order m , although the value of m can also be optimized.
We follow the block coordinate update proposed in Peyre et al.
[2016], i.e. , iteratively minimizing with respect to the couplings P i and updating the optimal cost matrix for the barycenter in a closed-form solution.
Given a set of coupling matrices P i , we can retrieve the optimal C by taking the partial derivative of (24)
[Equation]
So the optimal C under the current { P i } is C ∗ ← m [Equation] P [Equation]
D i P
[Equation]
i . Noting that OGW itself is still intractable, we resort to the tractable lower or upper bounds, replacing OGW in the deﬁnition of B ( C ) with its tractable bounds. In the case of OGW ub (resp. OGW o ), we simply adopt the locally (resp.
globally) optimal { P i } .
For OGW lb ,
GW OGW FGW OFGW a dd nod e f ea t u r e change N∩E into O∩E a dd nod e f ea t u r e change N∩E into O∩E we can rewrite B ( C ) in terms of Q 1 and Q 2 from (11) and (12), and then C can be updated using the optimal Q 1 and
Q 2 . More details are provided in Appendix A.7.
It is noteworthy that any optimal solution C for (24) leads to a set of optimal solutions { PCP (cid:62) : P ∈ O ∩ E} . This also resonates with the intuition retrievable from OGW o in (9), where only the eigenvalues of C matter. Therefore, additional “post-processing” is needed to pinpoint the optimal C from the equivalent class. Furthermore, the optimal
C ∗ found over O ∩ E does not guarantee elementwise nonnegativity.
Next, we present our method, named as spectral reconstruction , to ﬁnd the appropriate C .
3.1 SPECTRAL RECONSTRUCTION
To begin with, suppose C and all D i are graphs of order m , and we consider their projections to R ( m − 1) × ( m − 1) via ˆ C ∗ =
V (cid:62) C ∗ V and ˆ D i = V (cid:62) D i V .
Let their eigendecomposition be ˆ C ∗
= R (cid:62) Σ R and ˆ D i =
S (cid:62) i ∆ i S i .
Inspired by the above observation that OGW ( C , D i ) depends primarily on the eigenvalues of C and D i , we rebuild ˆ C ∗ by ˆ C ∗ recon =
(cid:80) i λ i S (cid:62)
i Σ S
i , i.e. , trusting and retaining the eigenvalues of the optimal solution ˆ C ∗ while pairing them with the eigenvectors of the sampled graphs.
For graphs with different sizes, the trick in Section 2.3, i.e. , padding on smaller graph, helps us to assemble the ˆC ∗ from the top m − 1 eigen system.
Noting that ˆ C ∗ is still on the projected domain R m − 1 × m − 1 , i.e. , there exist V such that ˆ C ∗
=
V (cid:62) C ∗ V .
Next, we bring ˆ C ∗ recon back to its original space R m × m via
[Equation]
where Y satisﬁes V (cid:62) YV = 0 , ensuring that V (cid:62) C ∗ recon V recovers ˆ C ∗ recon .
In addition, when OGW operates on dissimilarity matrices, we require diag( C ∗ recon ) = 0 , i.e., the dissimilarity between a node and itself is 0. A straightforward choice of Y satisfying the two conditions is
[Equation]
where d := diag( V ˆ C ∗ recon V (cid:62) ) .
[Equation]
[Equation]
To gain more intuition into the recipe, consider the barycenter problem with OGW lb [-= 
-]and only one sampled graph D 1 .
By Theorem 1, B ( C ) can be driven to 0 and it is attained when ˆ C shares the same eigenvalues as ˆ D 1 , i.e. , Σ = ∆ 1 . Then ˆ C ∗ recon
= ˆ D 1 by our construction. Furthermore, the proof of Theorem 1 indicates 1 (cid:62) C1 = 1 (cid:62) D 1 1 and (cid:13)(cid:13) V (cid:62) C1 (cid:13)(cid:13) = (cid:13)(cid:13) V (cid:62) D 1 1 (cid:13)(cid:13) .
It is then not hard to show that C ∗ recon is exactly D 1 .
We provide our experiments on both synthetic and real dataset in Section 5.3.
EXTENSION TO FUSED GW
Most applications carry features for each node.
To account for this important information, Vayer et al.
[2019a] proposed the fused GW ( FGW ), employing an additional matrix M whose ( i, k ) -th entry encodes the (cid:96) 2 distance between the features of node i in G and of node k in H .
M is asymmetric in general.
Then the vanilla FGW -distance was formulated by Vayer et al. [2019a] as
[Equation]
where α ∈ [0 , 1] is a trade-off between structure and feature measure.
For simplicity, we will only present the treatment for two graphs of the same size.
The extension to different sizes can be easily derived in the same way as in Section 2.3.
Similar to Q , with the additional linear term tr( M (cid:62) P ) , there is no closed-form solution even if we replace the domain of P by O .
However, we can still tighten the domain from O to O ∩ E , leading to our new approximation
[Equation]
The pipeline of construction is illustrated in Figure 2.
As α tends to zero, OFGW recovers OGW between only structures.
A number of favorable properties are enjoyed by OFGW , which are summarized in Theorem 3 below (proof deferred to Appendix A.3).
Although OGW must be nonnegative, OFGW is not guaranteed nonnegative for all M .
To see a counter-example, set C
= D = [-
-]0 and M = I .
Fortunately, for a large set of M , it still enjoys nonnegativity.
Theorem 3.
Suppose M satisﬁes 1 (cid:62) M1 ≥ n || ˆ M || ∗ .
Then
OFGW ( G, H, M ) ≥ 0 for all G , H , and is invariant to the (different) permutations of G and H .
When M = 0 , it degenerates to OGW .
Since M encodes the pairwise distance between two sets of node features { X i } ni =1 and { Y i } ni =1 with M ij = (cid:107) X i − Y j (cid:107) 2 , we can conﬁrm whether the above assumption holds a priori.
Interestingly, this is the case in all the datasets considered in our experiment.
For datasets without node attributes and labels, we take node degree as their features.
In the sequel, we will make this assumption on M .
Although OFGW is still intractable in general, local optimization can be performed very efﬁciently.
As will be shown in Section 4.1, it also admits a tight lower bound using the same relaxation technique as for OGW .
Now that OFGW is motivated by computational convenience, one may naturally wonder whether it captures as much graph structure as the original FGW does.
We veriﬁed this in the afﬁrmative by following Vayer et al.
[2019a], where FGW is used as a kernel function served in a support vector machine. In
Section 5.1, we will show that replacing FGW by OFGW achieves similar or better classiﬁcation accuracy on a variety of datasets, corroborating the effectiveness of OFGW .
4.1 UPPER AND LOWER BOUNDS OF OFGW
Following (7), plugging P = 1 n 11 (cid:62) + VQV (cid:62) into the optimization objective in (32) yields
[Equation]
where F := 2 αn C11 (cid:62) D − 1 − α 2 α M .
It reveals that compared with the expression of GW in (8), the additional linear term
M in the FGW formulation does not change the structure.
As a result, we can again derive the lower bound of Q , i.e. , upper bound of OFGW , by using the projected quasiNewton as before.
And a lower bound of OFGW can also be obtained by decoupling the Q in the two terms of (33).
Speciﬁcally, by using the Q ub deﬁned (11) via decoupling into Q 1 and Q 2 , we obtain
[Equation]
−
[Equation]  EXPERIMENTS We now demonstrate the empirical effectiveness of OGW and OFGW via two applications: graph classiﬁcation and barycenter problem.
We will also illustrate the tightness of our lower bound OGW lb . All the code and data are available at https://github.com/cshjin/ogw .
5.1 EFFECTIVENESS OF OGW / OFGW
Recall in Figure 2, N ∩ E was replaced by O ∩ E because the latter enjoys tight and efﬁciently computable upper and lower bounds.
So it is important to validate the resulting
OGW / OFGW as an equally good measure of comparing two graphs as the vanilla GW / FGW .
Datasets.
We experimented on six graph classiﬁcation datasets: BZR, COX2, MUTAG, PTC-MR, IMDB-Binary, and IMDB-Multi [TUDataset].
Their statistics are given in Appendix A.8.
The ﬁrst four datasets contain a collection of molecules ( e.g. , chemical compound and ligands), where the vertices represent atoms and edges are chemical bonds.
The class label represents a certain property of the molecules, e.g. , "mutagenic effect on a speciﬁc bacterium" (MUTAG) and carcinogenicity of compounds for male rats (PTC-MR).
BZR and COX2 consist collections of ligands for the benzodiazepine receptor and cyclooxygenase-2 inhibitors, respectively.
IMDB-Binary and IMDB-Multi are the movie collaboration dataset, where nodes represent actors/actresses who played roles in movies in IMDB, and one edge means two played in the same movie.
We group the dataset into three categories according to their feature property: vectorized, discrete, and no features.
For the datasets with no features, we take the node degrees as their features.
Settings.
In order to evaluate a discrepancy measure d , we follow Vayer et al.
[2019a] by studying the graph classiﬁcation accuracy of an SVM, whose kernel k ( G, H ) is computed by exp ( − γd ( G, H )) .
For both FGW and OFGW , the feature distance matrix M employed the squared Euclidean distance.
Since OFGW itself is intractable to evaluate, we resort to the lower bound of OFGW in (31) which has an analytic form and is nonnegative.
For the vanilla GW / FGW , we adopt the implementation from POT package [Flamary et al., 2021], which initiates the transition matrix by the outer product of marginal distributions.
And we instantiate the cost matrix c G by all-pair shortest path for each graph in the datasets, knowing that the structures are all connected.
We evaluate the models by cross-validation on the hyperparameters in SVM, setting γ from (cid:8) 2 − 10 , 2 − 9 , · · · , 2 10 (cid:9) and
C from 2 − 4 to 2 4 on evenly log scale with 15 steps.
Moreover, for the FGW / OFGW , we cross-validate the value of α from [0 , 1] with grid search.
In addition, we consider the graph kernel methods as the baselines.
More speciﬁcally, we adopt the implementation of shortest path (SP) kernel [Borgwardt and Kriegel, 2005] and graphlet sampling (GK) kernel [Przulj, 2007] from Siglidis et al.
SP kernel decomposes graphs into shortest paths and compares pairs of shortest paths according to their lengths and the labels of their endpoint, while GK kernel decomposes graphs into graphlets, i.e. , small subgraphs with k nodes where k ∈ { 3 , 4 , 5 , · · · } , and counts matching graphlets in the input graphs.
Results
The average accuracy achieved with 10-fold cross-validation is presented in Table 1.
To see the effectiveness of OGW and its lower bounds ( OGW lb and OGW o ), we compare their accuracy against that of the vanilla GW and its ﬁrst lower bound ( GW flb ). In addition, we also include FGW and OFGW that incorporate node features. As the table shows, the tractable lower bounds of our proposed [Table]
[Figure]
[Figure]
[Figure] [Figure] [Figure] metric OGW have comparable performance with GW and its variant.
With the additional information from node features (except IMDB datasets), the OFGW provides higher accuracy in general.
It corroborates that the chain in Figure 2 preserves important structures in graphs, making the tractable lower bounds of OGW / OFGW sound discrepancy measures for graphs.
Moreover, OGW lb achieves higher accuracy than OGW o on two datasets, and performs similarly on the other four datasets.
5.2 TIGHTNESS OF THE LOWER BOUND
The tightness of our lower bound OGW lb , with respect to
OGW , can be evaluated through its difference to the upper bound OGW ub , which is obtained by the projected quasiNewton method. Such a gap will be compared with its counterpart in GW distance, where the upper bound is served by a local optimizer based on the conditional gradient method, and the lower bounds are proposed by Memoli [2011], including FLB, SLB, and TLB (the best known lower bound).
Synthetic data.
To demonstrate the tightness in synthetic data, we generate a path graph with 20 nodes and randomly perturb δ g ∈ [1 , 10] edges for 50 times, so that we can measure the distance (dissimilarity) between the original graph and the perturbed graph under different measures. Only connected graphs are kept.
Figure 3a and 3b provide, respectively for GW and OGW , the average distance as a function of the number of perturbed edges.
The gap between OGW ub and OGW lb is much tighter than the best gap for the GW case, i.e. , GW − GW tlb .
Note that TLB of GW requires O ( n 5 ) computational time, while OGW lb only costs O ( n 3 ) . To verify it, we measure the running time by varying the graph sizes.
For each graph size that ranges from 10 to 1000, 20 Erd ˝ os-Rényi random graphs are generated, ensuring they are connected.
Then their average running time is reported in Figure 3c, which clearly matches the analyzed computational complexity.
Real-world data.
We also examine the gap between lower and upper bounds on the real-world dataset MUTAG by evaluating pairwise distance.
Figure 4a provides the distribution of the gaps from GW and OGW .
For GW , we report the gap between GW local optimizer and GW tlb .
Clearly, the gap between OGW ub and OGW lb centers around 0 .
1 , while that between the GW and its TLB concentrates around 1 .
In addition, we also compare in Figure 4b the tightness of OGW lb and OGW o , both as a lower bound for OGW . Each point in the scatter plot represents a graph in the MUTAG dataset, and the horizontal (resp. vertical) axis is the gap between the upper bound of OGW and OGW lb (resp.
OGW o ). The fact that the vast majority of the points lie above the diagonal conﬁrms the superior tightness achieved by OGW lb . This demonstrates the beneﬁt of employing E in the constraint and separating Q 1 and Q 2 in (11). More tightness results on other datasets are provided in Appendix A.9.
[Figure] [Figure]
5.3 BARYCENTER
In the last set of experiments, we evaluate the ability of
OGW lb and OGW ub to solve the barycenter problem.
Synthetic data.
To start with, we generate a set of cyclelike graphs with different sizes ranging from 15 to 25.
In addition, we also explicitly add random structural noise to the graphs separately, and pre-compute their shortest path as cost matrices.
All synthetic samples are plotted in Appendix A.10.
To initialize the barycenter, we ﬁx the number of nodes to be 20 and initiate a random symmetric C when starting the block coordinate descend to update
P .
For GW , we adopt the implementation from Peyre et al.
[2016] to ﬁnd the optimal cost matrix of the center. For the
OGW lb and OGW ub , we solve the problem by our proposed eigen projection method in Section 3.
To better visualize the results, we also reconstruct the adjacency matrix following a standard heuristic [Vayer et al., 2019a].
In particular, for a given threshold, a pair of nodes can be connected by an edge if and only if their corresponding entry in the cost matrix is below the threshold.
Then we perform a line search on the threshold to minimize the difference between the optimal cost matrix and the one corresponding to the threshold based adjacency matrix.
From Figure 5, we can see the reconstructed cost matrix for the barycenter from our OGW lb and OGW ub are well aligned to its node ordering.
We also recognize that the result from GW is just one of the local minimizers in the context of permutation. Due to the tightness between the lower and upper bounds, it is also hard to differentiate the structures found by OGW lb and OGW ub (i.e., sub-ﬁgures e and f).
Point cloud data.
We also explore the barycenter on point cloud dataset called MNIST-2D 1 . Refer to Appendix A. for the plots of the corresponding point cloud data. Clearly, the point clouds for 6 and 9 are quite similar up to rotation.
[Table]
The point cloud is modeled by a graph whose nodes correspond to non-zero (non-black) pixels, represented by their 2D coordinates.
Without constructing the explicit mesh connections between pixels, we take the Euclidean distance as their cost matrix D i .
After retrieving the optimal C ∗ of the barycenter, we further uncover the associated optimal coordinates.
A BFGS optimizer was used to seek the locally optimal coordinates such that the resulting Euclidean distance between pixels best reconstructs C ∗ . Clearly, such a recovery can only be up to the standard invariant transformations such as rotation and shift, and they are determined by the random initialization of BFGS.
Table 2 illustrates the optimal point clouds with labels to be 0 , 6 , and 9 .
We sample 5 different point clouds for each digit and set the weight λ
i to the uniform distribution.
Moreover, we ﬁx the number of nodes on the barycenter as the minimum size from the samples.
Compared with GW , OGW lb and OGW ub clearly ﬁnd better point clouds of the barycenter in the cases of digit 6 and 9.
Moreover, thanks to the tight gap between the lower bounds, the performance of OGW lb and OGW ub differ only indistinguishably.
Acknowledgements
We thank the reviewers for their constructive comments.
This work is supported by NSF grant RI:1910146.