A C ONVERGENT S INGLE -L OOP A LGORITHM FOR G ROMOV -W ASSERSTEIN IN G RAPH D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose the first provable single-loop algorithm for computing the GromovWasserstein (GW) distance.
We call it Bregman Alternating Projected Gradient (BAPG) method.
Our analysis is based on a surprising observation that the GW problem satisfies a so-called Luo-Tseng error bound condition (Luo & Tseng, 1992), which relates to estimating the distance of a point to the critical point set of the GW problem based on the optimality residual.
Armed with this error bound condition, we are able to show that BAPG converges to a critical point of the
GW problem asymptotically (as step size goes to infinity), and further give an approximation bound for the distance between the fixed-point set of BAPG and the critical point set of GW explicitly.
We conduct comprehensive numerical experiments to validate the effectiveness of BAPG on graph alignment and partition tasks.
In terms of both wall-clock time and quality of solutions, the proposed method achieves state-of-the-art results.
I NTRODUCTION
The Gromov-Wasserstein (GW) distance provides a flexible way to compare and couple probability distributions supported on different metric spaces.
As such, we have witnessed a fast-growing body of literature that applies the GW distance to various structural data analysis tasks, e.g., 2D/3D shape matching (Peyré et al., 2016; Mémoli & Sapiro, 2004; Mémoli, 2009), molecule analysis (Vayer et al., 2018; 2019a), graph alignment and partition (Chowdhury & Mémoli, 2019; Xu et al., 2019b;a;
Chowdhury & Needham, 2021; Gao et al., 2021), graph embedding and classification (Vincent-Cuaz et al., 2021b;
Xu et al., 2022), generative modeling (Bunne et al., 2019; Xu et al., 2021), to name a few.
Although the GW distance has received much attention in both machine learning and data science communities, there are still few results that focus on the design of practically efficient algorithms with provable convergence guarantees.
Existing algorithms either are double-loop (i.e., requiring another iterative algorithm as a subroutine at each iteration) or do not have any theoretical justification.
Recently, Solomon et al.
(2016) has proposed an entropy-regularized iterative sinkhorn projection algorithm called eBPG, which is proven to converge under the Kurdyka-Łojasiewicz framework (Attouch et al., 2010; 2013).
However, eBPG has several crucial drawbacks that prevent its widespread adoption.
First, instead of tackling the GW problem directly, eBPG addresses an entropic regularized
GW objective, whose regularization parameter affects the model performance dramatically.
Second, eBPG has to solve the entropic optimal transport problem (Peyré et al., 2019; Benamou et al., 2015) at each iteration via another iterative algorithm (Cuturi, 2013).
Thus, eBPG is neither computationally efficient nor practically robust.
To solve the GW problem directly, Xu et al. (2019b) proposes the
Bregman projected gradient (BPG).
Unfortunately, BPG is still a double-loop algorithm that also invokes the sinkhorn as its subroutine.
Notably, due to the lack of the entropic regularizer, the inner solver suffered from the numerical instability issue.
On another front, Vayer et al. (2019a); Mémoli (2011) have introduced the Frank-Wolfe (FW) method (see (Jaggi, 2013; Lacoste-Julien, 2016) for recent treatments) to solve the GW problem.
Nevertheless, in their implementation, they still rely on off-the-shelf linear programming solvers and line-search schemes, which are not well-suited for even medium size tasks.
it is only recently that Xu et al.
(2019b) has further developed a simple heuristic single-loop method (BPG-S) based on BPG, which presented an attractive empirical performance on the node correspondence task.
However, such a heuristic still lacks theoretical support, and thus it is not necessarily guaranteed to perform well (or even converge) under naturally noisy observations.
[Table]
To bridge the above theoretical-computational gap, we propose the first provable single-loop algorithm for the GW distance computation, which we call Bregman Alternating Projected Gradient (BAPG).
The computational stumbling block here is the Birkhoff polytope constraint (i.e., polytope of doubly stochastic matrices) for the coupling matrix, as the associated Bregman projection is either computationally expensive or practically sensitive to hyperparameters.
To address this issue, we are inspired to decouple the Birkhoff polytope as simplex constraints for rows and columns separately, and then execute the projected gradient descent in an alternating fashion.
By leveraging the closed-form Bregman projection of the simplex constraint, BAPG only involves the matrixvector/matrix-matrix multiplications and element-wise matrix operations at each iteration.
In short, the proposed single-loop algorithm (BAPG) enjoys a variety of convenient properties: GPU-friendly implementation, robustness concerning the step size (i.e., the only hyper-parameter), and low memory cost.
Nonetheless, the iterates generated by BAPG do not necessarily satisfy the Birkhoff polytope constraint and can only reach a critical point of the original GW problem asymptotically.
Fortunately, some learning tasks, e.g., graph alignment and partition, only care about the matching accuracy (i.e., performance measure) but not the sharpness of the matching coupling.
Thus, we can sacrifice some feasibility to gain both matching accuracy and computational efficiency.
For these practically relevant tasks, BAPG is already able to obtain the desirable performance and achieve a great balance between accuracy and efficiency.
The comparison of various algorithms is summarized in Table 1.
Next, we investigate the approximation bound and convergence behavior of the proposed BAPG.
A key fact here is that the GW problem is a symmetric nonconvex quadratic program with polytope constraints.
By fully exploring this problem-specific structure, it is surprisingly interesting to note that the GW problem satisfies the Luo-Tseng error bound condition (Luo & Tseng, 1992), which acts as a main technical tool to bound the distance between the fixed-point set of BAPG and the critical point set of the GW problem.
This is a notable departure from the usual approach to utilizing the Luo-Tseng error bound condition in establishing the local linear convergence rate for structured convex problems (Zhou & So, 2017).
As a result, we are able to quantify the approximation bound for the fixed-point set of BAPG explicitly and further establish the subsequent convergence result.
Lastly, we provide extensive experiment results to validate the effectiveness of BAPG on graph alignment and graph partition.
In terms of both wall-clock time and modeling performance (i.e., matching accuracy), we have shown that BAPG consistently achieves superior performance compared with other heuristic single-loop or theoretically sound double-loop methods.
We also conduct the sensitivity analysis of BAPG and justify the effectiveness of its GPU acceleration on both synthetic and real-world datasets.
All theoretical insights and results have been well-corroborated in the experiments.
P ROPOSED A LGORITHM
In this section, we first formulate the GW distance as a nonconvex quadratic problem with Birkhoff polytope constraints.
Subsequently, we provide the theoretical insights and computational properties of the proposed BAPG.
2. P ROBLEM S ETUP
The Gromov-Wasserstein (GW) distance was originally proposed in (Mémoli, 2011; 2014; Peyré et al., 2019) for quantifying the distance between two probability measures supported on heterogeneous metric spaces.
More precisely:
Definition 2.1 (GW distance) .
Suppose that we are given two unregistered compact metric spaces (X , d X ) , (Y , d Y ) accompanied with Borel probability measures µ , ν respectively.
The GW distance between µ and ν is defined as
[Equation]
where Π ( µ , ν ) is the set of all probability measures on X × Y with µ and ν as marginals.
Intuitively, the GW distance is trying to preserve the isometric structure between two probability measures under optimal derivation.
If a map pairs x → y and x ′ → y ′ , then the distance between x and x ′ is supposed to be close to the distance between y and y ′ .
In view of these nice properties, the
GW distance acts as a powerful modeling tool in structural data analysis, especially in graph learning; see, e.g., (Vayer et al., 2019b; Xu et al., 2019b;a; Solomon et al., 2016; Peyré et al., 2016) and the references therein.
To start with our algorithmic developments, we consider the discrete case for simplicity and practicality, where µ and ν are two empirical distributions, i.e., µ =
∑ ni = 1 µ i δ x i and ν =
∑ mj = 1 ν j δ y j .
Then, the
GW-distance admits the following reformulation:
[Equation]
where D X and D Y are two symmetric distance matrices.
2. B REGMAN A LTERNATING P ROJECTED G RADIENT (BAPG)
Next, we present the proposed Bregman alternating projected gradient (BAPG) method, which is the first provable single-loop algorithm tailored to the GW distance computation.
It is easy to observe that Problem (1) is a nonconvex quadratic program with polytope constraints.
Naturally, if we want to invoke (Bregman) projected gradient descent type algorithms to address (1), the key difficulty lies in the Birkhoff constraint with respect to the coupling matrix π .
In fact, all existing theoretically sound algorithms rely on an inner iterative algorithm to tackle the regularized optimal transport problem at each iteration, such as Sinkhorn (Cuturi, 2013) or semi-smooth Newton method (Li et al., 2020).
Arguably, one of the main drawbacks of the double-loop scheme is its computational burden, which is not GPU friendly.
To circumvent this drawback, we are motivated to handle the row and columns constraints separately (i.e., C 1 = { π ≥
∶ π 1 m = µ } and C 2 = { π ≥
∶ π T 1 n = ν } ).
The crux of BPAG is to take the alternating projected gradient descent step between C 1 and C 2 .
Moreover, by fully detecting the hidden structures, we are able to further benefit from the fact that the Bregman projection with respect to the negative entropy for the simplex constraint (e.g., C 1 and
C 2 ) can be extremely efficient computed (Krichene et al., 2015).
The above considerations lead to the closed-form updates in each iteration of BAPG:
[Equation]
where ρ is the step size and ⊙ denotes element-wise (Hadamard) matrix multiplication.
BAPG enjoys several nice properties that are extremely attractive for medium or large-scale graph learning tasks.
First, BAPG is a single-loop algorithm that only involves matrix-vector/matrix-matrix multiplications and element-wise matrix operations.
All these operations are GPU-friendly.
Second, different from the entropic regularization parameter in eBPG, BAPG is more robust to the step size ρ in terms of solution performance.
Third, BAPG only involves one memory operation of a large matrix with size nm .
Notably, even for large-scale optimal transport problems, the main bottleneck is not floating-point computations, but rather time-consuming memory operations (Mai et al., 2021).
Theoretical Insights for BAPG Now, we start to give the theoretical intuition of why BAPG will work well in practice.
For simplicity, we consider the compact form for illustration:
[Equation]
Here, f ( π ) = − Tr
( D X πD Y π T ) is a nonconvex quadratic function; g 1 ( π )
=
I { π ∈ C 1 } and g 2 ( π ) =
I { π ∈ C 2 } are two indicator functions over closed convex polyhedral sets.
To better understand the alternating projected scheme developed in (2), we adopt the operator splitting strategy to reformulate (3) as
[Equation]
where f ( π , w ) = − Tr
( D X πD Y w T ) .
Then, the BAPG update (2) can be interpreted as processing the alternating minimization scheme on the constructed penalized function, i.e.,
[Equation]
Here, D h (⋅ , ⋅) is the so-called Bregman divergence, i.e., D h ( x , y ) ∶=
h ( x ) −
[Equation]
( y ) − ⟨∇ h
( y ) , x − y ⟩ , where h (⋅) is the Legendre function, e.g., 12 ∥ x ∥ 2 , relative entropy x log x , etc.
For the k -th iteration, the BAPG update takes the form
[Equation]
When h is the relative entropy and we use the same variable to update π and π , (2) can be derived from (5).
It is worth noting that the way to invoke the operator splitting strategy is not for the algorithm design as usual but instead to provide an intuitive way to explain why BAPG will work.
As we shall see later, it plays an important role in deriving the approximation bound of the fix-point set of BAPG instead.
Moreover, similar to the quadratic penalty method (Nocedal & Wright, 2006), BAPG is an infeasible method that can only converge to a critical point of (1) in an asymptotic sense.
In other words, if we choose the parameter ρ as a constant, there is always an infeasibility gap.
Fortunately, BAPG is already able to achieve the desired performance for some graph learning tasks, especially those that care more about implementation efficiency and matching accuracy.
Recently, (Vincent-Cuaz et al., 2021a) has proposed a relaxed version of GW distance for the graph partition task, which further corroborates our investigation that it is acceptable and promising to sacrifice some feasibility to gain other benefits.
Moreover, Séjourné et al.
(2021) also proposed a closely related marginal relaxation.
That is, we make π = w and F ρ ( π , π ) is the objective introduced in (Séjourné et al., 2021).
Unfortunately, they did not develop any provable algorithms for the unbalanced GW distance.
In
Sections 4.2 and 4.3, extensive experiments have been conducted to demonstrate that BAPG has superior performance compared with other existing baselines on graph alignment and partition tasks.
T HEORETICAL R ESULTS
In this section, we present all theoretical results conducted in this paper, including the approximation bound of the fix-point set of BAPG and its convergence analysis.
At the heart of our analysis is that the following regularity condition holds for the GW problem (1).
Proposition 3.1 (Luo-Tseng Error
Bound Condition for (1)) .
There exist scalars ϵ > 0 and τ >such that
[Equation]
whenever ∥ π − proj
[Equation]
Y )
∥ ≤ ϵ , where X is the critical point set of (3) defined by
[Equation]
and N C ( π ) is the normal cone to C at π .
As the GW problem is a nonconvex quadratic program with polytope constraint, we can invoke
Theorem 2.3 in (Luo & Tseng, 1992) to conclude that the error bound condition (6) holds on the whole feasible set C 1 ∩ C 2 .
Proposition 3.1 extends (6) to the whole space R n × m .
This regularity condition is trying to bound the distance of any coupling matrix to the critical point set of the GW problem by its optimality residual, which is characterized by the difference for one step projected gradient descent.
It turns out that this error bound condition plays an important role in quantifying the approximation bound for the fixed points set of BAPG explicitly.
3. A PPROXIMATION B OUND FOR THE F IX -P OINT S ET OF BAPG
To start, we present one key lemma that shall be used in studying the approximation bound of BAPG.
Lemma 3.2.
Let C 1 and C 2 be convex polyhedral sets.
There exists a constant M > 0 such that
[Equation]
The proof idea follows essentially from the observation that the inequality can be regarded as the stability of the optimal solution for a linear-quadratic problem, i.e.,
[Equation]
The parameter r is indeed the perturbation quantity.
If r = 0 , we have p ( 0 ) =
q ( 0 ) =
proj C 1 ∩ C 2 ( x + y 2 ) ; by choosing r = proj C 1 ( x )− proj C 2 ( y ) , it is easy to see that ( p ( r ) , q ( r )) =
( proj C 1 ( x ) ,proj C 2 ( y )) .
Together with Theorem 4.1 in (Zhang & Luo, 2022), the desired result is obtained.
All the proof details are given in Appendix.
Equipped with Lemma 3.2 and Proposition 3.1, it is not hard to obtain the following approximation result.
Proposition 3.3 (Approximation Bound of the Fix-point Set of BAPG) .
The point ( π ⋆ , w ⋆ ) belongs to the fixed-point set X BAPG of BAPG if it satisfies
[Equation]
where p ∈ N C 1 ( π ⋆ ) and q ∈ N C 2 ( w ⋆ ) .
Then, the infeasibility error satisfies ∥ π ⋆ − w ⋆ ∥ ≤ τ 1 ρ and the gap between X BAPG and X satisfies
[Equation]
where τ 1 and τ 2 are two constants.
Remark 3.4.
If π ⋆ = w ⋆ , then X BAPG ( (8) ) is identical to X and BAPG can reach a critical point of the GW problem (1) .
Proposition 3.3 indicates that as ρ → +∞ , the infeasibility error term ∥
π ⋆ − w ⋆ ∥ shrinks to zero and thus BAPG converges to a critical point of (1) in an asymptotic way.
Furthermore, it explicitly quantifies the approximation gap when we select the parameter ρ as a constant.
The proof can be found in Appendix.
The explicit form of τ 1 and τ 2 only depend on the problem itself, including σ max ( D X ) σ max ( D Y ) , the constant for the Luo-Tseng error bound condition in Proposition 3.1 and so on.
3. C ONVERGENCE A NALYSIS OF BAPG
A further natural question is whether BAPG will converge or not.
We answer the question in the affirmative.
Specifically, we show that under several canonical assumptions, any limit point of BAPG belongs to X BAPG .
Towards that end, let us first establish the sufficient decrease property of BAPG based on the potential function F ρ (⋅) ,
Proposition 3.5.
Let {( π k , w k )} k ≥ 0 be the sequence generated by BAPG.
Suppose that D h (⋅ , ⋅) is symmetric on the whole sequence.
Then, we have
[Equation]
As F ρ (⋅) is coercive, we have ∑ ∞ k =
D
[Equation]
( π
[Equation]
, π
[Equation]
) <
+∞ .
Both { D h ( π k , π k + 1 )} k
≥and { D h ( w k , w k + 1 )} k ≥ 0 converge to zero.
Thus, the following convergence result holds.
Theorem 3.6 (Subsequent Convergence of BAPG) .
Any limit point of the sequence {( π k , w k )} k ≥generated by BAPG belongs to X BAPG .
[Figure]
Remark 3.7.
The symmetric assumption in Proposition 3.5 does not hold for the KL divergence in general.
However, if we add a mild condition that the asymmetric ratio is bounded for iterations, similar results can be achieved.
Moreover, when h is a quadratic function, we can further obtain the global convergence result under the Kurdyka-Lojasiewicz analysis framework developed in (Attouch et al., 2010; 2013).
To the best of our knowledge, the convergence analysis of alternating projected gradient descent methods has only been given under the convex setting, see (Wang & Bertsekas, 2016; Nedi ´ c, 2011) for details.
In this paper, by heavily exploiting the error bound condition of the GW problem, we take the first step and provide a new path to conduct the analysis of alternating projected descent method for nonconvex problems, which could be of independent interest.
E XPERIMENT R ESULTS
In this section, we provide extensive experiment results to validate the effectiveness of the proposed
BAPG on various representative graph learning tasks, including graph alignment and graph partition.
All simulations are implemented using Python 3.9 on a high-performance computing server running
Ubuntu 20.04 with an
Intel(R) Xeon(R)
Gold 6226R CPU and an NVIDIA GeForce RTX 3090 GPU.
For all methods conducted in the experiment part, we use the relative error ∥
π k + 1 − π k ∥ 2 /∥
π k ∥ 2 ≤ 1 e −and the maximum iteration as the stopping criterion, i.e., min { k ∈ Z ∶ ∥
π k + 1 − π k ∥ 2 /∥ π k ∥ 2 ≤ 1 e − 6 and k ≤ 2000 } .
4. T OY 2D M ATCHING P ROBLEM
In this subsection, we study a toy matching problem in 2D to corroborate our theoretical insights and results in Sec 2 and 3.
Fig.
1 (a) shows an example of mapping a two-dimensional shape without any symmetries to a rotated version of the same shape.
Here, we sample 300 and 400 points from source and target shapes respectively and use the Euclidean distance to construct the distance matrices D X and D Y .
Figs.1 (b)-(f) provide all color maps of coupling matrices to visualize the matching results.
Here, the sparser coupling matrices indicate sharper mapping.
All experiment results are consistent with
Table 1.
We can observe that both BPG and FW give us satisfactory solution performance as they aim at solving the GW problem exactly.
However, FW and BPG will suffer from a huge computation burden which will be further justified in Sec 4.2 and 4.3.
On the other hand, the performance of
BPG-S and eBPG are obviously harmed by the infeasibility issue.
The sharpness of BAPG’s coupling matrix is relatively not effected by its infeasibility issue too much although its coupling matrix is denser than BPG and FW ones.
As we shall see later, the effect of the infeasibility issue is minor
[Table]
when the penalty parameter ρ is not too small and will not even result in even a real cost for graph alignment and partition tasks, which only care about the matching accuracy instead of the sharpness of the coupling.
4. G RAPH A LIGNMENT
Graph alignment aims to identify the node correspondence between two graphs possibly with different topology structures (Zhang et al., 2021;
Chen et al., 2020).
Instead of solving the restricted quadratic assignment problem (Lawler, 1963; Lacoste-Julien et al., 2006), the GW distance provides the optimal probabilistic correspondence relationship via preservation of the isometric property.
Here, we compare the proposed BAPG with all existing baselines: FW (Vayer et al., 2019a), BPG (Xu et al., 2019b),
BPG-S (Xu et al., 2019b)
(i.e., the only difference between BPG and BPG-S is that the number of inner iterations for BPG-S is just one), ScalaGW
(Xu et al., 2019a), SpecGW (Chowdhury & Needham, 2021), and eBPG (Solomon et al., 2016).
Except for BPG and eBPG, others are pure heuristic methods without any theoretical guarantee.
Parameters Setup We utilize the unweighted symmetric adjacency matrices as our input distance matrices, i.e., D X and D Y .
Alternatively, SpecGW uses the heat kernel exp (− L ) where L is the normalized graph Laplacian matrix.
We set both µ and ν to be the uniform distribution.
For three heuristic methods — BPG-S, ScalaGW, and SpecGW, we follow the same setup reported in their papers.
As mentioned, eBPG is very sensitive to the entropic regularization parameter.
To get comparable results, we report the best result among the set { 0.1,0.01,0.001 } of the regularization parameter.
For BPG and BAPG, we use the constant step size 5 and ρ = 0.1 respectively.
For FW, we use the default implementation in the PythonOT package (see Appendix D).
All the experiment results reported here were the average of 5 independent trials over different random seeds and the standard deviation is collected in Appendix.
Database Statistics We test all methods on both synthetic and real-world databases.
Our setup for the synthetic database is the same as in (Xu et al., 2019b).
The source graph G s =
{V s , E s } is generated by two ideal random models, Gaussian random partition and Barabasi-Albert models, with different scales, i.e., ∣V s ∣ ∈ { 500,1000,1500,2000,2500 } .
Then, we generate the target graph
G t =
{V t , E t } by first adding q % noisy nodes to the source graph, and then generating q % noisy edges between the nodes in V t , i.e., ∣V t ∣ =
( 1 + q % )∣V s ∣ , ∣E t ∣ =
( 1 + q % )∣E s ∣ , where q ∈ { 0,10,20,30,40,50 } .
For each setup, we generate five synthetic graph pairs over different random seeds.
To sum up, the synthetic database contains 300 different graph pairs.
We also validate our proposed methods on other three real-world databases from (Chowdhury & Needham, 2021), including two biological graph databases Proteins and Enzymes , and a social network database Reddit .
Furthermore, to demonstrate the robustness of our method regarding the noise level, we follow the noise generating process (i.e., q = 10% ) conducted for the synthesis case to create new databases on top of the three real-world databases.
Towards that end, the statistics of all databases used for the graph alignment task have been summarized in Appendix D.
We match each node in G s with the most likely node in G t according to the optimized π ⋆ .
Given the predicted correspondence set S pred and the ground-truth correspondence set S gt , we calculate the matching accuracy by Acc = ∣S gt ∩ S pred ∣/∣S gt ∣ × 100% .
[Table]
Results of Our Methods Table 2 shows the comparison of matching accuracy and wall-clock time on four databases.
We observe that BAPG works exceptionally well both in terms of computational time and accuracy, especially for two large-scale noisy graph databases Synthetic and Reddit .
Notably,
BAPG is robust enough so that it is not necessary to perform parameter tuning.
As we mentioned in
Sec 2, the effectiveness of GPU acceleration for BAPG is also well corroborated on Synthetic and
Reddit .
GPU cannot further speed up the training time of Proteins and Reddit as graphs in these two databases are small-scale.
Additional experiment results to demonstrate the robustness of BAPG and its GPU acceleration will be given in Sec 4.4.
Comparison with Other Methods
The performance of eBPG and ScalaGW are influenced by the entropic regularization parameter and approximation error respectively, which accounts for their poor performance.
Moreover, it is easy to observe that SpecGW works pretty well on the small dataset but the performance degrades dramatically on the large one, e.g., synthetic .
The reason is that SpecGW relies on a linear programming solver as its subroutine, which is not well-suited for large-scale settings.
Besides, although ScalaGW has the lowest per-iteration computational complexity, the recursive K-partition mechanism developed in (Xu et al., 2019a) is not friendly to parallel computing.
Therefore, ScalaGW does not demonstrate attractive performance on multi-core processors.
4. G RAPH P ARTITION
The GW distance can also be potentially applied to the graph partition task.
That is, we are trying to match the source graph with a disconnected target graph having K isolated and self-connected super nodes, where K is the number of clusters (Abrishami et al., 2020).
Similarly, we compare the proposed BAPG with other baselines described in Section 4.2 on four real-world graph partitioning datasets.
Following (Chowdhury & Needham, 2021), we also add three non-GW methods specialized in graph alignment, including FastGreedy (Clauset et al., 2004), Louvain (Blondel et al., 2008), and
Infomap (Rosvall & Bergstrom, 2008).
Parameters Setup For the input distance matrices D X and D Y , we test our methods on both the adjacency matrices and the heat kernel matrices proposed in (Chowdhury & Needham, 2021).
For
BAPG, we pick up the lowest converged function value among ρ ∈ { 0.1,0.05,0.01 } for adjacency matrices and ρ ∈ { 0.01,0.005,0.001 } for heat kernel matrices.
The quality of graph partition results is quantified by computing the adjusted mutual information (AMI) score (Vinh et al., 2010) against the ground-truth partition.
Results of All Methods Table 3 shows the comparison of AMI scores among all methods on graph partition.
BAPG outperforms other GW-based methods in most cases and is more robust to the noisy setting.
Specifically, BAPG is consistently better than FW and SpecGW which both rely on the Frank-Wolfe method to solve the problem.
eBPG has comparable results using the adjacency matrices but is unable to process the spectral matrices.
The possible reason is that the adjacency matrix and the heat kernel matrix admit quite different structures, e.g., the former is sparse while
[Table] [Table]
[Figure]
1e
[Figure]
1e1e- 1e1e- 1eI n f e a s i b ili t y
E rr
o r =0.
=0.
=0. =0.
(b
[Figure]
the latter is dense.
BPG and BPG-S enjoy similar performances in most cases, but both are not as good as our proposed BAPG on all datasets.
BAPG also shows competitive performance compared to the specialized non-GW graph partition methods.
For example, BAPG outperforms Infomap and
Louvain in 6 out of 8 scores, respectively.
4. T HE E FFECTIVENESS AND R OBUSTNESS OF BAPG
At first, we target at demonstrating the robustness of BAPG on graph alignment, as it would be more reasonable to test the robustness of a method on a database (e.g., graph alignment) rather than a single point (e.g., graph partition).
Noise Level and Graph Scale
At the beginning, we report the sensitivity analysis of BAPG regarding the noise level q % and the graph scale ∣V s ∣ in Fig.
2 on the Synthetic Database.
Surprisingly, the solution performance of BAPG is robust to both the noise level and graph scale.
In contrast, the accuracy of other methods degrades dramatically as the noise level or the graph scale increases.
Step Size ρ Subsequently, we conduct the sensitivity analysis of BAPG with respect to the step size ρ on four real-world databases.
Table 4 showcases that the matching performance of BAPG is robust (not sensitive) regarding the step size.
Moreover, convergence curves of different ρ in Fig.corroborate Proposition 3.3 empirically and further guide us on how to choose the step size.
The larger ρ means the smaller infeasibility error but slower convergence speed.
Together with Table 4, we can conclude that if the resulting infeasibility error is not quite large (i.e., the step size ρ is not so small), the final matching accuracy is robust to the step size.
Under this situation, the step size only affects the convergence speed.
GPU Acceleration of BAPG.
We conduct experiment results to further justify that BAPG is
GPU-friendly.
In Table 5, We compare the acceleration ratio (i.e., CPU wall-clock time divided by GPU wall-clock time) of BAPG, eBPG, and BPG on two large-scale graph alignment datasets using the same computing server.
For eBPG, we use the official implementation in the PythonOT package, which supports running on GPU.
For BPG, we implement the GPU version by ourselves using Pytorch.
We can find that BAPG has a much higher acceleration ratio on the GPU compared with BPG and eBPG.
C LOSING R EMARK
In this paper, we conduct a systematic investigation on developing the provable single-loop algorithm (i.e., BAPG) for the GW distance computation.
By exploiting the error bound condition (i.e., never investigated in the GW literature before), we are able to conduct the convergence analysis of BAPG.
For a broader viewpoint, our work also takes the first step and provides a rather new path to study the alternating projected descent method for general nonconvex problems.
To the best of our knowledge, it is still an open question even in the optimization community.
On the computational side, the proposed single-loop algorithm BAPG still suffers from the cubic per-iteration computational complexity, which will further limit its applications in real-world large-scale settings.
A natural future direction is to consider sparse and low-rank structures of the matching matrix to decrease the per-iteration cost and further speed up our method (Scetbon et al., 2021).
Moreover, for simplicity, we assume that
D X and D Y are two symmetric distance matrices.
However, the proposed BAPG can be potentially applied to the non-symmetric case as the Luo-tseng error bound condition still holds and algorithmic steps keep the same.
A A PPENDIX
Overall, the norm ∥ ⋅ ∥ means ℓ 2 norm.
Both dist and proj are defined w.r.t.
ℓ 2 norm, i.e., for any convex set C and a point x ,
[Equation]
B P ROBLEM P ROPERTIES OF (1)
In this paper
[Equation]
we consider the discrete case
[Equation]
where only m , n samples are drawn from real distributions,
i
[Equation]
e., µ = ∑ ni = 1 µ i δ x i and ν = ∑ mj = 1 ν j δ y j .
The Gromov-Wasserstein distance between µ and ν can be recast into the following minimization problem:
[Equation]
where Π ( µ
[Equation]
ν ) is a generalized Birkhoff polytope
[Equation]
i
[Equation]
e.
[Equation]
Π
[Equation]
To further simplify the problem and detect the hidden structures
[Equation]
we reformulate it as a compact form,
[Equation]
∑ i =n ∑ j =m ∑ k =m ∑ l = 1 D X ( i , j ) D Y ( k , l ) π ik π jl =
n ∑ i =n ∑ j = 1 D 2 X ( i , j ) µ i µ j + m ∑ k
=m ∑ l = 1 D 2 Y ( k , l ) ν k ν l ·„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„‚„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„¶ constant −
Tr ( D X πD Y π T ) .
where D X ( i , j ) =
d X ( x i , x j ) and D Y ( k , l ) =
d Y ( y k , y l ) for i , j ∈ [ n ] and k , l ∈ [ m ] .
Thus, (1) is obtained.
Proof of Proposition 3.1 — Luo-Tseng Error Bound Condition for the GW Problem (1)
Proof.
D X and D Y are two symmetric matrices and C
∩
C 2 is a convex polyhedral set.
By invoking
Theorem 2.3 in [24], the Luo-Tseng local error bound condition (6) just holds for the feasible set
C 1 ∩ C 2 .
That is,
[Equation]
where π ∈ C 1 ∩ C 2 .
Then, we aim at extending (12) to the whole space.
Define ˜ π
= proj
C 1 ∩ C 2 ( π ) , we have
[Equation]
where the first inequality holds because of the triangle inequality, and the second one holds because of the fact ˜ π ∈ C 1 ∩ C 2 and (12).
Apply the triangle inequality again, we have
[Equation]
Since ˜ π = proj C 1 ∩ C 2 ( π ) , owing to the fact that the projection operator onto a convex set proj C 1 ∩ C 2 (⋅) is non-expensive, i.e., ∥ proj
[Equation]
− y ∥ holds for any x and y , we have ∥
[Equation] [Equation]
and
[Equation]
where σ max ( D X ) and σ max ( D Y ) denote the maximum singular value of D X and D Y , respectively.
By substituting (14) and (16) into (13), we get
[Equation]
where the last inequality holds because of the fact that
[Equation]
By letting τ = τσ max ( D X ) σ max ( D Y ) + 2 τ + 1 , we get the desired result.
C C ONVERGENCE A NALYSIS OF BAPG
Assumption C.1.
The critical point set X is non-empty.
Definition C.2 (Bregman Divergence) .
We define the proximity measure D h ∶ dom ( h ) × int ( dom ( h ))
→ R +
[Equation]
The proximity measure D h is the so-called Bregman Distance.
It measures the proximity between x and y .
Indeed, thanks to the gradient inequality, one has h is convex if and only if D h ( x , y ) ≥ 0, ∀ x ∈ dom
h , y ∈ intdom h .
Concrete Examples
• Relative entropy, or Kullback–Leibler divergence, h ( x )
=
∑ i x i log x i and the Bregman divergence is given as
[Equation]
• Quadratic function: h ( x ) =
∥ x ∥ 2 .
As such, the Bregman divergence D h ( x , y ) will reduce to the Euclidean distance.
For the completeness of our algorithmic development, we also give the details how the general BAPG (5) update implies (2) if we choose the Legendre function as relative entropy.
[Equation]
where P k = w k ⊙ exp
( D X w k D Y ρ ) .
[Equation]
where Z k = π k + 1 ⊙
exp
( D X π k + 1 D Y ρ ) .
The optimality conditions for each iteration follow,
[Equation] [Equation]
Proof of Lemma 3.
Proof.
To prove
[Equation]
we first convert the left-hand side of (19) to
[Equation]
Then, the inequality can be regarded as the stability of the optimal solution for a linear-quadratic problem,
i
[Equation]
e.,
[Equation]
When r =
[Equation]
the pair ( p ( 0 ) , q ( 0 )) satisfies p ( 0 ) =
q ( 0 ) =
proj C 1 ∩ C 2 ( x + y 2 ) .
Moreover, the parameter r itself can be viewed as the perturbation quantity, which is indeed the right-hand side of (19).
By invoking Theorem 4.1 in (Zhang & Luo, 2022), we can bound the distance between two optimal solutions by the perturbation quantity r , i
[Equation]
e.,
[Equation]
Remark C.3.
For any vector r , let ( p ( r ) , q ( r )) be the optimal solution to (20) .
On the one hand, choosing r = 0 , it is easy to see that p ( 0 ) =
q ( 0 ) = proj
[Equation]
On the other hand, by choosing r = proj C 1 ( x ) − proj C 2 ( y ) , it is easy to see that ( p ( r ) , q ( r )) =
( proj C 1 ( x ) ,proj C 2 ( y )) .
Since Theorem 4.1 in (Zhang & Luo, 2022) implies that for any r , we have
[Equation]
where M only depends on C 1 and C 2 .
By choosing r = proj C 1 ( x ) − proj C 2 ( y ) , we have
[Equation]
Proof of Proposition 3.3 — Approximation bound of fixed points of BAPG
Definition C.4 ( Bounded Linear Regularity (cf.
Definition 5.6 in Bauschke (1996))) .
Let
C 1 , ... , C N be closed convex subsets of R n with a non-empty intersection C .
We say that the collection { C 1 , ... , C N } is bounded linearly regular (BLR) if for every bounded subset B of R n , there exists a constant κ > 0 such that
[Equation]
If all of C i are polyhedral sets, the BLR condition will automatically hold.
Proof.
Recall the fixed-point set of BAPG:
[Equation]
Define ˆ π = proj
C 1 ∩ C 2 ( π ⋆ ) , we first want to argue the following inequality holds,
[Equation]
As the BLR condition is satisfied for the polyhedral constraint, see Definition C.4 for details, we have
[Equation]
Based on the stationary points defined in (8)
[Equation]
we have,
[Equation]
Summing up the above two equations,
[Equation]
where (♣) holds as (∇ h ( x ) − ∇ h
[Equation]
y )
[Equation]
+ D h
( y , x ) and (♠) follows from the property of the normal cone.
For instance, since q ∈ N C 1 ( π ⋆ ) , we have q T ( ˆ π − π ⋆ ) ≤ 0 (i.e., ˆ π ∈ C 1 ).
Therefore,
[Equation]
The next to last inequality holds as f (⋅) is a quadratic function and the effective domain C 1 ∩ Cis bounded.
Thus, the norm of its gradient will naturally have a constant upper bound, i.e., L f = σ max ( D X ) σ max ( D Y ) .
As h is a σ -strongly convex, we have
[Equation]
Together with this property, we can quantify the infeasibility error ∥
π ⋆ −
[Equation]
⋆ ∥
[Equation]
When ρ → +∞ , it is easy to observe that the infeasibility error term ∥ π ⋆ − w ⋆ ∥ will shrink to zero.
More importantly, if π ⋆ = w ⋆ , then X BAPG will be identical to X .
Next, we target at quantifying the approximation gap between the fixed-point set of BAPG and the critical point set of the original problem (1).
Upon (21), we have
[Equation]
where ∇ f (⋅) is a linear operator.
By applying the Luo-Tseng local error bound condition of (1), i.e.,
Proposition 3.1, we have dist
[Equation]
where (♣) holds due to the normal cone property,
that is, proj C ( x + z ) =
x for any x ∈ C and z ∈ N C ( x ) , and (♠) follows from Lemma 3.2.
Incorporating with (23), the approximation bound for
BAPG has been characterized quantitatively, i.e.,
[Equation]
Proof of Proposition 3.5 — Sufficient decrease property of BAPG
Proof.
We first observe from the optimality conditions of main updates, i
[Equation]
e.,
[Equation]
where g 1 ( π ) = I { π ∈ C 1 } and g 2 ( w )
=
I { w ∈ C 2 } .
Due to the convexity of g 1 (⋅) , it is natural to obtain,
[Equation]
As f ( π , w ) = − tr ( D X πD Y w T ) is a bilinear function
[Equation]
we have,
[Equation]
Consequently, we get
[Equation]
Similarly, based on the w -update
[Equation]
we obtain
[Equation]
Combine with (26) and (27), we obtain
[Equation]
The right-hand side can be further simplified,
[Equation]
Here, (♣) uses the fact that the three-point property of Bregman divergence holds,
i
[Equation]
e., For any y , z ∈ int dom h and x ∈ dom h ,
[Equation]
Moreover, (♠) holds as D h (⋅ , ⋅) is symmetric on the whole sequence.
To proceed, this together with (28) implies
[Equation]
Summing up (29) from k = 0 to +∞ , we obtain
[Equation]
As the potential function F ρ (⋅ , ⋅) is coercive and {( π k , w k )} k ≥ 0 is a bounded sequence, it means the left-hand side is bounded, which implies
[Equation]
Proof of Theorem 3.6 — Subsequence convergence result of BAPG
Proof.
Let ( π ∞ , w ∞ ) be a limit point of the sequence {( π k , w k )} k ≥ 0 .
Then, there exists a sequence { n k } k ≥ 0 such that {( π n k , w n k )} k ≥ 0 converges to ( π ∞ , w ∞ ) .
Replacing k by n k in (24) and (25), taking limits on both sides as k → ∞
[Equation]
Based on the fact that ∇ π f ( π ∞ , w ∞ ) = ∇ f ( w ∞ ) and ∇ w f ( π ∞ , w ∞ )
= ∇ f ( π ∞ ) , it can be easily concluded that ( π ∞ , w ∞ ) ∈
X BAPG .
More importantly, the above subsequence convergence result can be extended to the global convergence if the Legendre function h (⋅) is quadratic.
Theorem C.5 (Global Convergence of BAPG — Quadratic Case) .
The sequence {( π k , w k )} k ≥converges to a critical point of F ρ ( π , w ) .
Proof.
To invoke the Kurdyka-Lojasiewicz analysis framework developed in (Attouch et al., 2010; 2013), we have to establish two crucial properties — sufficient decrease and safeguard condition.
The first one has already been proven in Proposition 3.5.
Then, we would like to show that there exists a constant κ 2 such that,
[Equation]
At first, noting that
[Equation]
Again, together with the updates (24) and (25), this implies
[Equation]
where σ max (⋅) denotes the maximum singular value.
By letting κ 2 = √ σ 2 max ( D X ) σ 2 max ( D Y ) + ρ , we get the desired result.
D A DDITIONAL E XPERIMENT D ETAILS
[Table]
Source codes of all baselines used in this paper and their licenses
• BPG-S (GPL-3.0 license)
(Xu et al., 2019b):
https://github.com/HongtengXu/ gwl • ScalaGW
(No license)
(Xu et al., 2019a): https://github.com/HongtengXu/ s-gwl • SpecGW (MIT license) (Chowdhury & Needham, 2021):
https://github.com/trneedham/Spectral-Gromov-Wasserstein • eBPG (MIT license)
(Flamary et al., 2021):
https://github.com/PythonOT/POT
[Table]
D
[Table]
M ULTI -O MICS S INGLE -C ELL I NTEGRATION
To validate the application capability of BAPG beyond graph alignment and partition
[Table]
we further evaluate the proposed BAPG on an additional task (alignment/matching based), multi-omics singlecell integration.
It aims at integrating multiple molecular features at different modalities in a cell, e
[Table]
g., gene expressions and chromatin accessibility (Teichmann & Efremova, 2020; Wen et al., 2022; Cao et al., 2020; 2022).
This task offers opportunities for gaining holistic views of cells and is selected as “Method of the Year 2019” by Nature (Teichmann & Efremova, 2020).
Task Definition Suppose we have two single cell multi omics datasets, X
=
[ x 1 ;...; x n x ] ∈ R n x × d x and Y =
[ y 1 ;...; y n y ] ∈ R n x × d x across two modalities, where n x / n y and d x / d y are the number of cell samples and feature dimensions for X/Y, respectively.
Given X and Y, the tasks of multi-omics single-cell integration aims at calculating the joint representation of two modalities.
In the joint representation, the same cell types in X and Y should be clustered together.
Thus, aligning the same types of cells across X and Y in advance can improve the quality of the joint representation.
[Table]
Dataset and Evaluation
Following (Cao et al., 2020; 2022), we conduct experiments on two real-world single-cell multi-omics datasets: (1) the scGEM dataset for the single-cell analysis of genotype, expression and methylation data and (2) the scNMT dataset for the single-cell analysis of nucleosome, methylome and transcriptome data.
The dataset statistics is listed in Table 8.
We use the label transfer accuracy (i.e., the percentage of aligned cell pairs with the same cell type across two modalities) to measure the ability to transfer labels of the shared cells from one modality to another.
Baselines We compare the proposed BAPG with other GW methods in our paper by directly setting the distance matrix D X and D Y as the feature similarity matrix.
Besides, we add two baselines specified for this task, including UnionCom (Cao et al., 2020) and Pamona (Cao et al., 2022).
Pamona also relies on calculating the GW distance method and uses eBPG as the GW solver.
We replace eBPG in Panona with BAPG and obtain a new method named Pamona-BAPG.
Results of All Methods Table 9 summarizes the results of all compared methods on the scGWM and scNMT datasets.
Among all GW methods, BAPG achieves the best performance on 4 out of 6 scores and has the shortest running time.
UnionCom and Pamona are better than most pure GW methods as they involve additional pre-processing and post-processing techniques specified for this task.
Moreover, Pamona-BAPG outperforms Pamona on 5 out of 6 scores and reduces the running time to 16% 25%.
It indicates that the calculation of GW distance is a bottleneck for this task, and
a
[Table]
better GW solver can greatly facilitate related downstream applications.
As it is just a preliminary study to show the potential of applying BAPG to other tasks, we left other potential application-driven tasks as our future work
[Table]