Low-rank Optimal Transport: Approximation, Statistics and Debiasing
Abstract
The matching principles behind optimal transport (OT) play an increasingly important role in machine learning, a trend which can be observed when OT is used to disambiguate datasets in applications (e.g. single-cell genomics) or used to improve more complex methods (e.g. balanced attention in transformers or self-supervised learning).
To scale to more challenging problems, there is a growing consensus that
OT requires solvers that can operate on millions, not thousands, of points.
The lowrank optimal transport (LOT) approach advocated in Scetbon et al.
[2021] holds several promises in that regard, and was shown to complement more established entropic regularization approaches, being able to insert itself in more complex pipelines, such as quadratic OT.
LOT restricts the search for low-cost couplings to those that have a low-nonnegative rank, yielding linear time algorithms in cases of interest.
However, these promises can only be fulfilled if the LOT approach is seen as a legitimate contender to entropic regularization when compared on properties of interest, where the scorecard typically includes theoretical properties (statistical complexity and relation to other methods) or practical aspects (debiasing, hyperparameter tuning, initialization).
We target each of these areas in this paper in order to cement the impact of low-rank approaches in computational OT.
Introduction
Optimal transport (OT) is used across data-science to put in correspondence different sets of observations.
These observations may come directly from datasets, or, in more advanced applications, depict intermediate layered representations of data.
OT theory provides a single grammar to describe and solve increasingly complex matching problems (linear, quadratic, regularized, unbalanced, etc...), making it gain a stake in various areas of science such as as single-cell biology Schiebinger et al. [2019], Yang et al.
[2020], Demetci et al.
[2020], imaging Schmitz et al.
[2018], Heitz et al.
Zheng et al.
[2020] or neuroscience Janati et al.
[2020], Koundal et al.
Regularized approaches to OT.
Solving OT problems at scale poses, however, formidable challenges.
The most obvious among them is computational: the Kantorovich [1942] problem on discrete measures of size n is a linear program that requires O ( n 3 log n ) operations to be solved.
A second and equally important challenge lies in the estimation of OT in high-dimensional settings, since it suffers from the curse-of-dimensionality Fournier and Guillin [2015].
The advent of regularized approaches, such as entropic regularization [Cuturi, 2013], has pushed these boundaries thanks for faster algorithms [Chizat et al., 2020, Clason et al., 2021] and improved statistical aspects [Genevay et al., 2018a].
Despite these clear strengths, regularized OT solvers remain, however, costly as they typically scale quadratically in the number of observations.
Scaling up OT using low-rank couplings.
While it is always intuitively possible to reduce the size of measures (e.g. using k -means) prior to solving an OT between them, a promising line of work proposes to combine both [Forrow et al., 2019, Scetbon et al., 2021, 2022].
Conceptually, these low-rank approaches solve simultaneously both an optimal clustering/aggregation strategy with the computation of an effective transport.
This intuition rests on an explicit factorization of couplings into two sub-couplings.
This has several computational benefits, since its computational cost becomes linear in n if the ground cost matrix seeded to the OT problem has itself a low-rank.
While these computational improvements, mostly demonstrated empirically, hold several promises, the theoretical properties of these methods are not yet well established.
This stands in stark contrast to the Sinkhorn approach, which is comparatively much better understood.
Our Contributions.
The goal of this paper is to advance our knowledge, understanding and practical ability to leverage low-rank factorizations in OT.
This paper provides five contributions, targeting theoretical and practical properties of LOT: (i) We derive the rate of convergence of the low-rank OT to the true OT with respect to the non-nnegative rank parameter.
(ii)
We make a first step towards a better understanding of the statistical complexity of LOT by providing an upper-bound of the statistical error, made when estimating LOT using the plug-in estimator; that upper-bound has a parametric rate
O ( (cid:112) 1 /n )
that is independent of the dimension.
(iii)
We introduce a debiased version of LOT: as the Sinkhorn divergence [Feydy et al., 2018], we show that debiased LOT is nonnegative, metrizes the weak convergence, and that it interpolates between the maximum mean discrepancy [Gretton et al., 2012] and OT.
(iv) We exhibit links between the bias induced by the low-rank factorization and clustering methods.
(v) We propose practical strategies to tune the step-length and the initialization of the algorithm in [Scetbon et al., 2021].
Notations.
We consider ( X , d X ) and ( Y , d Y ) two nonempty compact Polish spaces and we denote
M +1 ( X ) (resp.
M +1 ( Y ) )
the space of positive Radon probability measures on X (resp.
Y ).
For all n ≥ 1 , we denote ∆ n the probability simplex of size n and ∆ ∗ n the subset of ∆ n of positive histograms.
We write 1 n ≜ (1 , . . .
, 1) T ∈ R n
and we denote similarly ∥ · ∥ 2 the Euclidean norm and the Euclidean distance induced by this norm depending on the context.
Background on Low-rank Optimal Transport
Let µ ∈ M +1 ( X ) , ν ∈ M +1 ( Y ) and c : X × Y → R + a nonnegative and continuous function.
The
Kantorovitch formulation of optimal transport between µ and ν is defined by
[Equation]
where the feasible set is the set of distributions over the product space X × Y with marginals µ and ν :
[Equation]
with P 1# π (resp.
P 2# π ), the pushforward probability measure of π using the projection maps
P 1 ( x, y )
=
x (resp.
P 2 ( x, y ) =
y ).
When there exists an optimal coupling solution of (1) supported on a graph of a function, we call such function a Monge map.
In the discrete setting, one can reformulate the optimal transport problem as a linear program over the space of nonnegative matrices satisfying the marginal constraints.
More precisely, let a and b be respectively elements of ∆ ∗ n and ∆ ∗ m and let also X ≜ { x 1 , . . .
, x n } and Y ≜ { y 1 , . . . , y m } be respectively two subsets of X and Y .
By denoting µ a, X ≜ (cid:80) ni =1 a i δ x i and ν b, Y ≜ (cid:80) mj
=1 b j δ y
j
the two discrete distributions associated and writing C ≜
[ c ( x i , y j )] i,j , the discrete optimal transport problem can be formulated as
OT
[Equation]
[Equation]
P
[Equation]
,b
[Equation]
C, P ⟩ where Π a,b ≜ { P ∈ R n × m +
[Equation] P [Equation]
Scetbon et al.
[2021] propose to constrain the discrete optimal transport problem to couplings that have a low-nonnegative rank:
Definition 1.
Given M ∈ R n × m + , the nonnegative rank of M is defined by: rk + ( M ) ≜ min { q | M = (cid:80) qi
=1 R i , ∀ i, rk ( R i ) = 1 , R i ≥ 0 } .
Note that for any M ∈ R n × m + , we always have that rk + ( M ) ≤ min( n, m ) .
For r ≥ 1 , we consider the set of couplings satisfying marginal constaints with nonnegative-rank of at most r as Π a,b ( r ) ≜ { P ∈ Π a,b , rk + ( P ) ≤ r } .
The discrete Low-rank Optimal Transport (LOT) problem is defined by:
[Equation]
To solve this problem, Scetbon et al.
[2021] show that Problem (3) is equivalent to
[Equation]
where C 1 ( a, b, r ) ≜ (cid:110) ( Q, R, g ) ∈ R n × r + ×
[Equation]
m [Equation] + × [Equation]
R
[Equation]
r s.
[Equation]
[Equation]
r = a,
[Equation]
1 r = b (cid:111) and C 2 ( r ) ≜ (cid:110)
[Equation]
Q, R,
[Equation]
[Equation] [Equation] g (cid:111) . They propose to solve it using a mirror descent scheme and prove the non-asymptotic stationary convergence of their algorithm.
While Scetbon et al.
[2021] only focus on the discrete setting, we consider here its extension for arbitrary probability measures.
Following [Forrow et al., 2019], we define the set of rank- r couplings satisfying marginal constraints by:
[Equation]
This more general definition of LOT between µ ∈ M +1 ( X ) and ν ∈ M +
( Y ) reads:
[Equation]
Note that this definition of LOT r,c is consistent as it coincides with the one defined in (3) on discrete probability measures.
Observe also that Π r ( µ, ν ) is compact for the weak topology and therefore the infimum in (5) is attained.
See Appendix A for more details.
Approximation Error of LOT to original OT as a function of rank
Our goal in this section is to obtain a control of the error induced by the low-rank constraint when trying to approximate the true OT cost.
We provide first a control of the approximation error in the discrete setting.
The proof is given in Appendix B.1.
Proposition 1.
Let n, m ≥ 2 , X ≜ { x 1 , . .
, x n } ⊂ X , Y ≜ { y 1 , . . .
y m } ⊂ Y and a ∈ ∆ ∗ n and b ∈ ∆ ∗ m .
Then for 2 ≤ r ≤ min( n, m ) , we have that
[Equation]
Remark 1.
Note that this result improves the control obtained in [Liu et al., 2021], where they obtain that | LOT r,c ( µ a, X , ν b, Y ) − OT c ( µ a, X , ν b, Y ) | ≲ ∥ C ∥ ∞ √ nm (min( n, m ) − r ) as we have for any z, z ′ ≥ 1 , | ln( z ) − ln
( z ′ )
| ≤ | z − z ′ | .
It is in fact possible to obtain another control of the approximation error by partitioning the space where the measures are supported.
For that purpose let us introduce the notion of entropy numbers.
Definition 2.
Let ( Z , d ) a metric space, W ⊂ Z and k ≥ 1 an integer.
Then by denoting B Z ( z, ε ) ≜ { y ∈ Z : d ( z, y ) ≤ ε } , we define the k -th (dyadic) entropy number of W as
[Equation]
, ε ) } . For example, any compact set W of R d admits finite entropy numbers, and by denoting R ≜ sup w ∈W ∥ w ∥ 2 , we have N k ( W , ∥ · ∥ 2 ) ≤ 4 R/ 2 k/d .
We obtain next a control of the approximation error of LOT r,c to the true OT cost using entropy numbers (see proof in Appendix B.2).
Proposition 2.
Let µ ∈ M +1 ( X ) , ν ∈ M +1 ( Y ) and assume that c is L -Lipschitz w.r.t.
x and y .
Then for any r ≥ 1 , we have
[Equation]
This results in the following bound for the p -Wasserstein distance for any p ≥ 1 on R d .
Corollary 1.
Let d ≥ 1 , p ≥ 1 , X a compact subspace of R d and µ,
ν ∈ M +
( X ) .
By denoting
R ≜ sup x ∈X ∥ x ∥ 2 , we obtain that for any r ≥ 1 ,
[Equation]
As per the Proof of Proposition 2 we can provide a tighter control, assuming a Monge map exists.
Corollary 2.
Under the same assumptions of Proposition 2 and by assuming in addition that there exists a Monge map solving OT c ( µ, ν ) , we obtain that for any r ≥ 1 ,
[Equation]
When X = Y are a subspaces of R d , a sufficient condition for a Monge map to exists is that either µ or ν is absolutely continuous with respect to the Lebesgue measure and that c is of the form h ( x − y ) where h : X → R + is a strictly convex function [Santambrogio, 2015, Theorem 1.17].
Therefore if µ is absolutely continuous with respect to the Lebesgue measure, we obtain for any r ≥ 1 and p >[-
-]
[Equation]
Sample Complexity of LOT
We now focus on the statistical performance of the plug-in estimator for LOT.
In the following we assume that X = Y for simplicity.
Given µ, ν ∈ M +1 ( X ) , we denote the empirical measures associated ˆ µ n ≜ 1 n (cid:80) ni
=
δ X i and ˆ ν n ≜ 1 n (cid:80) ni
=1 δ Y i , where ( X i , Y i ) ni
=1 are sampled independently from µ ⊗ ν .
We consider the plug-in estimator defined as LOT r,c
(ˆ µ n , ˆ ν n ) , and we aim at quantifying the rate at which it converges towards the true low-rank optimal transport cost LOT r,c ( µ, ν ) .
Before doing so, in the next Proposition we show that this estimator is consistent on compact spaces.
The proof is given in Appendix B.3.
Proposition
[-Let r ≥ 1 and µ, ν ∈ M +
-]
[Equation]
µ, ν ) a.s.
Next we aim at obtaining the convergence rates of our plug-in estimator.
In the following Proposition, we obtain a non-asymptotic upper-bound of the statistical error.
See Appendix B.4 for the proof.
Proposition 4.
Let r ≥ 1 and µ, ν ∈ M +
( X ) .
Then, there exists a constant K r such that for any δ > 0 and n ≥ 1 , we have, with a probability of at least 1 − 2 δ , that
[Equation]
This result is, to the best of our knowledge, the first attempt at providing a statistical control of low-rank optimal transport.
We provide an upper-bound of the plug-in estimator which converges towards LOT r,c at a parametric rate and which is independent of the dimension on general compact metric spaces.
While we fall short of providing a lower bound that could match that upper bound, and therefore provide a complete statistical complexity result, we believe this result might provide a first explanation on why, in practice, LOT r,c displays better statistical properties than unregularized
OT and its curse of dimensionality [Dudley, 1969].
In addition, that upper bound compares favorably to known results on entropic optimal transport.
The rate of entropy regularized OT does not depend on the ambient dimension with respect to n , but carries an exponential dependence in dimension with respect to the regularization parameter
ε
[Mena and Niles-Weed, 2019].
By contrast, the term associated with the nonnegative rank r in our bound has no direct dependence on dimension.
Our next aim is to obtain an explicit rate with respect to r and n .
In Proposition 4, we cannot control explicitly K r in the general setting.
Indeed, in our proof, we obtain that K r ≜ 14 / min i λ ∗ i where ( λ ∗ i ) ri =1 ∈ ∆ ∗ r are the weights involved in the decomposition of one optimal solution of the true
LOT r,c ( µ, ν ) .
Therefore the control of K r requires additional assumptions on the optimal solutions of LOT r,c ( µ, ν ) .
In the following Proposition, we obtain an explicit upper-bound of the plug-in estimator with respect to r and n in the asymptotic regime.
Proposition 5.
Let r ≥ 1 , δ > 0 and µ, ν ∈ M +
( X ) .
Then there exists a constant N r,δ such that if n ≥ N r,δ then with a probability of at least 1 − 2 δ , we have
[Equation]
Note that one cannot recover the result obtained in Proposition 5 from the one obtained in Propositionas we have that K r ≥ 14 r −−−−−→ r → + ∞ + ∞ .
In order to prove the above result, we use an extension of the McDiarmid’s inequality when differences are bounded with high probability [Kutin, 2002].
See proof in Appendix B.5 for more details.
Debiased Formulation of LOT
We introduce here the debiased formulation of LOT r,c and show that it is able to distinguish two distributions, metrize the convergence in law and can be used as a new objective in order to learn distributions.
We focus next on the debiasing terms involving measures with themselves LOT r,c ( µ, µ ) in this new divergence, and show that they can be interpreted as defining a new clustering method generalizing k -means for any geometry.
5.1 On the Proprieties of the Debiased Low-rank Optimal Transport
When it comes to learn (or generate) a distribution in ML applications given samples, it is crucial to consider a divergence that is able to distinguish between two distributions and metrize the convergence in law.
In general, LOT r,c ( µ, µ ) ̸ = 0 and the minimum of LOT r,c ( ν, µ ) with respect to ν will not necessarily recover µ .
In order to alleviate this issue we propose a debiased version of LOT r,c defined for any µ, ν ∈ M +
( X ) as
[Equation]
Note that DLOT r,c ( ν, ν ) = 0 .
In the next Proposition, we show that, as the Sinkhorn divergence [Genevay et al., 2018b, Feydy et al., 2018], DLOT r,c interpolates between the Maximum
Mean Discrepancy (MMD) and OT.
See proof in Appendix B.6.
Proposition 6.
Let µ, ν ∈ M +
( X ) .
Let us assume that c is symmetric, then we have
[Equation]
If in addition we assume the c is Lipschitz w.r.t to x and y , then we have
[Equation]
Next, we aim at showing some useful properties of the debiased low-rank OT for machine learning applications.
For that purpose, let us first recall some definitions.
Definition 3.
We say that the cost c : X × X → R + is a semimetric on X if for all x, x ′ ∈ X , c
( x, x ′ )
=
c ( x ′ , x ) and c ( x, x ′ )
=
if and only if x = x ′ .
In addition we say that c has a negative type if ∀ n ≥ 2 , x 1 , . . .
, x n ∈ X and α 1 , . .
. , α n ∈ R such that (cid:80) ni
=
α i = 0 , (cid:80) ni,j =
α i α j c
( x i , x j ) ≤ 0 .
We say also that c has a strong negative type if for all µ, ν ∈ M +
( X ) , µ ̸
= ν = ⇒ (cid:82) X 2 c ( x, y ) d [ µ − ν ] ⊗ [ µ − ν ] < 0 .
Note that if c has a strong negative type, then c has a negative type too.
For example, all Euclidean spaces and even separable Hilbert spaces endowed with the metric induced by their inner products have strong negative type.
Also, on R d , the squared Euclidean distance has a negative type [Sejdinovic et al., 2013].
We can now provide stronger geometric guarantees for DLOT r,c .
In the next Proposition, we show that for a large class of cost functions, DLOT r,c is nonnegative, able to distinguish two distributions, and metrizes the convergence in law.
The proof is given in Appendix B.8.
Proposition 7.
Let r ≥ 1 , and let us assume that c is a semimetric of negative type.
Then for all µ, ν ∈ M +1 ( X ) , we have that
[Equation]
In addition, if c has strong negative type then we have also that
[Equation]
where the convergence of the sequence of probability measures considered is the convergence in law.
Observe that when c has strong negative type, ν → DLOT r,c ( ν, µ ) ≥
and it admits a unique global minimizer at ν = µ .
Therefore, DLOT r,c has desirable properties to be used as a loss.
It is also worth noting that, in order to obtain the metrization of the convergence in law, we show the following
Proposition.
See proof in Appendix B.7.
Proposition 8.
Let r ≥ 1 and ( µ n ) n ≥
and ( ν n ) n ≥ 0 two sequences of probability measures such that µ n → µ and ν n → ν with respect to the convergence in law.
Then we have that
[Equation]
5.2 Low-Rank Transport Bias and Clustering
We turn next to the debiasing terms appearing in DLOT and exhibit links between LOT and clustering methods.
Indeed, in the discrete setting, the low-rank bias of a probability measure µ defined as
LOT k,c ( µ, µ ) can be seen as a generalized version of the k -means method for any geometry.
In the next Proposition we obtain a new formulation of LOT k,c ( µ, µ ) viewed as a general clustering method on arbitrary metric space.
See proof in Appendix B.9.
Proposition 9.
Let n ≥ k ≥ 1 , X ≜ { x 1 , . . .
, x n } ⊂ X and a ∈ ∆ ∗ n .
If c is a semimetric of negative type, then by denoting C
= ( c ( x i , x j ))
i,j , we have that
[Equation]
Let us now explain in more details the link between (6) and k -means.
When X is a subspace of R d , c is the squared Euclidean distance and a = 1 n , we recover exactly the k -means algorithm.
Corollary 3.
Let n ≥ k ≥ 1 and X ≜ { x 1 , . .
, x n } ⊂ R d .
We have that
[Equation]
In the general setting, solving LOT k,c ( µ a, X , µ a, X ) for a given geometry c , and a prescribed histrogram a offers a new clustering method where the assignment of the points to the clusters is determined by the matrix Q ∗ solution of (6).
Computing LOT: Adaptive Stepsizes and Better Initializations
We target in this section practical issues that arises when using [Scetbon et al., 2021, Algo.3] to solve (4).
Scetbon et al.
[2021] propose to apply a mirror descent scheme with respect to the KullbackLeibler divergence which boils down to solve at each iteration k ≥
the following convex problem using the Dykstra’s Algorithm [Dykstra, 1983]:
[Equation]
(7) where ( Q 0 , R 0 , g 0 ) ∈
C 1 ( a, b, r ) ∩ C 2 ( r ) , ξ k ≜ ( ξ (1) k , ξ (2) k , ξ (3) k ) , ξ (1) k ≜
Q k ⊙ exp( − γ k CR k
diag(1 /g k )) , ξ (2) k ≜
R k ⊙ exp( −
γ k C T Q k
diag(1 /g k )) , ξ (3) k ≜ g k ⊙ exp ( γ k ω k /g 2 k ) with [ ω k ] i ≜ [ Q Tk CR k
i,i for all i ∈ { 1 , . . .
, r } , KL( w , r ) ≜ (cid:80)
i w i log( w i /r i ) and ( γ k ) k ≥ 0 is a sequence of positive step sizes.
In the general setting, each iteration of their algorithm requires
O ( nmr ) operations and when the ground cost matrix C admits a low-rank factorization of the form
C = AB T where A ∈ R n × q and B ∈ R m × q with q ≪ min( n, m ) , then the total complexity per iteration becomes linear O (( n + m ) rq ) .
Note that for the squared Euclidean cost on R d , we have that q = d + 2 .
In the following we investigate two practical aspects of the algorithm: the choice of the step sizes and the initialization.
Adaptive choice of γ k .
Scetbon et al.
[2021] show experimentally that the choice of ( γ k ) k ≥ 0 does not impact the solution obtained upon convergence, but rather the speed at which it is attained.
Indeed the larger γ k is, the faster the algorithm will converge.
As a result, their algorithm simply relies on a fixed γ schedule.
However, the range of admissible γ depends on the problem considered and it may vary from one problem to another.
Indeed, the algorithm might fail to converge as one needs to ensure at each iteration k of the mirror descent scheme that the kernels ξ k do not admit 0 entries in order to solve (7) using the Dykstra’s Algorithm.
Such a situation can occur when the terms involved in the exponentials become too large which may depend on the problem considered.
Therefore, it may be of particular interest for practitioners to have a generic range of admissible values for γ independently of the considered problem, in order to alleviate parameter tuning issues.
We propose to consider instead an adaptive choice of ( γ k ) k ≥ 0 along iterations.
D’Orazio et al.
[2021], Bayandina et al.
[2018] have proposed adaptive mirror descent schemes where, at each iteration, the step-size is normalized by the squared dual-norm of the gradient.
Applying such a strategy in our case amounts to consider at each iteration
[Equation]
where the initial γ > 0 is fixed.
By doing so, we are able to guarantee a lower-bound of the exponential terms involved in the expression of the kernels ξ k at each iteration and prevent them from having 0 entries.
We recommend to set such as global γ ∈ [1 , 10] , and observe that this range works whatever the problem considered.
On the choice of the initialization.
As LOT r,c (4) is a non-convex optimization problem, the question of choosing an efficient initialization arises in practice.
Scetbon et al.
[2021] show experimentally that the convergence of the algorithm does not depend on the initalization chosen if no stopping criterion is used.
Indeed, their experimental findings support that only well behaved local minimas are attractive.
However, in practice one needs to use a stopping criterion in order to terminate the algorithm.
We do observe in many instances that using trivial initializers may result in spurious local minima, which trigger the stopping criterion early on and prevent the algorithm to reach a good solution.
Based on various experimentations, we propose to consider a novel initialization of the algorithm.
Our initialization aims at being close to a well-behaved local minimum by clustering the input measures.
When the measures are supported on Euclidean space, we propose to find r centroids ( z i ) ri =1 of one of the two input discrete probability measures using k -means and to solve the following convex barycenter problem:
[Equation]
where C X,Z = ( c ( x i , z j ))
i,j , C Y,Z
=
( c ( y i , z j ))
i,j , and H ( P ) = − (cid:80) i,j P i,j (log( P i,j − 1) .
In practice we fix ε = 1 / [-
-]10 and we then initialize LOT r,c using ( Q, R ) solution of (9) and g ≜
Q T 1 r (= R T 1 r ) .
Note that ( Q, R, g ) is an admissible initialization and finding the centroids as well as solving (9) requires O (( n + m ) r ) algebraic operations.
Therefore such initialization does not change the total complexity of the algorithm.
In the general (non-Euclidean) case, we propose to initialize the algorithm by applying our generalized k -means approach defined in (6) on each input measure where we fix the common marginal to be g = 1 r /r .
More precisely, by denoting C X,X =
( c ( x i , x j ))
i,j and C Y,Y =
( c ( y i , y j ))
i,j , we initialize the algorithm by solving:
[Equation]
Note that again the ( Q, R, g ) obtained is an admissible initialization and the complexity of solving (10) is of the same order as solving (4), thus the total complexity of the algorithm remains the same.
Experiments
In this section, we illustrate experimentally our theoretical findings and show how our initialization provide practical improvements.
For that purpose we consider 3 synthetic problems and one real world dataset to: (i) provide illustrations on the statistical rates of LOT r,c , (ii) exhibit the gradient flow of the debiased formulation DLOT r,c , (iii) use the clustering method induced by LOT r,c , and (iv) show the effect of the initialization.
All experiments were run on a MacBook Pro 2019 laptop.
[Figure] [Figure]
Statistical rates.
We aim at showing the statistical rates of the plug-in estimator of LOT r,c .
As
LOT r,c ( µ, µ ) ̸ = 0 and as we do not have access to this value given samples from µ , we consider instead the debiased version of the low-rank optimal transport, DLOT r,c .
In figure 1, we show that the empiricial rates match the theoretical bound obtained in Proposition 4.
In particular, we show that that these rates does not depend on the dimension of the ground space.
Note also that we recover our theoretical dependence with respect to the rank r : the higher the rank, the slower the convergence.
Gradient Flows using DLOT.
We illustrate here a practical use of DLOT for ML application.
In figure 6, we consider Y 1 , . . .
, Y n independent samples from a moon shape distribution in 2D, and by denoting ˆ ν n the empirical measure associated, we show the iterations obtained by a gradient descent scheme on the following optimization problem:
[Equation]
We initialize the algorithm using n = 1000 samples drawn from a Gaussian distribution.
We show that the gradient flow of our debiased version is able to recover the target distribution.
We also compare it with the gradient flow of the biased version (LOT) and show that it fails to reproduce the target distribution as it is learning a biased one with a low-rank structure.
Application to Clustering.
In this experiment we show some applications of the clustering method induced by LOT r,c .
In figure 3, we consider 6 datasets with different structure and we aim at recovering the clusters using (6) for some well chosen costs.
We compare the clusters obtained when considering either the squared Euclidean cost (which amounts at applying the k -means) and the
[Figure]
-2 initialization [Scetbon et al., 2021] and a random initialization where Q, R and g are drawn from Gaussians.
We compare both the cost value and the criterion value ( ∆ k ) along the iterations of the MD scheme.
Note that the curves obtained do not start at the same point in time as we start plotting the curves after obtaining the initial point which in some case requires more algebraic operations (e.g. kmeans methods).
First we observe that whatever the initialization considered, the algorithm converges toward the same value.
In addition, we observe that both k -means and general k -means are able to initialize well the algorithm by avoiding bad local minima at initialization while the two other initialization are close to spurious local minima at initialization.
shortest-path distance on the data viewed as a graph.
We show that our method is able to recover the clusters on these settings for well chosen costs and therefore the proposed algorithm in Scetbon et al. [2021] can be seen as a new alternative in order to clusterize data.
Effect of the Initialization.
Our goal here is to show the effect of the initialization.
In figure 4, we display the evolution of the cost as well as the value of the stopping criterion along the iterations of the MD scheme solving (4) when considering different initialization.
The x -axis corresponds to the total number of algebraic operations.
This number is computed at each iteration of the outer loop of the algorithm proposed in Scetbon et al.
and is obtained by computing the complexity of all the operations involved in their algorithm to reach it.
We consider this notion of time instead of
CPU/GPU time as we do not want to be architecture/machine dependent.
Recall also that the stopping criterion introduced in [Scetbon et al., 2021] is defined for all k ≥ 1 by
[Equation]
where (( Q k , R k , g k ))
k ≥ 0 is the sequence solution of (7).
First, we show that whatever the initialization chosen, the algorithm manages to converge to an efficient solution if no stopping criterion is used.
However, the choice of the initialization may impact the termination of the algorithm as some initialization might be too close to some spurious local minima.
Indeed, the initial points obtained using a “rank 2” or random initialization can be close to spurious and non-attractive local minima, which may trigger the stopping criterion too early and prevent the algorithm from continuing to run in order to converge towards an attractive and well behaved local minimum.
We show also that the initialization we propose in (9) and (10) are sufficiently far away from bad local minima and allow the algorithm to converge directly toward the desired solution.
The right figure of Fig.4 shows two main observations: (i) that the initial point obtained using a “rank 2” or random initialization can be close to spurious and non-attractive local minima, which may trigger the stopping criterion too early and prevent the algorithm from continuing to run in order to converge towards an attractive and well behaved local minimum.
(ii) When initialiazing the algorithm using kmeans methods, we show that our stopping criterion is a decreasing function of time meaning that the algorithm converges directly towards the desired solution.
Conclusion.
We assembled in this work theoretical and practical arguments to support low-rank factorizations for OT.
We have presented two controls: one concerning the approximation error to the true optimal transport and another concerning the statistical rates of the plug-in estimator.
The latter is showed to be independent of the dimension, which is of particular interest when studying OT in
ML settings.
We have motivated further the use of LOT as a loss by introducing its debiased version and showed that it possesses desirable properties: positivity and metrization of the convergence in law.
We have also presented the links between the bias induced by such regularization and clustering methods, and studied empirically the effects of hyperparameters involved in the practical estimation of LOT.
The strong theoretical foundations provided in this paper motivate further studies of the empirical behaviour of LOT estimator, notably on finding suitable local minima and on improvements on the convergence of the MD scheme using other adaptive choices for step sizes. Acknowledgements.
This work was supported by a "Chaire d’excellence de l’IDEX Paris Saclay".
The authors would also like to thank Gabriel Peyré and Jaouad Mourtada for enlightening conversations on the topics discussed in this work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Sections 3,4,5,6.
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sections 3,4,5.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 7.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 7.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 7. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary materials A On the Definition of LOT r,c
Let ( X , d X ) and ( Y , d Y )
two nonempty compact Polish spaces, µ ∈ M +1 ( X ) , ν ∈ M +1 ( Y ) two probability measures on these spaces and c : X × Y → R + a nonnegative and continuous function.
We define the generalized low-rank optimal transport between µ and ν as
[Equation]
where
[Equation]
As X and Y are compact, Π r ( µ, ν ) is tight, then Prokhorov’s theorem applies and the closure of Π r ( µ, ν ) is sequentially compact.
Let us now show that Π r ( µ, ν ) is closed.
Indeed, Let ( π n ) n ≥a sequence of Π r ( µ, ν ) converging towards π ∗ .
Then by definition there exists for all k ∈ [ | 1 , r | ] , ( µ ( k ) n ) n ≥ 0 , ( ν ( k ) n ) n ≥ 0 and ( λ ( k ) n ) n ≥ 0 such that for all n ≥[-
-]
[Equation]
However, ( µ ( k ) n ) n ≥ 0 and ( ν ( k ) n ) n ≥ 0 are also tight, and Prokhorov’s theorem applies, therefore we can extract a common subsequence such that for all k , µ ( k ) n → µ ( k ) ∗ and ν ( k ) n → ν ( k ) ∗
In addition as ( λ n ) n ≥ 0 live in the simplex ∆ r , we can also extract a sub-sequence, such that λ n → λ ∗ ∈ ∆ r .
Finally by unicity of the limit we obtain that
[Equation]
Finally, by denoting I ≜ { k : λ ( k ) ∗ > 0 } , and by considering i ∗ ∈
I , we obtain that
[Equation]
from which follows that π ∗ ∈ Π r ( µ, ν ) .
B Proofs B.1 Proof of Proposition
Proposition.
Let n, m ≥ 2 , X ≜ { x 1 , . .
, x n } ⊂ X , Y ≜ { y 1 , . . .
y m } ⊂ Y and a ∈ ∆ ∗ n and b ∈ ∆ ∗ m .
Then for 2 ≤ r ≤ min( n, m ) , we have that
[Equation]
Proof.
Let P ∈ argmin P ∈ Π a,b ⟨ C, P ⟩ .
As P is a nonnegative matrix, its nonnegative rank cannot exceed min( n, m ) .
Assume for simplicity, that n = m , then there exists ( R i ) n i =1 nonnegative matrices of rank 1 such that
P [Equation] n (cid:88) i =
R i .
As for all i ∈ [ | 1 , n | ] , R i is a rank 1 matrix, there exist ˜ q i , ˜ r i ∈ R n + such that R i =
˜ q i ˜
r Ti .
Then by denoting q i = ˜
q i / | ˜ q i | , r i =
˜
r i / | ˜ r i |
and λ i = | ˜ q
i ||
˜
r i | where for any h ∈ R n | h | ≜ (cid:80) ni
=1 h i , we obtain that
[Equation]
Without loss of generality, we can consider the case where λ 1 ≥ · · · ≥ λ n .
Let us now denote λ :=
( λ 1 , . . .
, λ n ) , and by using the fact the P is a coupling we obtain that λ ∈ ∆ n .
Also, by definition of λ , we have that for all k ∈ [ | 1 , n | ] , λ k ≤ 1 /k .
Let us now define
[Equation]
where
[Equation]
Remark that ˜ P ∈ Π a,b ( r ) , therefore we obtain that | LOT
[Equation]
(cid:32) n (cid:88) i =
r λ i (cid:33)
α r β
Tr ⟩ ≤ ∥ C ∥ ∞ n (cid:88) i =
r λ i ≤ ∥ C ∥ ∞ n (cid:88) i =
r i ≤ [Equation]
ln ( n [Equation] ( r [Equation]
B.2 Proof of Proposition
Proposition 10.
Let µ ∈ M +1 ( X ) , ν ∈ M +1 ( Y ) and let us assume that c is L -Lipschitz w.r.t.
x and y .
Then for any r ≥ 1 , we have
[Equation]
Proof.
As X and Y are compact, N ⌊ log 2 ( ⌊√ r ⌋ ) ⌋ ( X , d ) , N ⌊ log 2 ( ⌊√ r ⌋ ) ⌋ ( Y , d ) < + ∞ and then by denoting ε X ≜ N ⌊ log 2 ( ⌊√ r ⌋ ) ⌋ ( X , d X ) , there exists x 1 , . . .
, x ⌊√ r ⌋ ∈ X , such that X ⊂ (cid:83) ri =1 B X ( x i , ε ) from which we can extract a partition ( S i, X ) ⌊√ r ⌋ i =1 of X such that for all i ∈ [ | 1 , ⌊√ r ⌋| ] , and x, y ∈ S i, X , d X ( x, y ) ≤ ε X .
Similarly we can build a partition ( S i, Y ) ⌊√ r ⌋ i =1 of Y .
Let us now define for all k ∈ [ | 1 , ⌊√ r ⌋| ] , µ k ≜ µ | S k, X µ
[Equation]
S k,
[Equation]
and ν k ≜ ν | S k, Y [-) with the convention that 
 = 0 , we can define [Equation] i,j =-]ν
( S k, Y ) with the convention
that 00 = 0 , we can define
[Equation]
First remarks that π r ∈ Π r ( µ, ν ) .
Indeed we have for any measurable set B
[Equation]
similarly π r ( A × Y ) =
µ ( A ) and we have that ⌊√ r ⌋ 2 ≤ r .
Therefore we obtain that
[Equation]
from which the result follows.
Corollary.
Under the same assumptions of Proposition 2 and by assuming in addition that there exists a Monge map solving OT c ( µ, ν ) , we obtain that for any r ≥ 1 ,
[Equation]
Proof.
Let us denote T a Monge map solution of OT c ( µ, ν ) and as in the proof above, let us consider a partition of ( S i, Y ) ri
=1 of Y such that for all i ∈ [ | 1 , r | ] , and x, y ∈ S
i, Y , d Y ( x, y ) ≤ ε Y with ε Y ≜ N ⌊ log 2 ( r ) ⌋ ( Y , d Y ) .
Let us now define for all k ∈ [ | 1 , ⌊√ r ⌋| ] ,
[Equation]
with the convention that 0
=
0 , we can define
[Equation]
Again we have that π r ∈ Π r ( µ, ν ) , and we obtain that
[Equation]
from which the result follows.
Note that to obtain the above inequalities, we use the fact that π ∗ is supported on the graph of T , and therefore we have have for all k ∈ [ | 1 , r | ] ,
[Equation]
T −
[Equation]
( S k, Y )
× S k, Y )
=
µ
[Equation]
T − 1 ( S k, Y ))
=
ν ( S k, Y ) .
B.3 Proof of Proposition
Proposition.
Let r ≥ 1 and µ, ν ∈ M +
( X ) , then LOT r,c (ˆ µ n , ˆ ν n ) −−−−−→ n → + ∞ LOT r,c ( µ, ν ) a.s.
Proof.
Let π ∗ solution of LOT r,c ( µ, ν ) .
Then there exists λ ∗ ∈ ∆ ∗ r , ( µ ∗ i ) ri =1 , ( ν ∗ i ) ri =
1 ∈ M +1 ( X ) r such that
[Equation]
Note that by definition, we have that
[Equation]
Let us now define π µ and π µ both elements of M +1 ( X × [ | 1 , r | ]) as follows: π µ ( A × { k } ) ≜ λ k µ k
( A ) and π ν ( A × { k } ) ≜
λ k ν k
( A ) for any measurable set
A and k ∈ [ | 1 , r | ] .
Observe that the right marginals of π µ and π ν is the same and we will denote it ρ .
We can now define for all x, y ∈
X the family of kernels ( k µ ( · , x ))
x ∈X ∈ M +
([ | 1 , r | ]) X and ( k ν ( · , y ))
y ∈X ∈
M +1 ([ | 1 , r | ])
X corresponding to the disintegration with respect to the projection of respectively µ and ν .
Let us now consider n independent samples ( Z µi ) ni
=1 and ( Z νi ) ni =1 such that for all i ∈ [ | 1 , n | ] ,
Z µi ∼ k µ ( · , X i ) and Z νi ∼ k ν ( · , Y i ) and let us define for all k ∈ [ | 1 , r | ] [Equation] i = [Equation]
and ˜ [Equation] n (cid:88) i = 1 Z νi
[Equation] k δ Y i .
Let us now define
[Equation]
with the convention that 00 = 0 .
Now it is easy to check that
˜ π ∈ Π r (ˆ µ, ˆ ν ) , indeed we have that ˜ π
[Equation]
A × X ) =
[Equation]
in addition by construction we have that
[Equation]
and both ˆ µ −
[Equation]
r − 1 k =1 min( | ˜ µ k | , | ˜ ν k | ) | ˜ µ k |
[Equation]
and
[Equation]
˜ ν
k are positive measures.
Therefore we obtain that
[Equation]
Now we aim at showing at (cid:82) X 2 c ( x, y )
d ˜ π ( x, y )
→ LOT r,c ( µ, ν ) a.s. .
Indeed first observe that from the law of large numbers we have that for all k ∈ [ | 1 , r | ] , | ˜ µ k | → λ ∗ k and similarly | ˜ ν k | → λ ∗ k .
In addition, for all k, q we have that almost surely, ˜ µ k ⊗ ˜ ν
q converges weakly towards λ ∗ k λ ∗ q µ k ⊗ ν q .
Indeed one can consider the following algebra F ≜ (cid:8) ( x, y ) ∈
X 2 → f ( x ) g
( y ) f,
g ∈ C ( X ) (cid:9) , and then by Stone-Weierstrass, one obtains by density the desired result.
Now remark that (cid:90)
[Equation]
(cid:90) [Equation] (cid:18)
[Equation]
min( [Equation]
|
[Equation] (cid:19) (cid:18) [Equation] − min(
[Equation]
| ˜ ν q | (cid:19) c ( x, y ) d
˜ µ k ( x ) d ˜ ν q ( y ) from which follows directly that (cid:82) X 2 c ( x, y )
d ˜ π ( x, y )
→ LOT r,c ( µ, ν ) a.s. Let us now denote for all n ≥ 1 , π n a solution of LOT r,c (ˆ µ, ˆ ν ) .
Let ω ∈ Ω an element of the probability space where live the random variables ( X i )
i ≥
and ( Y i )
i ≥
such that (cid:82) X 2 c ( x, y )
d ˜ π ( ω ) ( x, y )
→ LOT r,c ( µ, ν ) .
As X is compact Thanks to Prokhorov’s Theorem, we can extract a sequence such that ( π ( ω ) n ) n ≥ 0 converge weakly towards
π ( ω ) ∈
Π r ( µ, ν ) .
In addition we have that for all n ≥[-
-]
[Equation]
And by considering the limit we obtain that
[Equation]
However π ( ω
) ∈
Π r ( µ, ν ) and by optimality we obtain that
[Equation]
This holds for an arbitrary subsequence of ( π ( ω ) n ) n ≥ 0 , from which follows that (cid:82)
[Equation] ( x, y ) dπ ( ω ) n ( x, y ) → LOT r,c ( µ, ν ) .
Finally this holds almost surely and the result follows.
B.4 Proof of Proposition
Proposition.
Let r ≥ 1 and µ, ν ∈ M +
( X ) .
Then, there exists a constant K r such that for any δ > 0 and n ≥ 1 , we have, with a probability of at least 1 − 2 δ , that
[Equation]
Proof.
We reintroduce the same notation as in the proof of Proposition 3.
Let π ∗ solution of
LOT r,c ( µ, ν ) .
Then there exists λ ∗ ∈ ∆ ∗ r , ( µ ∗ i ) ri =1 , ( ν ∗ i ) ri {+=
+}
∈ M {++
+}
( Z ) r such that
[Equation]
As before let us also consider π µ and π µ defined as π µ ( A × { k } ) ≜ λ k µ k ( A ) and π ν ( A × { k } ) ≜
λ k ν k
( A ) for any measurable set
A and k ∈ [ | 1 , r | ] and denote ρ their common right marginal.
We also consider n independent samples ( Z µi ) ni
=1 and ( Z νi ) ni =1 such that for all i ∈ [ | 1 , n | ] , Z µi ∼ k µ ( · , X i ) and Z ν i ∼ k ν ( · , Y i )
and we denote for all k ∈ [ | 1 , r | ]
[Equation]
Let us now define
[Equation]
Our goal is to control the following quantity:
[Equation]
First observe that
[Equation]
Moreover
[Equation]
we have that
[Equation]
from which follows that
[Equation]
Now let us define for all ( x i , z i ) ni
=1 , ( y i , z ′ i ) ∈
( Z × [ | 1 , r | ]) n , g (( x 1 , z 1 ) , . . .
, ( x n , z n ) , ( y 1 , z ′ 1 ) , . . .
, ( y n , z ′ n )) ≜ r (cid:88) q =  λ ∗ q n(cid:88)
i,j c ( x i , y j ) 1 z i = q 1 z ′ j = q , since Z is compact and c is continuous, we have that
[Equation]
Then by applying the McDiarmid’s inequality we obtain that for δ >
[Equation]
with a probability at least of 1 − δ , we have
[Equation]
Now we aim at building a coupling ˜
π ∈ Π r (ˆ µ, ˆ ν ) from ˆ π .
Let us consider the same as the one introduce in the proof of Proposition B.3, that is ˜ π ≜
[Equation]
with the convention that 00 = 0 .
Let us now expand the above expression, and by denoting ˜ λ r =
[Equation]
˜
[Equation]
| , | ˜ ν k | ) we obtain that
[Equation]
Now we aim at controlling the following quantity
[Equation]
and we observe that
[Equation]
Let us now control each term of the RHS of the above equality.
Let us first consider the term in
Eq. 11, remark that we have
[Equation]
[Table]
[Equation]
where we have denoted ˜ λ µ ≜ ( | ˜ µ k | ) rk =1 and ˜ λ ν ≜ ( | ˜ ν k | ) rk =1 .
Now observe that
[Equation]
where d K is the Kolmogorov distance.
In addition we have
[Equation]
Combining the two above controls, we obtain that for all δ >
[Equation]
with a probability of at least 1 − δ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:90) Z
[Equation]
Let us now consider the term in Eq.
12, we have that
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
However we have that
[Equation]
In addition we have that E ( ∥ λ ∗ − ˜ λ µ ∥ 1 ) ≤ (cid:112) rn and by applying the McDiarmid’s Inequality [Equation] we obtain that for all δ > 0 , with a probability of 1 − δ
[Equation]
Therefore we obtain that with a probability of at least 1 − δ ,
[Equation]
For the term in Eq. 13 and [Equation] we obtain that
[Equation]
Therefore we obtain that with a probability of at least 1 − δ [Equation] [Equation] λ r
[Equation]
Finally the last term in Eq.
15 can be controlled as the following:
[Equation]
x, [Table]
≤ ∥ c ∥ ∞ ˜ λ r r − 1 (cid:88) k,q = (cid:18) 1 − min( | ˜ µ k | , | ˜ ν k | ) | ˜ µ k | (cid:19) (cid:18) 1 − min( | ˜ µ q | , | ˜ ν q | ) | ˜ ν
q | (cid:19) | ˜ µ k || ˜
ν
q | ≤ ∥ c ∥ ∞ ˜ λ r r − 1 (cid:88) k
= ( | ˜ µ k | − min( | ˜ µ k | , | ˜ ν k | ))
r − 1 (cid:88) k
= ( | ˜ ν k | − min( | ˜ µ k | , | ˜ ν k | )) ≤ 3 ∥ c ∥ ∞ max( ∥ λ ∗ − ˜ λ µ ∥ 1 , ∥ λ ∗ − ˜ λ
ν ∥ 1 )
and we obtain that with a probability of at least 1 − δ ,
[Equation]
Then by applying a union bound we obtain that with a probability of at least 1 − δ
[Equation]
Now observe that
[Equation]
and by combining the two control we obtain that with a probability of at least 1 − 2 δ ,
[Equation]
where α ≜ min 1 ≤ i ≤ r λ ∗ i and the result follows.
B.5 Proof Proposition
Proposition.
Let r ≥ 1 , δ > 0 and µ, ν ∈ M +
( X ) .
Then there exists a constant N r,δ such that if n ≥ N r,δ then with a probability of at least 1 − 2 δ , we have
[Equation]
Proof.
We consider the same notations as in the proof of Proposition 4.
In particular let us define for all ( x i , z i ) ni
=1 , ( y i , z ′ i ) ∈
( Z × [ | 1 , r | ]) n , g (( x 1 , z 1 ) , . . .
, ( x n , z n ) , ( y 1 , z ′ 1 ) , . . . , ( y n
[Equation]
z ′ n )
[Equation]
≜
[Equation]
(cid:88)
q
[Equation]
λ [Equation] (cid:88)
[Equation]
q [Equation] Recall that we have
[Equation]
In fact if we have a control in probability of the bounded difference we can use an extension of the
McDiarmid’s Inequality.
For that purpose let us first introduce the following definition.
Definition 4.
Let ( X i ) mi
=1 , m independent random variables and g a measurable function.
We say that g is weakly difference-bounded with respect to ( X i ) mi =1 by ( b, β, δ ) if
[Equation]
with X ′ i
=
X i except for one coordinate k
where X ′ k is an independent copy of X k .
Furthermore for any ( x i ) mi =1 and ( x ′ i ) mi =1 where for all coordinate except on x j = x ′ j
[Equation]
Let us now introduce an extension of McDiarmid’s Inequality [Kutin, 2002].
Theorem 1.
Let ( X i ) mi
=1 , m independent random variables and g a measurable function which is weakly difference-bounded with respect to ( X i ) mi =1 by ( b, β/m, exp( − Km )) , then if 0 < τ ≤
T ( b, β, K ) and m ≥ M ( b, β, K, τ ) , then
[Equation]
where
[Equation]
Given the above Theroem we can obtain an asymptotic control of the deviation of g from its mean.
Let δ ′ >
and let us denote
[Equation]
Observe now that with a probability of at least 1 − exp( − Km )
[Equation]
Let us now fix δ > 0 and let us choose δ ′ such that δ ′ ≜ 4 /n
and τ ≜ β (cid:113) 4log(4 /δ )
n , then we obtain that for n sufficiently large (such that n ≥ M ( b, β, K, τ ) / 2 and τ ≤ T ( b, β, K ) ), we have that with a probability of at least 1 − δ
[Equation]
Recall also from the proof of Proposition 4, that we have with a probability of at least 1 − δ (cid:12)(cid:12)(cid:12)(cid:12)(cid:90)
[Equation]
[Table]
[Equation]
[Table]
Finally by imposing in addition that
[Equation]
we obtain that for n is large enough (such that (such that n ≥ M ( b, β, K, τ ) / 2 and τ ≤ T ( b, β, K ) ) and satysfing the above inequalities, we have with a probability of at least 1 − 2 δ that
[Equation]
B.6 Proof Proposition
Proposition.
Let µ, ν ∈ M +
( X ) .
Let us assume that c is symmetric, then we have
[Equation]
If in addition we assume the c is Lipschitz w.r.t to x and y , then we have
[Equation]
Proof.
When r = 1 , it is clear that for any µ, ν ∈ M +1 ( X ) , Π r [Equation] µ, ν ) = { µ ⊗ ν } and thanks to the symmetry of c , we have directly that
[Equation]
The limit is a direct consequence of Proposition 2.
B.7 Proof of Proposition
Proposition.
Let r ≥ 1 and ( µ n ) n ≥
and ( ν n ) n ≥ 0 two sequences of probability measures such that µ n → µ and ν n → ν with respect to the convergence in law.
Then we have that
[Equation]
Proof.
Let us denote π an optimal solution of LOT r,c ( µ, ν ) and let us denote ( µ ( i ) )
ri =1 , ( ν ( i ) )
ri = and ( λ ( i ) ) ri =1 the decomposition associated.
In the following Lemma, we aim at building specific decompositions of the sequences ( µ n ) n ≥ 0 and ( ν n ) n ≥ 0 .
Lemma 1.
Let r ≥ 1 , µ ∈ M + 1 ( X ) and ( µ ( i ) )
r i =1 ∈ M + 1 ( X ) and ( λ ( i ) )
r i =1 ∈ ∆ ∗ r such that µ = (cid:80) ri =
λ i µ ( i ) .
Then for any sequence of probability measures ( µ n ) ≥ 0 such that µ n → µ , there exist for all i ∈ [ | 1 , r | ] a sequence of nonnegative measures ( µ ( i ) n ) n ≥ 0 such that µ [Equation] i ) n
[Equation]
λ i µ ( i ) for all i ∈ [ | 1 , r |
[Equation]
and
[Equation]
for all n ≥
Proof.
For r = 1 the result is clear.
Let us now show the result for r = 2 .
Let us denote (˜ µ (1) n ) a sequence converging weakly towards λ 1 µ (1) .
Then by denoting µ (1) n ≜ µ n − ( µ n − ˜ µ (1) n ) + where ( · ) + correspond to the non-negative part of the measure, we have that [Equation] and
[Equation]
for all n ≥ which is the result.
Let r ≥ 2 and let us assume that the result holds for all 1 ≤ k ≤ r .
Let us now consider a decomposition of µ such that µ = (cid:80) r
+1 i =
λ i µ ( i ) .
By denoting ˜ µ (1) ≜ (cid:80) ri =
λ i µ ( i ) (cid:80) ri =
λ i , we obtain that
[Equation]
Then by recursion we have that there exists sequences of nonnegative measures (˜ µ (1) n ) and ( µ ( r +1) n ) such that
[Equation]
for all n ≥
Now observe that ˜ µ (1) n | ˜ µ (1) n | → ˜ µ (1)
=
(cid:80) ri = [Equation] ri {+=
+}
λ i µ ( i ) .
Therefore applying the recursion on this problem allows us to obtain a decomposition of ˜ µ (1) n of the form
[Equation]
µ ( [Equation] n[-where µ ( i ) n ≥ 0 and µ ( i ) n → λ i (cid:80) ri =
-]
[Equation] µ ( i ) n where
[Equation]
and µ
[Equation] i
[Equation] n →
[Equation]
Therefore we obtain that
[Equation]
1) n where µ ( i ) n ≥ 0 , | ˜ µ (1) n | µ ( i ) n → λ i µ ( i ) for all i ∈ [ | 1 , r | ] and µ ( r +1) n ≥ 0 , µ ( r +1) n → λ r +1 µ ( r +1) from which follows the result. Let us now consider such decompositions of ( µ n ) n ≥ 0 and ( ν n ) n ≥ 0 such that each factor converges toward the target decomposition of µ . Now let us build the following coupling: ˜
[Equation]
with the convention that 00 = 0 .
Now it is easy to check that ˜ π n ∈ Π r ( µ n , ν n ) , and we have that
[Equation]
and by Prokhorov’s theorem and the optimality of the limit of (˜ π n ) n ≥ 0 (up to an extraction) we obtain that LOT r,c ( µ n , ν n ) → LOT r,c ( µ, ν ) .
B.8 Proof Proposition
Proposition.
Let r ≥ 1 , and let us assume that c is a semimetric of negative type.
Then for all µ, ν ∈ M +1 ( X ) , we have that
[Equation]
In addition, if c has strong negative type then we have also that
DLOT
[Equation]
where the convergence of the sequence of probability measures considered is the convergence in law.
Proof.
Let π ∗ solution of LOT r,c ( µ, ν ) .
Then there exists λ ∗ ∈ ∆ ∗ r , ( µ ∗ i ) ri =1 , ( ν ∗ i ) ri =
[Equation]
M +1 ( X ) r such that
[Equation]
Note that by definition, we have that
[Equation]
By definition we have also that
[Equation]
similarly for LOT r,c
( ν, ν ) we have
[Equation]
Therefore we have
[Equation]
where for any any probability measures α [Equation] β on X we define
[Equation]
However, as c is assumed to have a negative type, we have that
[Equation]
In addition if we assume that c has a strong negative type, then we obtain directly that
[Equation]
Let us now show that DLOT r,c metrize the convergence in law.
The direct implication is a direct consequence of the Proposition 8.
Conversely, if DLOT r,c ( µ n , µ ) → 0 , then by compacity of X and thanks to the Prokhorov’s theorem we can extract a subsequence of µ n → µ ∗ , and thanks to
Proposition 8, we also obtain that DLOT r,c ( µ n , µ ) → DLOT r,c ( µ ∗ , µ ) .
Finally we deduce that
DLOT r,c ( µ ∗ , µ ) = 0 and µ ∗ = µ .
B.9 Proof Proposition
Proposition.
Let n ≥ k ≥ 1 , X ≜ { x 1 , . . .
, x n } ⊂ X and a ∈ ∆ ∗ n .
If c is a semimetric of negative type, then by denoting C
= ( c ( x i , x j ))
i,j , we have that
[Equation]
[Equation]
Proof.
First remarks that one can reformulate the LOT k,c problem as
[Equation]
where
[Equation]
Indeed the above optimization problem is just a reformulation of LOT k,c ( µ, µ ) where we have vectorized the couplings in a column-wise order.
Let us now show the following lemma from which the result will follow.
Lemma 2.
Under the same assumption of Proposition 9 we have that for all g ∈ ∆ ∗ k
[Equation]
Proof.
Let ( x ∗ , y ∗ ) solution of the LHS optimization problem.
Then we have that
[Equation]
Therefore we obtain that
[Equation]
g
[Equation]
Then by symmetry of C , we obtain by adding the two terms that
[Equation]
However, thanks to the linear constraints, we have that for all i ∈ [ | 1 , k | ] , n − 1 (cid:88)
[Equation]
Therefore ( x ∗ i − y ∗ i ) T 1 n =
and thanks to the negativity of the cost function c we obtain that
[Equation]
Therefore we have that
[Equation]
from which follows that
[Equation]
and the result follows.
As the above result holds for any g ∈ ∆ ∗ k , we obtain that
[Equation]
Then by formulating back this problem in term of matrices, we obtain that
[Equation]
from which the result follows.
C Additional Experiments C.1 Comparison of the γ schedules
[Figure]
C.2 Gradient Flows between two Moons
[Equation]
descent direction.
We show that the debiased version allows to recover the target distribution while LOT r,c is learning a biased version with a low-rank structure.