Parameter tuning and model selection in Optimal Transport with semi-dual Brenier formulation
Abstract
Over the past few years, numerous computational models have been developed to solve Optimal Transport (OT) in a stochastic setting, where distributions are represented by samples and where the goal is to ﬁnd the closest map to the ground truth
OT map, unknown in practical settings.
So far, no quantitative criterion has yet been put forward to tune the parameters of these models and select maps that best approximate the ground truth.
To perform this task, we propose to leverage the
Brenier formulation of OT.
Theoretically, we show that this formulation guarantees that, up to sharp a distortion parameter depending on the smoothness/strong convexity and a statistical deviation term, the selected map achieves the lowest quadratic error to the ground truth.
This criterion, estimated via convex optimization, enables parameter tuning and model selection among entropic regularization of OT, input convex neural networks and smooth and strongly convex nearestBrenier (SSNB) models.
We also use this criterion to question the use of OT in
Domain-Adaptation (DA).
In a standard DA experiment, it enables us to identify the potential that is closest to the true OT map between the source and the target.
Yet, we observe that this selected potential is far from being the one that performs best for the downstream transfer classiﬁcation task.
Introduction
Optimal transport (OT) is a tool to compare probability distributions that has found numerous applications ranging from economics [Galichon, 2016, Chiappori et al., 2010], unsupervised learning [Sim et al., 2020], shape matching [Feydy et al., 2017], NLP [Chen et al., 2019, Alvarez-Melis and
Jaakkola, 2018] and biology [Schiebinger et al., 2019, Tong et al., 2020].
In its dual form, OT is a linear maximization problem on functions, which are called potentials, subject to a cost constraint.
When the cost is chosen to be quadratic, the solutions of this problem are convex and their gradient provide optimal maps that transport one distribution onto the other.
In a signiﬁcant part of the OT applications, the transport map itself is the object of interest.
For instance in Domain-Adaptation, the source distribution is transported on the target [Courty et al., 2017], for color transfer one color histogram is transported on the other [Rabin et al., 2014] and in biology, the RNA cell expression proﬁle is interpolated in time using OT maps [Schiebinger et al., 2019].
Over the past few years, many models and computational methods [Cuturi, 2013, Genevay et al., 2016, Seguy et al., 2018,
Bonneel and Coeurjolly, 2019, Vacher et al., 2021] were proposed and implemented to estimate these optimal transport maps.
Under regularity assumptions, some of these models were shown to accurately estimate the original transport map provided the models use optimal parameters [Pooladian and Niles-Weed, 2021, Manole et al., 2021].
When such results exist, either the parameters to use are explicit but they are impractical as they rely on generic worst-case bounds, either they involve unavailable constants.
To the best of our knowledge, no quantitative criterion has yet been devised to tune the parameters of OT models and later discriminate between calibrated models.
The setting 36th we are interested in is standard in statistical and ML applications, in which probability measures are only accessible via samples in Euclidean spaces.
The goal is to recover, for the quadratic cost, a potential chosen among different models/parameters that is the closest to the unknown ground truth.
For achieving this task, we put forward the use of the semi-dual functional of OT that we now deﬁne.
The semi-dual (Brenier) objective.
The quantitative criterion that we propose is the so-called semidual Brenier objective of OT.
It is a convex functional on the space of functions f : R d → R and is deﬁned for µ, ν two probability measures on the Euclidean space by, denoting " · , · # the pairing between Radon measures and continuous functions, J µ, ν ( f ) := " f, µ # + " f ∗ , ν # with f ∗ the FenchelLegendre transform of f given by f ∗ ( y ) :
= sup x ∈ R d x # y − f ( x ) .
Note that for a general cost c , this new objective can be obtained by replacing one potential by its c -transform in the Kantorovitch dual formulation.
Now, whenever f is convex, its legendre transform can be computed pointwise with an arbitrary precision.
Furthermore, if ν is a ﬁnite sum of Dirac masses, the resulting convex problems of J ( f ) are independent and can be solved in parallel.
Related works.
The problem of evaluating OT models was recently studied by Korotin et al.
They proposed to generate synthetic ground truth optimal maps using an input convex neural network.
Then, they calibrate various OT models on these ground truth OT maps and compare the performance of each calibrated OT model by measuring the natural L 2 distance between the estimated map and the ground truth.
Their paper gives an interesting perspective on comparing current OT models.
However, their setup requires the knowledge of the ground truth to calibrate the
OT models.
This limitation can be overcome for convex potentials as shown in our work.
The use of the Fenchel-Legendre transform can be found in the pioneering paper Brenier [1991].
It can be shown that this new formulation retains more convexity than the Kantorovitch formulation.
On the theoretical side, this gain was leveraged for uses as diverse as sharp bounds for the problem of statistical map estimation [Hütter and Rigollet, 2021] or quantitative stability results of the transport map with respect to the measures [Delalande and Merigot, 2021].
On the numerical side, since the Fenchel-Legendre transform has linear cost on a grid (Fast Legendre Transform), the semi-dual is used to design efﬁcient numerical algorithms in low dimension [Jacobs and Léger, 2020].
In the machine learning community, the semi-dual was proposed by Taghvaei and Jalali [2019] to estimate convex transport maps parametrized by Input Convex Neural Networks [Amos et al., 2017].
When n is the sample size, they noticed that instead of the classical O ( n 2 ) complexity of OT, this new formulation leads to an O ( n ) complexity per iteration as it only requires n -independent computations of the Legendre transform.
Our contributions.
The goal of our paper is to answer the following question: given ( f 1 , · · · , f p ) , p convex potentials, how to select the one that minimizes the quadratic error, denoting ∇ f i the gradient of f i , e µ ( f i ) =
[Equation] ) , where T 0 is the true, unknown, OT map from µ to ν ?
A concrete example of this problem would be: we train a Sinkhorn model [Cuturi, 2013] on samples (ˆ µ train , ˆ ν train ) with different temperatures ( ε 1 , · · · , ε p ) , giving us empirical potentials ( ˆ f ε 1 , · · · , ˆ f ε p ) .
Given test samples (ˆ µ test , ˆ ν test ) how to choose the Sinkhorn empirical potential that minimizes the unknown error ( e µ ( ˆ f ε i )) 1 ≤ i ≤ p ?
We give an illustration of this problem in Fig.
The main contribution of the paper is to use the empirical semi-dual to answer this question; to the best of our knowledge, it is the ﬁrst time that the model selection problem in OT has been addressed.
From a theoretical point of view, in the strongly-convex and smooth setting, we prove that the potential f i minimizing the empirical semi-dual is the one minimizing e µ up to a sharp multiplicative factor that depends on the smoothness of the potential and up to an additive statistical deviation term.
From an experimental point of view, we showcase on three synthetic experiments, with an ICNN model [Taghvaei and Jalali, 2019], a Sinkhorn model [Cuturi, 2013] and a SSNB model [Paty et al., 2020], that the potential achieving the lowest quadratic error is nearly always selected by the semi-dual.
This consistent behavior enables us to conclude on a concrete ML application of our work: in Domain Adaptation, a widely accepted approach is to seek for an OT map between a source and a target using various models/parameters in order to transfer the source’s labels on the unknown target’s labels.
The question whether the OT model is truly relevant for this task has remained unanswered because of the impossibility to discriminate between candidates close to the "true" OT map and those "far" from it.
In our last experiment, we bring a partial negative answer to this question: we observe that the map minimizing the semi-dual, hence the closest to the ground truth OT map, is often far from being the one that achieves the best label transfer.
Assumptions and notations.
In this paper X, Y are compact subsets of R d , µ
and ν are probability measures over X and Y respectively with their n -samples empirical counterparts ˆ µ , ˆ ν .
We shall denote by supp ( µ ) , supp ( ν ) the support of µ and ν respectively.
The cost we consider is the quadratic cost c ( x, y ) = & x − y & 2 / 2 and shall denote q the quadratic function q ( x ) = & x & 2 / 2 .
Finally, we shall call M -smooth any function with an M -Lipschitz gradient.
Brenier formulation of OT
In its dual formulation, the OT problem optimizes over a pair of continuous functions, called Kantorovitch potentials ( φ , ψ ) subject to a cost constraint as OT c ( µ, ν )
=
sup ( φ , ψ ) " φ , µ # + " ψ , ν # + ι ( φ ⊕ ψ ≤ c ) where φ ⊕ ψ is deﬁned as ( φ ⊕ ψ )( x, y ) =
φ ( x ) + ψ ( y ) and ι is the convex indicator function.
When c is the Euclidean squared distance, we simply denote it by OT.
In this case, if one of the two measures has density w.r.t.
the Lebesgue measure, Brenier’s theorem [Brenier, 1991] shows that a unique optimal map sending µ to ν exists and is given by the gradient of a convex function.
If one further assumes regularity of the underlying densities and convexity of the support of the distributions, the optimal map gains in regularity.
Theorem 1 (Caffarelli [2000] Theorem 2b.) .
Assume that µ and ν have C 1 densities bounded from below and above.
If µ , ν have compact and convex support, then, deﬁning the Brenier potentials ( f, g ) =
( q − φ , q − ψ ) , f and g are C 2 convex functions such that ∇ f # ( µ ) =
ν and ∇ g # ( ν ) = µ , where T # ( η ) is the pushforward of the distribution η by the map T deﬁned as T # ( η )( A )
=
η ( T − 1 ( A )) for all Borel A .
Hence under the assumptions of Theorem 1, the optimal Brenier potentials are both smooth.
In particular, since they are mutual Legendre transform of one and other, they are both also strongly convex.
The semi-dual Brenier objective for convex potential selection.
In the context of statistical OT, the transport Kantorovitch potentials ˆ φ , ˆ ψ are usually estimated using the dual formulation of OT.
To compare the obtained potentials, one may be tempted to simply evaluate the Kantorovitch linear objective on a test set K ˆ µ, ˆ ν (ˆ φ , ˆ ψ ) = "
ˆ φ , ˆ µ # + " ˆ ψ , ˆ ν # + ι (ˆ φ ⊕ ˆ ψ ≤ c ) where ˆ µ, ˆ ν represent independent samplings of µ, ν .
However in numerous OT models, the learned potentials (ˆ φ , ˆ ψ ) usually do not respect the cost constraint and the ι (ˆ φ ⊕ ˆ ψ ≤ c ) term diverges.
For instance, in the entropic regularization of OT the constraint is "loosely" satisﬁed on the train set since it replaces the hard inequality constraint by the soft penalization ε " e φ ⊕ ψ − c ε , ˆ µ ⊗ ˆ ν # .
It is possible though to remove the cost constraint from the objective in order to evaluate candidate potentials.
Rewriting the Kantorovich dual with the
Brenier potentials gives OT ( µ, ν ) = " q, µ + ν # − inf ( f,g ) " f, µ # + " q, ν # + ι ( f ( x ) + g ( y ) ≥ x # y ) where the optimization is done on f, g ∈ C 0 , the space of continuous functions.
The inequality costraint implies that g ( y ) ≥ sup x x # y − f ( x ) for every x which shows that we can replace g by f ∗ , the Fenchel-Legendre transform of f .
Therefore, up to moment terms, we get the semi-dual
Brenier formulation inf f J µ, ν ( f ) = " f, µ # + " f ∗ , ν # .
This new nonlinear objective gains in con- vexity with respect to the Kantorovitch formulation (see Sec. 3) and now, whenever f is strongly convex, J µ, ν ( f ) is ﬁnite and can be efﬁciently computed on discrete measures using standard convex optimization algorithms.
Hence, if we restrict ourselves to convex f , possibly regularized with the addition of a small quadratic term, we are provided with a tractable and well-behaved selection criterion: the potential that minimizes J ˆ µ, ˆ ν .
We show in the next section that, thanks to the extra convexity, the minimization of this objective coincides, up to a stochastic term, with the minimization of the quadratic error e µ ( f ) =
[Equation]
Potentials selection
We give in this section our main theoretical guarantee on model selection via the semi-dual.
The setting we need to consider is the case of M -smooth and γ -strongly convex potentials, assumption which is discussed below.
Let us ﬁrst recall a stability result which shows that the semi-dual formulation is upper-bounded and lower-bounded by the quadratic error e µ .
When no confusion is possible, we shall from now on denote J µ, ν by J .
Lemma 1.
Assuming that an optimal convex potential f 0 such that T 0 = ∇ f 0 pushes µ onto ν exists, then if f is M -smooth, denoting J
= J ( f 0 ) , we have 12 M e µ ( f ) ≤ ( J ( f ) − J 0 ) where e µ ( f ) = &∇ f − T 0 & 2 L 2 ( µ ) .
Conversely, if f is a continuous γ -strongly convex function then J ( f ) −
J 0 ≤ 12 γ e µ ( f ) .
This result is derived from Muzellec et al.
[2021, Proposition 1], see Appendix for more details.
A similar result can be found in Hütter and Rigollet [2021, Proposition 10] in a slightly less general form and in Makkuva et al.
[2020, Theorem 3.6].
Importantly, note that there is no smoothness assumption made on the optimal map T 0 .
In contrast, we prove that the smoothness and strongconvexity assumptions on the candidate f are necessary to lower and upper bound J ( f ) − J 0 with respect to e µ .
Proposition 1.
Take µ ∼ U ([ − 12 , 12 ]) and f 0 of the form
λ q ( x ) + x with λ ≥ 0 .
The potential f ( x ) = x is indeed convex with Lipschitz gradient and is such that J ( f ) =
+ ∞ yet e µ ( f )
= λ 2 4 → 0 .
Conversely, deﬁne the potential g 0 ( x ) =
| x | + q ( x ) and for 0 ≤ λ ≤ 12 , deﬁne g λ
= g 0 ( · − λ ) which is indeed strongly convex (and even locally Lipschitz).
The difference of semi-duals reads
J ( g λ ) − J ( g 0 ) =
2 λ [-
-]2 − λ 3 ∼ λ →
0 2 λ 2 and yet e µ ( g λ )
= 4 λ + λ
∼ λ → 0 4 λ .
The detailed computations are left in Appendix.
Equipped with this lemma, we can derive our result.
Proposition 2.
Let ( f 1 , · · · , f p ) be p potentials and (ˆ µ, ˆ ν ) be the n -samples empirical counterparts of ( µ, ν ) .
Let i 0 be the index of the map that minimizes the empirical semi-dual, i 0 =
arg min i J ˆ µ, ˆ ν ( f i ) and similarly i 1 = arg min i e µ ( f i ) .
If f i 0 is M -smooth and f i 1 is γ -strongly convex and if an OT map from µ to ν exists, then for all 0 < δ <
we have with probability at least 1 −
[Equation]
where C = max( C i 0 , C i 1 ) with C i = max( & f i & X,o , & f ∗ i & Y,o ) and & . & Z,o is deﬁned as & g & Z,o = sup z
[Equation]
The proof is left in Appendix and is a direct application of the previous stability estimate and the
Hoeffding lemma.
The following proposition shows that in the non-stochastic regime n = + ∞ , our bound M γ is tight.
Proposition 3.
Take µ ∼ U [0 , 1] , f 0 ≡ 0 , g ( x ) =
Mq ( x ) + x and for * > 0 , take h ’ ( x )
= γ q ( x ) + ( * + α M, γ ) x with α M,
γ = γ#$ 1 + 4( M − γ ) 3 γ − 1 % , and where we assumed M ≥ γ .
The potential g is indeed M -smooth and h is indeed γ -strongly convex and we have e µ ( g ) e µ ( h $ ) → ’ →
M γ yet
J ( g [Equation]
The computations are left in Appendix.
The bounds of Proposition 2 being tight in the non-stochastic setting, it is natural to question the assumptions on the class of potentials, namely M -smoothness and γ -strong convexity.
Let us ﬁrst mention Theorem 1 readily ensures that in the case where µ , ν have C 1 densities with convex compact support then the optimal potentials satisfy the smoothness assumptions.
Under such conditions on the measures, one should indeed benchmark potentials which are already smooth and strongly convex.
However in practice, we may not have access to potentials satisfying these conditions.
A simple idea to satisfy the smoothness assumption is to use the Moreau-Yosida transform M τ ( f ) =
inf y f ( y )
+ 1 τ q ( x − y ) , which veriﬁes f − L 2 τ≤ M τ ( f ) ≤ f where L is the Lipschitz constant of f on ν .
The semi-dual shall read 12 τ e µ ( M τ ( f ))
≤ J ( M τ ( f ))
− J 0 , however without further assumptions, we cannot upper-bound the original error e µ ( f ) as | e µ ( M τ ( f )) − e µ ( f ) | may not be in o (1) .
The case of strong-convexity has a more favorable behavior.
Denoting Q
δ
( f ) =
f
+ δ q , we can apply Lemma 1 and obtain
J ( Q δ ( f )) − J 0 ≤ 12 δ e µ ( Q δ ( f )) .
The error can be upper bounded by e µ ( f ) + 4 δ 2 " q, µ # .
Under a local Lipschitz assumption on f ∗ , J ( Q δ ( f )) − J 0 is O ( δ ) , hence we observe there is a trade-off between the deterioration of the factor in O ( 1 δ ) and the bias in O ( δ ) .
As a consequence, we can derive a new stability result, proved in Appendix, when the optimal δ is chosen.
The strong-convexity assumption is relaxed with a local Lipschitz assumption of the conjugate at the cost of loosing on exponent on the error.
Proposition 4.
If f is s.t.
f ∗ is L -Lipschitz on the support of ν then the semi-dual reads J ( f ) − J 0 ≤
[Equation]
Note that at optimality, f ∗ 0 is indeed Lipschitz on the support of ν , yet this condition may be difﬁcult to enforce in practice.
We believe though that sharper/less restrictive stability results could be obtained via a reﬁned analysis and we postpone this interesting question for future works.
Sinkhorn potentials
The Sinkhorn model [Cuturi, 2013], deﬁned as S ε ( µ, ν )
= sup φ , ψ ∈ C 0 " φ , µ # + " ψ , ν # − ε " e φ ⊕ ψ − c ε , µ ⊗ ν # is very popular in the OT community.
We show in this section that the discrete potentials provided by the empirical Sinkhorn model can indeed be extended to continuous, convex and smooth potentials.
However, their extension is not strongly convex and we prove that the semidual diverges almost surely when evaluted on empirical Sinkhorn potentials; hence we propose to quadratically regularize the Sinkhorn potentials.
From a numerical point of view, we show how to compute efﬁciently the semi-dual on this regularized model with a fast converging scheme.
A convex smooth model.
Recall that the ﬁrst-order optimality condition on the Sinkhorn potential φ ε gives φ ε ( x ) = − ε log( !
y e
[Equation]
[Equation]
d ν ( y )) .
As done for instance in Berman [2018], Pooladian and Niles-Weed [2021], we use this conidtion to extend the empirical Sinkhorn potentials (ˆ φ ε , ˆ ψ ε ) =
arg min ( φ , ψ ) S ε (ˆ µ, ˆ ν ) on the whole domain R d .
Deﬁning the associated Brenier potentials as ˆ f ε = q − ˆ φ ε , we obtain ˆ f ε ( x ) = ε log(
[Equation]
y )) , where we deﬁned ˆ β ε ( y ) =
e (1 / ε )( ˆ ψ ε ( y ) − q ( y )) .
This shows that the potentials ( ˆ f ε , ˆ g ε ) are Log-Sum-Exp (LSE) functions and in particular, they are convex.
Proposition 5.
ˆ f ε is D 2 ( supp [Equation]
ε smooth where D ( supp (ˆ ν )) is the diameter of the support of ˆ ν .
The proof is left in Appendix.
While LSE functions are indeed smooth, they are not strongly convex.
The following proposition shows that the semi-dual diverges almost surely on ˆ f ε .
Proposition 6.
Let ( ˆ f ε , ˆ g ε ) =
( q − ˆ φ ε , q − ˆ ψ ε ) .
If ν has continuous density with respect to the
Lebesgue measure we have almost surely " ˆ f ∗ ε , ν # = + ∞ .
The proof is left in Appendix.
This Proposition implies that evaluating the semi-dual on the empirical
Sinkhorn potentials is ill-posed and it justiﬁes their strong-convexity regularization.
To this end, we consider Q δ ( ˆ f ε ) =
ˆ f ε + δ q with δ > 0 small and shall now read J ( Q δ ( ˆ f ε ))
− J 0 ≤ 12 δ e µ ( ˆ f ε )
+ O ( δ ) .
We show in the experiments of Sec. 5 that despite this small bias, the selection of empirical Sinkhorn potentials via the semi-dual criterion still provides satisfactory results.
Self-concordant potentials.
Now, even if the Legendre transform can be computed via a standard convex ﬁrst order minimization
algorithm, it will not be effective in practice as the problem is conditioned by O ( 1 δε ) .
One way to circumvent poor conditioning is to employ second order methods.
In the standard cases, they require costly line-searches however if the function has a generalized self-concordant structure, we can use a second order algorithm of the form x k +
= x
k − α k ( ∇ 2 f ( x k )) − 1 ∇ f ( x k ) , where α k is an explicit step size given in Sun and TranDinh [2019, Theorem 2] and that provably yields a super-linearly convergent algorithm.
Deﬁnition 1.
Let α > 0 and f be a C 3 convex function.
The function f is said to be ( α , M f ) self-concordant if for all ( x, u, v ) , | ( ∇ 3 f ( x )[ v ] u ) # u | ≤ M f & u & 2 x & v & α − 2 x & v & 3 − α , where & u & 2 x = ( ∇ 2 f ( x ) u ) # u , ∇ 3 f ( x ) is the tensor ( ∂ 3 f ∂ x i x j x k ) 1 ≤ ijk ≤ d and for a tensor T = ( t ijk )
1 ≤ i,j,k ≤ n , T [ v ] = & pi = v
i T i with T i the matrix ( t ijk ) 1 ≤ j,k ≤ n .
Informally, the self-concordance measures how fast the Hessian varies with respect to the metric it induces.
Proposition 7.
The Sinkhorn Brenier potential ˆ f ε is (2 , D (ˆ ν ) ε ) self-concordant where the diameter
D is deﬁned as D (ˆ ν ) =
sup y ∈
ˆ ν ,z ∈
ˆ ν & y − z & 2 .
The proof is left in Appendix.
Fig.
2 shows that the second order scheme is much faster to compute the Fenchel conjugate on n = 1000 points, with learning rate O ( 1 ε ) for the ﬁrst-order method.
Numerical experiments
We ﬁrst introduce two other transport models on which we perform our numerical experiments.
5.1 Other models
Input Convex Neural Network (ICNN).
The convex Brenier potentials are modeled by ( f θ 1 , g θ 2 ) two ICNN.
Then the model is trained using a minimax objective
[Equation] " f θ 1 , ( ∇ g θ 2 ) # (ˆ ν )
# The maximization aims at recovering g θ
≈ f ∗ θ 1 and the minimization approximately solves the semi-dual.
The implementation is based on the code of the authors 1 .
Softplus activation layers were used instead of ReLu to obtain less degenerated maps.
Note that since the weights θ 1 , θ 2 are not controlled, we expect this model to provide lowly regular maps.
Smooth Strongly convex Nearest Brenier (SSNB).
This model estimates the potential f by approximately solving inf f ∈ F ( l,L ) W 2 (( ∇ f ) # (ˆ µ ) , ˆ ν ) , where F ( l,L ) is the set of L -smooth, l -strongly convex functions.
As opposed to the previous model, SSNB provides maps that are very regular (in a bi-Lipschitz sense).
5.2 The experiments
Synthetic XP.
We compare the ability of the models to recover the ground truth transport map using the semi-dual criterion map for three different distributions in a medium dimension setting d = 8 .
In all three cases, the distribution µ is uniform on the cube [0 , 1] d and ν is given by ( ∇ f ) # ( µ ) where f is a convex function ; in virtue of Brenier’s theorem, T = ∇ f is the ground truth OT map between µ and ν .
The function f has 3 different forms.
(i) Quadratic: f ( x ) = 12 x # Qx + x # b where Q =
O # DO + 0 .
25 Id where O is a randomly chosen orthogonal matrix, D is a random diagonal matrix whose entries are uniform in [0 , 1] and b is a random d -dimensional gaussian.
This is a standard benchmark which simply aims at recovering a translation.
(ii)
Tensorized: T 0 ( x ) = x
+ (6 − cos(6 π x ) − 0 . 2) − 1 and T ( x ) = & dk =1 T 0 ( x k ) .
The map to learn is more complex but has a low dimensional structure as it pushes independently each directions.
(iii) (Regularized) Log-Sum-Exp: f ( x ) =
t LSE( Ct x + b ) + δ q ( x ) where the matrix
C is comprised of 10 centers uniformly chosen in [ − 1 , 1] d , the shift b is a random d -dimensional gaussian, the temperature t was ﬁxed at 0 .
3 and the regularizer δ = 0 .
Note that any convex function can be approximated by such a Log-Sum-Exp
[Calaﬁore et al., 2020].
However, because of this parametric structure, we expect the Sinkhorn model to be favored.
The training of the models, the semi-dual estimation and the Monte-Carlo approximation of the {+error error 
+} e ˆ µ ( f i ) = !
&∇ f i ( x ) − T 0 ( x ) & 2 dˆ µ ( x ) are done with 3 independent batches of size n = 1024 .
For the computation of the semi-dual, we regularized with + δ q the ICNN and Sinkhorn models taking δ = 1 e − 3 . Note that the errors are computed on the original potentials, without the Concentrates n ) toward the true" error.
[Table]
Sinkhorn
[Table]
[Table]
f [Table] regularization term. Forty-eight combinations of parameters were tested for the ICNN model, ﬁve for the Sinkhorn model and eleven for the SSNB model.
More details on the parameters and on the implementation are given in Appendix.
The results were averaged on ﬁfteen independent runs and are reported on Table 1.
The parameters of each model are tuned with the semi-dual criterion .
We denote θ i {+
 the parameter+} that minimizes semi-dual and θ i 1 the parameter that minimizes the error e ˆ µ .
The numerator between brackets corresponds to the rank of the selected parameter with respect to the error e ˆ µ ; the closer to one the better.
In particular, if θ i
=
θ i 1 we obtain the rank 1 .
The denominator is number of parameters: for instance, we tested the Sinkhorn model with ﬁve different ε , hence the denominator is 5 .
[Table]
the three models are tuned with the semi-dual, we select one among the three again with the semi-dual criterion.
Note that the denominator is now equal to the sum of the number of parameters that we tested for each model.
In the case of SSNB where the smoothness and strong convexity parameters are explicitly controlled, the best parameter is almost always chosen.
In the case of the Sinkhorn model, the regularity decreases for small values of ε yet the selected potential remains in the top 40% for the Quadratic and Log-Sum-Exp experiments.
Conversely, the regularity is not controlled in the ICNN model yet the selected parameters remains in the top-tier for the Quadratic and Log-Sum-Exp experiments; as for the Tensorised experiment, the best potential is almost always selected.
Once the three models are calibrated with the semi-dual, we select one among them with the same criterion (see Table 2); note the denominator between brackets is now equal to the sum of the number of parameters considered for each model.
We observe that the criterion selects the best or nearly-best transport map.
This shows that the Brenier criterion can be both used for calibration and selection.
Fig.
3 plots the semi-dual values against the error in the Log-Sum-Exp experiment and empirically suggests an even better behavior than best model selection.
For the Sinkhorn and SSNB models, the error strictly increases with the semi-dual value, hence the semi-dual can not only select but can also directly rank the potentials with respect to the error e µ .
For the ICNN model, we do not observe the same monotone behavior
but we still get a positive correlation between the error and the semi-dual value.
This less consistent behavior can be explained once again by the lack of control on the regularity of the potentials given by the model.
Ideally, should strictly
Overall, when we compare the models after being calibrated with the semi-dual we observe that ICNN always has the poorest performance.
We may not have chosen the hyperpameters and the network structures in the best possible way and the ground truth may not favour this model.
The SSNB model performs better by almost an order of magnitude than
Sinkhorn on the Quadratic and Tensorized experiments.
Conversely, as expected, the Sinkhorn model is the best one on the Log-Sum-Exp experiment.
In terms of computation time, the training of the SSNB model takes between one and three hours and between 30 minutes and one hour for the semi-dual computation on a 120 GB RAM CPU.
ICNN and Sinkhorn take a few minutes and a few seconds respectively for the training and semi-dual computation on a RTX6000 GPU.
[Table]
n
The numerator of the Rank corresponds to the rank of the potential calibrated with the semi-dual criterion with respect to the error e µ : the closer to one, the better.
The denominator corresponds to the number of parameters.
We observe that the semi-dual accurately tunes the
Sinkhorn model as the best parameter is always selected.
Thanks to the high scalabilty of Sinkhorn, we repeated those three experiments on larger batches with n =averaged on 10 runs.
As shown on Table 3, Sinkhorn recovers the same behavior as SSNB with respect to the semi-dual.
Not only the best parameter ε is always chosen but also, we show in Appendix that in the Quadratic and
LSE settings, the error increases with the semi-dual.
Before ending this paragraph, we brieﬂy discuss the effect of the quadratic regularization for the ICNN and
Sinkhorn models.
We observed experimentally that without it, the Legendre transform indeed diverged over several points.
On the other hand, the experiments did show that the semi-dual accurately selects the best potential even in the presence of the regularizer.
We propose the following informal explanation: we show in Appendix that J
[Equation]
( f )
[Equation]
( T 0 ( x )) −
Q δ ( f ) ∗ ( T ( x )+ δ x )+ x # ( T ( x )+ δ x − T 0 ( x ))
d µ ( x ) .
We can decompose the integration on areas where the original potential f ∗ is O (1) Lipschitz, where we can expect | f ∗ ( T ( x ))
− Q δ ( f ) ∗ ( T ( x ) + δ x ) | =
O ( δ ) and on the remaining area, the integrand behaves like O ( 1 δ ) .
For instance in the case of the empirical Sinkhorn model whose conjugate is Lipschitz on the interior of the convex hull of ˆ ν , we conjecture that J ( Q δ ( ˆ f ε
[Equation] [Equation]
where α > 0 3 .
Domain Adaptation.
(DA)
The goal of DA is to infer unknown labels of a target distribution X t using a shifted source distribution X s with known labels Y s .
In the work of Courty et al.
[2017] and many others [Redko et al., 2019, Xu et al., 2020], a map T is sought between X s and X t .
Then a classiﬁer c is learned on ( T ( X s ) , Y s ) and is used to predict the unknown labels of the target as c ( X t ) .
Their core assumption is that T should be close to the OT map between X s , X t .
Question: is this assumption valid?
Is the "true" OT map between X s and X t the one that will achieve the best knowledge transfer?
Problem: among all proposed models, how to assess which map is the closest to the "true" unknown OT map?
Using our criterion, we can now select the parameters and the model that will be closest to this ground truth.
We use the Caltech-ofﬁce dataset which is a set of images of objects from ten distinct categories coming from four different sources of various quality: objects found in the online Amazon catalog (A), objects whose pictures have been taken with a webcam (W), with a high resolution digital SLR camera (D) and the Caltech-256 dataset (C) which is comprised of Google images.
We use all the nine distinct pairs as source/domain data.
As in Courty et al.
[2017], in order for the quadratic distance to be meaningful, we do not use the raw images but feed them to a Decaf [Donahue et al., 2014] network and extract the features of the last layer and we use a 1 -Nearest Neighbors as the classiﬁer.
The models and parameters we use are the same as in the Synthetic experiment.
In our setting, the transport map T is learned on train sets X trains , X traint and the semi-dual is evaluated on test sets X tests , X testt , with 70% of the data for the train and 30% for the test.
The results are reported on Table 4 where only (A) and (W) are used as sources, the rest of the results are reported in Appendix and exhibit a similar pattern.
We denote by i
the index of the potential minimizing the empirical semi-dual criterion and by i 1 the potential achieving the highest accuracy.
The numerator between bracket corresponds to the rank of the selected potential with respect to the accuracy obtained when the classiﬁer is learned on ( ∇ f ( X s ) , Y s ) ; the closer to 1 , the better and [Table] in particular, rank ( f i 1 ) = 1 .
We observe that the potential having the lowest accuracy is regularly selected by the semi-dual, even in the case of the SSNB model for which the semi-dual indicates very reliably the quality of the transport map.
Hence we conclude that for DA, the best mapping for label transfer is not close to an optimal transport map.
We remark that this conclusion is similar to the results of Korotin et al.
[2021] and Stanczuk et al.
[2021] who observed that the transport models which performed best for various ML tasks were not the ones that recover the sharpest OT maps.
Conclusion
The semi-dual Brenier formulation of quadratic OT provides us with a feasible criterion for convex potential selection.
If the potentials are convex, this criterion can be computed numerically.
Theoretically and experimentally, we showed that up to a sharp distortion parameter, the potential that minimizes the semi-dual is the one whose gradient minimizes the squared error to the ground truth map.
Hence this criterion gives a fair and accurate procedure to benchmark convex OT models and provides a ﬁrst step towards parameter tuning for stochastic OT.
Regarding ML applications, this criterion allowed us to bring a ﬁrst partial negative answer to the question of relevance of OT in
DA: indeed, one may achieve good results in DA using models borrowed from OT literature, yet we claim that their performance is not correlated with how well they approximate the ground truth OT map between the source and the target.
Possible extensions of our work could include more general cost c and more general potentials.
We believe M -smoothness and γ -strong convexity assumptions could be alleviated by a careful analysis of the quadratic and Moreau-Yosida regularizations.
If not possible, the question of smoothness parameters selection in stochastic OT is widely open.