S PARSITY -C ONSTRAINED O PTIMAL T RANSPORT
A BSTRACT
Regularized optimal transport (OT) is now increasingly used as a loss or as a matching layer in neural networks.
Entropy-regularized OT can be computed using the Sinkhorn algorithm
but it leads to fully-dense transportation plans, meaning that all sources are (fractionally) matched with all targets.
To address this issue, several works have investigated quadratic regularization instead.
This regularization preserves sparsity and leads to unconstrained and smooth (semi) dual objectives, that can be solved with off-the-shelf gradient methods.
Unfortunately, quadratic regularization does not give direct control over the cardinality (number of nonzeros) of the transportation plan.
We propose in this paper a new approach for OT with explicit cardinality constraints on the transportation plan.
Our work is motivated by an application to sparse mixture of experts, where OT can be used to match input tokens such as image patches with expert models such as neural networks.
Cardinality constraints ensure that at most k tokens are matched with an expert, which is crucial for computational performance reasons.
Despite the nonconvexity of cardinality constraints, we show that the corresponding (semi) dual problems are tractable and can be solved with first-order gradient methods.
Our method can be thought as a middle ground between unregularized OT (recovered in the limit case k = 1 ) and quadratically-regularized OT (recovered when k is large enough).
The smoothness of the objectives increases as k increases, giving rise to a trade-off between convergence speed and sparsity of the optimal plan.
I NTRODUCTION
Optimal transport (OT) distances (a.k.a.
Wasserstein or earth mover’s distances) are a powerful computational tool to compare probability distributions and have found widespread use in machine learning (Solomon et al., 2014; Kusner et al., 2015; Arjovsky et al., 2017).
While OT distances exhibit a unique ability to capture the geometry of the data, their applicability has been largely hampered by their high computational cost.
Indeed, computing OT distances involves a linear program, which takes super-cubic time to solve using state-of-the-art network-flow algorithms (Kennington & Helgason, 1980; Ahuja et al., 1988).
In addition, these algorithms are challenging to implement and are not GPU or TPU friendly.
An alternative approach consists instead in solving the so-called semi-dual using (stochastic) subgradient methods (Carlier et al., 2015) or quasi-Newton methods (M´erigot, 2011; Kitagawa et al., 2019).
However, the semi-dual is a nonsmooth, piecewise-linear function, which can lead to slow convergence in practice.
For all these reasons, the machine learning community has now largely switched to regularized OT.
Popularized by Cuturi (2013), entropy-regularized OT can be computed using the Sinkhorn algorithm (1967) and is differentiable w.r.t.
its inputs, enabling OT as a differentiable loss (Cuturi, 2013;
Feydy et al., 2019) or as a layer in a neural network (Genevay et al., 2019;
Sarlin et al., 2020; Sander et al., 2022).
A disadvantage of entropic regularization, however, is that it leads to fully-dense transportation plans.
This is problematic in applications where it is undesirable to (fractionally) match all sources with all targets, e.g., for interpretability or for computational cost reasons.
To address this issue, several works have investigated quadratic regularization instead (Dessein et al., 2018;
Blondel et al., 2018; Lorenz et al., 2021).
This regularization preserves sparsity and leads to unconstrained and smooth (semi) dual objectives, solvable with off-the-shelf algorithms.
Unfortunately, it does not give direct control over the cardinality (number of nonzeros) of the transportation plan.
−
Source Target Unregularized Entropy regularized k= k= Squared
[Figure]
Sparsity constrained (ours)
In this paper, we propose a new approach for OT with explicit cardinality constraints on the columns of the transportation plan.
Our work is motivated by an application to sparse mixtures of experts, in which we want each token (e.g. a word or an image patch) to be matched with at most k experts (e.g., multilayer perceptrons).
This is critical for computational performance reasons, since the cost of processing a token is proportional to the number of experts that have been selected for it.
Despite the nonconvexity of cardinality constraints, we show that the corresponding dual and semidual problems are tractable and can be solved with first-order gradient methods.
Our method can be thought as a middle ground between unregularized OT (recovered in the limit case k = 1 ) and quadratically-regularized OT (recovered when k is large enough).
We show that the dual and semidual are increasingly smooth as k increases, giving rise to a trade-off between convergence speed and sparsity.
The rest of the paper is organized as follows.
• We review related work in §2 and existing work on OT with convex regularization in §3.
• We propose in §4 a framework for OT with nonconvex regularization, based on the dual and semidual formulations.
We study the weak duality and the primal interpretation of these formulations.
• We apply our framework in §5 to OT with cardinality constraints.
We show that the dual and semidual formulations are tractable and that smoothness of the objective increases as k increases.
We show that our approach is equivalent to using squared k -support norm regularization in the primal.
• We validate our framework in §6 and in Appendix A through a variety of experiments.
Notation and convex analysis tools.
Given a matrix T ∈ R m × n , we denote its columns by t j ∈
R m for j ∈ [ n ] .
We denote the non-negative orthant by R m + and the non-positive orthant by R m − .
We denote the probability simplex by △ m := { p ∈ R m + : ⟨ p , 1 ⟩ = 1 } .
We will also use b △ m to denote the set { t ∈ R m + : ⟨ t , 1 ⟩ = b } .
The convex conjugate of a function f : R m → R ∪ {∞} is defined by f ∗ ( s ) :
= sup t ∈ dom( f ) ⟨ s , t ⟩ − f ( t ) .
It is well-known that f ∗ is convex (even if f is not).
If the solution is unique, then its gradient is ∇ f ∗ ( s ) =
argmax t ∈ dom( f ) ⟨ s , t ⟩ − f ( t ) .
If the solution is not unique, then we obtain a subgradient.
We denote the indicator function of a set C by δ C , i.e., δ C ( t )
=
if t ∈ C and δ C ( t ) = ∞ otherwise.
We denote the Euclidean projection onto the set C by proj C ( s ) =
argmin C
∥ s − t ∥ 22 .
The projection is unique when C is convex, while it may not be when C is nonconvex.
We use [ · ] + to denote the non-negative part, evaluated element-wise.
Given a vector s ∈ R m , we use s
R ELATED WORK
Sparse optimal transport.
OT with arbitrary strongly convex regularization is studied by Dessein et al.
and Blondel et al.
More specifically, quadratic regularization was studied in the discrete (Blondel et al., 2018; Roberts et al., 2017) and continuous settings (Lorenz et al., 2021).
Although it is known that quadratic regularization leads to sparse transportation plans, it does not enable explicit control of the cardinality (maximum number of nonzero elements), as we do.
In this work, we study the nonconvex regularization case and apply it to cardinality-constrained OT.
Sparse projections.
In this paper, we use k -sparse projections as a core building block of our framework.
Sparse projections on the simplex and on the non-negative orthant were studied by
Kyrillidis et al.
and Bolte et al.
(2014), respectively.
These studies were later extended to more general sets (Beck & Hallak, 2016).
On the application side, sparse projections on the simplex were used for structured prediction (Pillutla et al., 2018; Blondel et al., 2020), for marginalizing over discrete variables (Correia et al., 2020) and for Wasserstein K -means (Fukunaga & Kasai, 2021).
Sparse mixture of experts (MoE).
In contrast to usual deep learning models where all parameters interact with all inputs, a sparse MoE model activates only a small part of the model (“experts”) in an input-dependent manner, thus reducing the overall computational cost of the model.
Sparse
MoEs have been tremendously successful in scaling up deep learning architectures in tasks including computer vision (Riquelme et al., 2021), natural language processing (Shazeer et al., 2017; Lewis et al., 2021;
Lepikhin et al., 2021; Roller et al., 2021; Fedus et al., 2022b; Clark et al., 2022), speech processing (You et al., 2022), and multimodal learning (Mustafa et al., 2022).
In addition to reducing computational cost, sparse MoEs have also shown other benefits, such as an enhancement in adversarial robustness (Puigcerver et al., 2022).
See Fedus et al.
(2022a)
for a recent survey.
Crucial to a sparse MoE model is its routing mechanism that decides which experts get which inputs.
Transformer-based MoE models typically route individual tokens (embedded words or image patches).
To balance the assignments of tokens to experts, recent works cast the assignment problem as entropy-regularized OT (Kool et al., 2021; Clark et al., 2022).
We go beyond entropy-regularized
OT and show that sparsity-constrained OT yields a more natural and effective router.
O PTIMAL TRANSPORT WITH CONVEX REGULARIZATION
In this section, we review OT with convex regularization, which also includes the unregularized case.
For a comprehensive survey on computational OT, see (Peyr ´ e & Cuturi, 2019).
Primal formulation.
We focus throughout this paper on OT between discrete probability distributions a ∈ △ m and b ∈ △ n .
Rather than performing a pointwise comparison of the distributions,
OT distances compute the minimal effort, according to some ground cost, for moving the probability mass of one distribution to the other.
Recent applications of OT in machine learning typically add regularization on the transportation plan T .
In this section, we apply convex regularization Ω: R m + → R + ∪{∞} separately on the columns t j ∈ R m + of T and consider the primal formulation
[Equation]
where C ∈ R m × n + is a cost matrix and U ( a , b ) :
= { T ∈ R m × n + : T 1 n = a , T ⊤ 1 m = b } is the transportation polytope, which can be interpreted as the set of all joint probability distributions with marginals a and b .
It includes the Birkhoff polytope as a special case when m = n and a = b = 1 m m .
Dual and semi-dual formulations.
Let us denote
[Equation]
[Equation]
and
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Unregularized
[Table]
[Equation]
Negentropy
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
The dual and semi-dual corresponding to (1)
can then be written (Blondel et al., 2018) as
D Ω
[Equation]
a ,
[Equation]
, C
[Equation]
[Equation]
and
S Ω ( a , b , C [Equation] = max [Equation]
max
[Equation]
− n (cid:88) [Equation] = Ω ∗ b j ( α − c j ) , (5) where P ∗ Ω denotes the conjugate in the first argument.
When Ω is convex (which also includes the unregularized case Ω = 0 ), by strong duality, we have that P Ω ( a , b , C ) =
D Ω ( a , b , C ) =
S Ω ( a , b , C ) for all a ∈ △ m , b ∈ △ n and C ∈ R m × n + .
Computation.
With Ω = 0 (without regularization), then (2) becomes the indicator function of the non-positive orthant, leading to the constraint
α i + β j ≤ c
i,j .
The dual (4) is then a constrained linear program and the most commonly used algorithm is the network flow solver.
On the other hand, (3) becomes a max operator, leading to the so-called c -transform β j = min i ∈ [ m ] c i,j − α i for all j ∈ [ n ] .
The semi-dual (5) is then unconstrained, but it is a nonsmooth piecewise linear function.
The key advantage of introducing strongly convex regularization is that it makes the corresponding (semi) dual easier to solve.
Indeed, (2) and (3) become “soft” constraints and max operators.
In particular, when Ω is Shannon’s negentropy Ω( t ) = γ ⟨ t , log t ⟩ , where γ controls the regularization strength, then (2) and (3) rely on the exponential and log-sum-exp operations.
It is well known that the primal (1) can then be solved using Sinkhorn’s algorithm (Cuturi, 2013), which amounts to using a block coordinate ascent scheme w.r.t.
α ∈ R m and β ∈ R n in the dual (4).
As pointed out in Blondel et al.
(2018), the semi-dual is smooth (i.e., with Lipschitz gradients) but the dual is not.
When Ω is the quadratic regularization Ω( t ) = γ 2 ∥ t ∥ 22 , then as shown in Blondel et al.
Table 1), (2) and (3) rely on the squared relu and on the projection onto the simplex.
However, it is shown empirically that a block coordinate ascent scheme in the dual (4) converges slowly.
Instead,
Blondel et al.
(2018) propose to use LBFGS both to solve the dual and the semi-dual.
Both the dual and the semi-dual are smooth (Blondel et al., 2018), i.e., with Lipschitz gradients.
For both types of regularization, when γ → 0 , we recover unregularized OT.
Recovering a plan.
If Ω is strictly convex, the unique optimal solution T ⋆ of (1) can be recovered from an optimal solution ( α ⋆ , β ⋆ ) of the dual (4) by
[Equation]
or from an optimal solution α ⋆ of the semi-dual (5) by
[Equation]
If Ω is convex but not strictly so, any subgradient of Ω ∗ + and Ω ∗ b j gives a solution t ⋆j .
With Ω = 0 (unregularized case), given α ⋆ , we can always recover β ⋆ by the c -transform β ⋆j = min i ∈ [ m ] c i,j − α ⋆ i for all j ∈ [ n ] .
Given ( α ⋆ , β ⋆ ) , we can recover T ⋆ as follows.
From the complementary slackness conditions (Peyr´e & Cuturi, 2019, Proposition 3.3), we know that if t ⋆i,j > 0 , then α ⋆i + β
⋆j
=
c
i,j .
Otherwise, if α ⋆i + β ⋆j < c
i,j , then t ⋆i,j = 0 .
We can therefore use ( α ⋆ , β ⋆ ) to identify the support of T ⋆ .
The size of that support is at most m + n − 1 (Peyr´e & Cuturi, 2019,
Proposition 3.4).
Using the primal equality constraints T 1 n = a and T ⊤ 1 m = b , we can therefore form a system of linear equations of size m + n to recover T ⋆ .
O PTIMAL TRANSPORT WITH NONCONVEX REGULARIZATION
In this section, we again focus on the primal formulation (1), but now study the case when the regularization Ω: R m + → R + ∪ {∞} is nonconvex .
Concavity.
It is well-known that the conjugate function is always convex, even when the original function is not.
As a result, even if the conjugate expressions (2) and (3) involve nonconcave maximization problems in the variable t , they induce convex functions in the variable s .
We can therefore make the following elementary remark: the dual (4) and the semi-dual (5) are concave maximization problems, even if Ω is nonconvex.
This means that we can solve them to arbitrary precision as long as we know how to compute the conjugate expressions (2) and (3).
This is generally hard but we will see in §5 a setting in which these expressions can be computed exactly .
We remark that the identity S Ω ( a , b , C ) = max α ∈ R m ⟨ α , a ⟩ − P ∗ Ω ( α , b , C ) still holds even when Ω is nonconvex.
The semi-dual upper-bounds the dual.
Of course, if Ω is nonconvex, only weak duality holds, i.e., the dual (4) and semi-dual (5) are lower bounds of the primal (1).
The next proposition clarifies that the semi-dual is actually an upper-bound for the dual (a proof is given in Appendix B.1).
Proposition 1.
Weak duality
Let Ω: R m + → R + ∪ {∞} (potentially nonconvex).
For all a ∈ △ m , b ∈ △ n and C ∈ R + m × n
[Equation]
Therefore, if the goal is to compute approximately P Ω ( a , b , C ) , which involves an intractable nonconvex problem in general, it may be advantageous to use S Ω ( a , b , C ) as a proxy, rather than
D Ω ( a , b , C ) .
However, for the specific choice of Ω in §5, we will see that D Ω ( a , b , C ) and
S Ω ( a , b , C ) actually coincide, i.e., D Ω ( a , b , C ) =
S Ω ( a , b , C ) ≤ P Ω ( a , b , C ) .
Recovering a plan.
Many times, the goal is not to compute the quantity P Ω ( a , b , C ) itself, but rather the associated OT plan.
If Ω is nonconvex, this is again intractable due to the nonconvex nature of the problem.
As an approximation, given an optimal solution ( α ⋆ , β ⋆ ) of the dual or an optimal solution α ⋆ of the semi-dual, we propose to recover a transportation plan with (6) and (7), just like we would do in the convex Ω case.
The following proposition clarifies that the optimal transportation plan T ⋆ that we get corresponds to a convex relaxation of the original nonconvex problem (1).
A proof is given in Appendix B.2.
Proposition 2.
Primal interpretation
Let Ω: R m + → R + ∪ {∞} (potentially nonconvex).
For all a ∈ △ m , b ∈ △ n and C ∈ R m × n +
[Equation]
In the above, f ∗∗ denotes the biconjugate of f , the tightest convex lower bound of f .
When Ω is nonconvex, deriving an expression for Ω ∗∗ + and Ω ∗∗ b j could be challenging in general.
Fortunately, for the choice of Ω in §5, we are able to do so.
When a function is convex and closed, its biconjugate is itself.
As a result, if Ω is a convex and closed function, we recover P Ω ( a , b , C ) =
D Ω ( a , b , C ) =
S Ω ( a , b , C ) for all a ∈ △ m , b ∈ △ n and C ∈ R m × n + .
Let
Summary: proposed method.
To approximately solve the primal OT objective (1) when Ω is nonconvex, we proposed to solve the dual (4) or the semi-dual (5), which by Proposition 1 lower-bound the primal.
We do so by solving the concave maximization problems in (4) and (5) by gradientbased algorithms, such as LBFGS (Liu & Nocedal, 1989) or ADAM (Kingma & Ba, 2015).
When a transportation plan is needed, we recover it from (6) and (7), as we would do with convex Ω .
Proposition 2 clarifies what objective function this plan optimally solves.
When learning with OT as a loss, it is necessary to differentiate through S Ω ( a , b , C ) .
From Danskin’s theorem, the gradients ∇ a S Ω ( a , b , C ) ∈ R m and ∇ C S Ω
( a , b , C ) ∈
R m × n are given by α ⋆ and T ⋆ from (7), respectively.
Q UADRATICALLY - REGULARIZED OT WITH SPARSITY CONSTRAINTS
In this section, we build upon §4 to develop a regularized OT formulation with sparsity constraints.
Formulation.
Formally, given t ∈ R m , let us define the ℓ 0 pseudo norm by
[Equation]
i.e., the number of nonzero elements in t .
For k ∈ { 1 , . . .
, m } , we denote the ℓ 0 level sets by
[Equation]
Our goal in this section is then to approximately solve the following quadratically-regularized optimal transport problem with cardinality constraints on the columns of T
[Equation]
where γ > 0 controls the regularization strength (we use γ = 1 throughout our experiments) and where k is assumed large enough to make the problem feasible (it is infeasible to be sparser than unregularized OT).
Problem (8) is a special case of (1) with the nonconvex regularization
[Equation]
We can therefore apply the methodology outlined in §4.
If the cardinality constraints need to be applied to the rows instead of to the columns, we simply transpose the problem.
Computation.
We recall that in order to solve the dual (4) or the semi-dual (5), the main quantities that we need to be able to compute are the conjugate expressions (2) and (3), as well as their gradients.
While this is intractable for general nonconvex Ω , with the choice of Ω in (9), we obtain
[Equation]
where, without loss of generality, we assumed γ = 1 .
Indeed, when γ ̸ = 1 , we can simply use the property ( γf ) ∗ = γf ∗ ( · /γ ) .
From the envelope theorem of Rockafellar & Wets (2009, Theorem 10.31), the gradients are given by the corresponding argmax problems and we obtain
[Equation]
Therefore, computing an optimal solution t ⋆ reduces to the k -sparse projections of s onto the nonnegative orthant and onto the simplex (scaled by b > 0 ), respectively.
We note that when the solution t ⋆ is not unique, the argmax is set-valued and we get sub-gradients instead.
Fortunately, despite the nonconvexity of the set B k , it turns out that both k -sparse projections can be computed exactly (Kyrillidis et al., 2013; Bolte et al., 2014; Beck & Hallak, 2016) by composing the unconstrained projection onto the original set with a top- k operation:
[Equation]
for some normalization constant τ ∈ R , such that the solution sums to b .
Here, topk ( s ) is defined such that [ topk ( s )]
i =
s i if s i is in the top- k elements of s and [ topk ( s )]
i =
−∞ otherwise.
In case of ties, we break ties arbitrarily.
The k -sparse projection on the simplex is also known as top- k sparsemax (Pillutla et al., 2018;
Blondel et al., 2020; Correia et al., 2020).
Plugging these solutions back into ⟨ s , t ⟩ − 12 ∥ t ∥ 22 and after some simple algebraic manipulations, we obtain the expressions in Table 1 (a proof is given in Appendix B.3).
Computing (11) or (12) requires a top- k sort and the projection of a vector of size at most k .
A top- k sort can be computed in O ( m log k ) time, proj R m + simply amounts to the non-negative part [ · ] + and computing τ , as needed for proj b △ m , can be computed in O ( k ) time (Michelot, 1986; Duchi et al., 2008), by reusing the top- k sort.
We have thus obtained an efficient way to compute the conjugates (2) and (3).
The total complexity per LBFGS or ADAM iteration is O ( mn log k ) .
Once we have computed the optimal dual or semidual solutions, because we compute the corresponding transportation plan using (6) and (7), each column t j of the obtained plan is guaranteed to contain at most k nonzero elements, by construction.
Biconjugates and primal interpretation.
As we discussed in §4 and Proposition 2, the biconjugates Ω ∗∗ + and Ω ∗∗ b allow us to formally define what primal objective the transportation plans obtained by (6) and (7) optimally solve when Ω is nonconvex.
Fortunately, for the case of Ω defined in (9), we are able to derive an actual expression.
Let us define the squared k -support norm by
[Equation]
The k -support norm is known to be the tightest convex relaxation of the ℓ 0 pseudo norm over the ℓunit ball and can be computed in O ( m log m ) time (Argyriou et al., 2012; McDonald et al., 2016).
We then have the following proposition, proved in Appendix B.5.
Proposition 3.
Biconjugate and primal interpretation
With Ω defined in (9) , we have
[Equation]
Therefore, with Ω defined in (9) , we have for all a ∈ △ m , b ∈ △ n and C ∈ R m × n +
[Equation]
In other words, our dual and semi-dual approaches based on the nonconvex Ω are equivalent to using the convex relaxation Ψ as regularization in the primal!
We believe that the biconjugate expressions in Proposition 3 are of independent interest and could be useful in other works.
For instance, it shows that top- k sparsemax can be alternatively viewed as an argmax regularized with Ψ .
[Figure]
Limit cases and smoothness.
Let T ⋆ be the solution of the quadratically-regularized OT (without cardinality constraints).
If k ≥ ∥ t ⋆j ∥ 0 for all j ∈ [ n ] , then the constraint ∥ t j ∥ 0 ≤ k in (8) is vacuous, and therefore our formulation recovers the quadratically-regularized one.
Since Ω is strongly convex in this case, both conjugates Ω ∗ + and Ω ∗ b are smooth (i.e., with Lipschitz gradients), thanks to the duality between strong convexity and smoothness (Hiriart-Urruty & Lemar ´ echal, 1993).
On the other extreme, when k = 1 , we have the following (a proof is given in Appendix B.6).
Proposition 4.
Limit cases
With Ω defined in (9) and k = 1 , we have for all s ∈ R m
[Equation]
We then have for all a ∈ △ m , b ∈ △ n and C ∈ R m × n + ,
D Ω ( a
[Equation]
When m < n , it is infeasible to satisfy both the marginal and the 1 -sparsity constraints.
Proposition 4 shows that our (semi) dual formulations reduce to unregularized OT in this “degenerate” case.
As illustrated in Figure 2, the conjugates Ω ∗ + and Ω ∗ b become increasingly smooth as k increases.
We therefore interpolate between unregularized OT (limit case k = 1 ) and quadratically-regularized
OT ( k large enough), with the dual and semi-dual being increasingly smooth as k increases.
E XPERIMENTS 6. S OLVER AND OBJECTIVE COMPARISON
We compared two solvers, LBFGS (Liu & Nocedal, 1989) and ADAM (Kingma & Ba, 2015) for maximizing the dual and semi-dual objectives of our sparsity-constrained OT.
Results are provided in Figure 3.
Compared to ADAM, LBFGS is a more convenient option as it does not require the tuning of a learning rate hyperparameter.
In addition, LBFGS empirically converges faster than
ADAM in the number of iterations (first row of Figure 3).
That being said, when a proper learning rate is chosen, we find that ADAM converges either as fast as or faster than LBFGS in wallclock time (second row of Figure 3).
In addition, Figure 3 shows that dual and semi-dual objectives are very close to each other toward the end of the optimization process.
This empirically confirms
Proposition 3, which states that the dual and the semi-dual are equal at their optimum.
[Table]
We have seen that a greater k leads to a smoother objective landscape (Figure 2).
It is known that a smoother objective theoretically allows faster convergence.
We validate this empirically in
Appendix A.2, where we see that a greater k leads to faster convergence.
6. S PARSE MIXTURES OF EXPERTS
We applied sparsity-constrained OT to vision sparse mixtures of experts (V-MoE) models for largescale image recognition (Riquelme et al., 2021).
A V-MoE model replaces a few dense feedforward layers MLP : x ∈ R d (cid:55)→ MLP ( x ) ∈ R d in Vision Transformer (ViT)
(Dosovitskiy et al., 2021) with the sparsely-gated mixture-of-experts layers:
[Equation]
(13) where Gate r : R d → R + is a sparse gating function and feedforward layers { MLP r } nr =1 are experts .
In practice, only those experts MLP r ( · ) corresponding to a nonzero gate value Gate r ( x ) will be computed – in this case, we say that the token x is routed to the expert r .
Upon a minibatch of m tokens { x 1 , . . .
, x m } , we apply our sparsity-constrained OT to match tokens with experts, so that the number of tokens routed to any expert is bounded.
Following Clark et al.
(2022), we backprop the gradient only through the combining weights Gate r ( x ) , but not through the OT algorithm (details in Appendix A.5), as this strategy accelerates the backward pass of V-MoEs.
Using this routing strategy, we train the B/32 and B/16 variants of the V-MoE model: They refer to the “Base” variants of V-MoE with 32 × 32 patches and 16 × 16 patches, respectively.
Hyperparameters of these architectures are provided in Riquelme et al.
(2021, Appendix B.5).
We train on the JFT-300M dataset (Sun et al., 2017), which is a large scale dataset that contains more than 305 million images.
We then perform 10-shot transfer learning on the ImageNet dataset (Deng et al., 2009).
Additional V-MoE and experimental details in Appendix A.5.
Table 2 summarizes the validation accuracy on JFT300M and 10-shot accuracy on ImageNet.
Compared to baseline methods, our sparsity-constrained approach yields the highest accuracy with both architectures on both benchmarks.
C ONCLUSION
We presented a dual and semi-dual framework for OT with general nonconvex regularization.
We applied that framework to obtain a tractable lower bound to approximately solve an OT formulation with cardinality constraints on the columns of the transportation plan.
We showed that this framework is formally equivalent to using squared k -support norm regularization in the primal.
Moreover, it interpolates between unregularized OT (recovered when k = 1 ) and quadratically-regularized OT (recovered when k is large enough).
The (semi) dual objectives were shown to be increasingly smooth as k increases, enabling the use of gradient-based algorithms such as LBFGS or ADAM.
We illustrated our framework on a variety of tasks; see §6 and Appendix A. For training of mixtureof-experts models in large-scale computer vision tasks, we showed that a direct control of sparsity improves the accuracy, compared to top-k and Sinkhorn baselines.
Beyond empirical performance, sparsity constraints may lead to more interpretable transportation plans and the integer-valued hyperparameter k may be easier to tune than the real-valued temperature parameter γ . A CKNOWLEDGMENTS
We thank Carlos Riquelme, Antoine Rolet and Vlad Niculae for feedback on a draft of this paper, as well as Aidan Clark and Diego de Las Casas for discussions on the Sinkhorn-Base router.
We are also grateful to Basil Mustafa, Rodolphe Jenatton, Andr´e Susano Pinto and Neil Houlsby for feedback throughout the project regarding MoE experiments
A E XPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS A. I LLUSTRATIONS
OT between 2D points.
In Figure 1, we visualize the transportation plans between 2D points.
These transportation plans are obtained based on different formulations of optimal transport, whose properties we recall in Table 3.
The details of this experiment are as follows.
We draw 20 samples from a Gaussian distribution N ([ 00 ] , [ 1 0 0 1 ]) as source points; we draw 20 samples from a different
Gaussian distribution N ([ 44 ] , (cid:2) 1 − 0 .
8 − 0 .
6 1 (cid:3) ) as target points.
The cost matrix in R 20 × 20 contains the
Euclidean distances between source and target points.
The source and target marginals a and b are both probability vectors filled with values 1 / 20 .
On the top row of Figure 1, blue lines linking source points and target points indicate nonzero values in the transportation plan obtained from each optimal transport formulation.
These transportation plans are shown in the second row.
Figureconfirms that, by varying k in our sparsity-constrained formulation, we control the columnwise sparsity of the transportation plan.
[Figure]
OT between two Gaussians.
In Figure 4 above, we show transportation plans between two Gaussian distributions.
The concrete set up of this experiment is as follows.
We let Y and Z be categorical random variables taking values in { 0 , . . .
The realizations of Y and Z are the source and target locations on a 1D grid, respectively.
Let ϕ
( z ; m, s ) :
= exp (cid:0) − ( z − m ) 2 2 s(cid:1) , where z, m, s are all real scalars with s ̸ = 0 .
The source distribution is set to be P ( Y = y ) :
= ϕ
( y ; 10 , 4) /c
Y with a normalizing constant c Y := (cid:80) 31 y =
ϕ ( y ; 10 , 4) ; the target distribution is set to be P ( Z = z ) :
= ϕ
( z ; 16 , 5) /
c
Z with a normalizing constant c Z := (cid:80) 31 z =0 ϕ
( z ; 16 , 5) .
The cost matrix C contains normalized squared Euclidean distances between source and target locations: C ij =
( i − j ) 2 / 31 2 ∈ [0 , 1] .
By setting k = 2 in our sparsity-constrained OT formulation, we obtain a transportation plan that contains at most two nonzeros per column (right-most panel of Figure 4).
OT between Gaussian and bi-Gaussian.
Similar to Figure 4, we show transportation plans between a Gaussian source marginal and a mixture of two Gaussians target marginal in Figure 5.
We set the source distribution as P ( Y = y ) :
= ϕ
( y ; 16 , 5) /c
Y , where c Y :=
(cid:80) 31 y =
ϕ
( y ; 16 , 5) is the normalizing constant; we set the target distribution as P ( Z = z ) :
=
(cid:0) ϕ
( z ; 8 , 5) + ϕ ( z ; 24 , 5) (cid:1) /c
Z , where c Z = (cid:80) 31 z =0 (cid:0) ϕ
( z ; 8 , 5) + ϕ ( z ; 24 , 5) (cid:1) is the normalizing constant.
Apart from that, we use the same settings as Figure 4,
[Figure]
A. S OLVER COMPARISON WITH AN INCREASED CARDINALITY
We have seen that an increased k increases the smoothness of the optimization problem (Figure 2).
This suggests that solvers may converge faster with an increased k .
We show this empirically in
Figure 6, where we measure the gradient norm at each iteration of the solver and compare the case k = 2 and k = 4 . [Figure] n [Figure]
r m
[Figure]
A. C OLOR TRANSFER
We apply our sparsity-constrained formulation on the classical OT application of color transfer (Piti´e et al., 2007).
We follow exactly the same experimental setup as in Blondel et al.
(2018, Section 6).
Figure 7 shows the results obtained from our sparsity-constrained approach.
Similar to well-studied alternatives, our yields visually pleasing results.
Plan sparsity: 99.
[Figure]
Reference
Sq.
l2 norm reg.
γ = 0.
Sparsity constrained k=
[Figure]
Plan
[Figure]
[Figure]
[Figure]
A. S UPPLY - DEMAND TRANSPORTATION ON SPHERICAL DATA
We follow Amos et al.
(2022) to set up a synthetic transport problem between 100 supply locations and 10,000 demand locations worldwide.
Transport costs are set to be the spherical distance between the demand and supply locations.
This transportation problem can be solved via the entropy regularized optimal transport as in Amos et al.
We visualized this entropy-regularized transport plan in panel (a) of Figure 8.
Building upon the setting in Amos et al.
(2022), we additionally assume that each supplier has a limited supplying capacity .
That is, each supplier can transport goods to as many locations as possible up to a certain prescribed limit.
This constraint is conceivable, for instance, when suppliers operate with a limited workforce and cannot meet all requested orders.
We incorporate this constraint into our formulation of sparsity-constrained optimal transport by specifying k as the capacity limit.
The panel (b) of Figure 8 is the obtained transportation plan with a supplying capacity of k =(each supplier can transport goods to at most 100 demand locations).
Comparing panels (a) and (b) of Figure 8, we recognize that derived plans are visibly different in a few ways.
For instance, with the capacity constraint on suppliers, demand locations in Europe import goods from more supply locations in North America than without the capacity constraint.
Similar observations go to demand locations in pacific islands: Without the capacity constraint, demand locations in Pacific islands mostly rely on suppliers in North America; with the capacity constraint, additional suppliers in South America are in use.
[Figure]
A. V-MOE EXPERIMENT
Our experiment is based on the vision MoE (V-MoE) architecture (Riquelme et al., 2021), which replaces a few MLP layers of the vision Transformer (Dosovitskiy et al., 2021) by MoE layers.
In this subsection, we review the background of V-MoE, with a focus on the router , which decides which experts get which input tokens.
We introduce a few notations that will be used throughout this subsection.
Let { x 1 , . . .
, x m } ⊂ R d be a minibatch of m tokens in R d , and let X ∈ R m × d be a corresponding matrix whose rows are tokens.
Let W ∈ R d
× n be a learnable matrix of expert weights , where each column of W is a learnable feature vector of an expert.
Common to different routing mechanisms is an token-expert affinity matrix Π :=
XW ∈ R m × n : Its ( i, j ) -th entry is an inner-product similarity score between the i -th token and the j -th expert.
The TopK router.
To route tokens to experts, the TopK router in Riquelme et al.
(2021) computes a sparse gating matrix Γ that has at most κ nonzeros per row, through a function top κ : R n → R n that sets all but largest κ values zero:
[Equation]
with
[Equation]
[Equation]
Note that the integer κ is not to be confused with k used in the main text – κ here refers to the number of selected expert for each token and it can differ from the cardinality-constraint k used in the main text in general.
The vector ϵ ∼ N (0 , I ) in (14) is a noise injected to the token-expert affinity matrix
XW with σ ∈ R controlling the strength of noise.
In practice, σ is set to be 1 /n during training and 0 in inference.
To ensure that all experts are sufficiently trained, the gating matrix Γ in (14) is regularized by auxiliary losses that encourage experts to taken a similar amount of tokens in a minibatch.
A detailed description of these auxiliary losses is presented in Riquelme et al.
Section A).
For an efficient hardware utilization, Riquelme et al.
(2021) allocate a buffer capacity of experts, which specifies the number of tokens each expert can at most process in a minibatch.
With a specified buffer capacity and a computed gating matrix Γ , the TopK router goes over the rows of Γ and assign each token to its top-chosen expert as long as the chosen expert’s capacity is not full.
This procedure is described in Algorithm 1 of Riquelme et al.
(2021, Section C.1).
Finally, the outcomes of experts are linearly combined using the gating matrix Γ as in (13).
The S-BASE router.
Clark et al.
(2022) cast the token-expert matching problem as an entropyregularized OT problem, solved using the Sinkhorn algorithm.
This approach, dubbed as the
Sinkhorn-BASE (S-BASE) router, was originally designed for language MoEs that take text as input.
In this work, we adapt it to vision MoEs.
In direct parallel to the TopK gating matrix in (14), the gating matrix of entropy-regularized OT is set to be
[Equation]
(15) where
[Equation]
with a = 1 n and b = ( m/n ) 1 n .
The optimization plan Π ent in (16) can be obtained using the Sinkhorn algorithm (Sinkhorn &
Knopp, 1967).
Note that while we formulated optimal transport problems with non-negative cost matrices in the main text, values in the cost matrix C = − Π in (16) can be both positive and negative, following Clark et al.
Since Π ent is a dense matrix, a heuristic is needed to select only κ experts to form the gating matrix Γ ent – this is achieved by using a top κ in (15).
With a computed gating matrix Γ ent , the S-BASE router assigns each token to its top-chosen expert in the same way of the TopK router.
This process allocates each expert an amount of tokens, up to a certain upper bound specified by the buffer capacity as in the case of TopK. As in Clark et al.
, we linearly combine the output of experts using a softmax matrix softmax(Π) .
In this way, the backward pass of gradient-based training does not go through the Sinkhorn algorithm, can be faster and more numerically stable 1 .
The Sparsity-constrained router.
We cast the token-expert matching problem as a sparsityconstrained OT problem.
With a prescribed buffer capacity k , our goal is to upper-bound the number of tokens assigned to each expert by k .
This amounts to adding a cardinality constraint to each column of the gating matrix:
[Equation]
with a = 1 n and b = ( m/n ) 1 n
[Equation]
17) with Π softmax = softmax( XW ) .
The purpose of the softmax function here is to obtain a cost matrix containing values of the same sign.
Otherwise, if a cost matrix contains both positive and negative values, then the obtained plan from sparsity-constrained optimal transport may contain zero at all entries corresponding to positive values in the cost matrix, so as to minimize to the objective.
In that case, columns of this transportation may contain much fewer nonzeros than k – this is an undesirable situation as it under-uses the buffer capacity of experts.
Note that, however, this was not an issue in the S-BASE router – a cost matrix there can contain both positive and negative values (Clark et al., 2022) – because values of the transportation plan yielded by the Sinkhorn’s algorithm are strictly positive.
The sparse transportation plan Γ sparse in (17), allocates each expert an amount of tokens up to k .
As in the S-BASE router, we linearly combine the output of experts using the matrix Π softmax .
To approximate Γ sparse , we optimize its semi-dual proxy as introduced in Section 4.
We do so by using an ADAM optimizer with a learning rate of 10 − 2 for 50 steps.
V-MoE architecture.
We use the S-BASE router and our proposed sparsity-constrained router as drop-in replacements of the TopK router in otherwise standard V-MoE architectures (Riquelme et al., 2021).
We focus on the V-MoE B/32 and B/16 architectures, which use 32 × 32 and 16 ×patches, respectively.
We place MoEs on every other layer, which is the Every-2 variant in Riquelme et al.
We fix the total number of experts n = 32 for all experiments.
In the TopK and SBASE router, we assign 2 experts to each expert, that is, κ = 2 in (14) and (15).
The buffer capacity is set to be n/κ = 32 / 2 = 16 , that is, each expert can take 16 tokens at most.
To match this setting, we use k = 16 in (17) for our sparsity-constrained router.
Upstream training and evaluation.
We follow the same training strategy of Riquelme et al.
(2021) to train B/32 and B/16 models on JFT-300M, with hyperparameters reported in Riquelme et al.
(2021, Table 8).
JFT-300M has around 305M training and 50,000 validation images.
Since labels of the JFT-300M are organized in a hierarchical way, an image may associate with multiple labels.
We report the model performance by precision@1 by checking if the predicted class with the highest probability is one of the true labels of the image.
Downstream transfer to ImageNet.
For downstream evaluations, we perform 10-shot linear transfer on ImageNet (Deng et al., 2009).
Specifically, with a JFT-trained V-MoE model, we freeze the model up to its penultimate layer, re-initialize its last layer, and train the last layer on ImageNet.
This newly initialized layer is trained on 10 examples per ImageNet class (10-shot learning).
[Table]
Comparing the speed of routers.
We note that the sparsity-constrained router is slightly slower than baseline routers.
One reason is that the topk function used for k -sparse projection steps.
To further speedup the sparsity-constrained router, an option is to use the approximated version of topk (Chern et al., 2022), which we did not use in this study.
This approximated topk may be especially useful on large models like B/16, where the number of tokens is large.
Another way to accelerate
E-Step
M-Step
K-Means t i, : ← e argmin j [ C ( s ) ]
i,j µ ( s +1) ← ( XT ) ⊘ ( 1 d × m T )
Soft K-Means t i, : ← softmax (cid:0) − [ C ( s ) ] i, : (cid:1)
Negentropy
Solve (1) with Ω( t ) = ⟨ t , log t ⟩
Squared 2-norm
Solve (1) with Ω( t ) = 12 ∥ t ∥
Sparsity-constrained
Solve (4), (5) with Ω( t ) = 12 ∥ t ∥ 22 + δ B k
( t ) the sparsity-constrained router is to explore different optimizers.
Currently, we run the ADAM optimizer for 50 steps using a learning rate 10 − 2 .
We suspect that with a more careful tuning of the optimizer, one can reduce the number of steps without harming the performance.
Variants of accelerated gradient-based methods (An et al., 2022) may also be applicable.
A. S OFT BALANCED CLUSTERING
OT viewpoint.
Suppose we want to cluster m data points x 1 , ..., x m ∈ R d into n clusters with centroids µ 1 , . . . , µ n ∈ R d .
We let X ∈ R
d
× m be a matrix that contains data points x 1 , ..., x m as columns.
Similarly, we let µ ∈ R d
× n be a matrix of centroids.
The K-Means algorithm can be viewed as an OT problem with only one marginal constraint, min
[Equation]
T, C ⟩ , where [ C ] i,j = ∥ x i − µ j ∥ 22 and a = 1 m /m .
Lloyd’s algorithm corresponds to alternating minimization w.r.t.
T (updating centroid memberships) and w.r.t.
µ (updating centroid positions).
This viewpoint suggests two generalizations.
The first one consists in using two marginal constraints
[Equation]
This is useful in certain applications to impose a prescribed size to each cluster (e.g., b = 1 n /n ) and is sometimes known as balanced or constrained K-Means (Ng, 2000).
The second generalization consists in introducing convex regularization Ω
[Equation]
This moves the optimal plan away from the vertices of the polytope.
This corresponds to a “soft” balanced K-Means, in which we replace “hard” cluster memberships with “soft” ones.
We can again alternate between minimization w.r.t.
T (solving a regularized OT problem) and minimization w.r.t.
µ .
In the case of the squared Euclidean distance, the closed form solution for the latter is µ i ∝ (cid:80) nj =1 t i,j x j for all i ∈ [ m ] .
When Ω is nonconvex, we propose to solve the (semi) dual as discussed in the main text.
Results on MNIST.
MNIST contains grayscale images of handwritten digits, with a resolution of 28 × 28 pixels.
The dataset is split in 60 000 training and 10 000 test images.
As preprocessing,
[Table]
we simply put the pixel values in the range [ − 1 , 1] and “flatten” the images to obtain vectors ofelements.
We use the training set to estimate the centers of the clusters using different algorithms.
We use an
EM-like algorithm to estimate the cluster centers in all cases, as described in Table
(we perform 50 update steps).
In particular, notice that only the E-step changes across different algorithms, as described in Table 5.
Since there are 10 digits, we use 10 clusters.
We evaluate the performance on the test set.
Since some of the algorithms produce a “soft” clustering (all except K-Means), represented by the matrix T , for each test image i we assign it to the cluster j with the largest value in T =
( t i,j ) .
We measure the average cost (i.e. average squared distance between each image and its selected cluster), and the KL divergence between the empirical distribution of images per cluster and the expected one (a uniform distribution).
The centers are initialized from a normal distribution with a mean of 0 and a standard deviation of 10 − 3 .
Algorithms employing an OT-based approach perform 500 iterations to find T , using either the Sinkhorn algorithm (with the Negentropy method) or LBFGS (used by the rest of OT-based methods).
We use a sparsity-constraint of k = ⌈ 1 . 15 · mn ⌉ (recall that k is the maximum number of nonzeros per column).
Notice that using k = mn , and assuming that n is a divisor of m , would necessary require that the number of nonzeros per row is 1.
Thus, our minimization problem would be equivalent to that of the unregularized OT.
Thus, we slightly soften the regularization.
Table 6 shows the results of the experiment, averaged over 20 different random seeds.
The best cost is achieved by the Soft K-Means algorithm, but the resulting clustering is quite unbalanced, as reported by the KL divergence metric.
On the other hand, all OT-based approaches achieve similar costs, but the algorithm based on §5 obtains a significantly better balanced clustering.
B P ROOFS B. W EAK DUALITY (P ROPOSITION 1)
Recall that
[Equation]
We add Lagrange multipliers for the two equality constraints but keep the constraint T ∈ R m × n + explicitly.
The Lagrangian is then
[Equation]
Using the inequality min u max v f ( u , v ) ≥ max v min u f ( u , v ) twice, we have
[Equation]
For the first inequality, we have
[Equation]
For the second inequality, we have
[Equation]
To summarize, we showed P Ω ( a , b , C ) ≥ S Ω ( a , b , C ) [Equation] D Ω ( a , b , C ) [Equation]
B. P RIMAL INTERPRETATION (P ROPOSITION 2)
For the semi-dual, we have
[Equation]
where we used that strong duality holds, since the conjugate is always convex, even if Ω is not.
Likewise, [Table] , we have
[Equation]
B. C LOSED - FORM EXPRESSIONS (T ABLE 1)
The expressions for the unregularized, negentropy and quadratic cases are provided in (Blondel et al., 2018, Table 1).
We therefore focus on the top- k case.
Plugging (11) back into ⟨ s , t ⟩ − 12 ∥ t ∥ 22 , we obtain
[Equation]
Plugging (12) back into ⟨ s , t ⟩ − 12 ∥ t ∥ 22 , we obtain
[Equation]
B. U SEFUL LEMMAS
Lemma 1.
Conjugate of the squared k -support norm
Let us define the squared k -support norm for all t ∈ R m by
[Equation]
Its conjugate for all s ∈ R m is the squared k -support dual norm:
[Equation]
Proof.
This result was proved in previous works (Argyriou et al., 2012; McDonald et al., 2016).
We include here an alternative proof for completeness
[Equation]
Using Lapin et al.
(2015, Lemma 1), we have for all a ∈ R m
[Equation]
We therefore obtain the variational formulation
[Equation]
We rewrite the problem in constrained form
[Equation]
We introduce Lagrange multipliers λ ∈ R m + , for the inequality constraints but keep the non-negative constraints explicitly
[Equation]
Using strong duality
[Equation]
we have
[Equation]
We therefore obtain
[Equation]
s
[Equation]
⟨ λ , 1 ⟩ = k, λ
[Equation]
≤ 1 ∀ i ∈ [
[Equation]
[Equation]
Using Ψ( t ) =
12 ψ ∗ (2 t ) gives the desired result.
Lemma 2.
For all s ∈ R m and b >
[Equation]
with optimality condition (cid:80) mi =1 [ s i
[Equation]
= b .
Proof.
[Equation]
From which we obtain the optimality condition
[Equation]
B. B ICONJUGATE EXPRESSIONS (P ROPOSITION 3) Derivation
of Ω ∗∗ + Recall that
[Equation]
Using Lemma 1, we then have
[Equation]
Note that ξ is not constrained to be non-negative because it is squared in the objective.
Derivation of Ω ∗∗ b .
Recall that
[Equation]
Using Lemma 1, Lemma 2 and (12), we obtain
[Equation]
We then have
[Equation]
R
[Equation]
= (cid:26) Ψ( t )
if t ∈ b △ m ∞ , otherwise .
B. L IMIT CASES (P ROPOSITION 4)
In the limit case k = 1 with Ω defined in (9), we have
[Equation]
We therefore get
[Equation]
Likewise, we have
[Equation]
We therefore get
[Equation]
From Proposition
[Equation]
we have D Ω = S Ω = S
[Equation]
b
[Equation]