Fast Stochastic Composite Minimization and an Accelerated Frank-Wolfe Algorithm under Parallelization
Abstract
We consider the problem of minimizing the sum of two convex functions.
One of those functions has Lipschitz-continuous gradients, and can be accessed via stochastic oracles, whereas the other is “simple”.
We provide a Bregman-type algorithm with accelerated convergence in function values to a ball containing the minimum.
The radius of this ball depends on problem-dependent constants, including the variance of the stochastic oracle.
We further show that this algorithmic setup naturally leads to a variant of Frank-Wolfe achieving acceleration under parallelization.
More precisely, when minimizing a smooth convex function on a bounded domain, we show that one can achieve an ϵ primal-dual gap (in expectation) in ˜ O
(1 / √ ϵ ) iterations, by only accessing gradients of the original function and a linear maximization oracle with O (1 / √ ϵ ) computing units in parallel.
We illustrate this fast convergence on synthetic numerical experiments.
Introduction
We consider the composite minimization problem
[Equation]
where V is some real vector space equipped with a norm ∥·∥ and an inner product ⟨· , ·⟩ , see Sectionfor details.
For instance, one can consider V =
R d or V
= R p × q equipped with the standard
Euclidean inner product and norm.
The function G is convex and β -smooth and the function H is µ -strongly convex, both with respect to (w.r.t.)
some norm ∥ .
∥ on V (see Section 2 for precise definitions).
We assume that H is “simple”, meaning that one can efficiently solve
[Equation]
where D w is the Bregman divergence induced by a strongly convex function w .
For instance, when w is the standard squared Euclidean norm, this amounts to computing the proximal operator of H .
When deterministic gradients of G are available, an accelerated method relying on (2) and achieving rates of the form O (cid:16) exp (cid:16) − k (cid:113) µβ (cid:17)(cid:17) was proposed by Diakonikolas and Guzmán [13, AGD+].
In this work, we are interested in the case where G is only available through a stochastic oracle.
In particular, we provide an accelerated algorithm converging in function values to a neighborhood of the minimum with the same rate as above.
The size of the neighborhood is of order O (cid:16) σ 2 √ µβ (cid:17) where σ 2 is the variance of the stochastic gradients.
The dependence of the noise term on 1 / √ µβ is similar to that of previous stochastic accelerated methods in simpler settings, see, e.g., [3].
Although our rate is an extension of the result in [13], the parameters are different and tailored for the stochastic setup.
In Section 4, we focus on minimization problems over a compact convex set K for which we have access to a linear optimization oracle, just as in the Frank-Wolfe algorithm.
Formally, we consider solving min x ∈ K f ( x ) where f is convex with Lipschitz-continuous gradients.
In that case, the dual problem can be seen as a particular instance of (1) where H :
= f ∗ is the Fenchel conjugate of f , and
G is a smoothed version of the support function of K .
Such smoothed functions can only be accessed through a stochastic oracle whose computation boils down to solving a linear optimization problem on K .
An appropriate choice of w allows us to use the algorithmic framework by only computing gradients of f and solving linear optimization subproblems over K .
In short, minimizing f over the set K with an ϵ primal-dual gap requires ˜ O (cid:16) max (cid:110) 1 √ ϵ , 1 ϵm (cid:111)(cid:17) iterations when m computing units can be used in parallel (each of them only solving linear optimization subproblems on K ).
In particular, when m = 1 / √ ϵ , this effectively yields an accelerated rate of ˜ O
( 1 / √ ϵ ) iterations with a single gradient evaluation per iteration to achieve the required accuracy.
We emphasize that the first contribution of this work is to provide an accelerated method for general stochastic composite problems (in Section 3).
In Section 4, we then show that it can be used for obtaining an accelerated Frank-Wolfe algorithm.
1.1 Related Work
As previously emphasized, the main algorithmic ingredients which inspired this work were developped by [13] and [19].
For further references on acceleration and gradient-based composite convex minimization, we refer to the original works [38, 40] or to a recent survey [15].
For the stochastic setup, we refer to [32].
We refer to [39, 12] for further references and pointers on those topics.
Let us briefly describe the main differences between this work and [13, 19].
First, Gasnikov and
Nesterov [19] obtain accelerated rates similar to ours when the strong convexity assumption is placed on G instead of H here (which has practical consequences, including well-posedness of (2) and the size of the constants β and µ , see discussion in [13, Introduction]).
When the gradients of G are stochastic, [19] provides convergence rates, but only when the underlying norm is the Euclidean norm.
Finally, whereas Diakonikolas and Guzmán [13] consider more general assumptions in the deterministic non-Euclidean case, our work also covers stochastic approximations.
We emphasize that each assumption we make (namely accelerated, stochastic, proximal Bregman methods) is necessary to yield acceleration of Frank-Wolfe under parallelization in Section 4.
Two key observations underlying this work are on the one hand the duality link between Frank-Wolfe methods and Bregman methods [4], and on the other hand randomized smoothing techniques [41, 14, 1, 2, 7, 8] which can be naturally computed using linear optimization steps.
The main question of interest was whether accelerated Bregman methods should naturally give rise to accelerated
Frank-Wolfe methods on the dual.
The Frank-Wolfe (FW) method (a.k.a.
conditional gradient method) and its variants were first introduced by Frank and Wolfe [16] (see also [27, 26] for more modern presentations).
When considering optimization over a convex set K , classical first-order methods are often naturally embedded with a projection operator (onto K ).
Depending on K , those projections are potentially costly.
An alternate approach for taking constraints into account within first-order method consists in using linear optimization oracles (a.k.a.
Frank-Wolfe techniques).
In many applications, such linear minimizations are much cheaper than projecting onto the feasible set, see for example [17, 23].
Despite its wide use in practical applications, the main drawback of the Frank-Wolfe method lies in its slow convergence rate of O ( 1 / ϵ ) , standing in sharp contrast with the O ( 1 / √ ϵ ) convergence of
Nesterov’s accelerated gradient descent [38] relying on projections.
As “purely accelerated” rates of convergence are out of reach for vanilla Frank-Wolfe methods (see lower complexity bound in [33]), most works on accelerating Frank-Wolfe have focused on exploiting specific additional assumptions on the problems at hand.
Common such assumptions include strong convexity of the feasible set [11, 35, 18], and strong convexity of the objective function along with the assumption that the minimizer lies in the interior of the feasible set [21, 31].
In both cases,
Frank-Wolfe is known to improve on the O ( 1 / ϵ ) rate.
Some efforts have also gone into finding rates matching performances of Nesterov’s accelerated gradient descent [38] without strong convexity.
In particular, when K is a polytope and when a certain type of constraint qualification is satisfied,
Frank-Wolfe converges asymptotically at rate O
( 1 / k 2 ) [5].
Adding momentum to Frank-Wolfe has also been studied, and accelerated rates are attained on some ℓ p -norm balls when the minimizer is on the boundary of the feasible set [36].
Our approach in this work is orthogonal to the aforementioned results, in that we do not make additional assumptions on the objective function or the feasible set, but instead show that parallelization can help reaching accelerated rates for a variant of Frank-Wolfe.
Note that [34] manages to reach a similar rate of O ( 1 / √ ϵ ) iterations for a variant of Frank-Wolfe, where each iteration requires one gradient evaluation and O ( 1 / √ ϵ ) calls to a linear optimization oracle.
However, in contrast with our algorithm, their approach is non-parallelizable as the calls to the linear optimization oracle need to be made sequentially.
The rest of the paper is organized as follows.
In Section 2 we define notations and review some classical definitions from convex analysis.
In Section 3 we provide worst-case rates for a stochastic
Bregman dual-averaging-type algorithm, along with some intuitions and proof sketches.
In Sectionwe show how a Frank-Wolfe method directly fits within this framework, thereby obtaining accelerated worst-case guarantees on the primal-dual gap.
In Section 5, we illustrate our theoretical findings on a set of simple numerical examples.
Notation and definitions
Formally, we consider a real finite-dimensional vector space V and its dual space V ∗ consisting of all linear functions on V , as well as a dual pairing denoted by ⟨· , ·⟩ : V ∗ × V
→ R , and a norm ∥·∥ :
V → R .
We also consider the corresponding dual norm ∥·∥ ∗ : V ∗ →
R induced by the choice of ∥·∥ and ⟨· , ·⟩ using ∥ z ∥ ∗ = sup
∥ x ∥≤ 1 ⟨ z, x ⟩ .
For instance, one can use V = V ∗ =
R d or
V =
V ∗ =
R p × q equipped with the standard Euclidean inner product and norm.
We insist on the fact that the formal notation V and V ∗ is used mostly for emphasizing differences between primal and dual spaces/problems below.
For a closed, proper, convex function Ψ : V → R , we denote ∂ Ψ( x ) the set of all subgradients of Ψ at x .
When ∂ Ψ( x ) is a singleton, we denote its single element by ∇ Ψ( x ) .
Definition 1.
Ψ :
V → R is L -smooth w.r.t.
∥·∥ if for all x, y ∈ V ,
[Equation]
The following proposition will be helpful throughout the paper (see [39] for a proof).
Proposition 1.
Ψ :
V → R is L -smooth w.r.t.
∥·∥ if and only if for all x, y ∈ V ,
[Equation]
Definition 2.
Ψ :
V → R is µ -strongly convex w.r.t.
∥·∥ if for all x, y ∈ V and any g Ψ ( y ) ∈
∂ Ψ( y ) ,
[Equation]
Definition 3.
For a closed, proper, convex function Ψ : V → R , its conjugate function is defined as
[Equation]
For later references, we need a few results related to smoothing the support function of a set K ⊂ V ∗
( K will appear as a convex set in the dual problem to (1)).
Definition 4.
For a set K ⊂ V ∗ , we define I K as the indicator function of K , i.e., I K ( x )
=
for x ∈ K
and I K ( x ) = ∞ for x ∈ V ∗ \ K .
The support function of K is defined as
[Equation]
If K is non-empty, convex and closed, the indicator and support functions of K are conjugates of each other, i.e., ( I K ) ∗ =
s and s ∗ =
( I K )
∗∗ = I K [42].
The support function is convex yet not differentiable in general applications.
For the purpose of our work, we consider smoothing the support function via randomization.
Such stochastic smoothing has recently gained popularity in the optimization literature, see for example [41, 14, 1, 2, 7].
We define the main tools and state the relevant properties behind this idea.
Definition 5.
For a set K ⊂ V ∗ , a scalar α > 0 and a random variable ∆ , we define the smoothed support function of K as
[Equation]
We will use the following proposition on the smoothed support function, proved in [7].
Proposition 2.
Suppose K ⊂ V ∗ is convex compact and let R K = max x ∈ K
∥ x ∥ ∗ .
Suppose the random variable ∆ has positive differentiable density dπ ( z ) ∝ exp( − η ( z ))
dz for some function η ( · ) , and let M 2 = E (cid:104) ∥∇ z η ( Z ) ∥ 2 ∗ (cid:105) .
Then s α is convex,
R K
Mα -smooth w.r.t.
∥·∥ and for any y ∈ V ,
[Equation]
arg
[Equation]
[Equation]
and s ( y ) ≤ s α ( y ) ≤ s ( y )
+ αs
[Equation] [Equation]
Fast Stochastic Composite Minimization
In this section we focus on solving the composite problem
[Equation]
8) where G is convex and β -smooth, and H is µ -strongly convex w.r.t.
∥·∥ .
Algorithm 1 summarizes our proposed algorithm, which resembles the general AGD+ algorithm from [13].
Notable differences include access to gradients of G through a stochastic oracle, and explicit dependencies on the smoothness and strong convexity constants for the different updates.
Moreover, the presented analysis is different and specifically tailored to handle stochasticity in the gradients of G .
An actual implementation of Algorithm 1 requires the ability to efficiently solve the minimization step (13), which should be well-defined.
This intermediate minimization subproblem is often referred to as a Bregman proximal problem, and a sufficient condition for this operation to be well-defined is to require w to be strongly convex.
In the case where w is the Euclidean norm, (13) amounts to computing the proximal operator of H .
Considering a general regularizer w has several benefits, in that the Euclidean norm might not well capture the geometry of the problem, and because particular choices of w might make (13) easily solvable.
The latter will become particularly clear in Section 4.
Before we move on to the analysis, let us emphasize that the first step of Algorithm 1 consisting in picking z 0 ∈ arg min y w ( y ) is not restrictive.
Indeed one could instead pick any z 0 ∈ V and set ˜ w ( y ) =
w ( y ) − ⟨ g w ( z 0 ) , y ⟩ where g w ( z 0 ) ∈
∂w ( z 0 ) .
One could then run Algorithm 1 with the “shifted” ˜ w instead of w .
Doing so does not change the complexity of the minimization step (13), and the following analysis remains unchanged.
We are now ready to analyse Algorithm 1.
For this purpose let us define
[Equation]
where the sequences { A k } k ∈ N and { y k } k ∈ N are defined in Algorithm 1.
We also use the notation E k for denoting the expectation at iteration k conditioned on the previous iterations (that is, E k shortens
E k [ · ] =
E [ · | y k , z k , d k , c k ] ), while E denotes the total expectation.
Before we can state our results, we need one more assumption on the stochastic gradients.
Assumption 1.
For any k ∈ N , the stochastic gradients satisfy E k
[ g k ] = ∇ G ( v k ) and
[Equation]
The assumption on the variance of the stochastic gradients is common when studying stochastic first-order methods, and allows proving the next proposition, which relates consecutive iterations.
Input :
[Equation]
function G
[Equation]
-strongly convex function H
[Equation]
ν
[Equation]
strongly convex function w , all w.r.t.
the same norm ∥·∥ .
Pick z 0 ∈ arg min y w ( y ) and set y
= z 0 .
Set A
= 0 and d 0 = 0 .
for k = 0 , 1 , . . .
do
[Equation]
Compute a stochastic gradient g k of function G at iterate v k
[Equation]
end
Proposition 3.
Suppose Assumption 1 holds and let m k ( y ) :
= ⟨ d k , y ⟩ + c k + A k H ( y ) + βw ( y ) .
At iteration k , the iterates of Algorithm 1 satisfy
[Equation]
We defer the full proof to Appendix A, but highlight the main steps here.
This way we also hope to shed light on the update of A k +1 .
The first step is to compute an upper bound on A k +1 G ( y k +1 ) depending on A k G ( y k ) .
The second step is similar, and computes an upper bound on A k +1 H ( y k +1 ) depending on A k H ( y k ) .
Summing up the two inequalities and taking expectations yields inequality (15) with an additional term in ∥ z k +1 − z k ∥ 2 .
To exactly obtain inequality (15) we set A k
+
so as to cancel out the coefficient multiplying ∥
z k +1 − z k ∥ 2 .
This turns out to be equivalent to setting A k +1 as the root of a quadratic polynomial, explaining the form of update (9) in Algorithm 1.
Proposition 3 allows us to get the final rate of convergence of Algorithm 1.
We again defer the proof to Appendix A.
The idea here is first to unroll the recursion in (15) so as to get a constant term on the right-hand side of the inequality.
We then relate m N ( z N ) to m N ( y ⋆ ) and to the minimum of F and finally, we show that A k +1 ≥
A k
[Equation]
, which gives the exponential decay term.
Theorem 1.
Suppose Assumption 1 holds, let y ⋆ ∈ arg min F and define D w ( y ⋆ , y 0 )
=
w ( y ⋆ )
− w ( y 0 ) ≥ 0 .
The convergence rate of Algorithm 1 after k iterations is
[Equation]
This rate is a typical accelerated rate.
In the (usual) case where µ ≤ β , the exponential decay is bounded above by exp (cid:16) − k 4 (cid:113) µβ (cid:17) , which shows the natural dependence on (cid:113) µβ .
In addition, the neighborhood term is of the form O (cid:16) σ 2 √ µβ (cid:17) , which is again typical of accelerated stochastic methods [3].
Note that w ( y ⋆ ) − w ( y 0 ) is equal to the Bregman divergence D w ( y ⋆ , y 0 ) :
= w ( y ⋆ )
− w ( y 0 ) − ⟨ g w ( y 0 ); y ⋆ − y 0 ⟩ (with g w ( y 0 )
∈ ∂w ( y 0 ) ) through the choice g w ( y 0 ) =
∈ ∂w ( y 0 ) , which is valid as y 0 minimizes w ( · ) .
We emphasize that we do not require differentiability of the function w anywhere.
In the next section, we show how the above algorithm can be directly applied to a smoothed dual of a minimization problem over a compact convex set, yielding a variant of the Frank-Wolfe algorithm which can achieve accelerated rates under parallelization.
Accelerating Frank-Wolfe with parallelization
We consider the following minimization problem over a compact convex set K ⊂ V ∗
[Equation]
where f is convex and L -smooth w.r.t.
∥·∥ ∗ .
Its Fenchel-Rockafellar dual [42] reads
[Equation]
The smoothness of f implies that f ∗ is 1 / L -strongly convex w.r.t.
∥·∥ [43, Proposition 12.60].
The term in s is however not smooth w.r.t.
∥·∥ , which prevents us from directly applying results from the previous section.
Instead, we choose some smoothing parameter α > 0 and, following Definitionand Proposition 2, we consider the smoothed minimization problem
[Equation]
Problem (19) fits within the framework of (8) with G ( y ) =
s α ( − y ) , H ( y )
=
f ∗ ( y ) , µ = 1 / L and β =
R K Mα .
Using Proposition 2, we see that an unbiased stochastic gradient g of G at some point y can be obtained by sampling ∆ and computing g = − arg max u ∈ K ⟨ u, − y + α ∆ ⟩ .
This boils down to a linear optimization oracle over K , the same oracle as that of the Frank-Wolfe algorithm.
We now show how to pick the distance generating function w such that the minimization step of Algorithm 1 has a closed form.
Recalling that z 0 must minimize w , we set w ( y ) =
f ∗ ( y ) − ⟨ g f ∗ ( z 0 ) , y ⟩ where g f ∗ ( z 0 ) ∈
∂f ∗ ( z 0 ) .
Clearly, w is 1 / L -strongly convex w.r.t.
∥·∥ and is minimized at z 0 .
Moreover, first-order optimality conditions for the minimization step (13) of Algorithm 1 read
[Equation]
Choosing some x 0 ∈ K and setting z k = ∇ f ( x k ) for all k , one can replace (21) by
[Equation]
Doing this allows to avoid computing subgradients of f ∗ .
Instead, whenever the value of z k is needed, we compute ∇ f ( x k ) .
We summarize this fully primal algorithm in Algorithm 2.
Note that similar tricks to obtain “dual-free” methods have already been used, see for example [32, 46, 25, 10, 28].
Observe that each iteration of Algorithm 2 requires the computation of one gradient of f , and m calls to a linear maximization oracle.
When m = 1 , an iteration of Algorithm 2 is as costly as one iteration of Frank-Wolfe.
We now show how to further exploit parallelization.
To do so, we need to ensure that using multiple samples appropriately improves the quality of the approximation of the true gradient.
This the point of the following assumption.
Assumption 2.
There exists a norm-dependent constant ρ ∥·∥ ∗ such that the variance verifies
[Equation]
Remark 1.
A few examples are in order.
For the standard Euclidean norm ∥ · ∥ = ∥ · ∥ ∗ = ∥ · ∥ 2 it holds that ρ ∥·∥ 2 =
For ℓ p -norms on a space of dimension d , we
have
[Equation]
In the general case, as all norms are equivalent in finite dimensional spaces, there exist constants c, C > 0 such that c ∥·∥ 2 ≤ ∥·∥ ∗ ≤ C ∥·∥ 2 .
Then ρ ∥·∥ ∗ ≤ C 2 c 2 .
We refer to Appendix B.3 for more details and references.
Input : ( L, x 0 , R K , M, m, α, T ) .
L -smooth convex function f w.r.t.
∥·∥ ∗ , x 0 ∈ K , R K = max x ∈ K ∥ x ∥ ∗ , distribution with density dπ
( z ) ∝ exp( − η ( z ))
dz such that M 2 =
E ∆ ∥∇ η (∆) ∥ 2 ∗ , m computing units in parallel, smoothing parameter α , number of iterations T .
Set A 0 = 0 , d 0 = 0 , β =
R K
Mα , µ = ν = 1 L , y
= ∇ f ( x 0 ) .
for k = 0 , 1 , . . .
, T − 1 do
[Equation]
For all i ∈ [ m ] in parallel, sample ∆ i ∼ dπ , compute g k,i = − arg max u ∈ K ⟨ u, − v k + α ∆ i ⟩ g
[Equation]
i
[Equation]
g
[Equation] [Equation]
end
As mentioned earlier, one can directly obtain convergence of the dual problem (19) by applying Theorem 1.
The next theorem states that not only are the iterates { x k } k ∈ N always feasible, but also that the previous rate of convergence also holds for the primal-dual gap between the iterates { x k } k ∈ N and { y k } k ∈ N .
For simplicity of the exposition we assume that R K Mα ≥ 1 L .
In practice this is not an issue since we will choose small values of α .
Theorem 2.
Suppose f is convex and L -smooth w.r.t.
∥·∥ ∗ and let x 0 ∈ K .
Under Algorithm 2, the primal iterates are always feasible, i.e., x k ∈ K for all k ∈ N .
Moreover, after k iterations, we have
[Equation]
The above convergence result is obtained by first plugging the smoothness and strong convexity constants of this section within the result of Proposition 3.
Then, one has to relate m k
( z k ) to the primal objective f ( x k ) .
The first term in the upper bound then follows by recursion.
The second term comes from the variance, which can be bounded by 4 R 2 K ρ ∥·∥ m when m stochastic gradients are computed in parallel.
The last term αs 1 (0) is due to the error induced by the smoothing of the dual.
The full proof can be found in Appendix B. From Theorem 2 we are able to show that Algorithmachieves acceleration under parallelization.
Theorem 3.
Suppose f is convex and L -smooth w.r.t.
∥·∥ ∗ and let x 0 ∈ K .
Under Algorithm 2 with α = min (cid:26)
[Equation]
, the number of iterations required to achieve an ϵ primal-dual gap, i.e., E [ f ( x k ) − d ( y k )] ≤ ϵ , is
[Equation]
The complexity is therefore ˜ O (cid:16)
[Table]
The above complexity result shows that (i) when m = 1 / √ ϵ , we get an accelerated rate ˜ O
( 1 / √ ϵ ) , (ii) when m = 1 , we recover the classical ˜ O ( 1 / ϵ ) complexity of Frank-Wolfe (up to logarithmic terms), (iii) there is no theoretical gain in going beyond m = 1 / √ ϵ computing units in parallel.
Moreover, while the total number of calls to the linear optimization oracle to reach an ϵ primal-dual gap is the same as in Frank-Wolfe for any value of m , in the case where m > 1 , the number of required gradients of f is strictly less than in Frank-Wolfe.
This is because in each iteration of Algorithm 2 we compute one gradient of f no matter what the value of m is.
In particular, when m = 1 / √ ϵ , we need only ˜ O
( 1 / √ ϵ ) gradients of f in total, compared to O ( 1 / ϵ ) in Frank-Wolfe.
One can note that the proposed algorithm in not as universal as the classical Frank-Wolfe method, as it requires upper bounds on several problem-specific constants.
However, acceleration of Frank-Wolfe has been extensively studied in the literature (see Section 1) and to the best of our knowledge our work is the first to provide accelerated rates (under parallelization) without further assumptions on the constraint set and/or the objective function.
We argue next that estimating upper bounds on most parameters does not pose significant challenges.
Indeed, as the user is free to pick any distribution to sample ∆ from (up to the assumptions of Proposition 2), one can choose a distribution for which the constant M is easy to compute (we give two examples in Section 5).
Moreover, the user typically knows the set over which the optimization is carried over, and from such knowledge an upper bound on the diameter can often be easily computed.
Finally, we also need to upper bound
the
Lipschitz constant L of the gradient.
This seems to be more of a limiting factor compared to classical
Frank-Wolfe, although an upper bound on L is often (but not always) necessary to implement other accelerated versions of Frank-Wolfe [18, 34] as well.
While it might be possible to circumvent this problem using some sort of line search within our method, it is not straightforward as the smoothing of the dual induces stochasticity, and obtaining theoretical rates for line search techniques on stochastic objectives is notoriously tedious [9, 44, 45].
Finally, note that it might seem like a small modification to Algorithm 2 could give a variant of FrankWolfe capable of handling stochastic gradients of f (for previous works on stochastic Frank-Wolfe, see [24, 20, 37] and references therein).
Indeed, one could see the noise in ∇ f ( x k ) in the update of v k as the random variable ∆ itself and study the resulting algorithm.
However, the current analysis would only work if the noise on ∇ f ( x k ) does not depend on the current iterate.
Moreover, even under this simplifying assumption, the result does not follow immediately as the stochasticity would appear in both the updates of v k and y k +1 , in contrast with the current analysis.
Experiments
We first consider minimizing a quadratic function in R d over the simplex
[Equation]
where A ∈ R n × d and b ∈ R n .
We choose ∆ to have the Gumbel distribution [22] with location and scale parameters equal to 0 and 1 respectively.
In this case R K 1 = 1 and M = √ d
(see Appendix C for details).
We set n = 200 , d = 50 and compare the bound from Theorem 2 with the practical performance of Algorithm 2 for both m = 1 and m = 1 / √ α parallel computer(s).
Fig.
1 shows that our upper bound captures well the speed of convergence to a neighborhood.
[Figure]
Next, in order to circumvent the inevitable stalling due to the fixed value of α observed above, we suggest a restarted algorithm which decreases the value of α during training.
Starting from α = 1 and some x 0 ∈ K , we run Algorithm 2 for T α iterations with m α computers in parallel to obtain some approximate solution x α ∈ K .
We then decrease α by a constant factor c < 1 and run Algorithmagain with the new value of α starting from x α .
We repeat this process until a satisfying solution is obtained.
We formalize this procedure in Algorithm 3.
In practice we choose c = 0 .
5 and set
T α = (cid:113) Lα log 1 α as a simplified version of the bound (25), and run the above procedure for both m α =
and m α = 1 / √ α computers in parallel.
Input : ( L, x 0 , R K , M, c ) .
L -smooth convex function f , x 0 ∈ K , R K = max x ∈ K
∥ x ∥ ∗ , distribution with density dπ ( z ) ∝ exp( − η ( z ))
dz such that M 2 =
E ∆ ∥∇ η (∆) ∥ 2 ∗ , decreasing factor c .
Set α = 1 .
for i = 0 , 1 , . . .
do
Set T α = (cid:113) Lα log 1 α and either set m α = 1 √ α or m α = 1 .
x α = Algorithm 2 ( L, x 0 , R K , M, m α , α, T α )
Set x 0 = x
α and α = cα .
end
Return x α .
We test the restarted scheme on problem (26) as well as on a generalized matrix completion problem over the trace norm ball, min
[Equation]
F } where K 2 = (cid:8)
[Equation]
|
[Equation]
(cid:9) , (27) where C ∈ R p × p and D ∈ R p × q .
Here ∥·∥ F stands for the Frobenius norm and ∥·∥ tr is the trace, or nuclear, norm.
We choose ∆ to have entries distributed according to a standard normal distribution.
In this case R K 2 = 1 and M = √ pq .
We compare the restarted scheme with the Frank-Wolfe algorithm with step-size η k =
k
+1 (denoted FW in the plots) and with exact line search (denoted FW-LS in the plots).
We set p = 10 , q = 8 and plot the results in Fig.
For both problems, we observed that using M = 1 instead of the theoretical value ( M = √ d for (26) and M = √ pq for (27)) led to significant speedups.
We consequently plot both strategies.
We observe that for m α = 1 / √ α ,
Algorithm 3 significantly outperforms Frank-Wolfe, and for m α = 1 the performance is similar to
Frank-Wolfe, which was expected from the discussion at the end of Section 4.
Full code to reproduce the experiments can be found at the following link: https://github.com/bpauld/PFW .
[Figure]
Conclusion
In this work we introduced a stochastic proximal Bregman algorithm able to converge to a neighborhood of the solution at the same accelerated rate as its deterministic counterpart.
We then used it to design a variant of the Frank-Wolfe algorithm able to achieve accelerated rates under parallelization.
One drawback is that the resulting algorithm is not "any time", in the sense that the desired precision ϵ must be given as an input to the algorithm.
We circumvent this by designing a simple heuristic which slowly decreases the value of ϵ .
We plan to investigate the theory for this restarted algorithm in the future.
We also hope to use randomized smoothing techniques similar to the ones used in the work to obtain stochastic Frank-Wolfe methods.
Acknowledgements
The authors acknowledge support from the European Research Council (grant SEQUOIA 724063)
This work was funded in part by the french government under management of Agence Nationale de la recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA- (PRAIRIE 3IA Institute).
The authors would like to warmly thank Alexander Gasnikov for very constructive remarks and for pointing out the existence and relationship of our work with [34, 20, 28, 10].
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See proposition and theorem statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is available at https://github.com/bpauld/PFW .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proofs for Section
B Proofs for Section
C Experimental details
A Proofs for Stochastic Composite Minimization A.1 Proof of Proposition
We break down the proof of Proposition 3 into different lemmas.
The first one bounds the improvement on G over an iteration, the second one bounds the improvement H over an iteration, the third one combines the first two results and further exploits the structure of the functions at play.
The proof of the proposition then follows by carefully choosing A k +1 as a function of A k .
In the remainder of the section, we work with the assumptions stated in Proposition 3 and do not restate them in the statements of the lemmas.
Lemma
[Equation]
Proof.
By smoothness
[Equation]
where the last inequality follows by strong convexity of G , Now, since y k
+1 =
A k A k +1 y k
+ A k
+1 −
A k A k +
z
k +1 , we can simplify to
[Equation]
Finally,
[Equation]
which concludes the proof
[Equation]
Lemma 2.
For any g H ( z k ) ∈ ∂H
( z k ) ,
[Equation]
Proof.
By strong convexity of H , for any g H ( y k +1 ) ∈ ∂H
( y k +1 )
we have
[Equation]
Now observe that
[Equation]
and thus we see that the two inner product terms cancel out.
Moreover we can also simplify the norms and get
[Equation]
Finally
[Equation]
observe that by strong convexity of H again, for any g H ( z k ) ∈ ∂H
( z k ) ,
[Equation]
2 , which concludes the proof.
Lemma 3.
Defining m k
( x ) = ⟨ d k , x ⟩ + c k + A k H ( x ) + βw ( x ) , we have
[Equation]
Proof.
Summing the inequalities in the above two lemmas above we have that for any g H ( z k ) ∈
∂H ( z k ) ,
[Equation]
Substracting ⟨ d k +1 , z k +1 ⟩ on both sides and adding/substracting ⟨ d k , z k +1 ⟩ and ⟨ d k , z k ⟩ on the right-hand side, we get
[Equation]
Now, by first-order optimality conditions of (13), 0 ∈ d k + A k ∂H ( z k ) + β∂w ( z k ) .
Therefore there exist subgradients g ′ H ( z k ) ∈ ∂H
( z k ) and g ′ w ( z k ) ∈
∂w ( z k ) such that d k + A k g ′ H ( z k ) =
− βg ′ w ( z k ) .
Since the above inequality is true for any g H ( z k ) ∈ ∂H
( z k ) , it is in particular true for g ′ H ( z k ) , and thus we have
[Equation]
Since d k − d k +
=
− ( A k +1 − A k )
g k we get
[Equation]
Finally
[Equation]
by strong convexity of w we have
[Equation]
and thus the previous inequality becomes
[Equation]
Those three lemmas allow us to prove Proposition 3.
Proof of Proposition 3 .
We can rewrite the previous result as
[Equation]
Taking expectation at iteration k conditioned on the previous iterations
[Equation]
we have E k
[( A k +1 −
A k ) ⟨∇ G ( v k ) − g k , z k ⟩ = 0] .
Moreover
[Equation]
by Fenchel-Young inequality we have that for any ρ > 0 ,
[Equation]
Taking expectation and using Assumption
[Equation]
Thus we have
[Equation]
A k +1 )
− β
A 2 k
A k +1 − ρ
( A k +1 − A k ) (cid:19)
E k ∥
z k +1 − z k ∥− µ 2 A k (cid:18) 1 −
A k
A k + (cid:19)
E k ∥
z k +1 − y k
∥ 2 .
Now, observe that since 0 ≤ A k /A
k +1 ≤ 1 , the term in E k ∥
z k +1 − y k
∥ 2 is non-positive.
Therefore, to obtain the final result, it suffices to set A k +
so that the term in E k ∥
z k +1 − z k ∥ 2 cancels out.
In other words, we require
A
[Equation]
⇐⇒
A
[Equation]
+1 ( β + ρ )
[Equation]
A k ( µ
[Equation]
+ ρ ) − βν
[Equation]
β A
[Equation]
A
[Equation]
⇐⇒
A 2 k +1 ( β + ρ ) − A k
+
( A k ( µ + 2 β
+ ρ ) + βν ) + βA
k
=
⇐ A k
+1 =
A k ( µ + 2 β + ρ ) + βν + (cid:113) ( A k ( µ + 2 β + ρ ) + βν ) 2 − 4( β + ρ ) βA 2 k 2( β + ρ ) ⇐⇒
A k
+1 =
A k ( µ + 2 β + ρ ) + βν + (cid:112) β 2 ν 2 + 2 βνA k ( µ + 2 β + ρ ) + A 2 k µ 2 + A 2 k ρ
+
A 2 k µρ + 4 A 2 k
µβ 2( β + ρ ) ⇐⇒
A k
+1 =
A k ( µ + 2 β + ρ ) + βν + (cid:112) ( βν + µA k )
+ 4 A k ( β 2 ν + A k µβ ) + 2 βνA k ρ + A 2 k ρ
+ 2 A 2 k µρ 2( β + ρ ) .
Setting ρ = √ µβ yields the update for A k +1 in Algorithm 1 and proves the proposition.
A.2 Proof of Theorem
Proof.
Unrolling the recursion in Proposition 3 and taking total expectation, we have
[Equation]
Now,
[Equation]
Taking total expectation on the g t we get
[Equation]
where the inequality is by convexity of G .
Moreover, m k
( z k ) ≤ m k
( y ⋆ ) by construction and thus
[Equation]
and thus
[Equation]
as y
= z 0 and w ( y ⋆ ) − w ( y 0 ) is equal to D w ( y ⋆ , y 0 ) :
= w ( y ⋆ )
− w ( y 0 ) − ⟨ g w ( y 0 ); y ⋆ − y 0 ⟩ (with g w ( y 0 )
∈ ∂w ( y 0 ) ) through the choice g w ( y 0 ) =
∈ ∂w ( y 0 ) , which is valid as y 0 minimizes w ( · ) .
Finally, we can bound A k +1 as
[Equation]
+
[Equation]
(cid:18)
[Equation] [Equation]
(cid:19)
=
A
[Equation]
(cid:18)
[Equation]
) (cid:19) and thus
E [
[Equation]
B Proofs for Accelerated Frank-Wolfe B.1 Proof of TheoremB.1.1 Proof of feasibility
First we show that x k ∈ K for all k ∈ N .
Proof.
We prove this by induction.
By assumption x 0 ∈ K .
Now suppose x k ∈ K for some k ∈ N .
We then have
[Equation]
By induction hypothesis, x k ∈ K .
We also have − g k ∈ K since
[Equation]
In other words, − g k is a convex combination of elements of K and is thus in K .
Therefore x k +1 ∈ K as a convex combination of elements of K .
B.1.2 Proof of dual gap convergence
Proof.
With H ( y ) =
f ∗ ( y ) , G ( y ) =
s α ( − y ) , β =
R K
Mα and µ = 1 L , we can apply Proposition 3 to
F = H + G and get
E
[Equation] [Equation]
k
[Equation]
m k
[Equation] [Equation]
) σ
[Equation]
µβ where m k ( y ) = ⟨ d k , y ⟩ + c k + A k H
( y ) + βw ( y ) , and z k = ∇ f ( x k ) .
Unrolling the recursion as before and taking total expectation we have
[Equation]
Recall that we set w ( y ) =
f ∗ ( y ) − ⟨ x 0 , y ⟩ .
Plugging in the function H = f ∗ , and recalling that
z k minimizes m k ( y ) , we get
[Equation]
Thus we can conclude that
[Equation]
and in particular
[Equation]
Now,
[Equation]
For any v ∈ V , G ( v ) = s α ( − v ) and thus
[Equation]
From Fenchel-Young, ⟨∇ s α ( − v ) , −
[Equation]
s
[Equation]
( − v )
+ s
[Equation]
α ( ∇ s α ( − v )) , and thus
[Equation]
Now, for all u ∈ V ∗ ,
[Equation]
where the inequality is from Proposition 2.
In particular, since ∇ s α ( − v ) is always feasible, we have s ∗ α ( ∇ s α ( − v )) ≥ − αs 1 (0) .
and thus
[Equation]
We can then rewrite (28) as
[Equation]
Now,
[Equation]
by Fenchel-Young and since z 0 = ∇ f ( x 0 ) .
Thus
[Equation]
Finally, from Proposition
[Equation]
for any y .
We can then conclude
[Equation]
Bounding A k as in Theorem 1 yields
[Equation]
It remains to bound the variance σ 2 .
Using Assumption 2 we have
[Equation]
Plugging this back into equation (29)
[Equation]
and plugging in the values of β =
R K
Mα and µ = 1 L gives
[Equation]
Assuming R K
Mα ≥ 1 L yields the result.
B
Proof.
Setting
[Equation]
it is easy to verify that
[Equation]
and that
[Equation]
It remains to compute k such that the first term in the bound of Theorem 2 is also smaller than ϵ/ 3 .
This gives
[Equation]
The ˜ O -complexity follows directly from plugging the value of α in the bound.
B.3 Dependence on the norms
In this section, we compute the value of ρ ∥·∥ ∗ for different ℓ p norms when the underlying vector space has dimension d .
B.3.1 Euclidean norm
In the case of the Euclidean norm, we have ∥·∥ = ∥·∥ ∗ = ∥·∥ 2 and thus
E
[Equation] [Equation]
where the second equality simply comes from the properties of the Euclidean norm, and the third one comes from the fact that g k,i + ∇ s α ( − v k ) and g k,j + ∇ s α ( − v k ) are zero-mean independent random variables for all i ̸
=
j .
Finally, arg max u ∈ K ⟨ u, v ⟩ ∈ K for any v , and similarly ∇ s α ( v ) =
E [arg max u ∈ K ⟨ u, v + α ∆ ⟩ ] ∈
K for all v .
Therefore we have
[Equation]
and we see that in this case ρ ∥·∥ 2 = 1 .
B.3.2 ℓ p -norms for 2 ≤ p < ∞
When ∥·∥ ∗ = ∥·∥ p , we have ∥·∥ = ∥·∥ q for 1 p + 1 q = 1 .
Since q ∈ (1 , 2] , from [6] we know that 12 ∥·∥ 2 q is ( q − 1) -strongly convex with respect to ∥·∥ q .
Therefore 12 ∥·∥ 2 p is 1 q − 1 -smooth with respect to ∥·∥ p [29].
We have
[Equation]
so 12 ∥·∥ 2 p is ( p − 1) -smooth with respect to ∥·∥ p .
We now closely follow the proof from [30, Lemma 2].
Let F = 12 ∥·∥ 2 p and let Z i =
g k,i − ∇ G ( v k ) so that E ∥ Z i ∥ 2 p ≤ 4 R 2 K .
Let S i
= (cid:80) i − 1 j =1 S j .
By smoothness of F we have
[Equation]
Taking conditional expectation with respect to Z 1 , .
, Z i − 1 , since E [ Z i ] =
we have
[Equation]
Thus, F ( S i ) − i ( p − 1)is a supermartingale and therefore
[Equation]
which shows that ρ ∥∥ p = p − 1 .
B.3.3 ℓ p -norms for 1 ≤ p <
Recall that for ∞ ≥ q > r ≥ 1 ,
[Equation]
If the norm of interest is ∥·∥ ∗ = ∥·∥ p for 1 ≤ p < 2 , we have
[Equation]
where the second inequality comes from the derivation for the Euclidean norm in Appendix B.3.1.
Using (30) we get
E
[Equation]
m and thus ρ ∥·∥ p =
d (2 /p − 1) .
B.3.4 ℓ ∞ -norm
When ∥·∥ ∗ = ∥·∥ ∞ , we use inequality (30) to get that for any 1 ≤ r < ∞ ,
E
[Equation]
m
[Equation]
) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) r .
Now, if r ≥ 2 , from Appendix B.3.2 we have that E
[Equation]
m m (cid:88) i
= g k,i − ∇ G ( v k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) r ≤ r −m max u,v ∈ K ∥
u − v ∥ 2 r ≤ ( r − 1) m d 2 /r max u,v ∈ K ∥
u − v ∥ 2 ∞ =
( r − 1)4 R 2 K m d 2 /r .
Taking r = 2 + log d (so that r ≥ 2 ), we then have
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)m m (cid:88) i =
g k,i − ∇ G ( v k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∗ ≤ ( r − 1)4 R
K m e 2 r log( d ) = 4 R 2 K m (log d + 1) e 2log d 2+log d ≤ 4 R 2 K m 2(log d
+
and thus we have ρ ∥·∥ ∞ = 2(log d + 1) .
B.3.5 General norm
For a general norm, since we are in a finite-dimensional space, there exist constants c, C > 0 such that c ∥·∥ 2 ≤ ∥·∥ ∗ ≤ C ∥·∥ 2 .
We then have
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)m m (cid:88) i =
g k,i − ∇ G ( v k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∗ ≤
C 2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 m m (cid:88) i =
g k,i − ∇ G ( v k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where the second inequality comes from the derivation in Appendix B.3.1.
Thus we ρ ∥·∥ ∗ ≤ C 2 c 2 .
C Experimental Details
In the experiments we only consider Euclidean norms so that ∥·∥ = ∥·∥ ∗ .
If the entries of z are independently distributed according to a Gumbel distribution with locationand scale 1, the probability density function reads
[Equation]
so that η ( z ) = (cid:80) di
=
z i + e − z i .
Thus we have
[Equation]
and
[Equation]
Algorithm 2 and Algorithm 3 require a bound on the value of M .
We compute it now.
[Equation]
For any i ≥ 2 ,
[Equation]
Moreover, one can check that
[Equation]
where C is some constant.
Taking limits one gets
[Equation]
and thus
[Equation]
The derivation in the case of a multivariate normal distribution is similar.