Distributionally Robust Optimization via Ball Oracle Acceleration
Abstract
We develop and analyze algorithms for distributionally robust optimization (DRO) of convex losses.
In particular, we consider group-structured and bounded fdivergence uncertainty sets.
Our approach relies on an accelerated method that queries a ball optimization oracle, i.e., a subroutine that minimizes the objective within a small ball around the query point.
Our main contribution is efﬁcient implementations of this oracle for DRO objectives.
For DRO with N non-smooth loss functions, the resulting algorithms ﬁnd an (cid:15) -accurate solution with (cid:101) O (cid:0) N(cid:15) − 2 / 3 + (cid:15) − 2 (cid:1) ﬁrst-order oracle queries to individual loss functions.
Compared to existing algorithms for this problem, we improve complexity by a factor of up to (cid:15) − 4 / 3 .
Introduction
The increasing use of machine learning models in high-stakes applications highlights the importance of reliable performance across changing domains and populations [11, 46, 35].
An emerging body of research addresses this challenge by replacing Empirical Risk Minimization (ERM) with Distributionally Robust Optimization (DRO) [6, 50, 49, 22, 40], with applications in natural language processing [47, 61, 35], reinforcement learning [17, 54] and algorithmic fairness [27, 56].
While
ERM minimizes the average training loss, DRO minimizes the worst-case expected loss over all probability distributions in an uncertainty set U , that is, it minimizes
[Equation]
[Equation]
where (cid:96) S ( x ) is the loss a model x ∈ X incurs on a sample S and X is a closed convex set with bounded Euclidean diameter.
This work develops new algorithms for DRO, focusing on formulations where U contains distributions supported on N training points, where N is potentially large.
We consider two well-studied DRO variants: (1) Group DRO [57, 31, 49], and (2) f -divergence DRO [19, 6, 22].
Group DRO Machine learning models may rely on spurious correlations (that hold for most training examples but are wrongly linked to the target) and therefore suffer high loss on minority groups where these correlations do not hold [30, 27, 11].
To obtain high performance across all groups, Group
DRO minimizes the worst-case loss over groups.
Given a set U
= { w 1 , . . .
, w M } of M distributions over [ N ] , the Group DRO objective is
[Equation]
[Table] [Table]
If we deﬁne the loss of group i as L i ( x ) := (cid:80) Nj =1 w ij (cid:96) j ( x ) then objective (2) is equivalent to max q ∈ ∆ M (cid:80) i ∈
[ M ] q i L i ( x ) with ∆ M := { q ∈ R M ≥ [-
-]0 | 1 T q = 1 } .
Note that, unlike ERM, Group
DRO requires additional supervision in the form of subgroup identities encoded by { w i } .
DRO with f -divergence Another approach to DRO, which requires only as much supervision as
ERM, takes U to be an f -divergence ball around the empirical (training) distribution.
For every convex function f : R + → R ∪ { + ∞} such that f (1) = 0 , f (0 / 0)
=
and the f -divergence between distributions q and p over [ N ] is D f ( q, p ) :
= (cid:80) i ∈
[ N ] p i f
( q i /p i ) .
The f -divergence DRO problem corresponds to the uncertainty set U = { q ∈ ∆ N : D f ( q, 1 N 1 )) ≤ 1 } , i.e.,
[Equation]
Several well-studied instances of DRO are a special case of this formulation, with the two most notable examples being conditional value at risk (CVaR) and χ 2 uncertainty sets.
CVaR at level α corresponds to f ( x )
= 1 { x< 1 α } such that U = (cid:8) q ∈ ∆ N s.t (cid:107) q (cid:107) ∞ ≤ 1 / ( αN ) (cid:9) , and has many applications in ﬁnance such as portfolio optimization and credit risk evaluation [48, 37] as well as in machine learning [47, 39, 20, 60, 17, 54].
The χ 2 uncertainty set with size ρ > 0 corresponds to f ( x ) :
= 12 ρ ( x − 1) 2 and the resulting DRO problem is closely linked to variance regularization [21] and has been extensively studied in statistics and machine learning [42, 27, 21, 39, 61].
Complexity notion In this paper, we design improved-complexity methods for solving the convex problems (2) and (3) under the assumption that the loss (cid:96) i is convex and Lipschitz for all i .
We measure complexity by the (expected) required number of (cid:96) i ( x ) and ∇ (cid:96) i ( x ) evaluations to obtain (cid:15) -suboptimal solution, i.e., return x such that L DRO ( x ) − min
x
(cid:63) ∈X L DRO ( x (cid:63) ) ≤ (cid:15) with constant probability.
Table 1 summarizes our complexity bounds and compares them to prior art.
Throughout the introduction we assume (for simplicity) a unit domain size ( (cid:107) x − y (cid:107) ≤ 1 for all x, y ∈ X ) and that each loss is 1 -Lipschitz.
Prior art Let us review existsing methods that solve Group DRO and f -divergence DRO problems (see Section 1.1 for extended discussion).
For a dataset with N training points, the subgradient method [45] ﬁnds an (cid:15) approximate solution in (cid:15) − 2 iterations.
Computing a single subgradient costs
N functions evaluations (since we need to ﬁnd the maximizing q ).
Therefore, the complexity of this method is O (cid:0) N(cid:15) − 2 (cid:1) .
DRO can also be viewed as a game between a minimizing x -player and a maximizing q -player, which makes it amenable to primal-dual methods [43, 42, 49].
If we further assume that the losses are bounded then, for q ∈ ∆ m , stochastic mirror descent with local norms obtains a regret bound of
O (cid:0)(cid:112)
m log( m ) /T
(cid:1) (see Appendix A.1).
As a consequence, for Group DRO (where m = M ) the complexity is (cid:101) O (cid:0) M(cid:15) − 2 (cid:1) , and for f -divergence DRO ( m = N ) the complexity is (cid:101) O (cid:0) N(cid:15) − 2 (cid:1) .
Levy et al.
[39] studied χ 2 -divergence and CVaR DRO problems, and proposed using standard gradient methods with a gradient estimator based on multilevel Monte Carlo (MLMC)
For χ 2 -divergence with ball of size ρ they proved a complexity bound of (cid:101) O (cid:0) ρ(cid:15) − 3 (cid:1) , and for CVaR at level α they established complexity (cid:101) O (cid:0)
α − 1 (cid:15) − 2 (cid:1) .
However, for large uncertainty sets (when ρ or α −approach N ) their method does not improve over the subgradient method.
Stronger complexity bounds are available under the weak smoothness assumption that each (cid:96) i has O (cid:0) (cid:15) − 1 (cid:1) -Lipschitz gradient.
Note that this is a weak assumption since if a function (cid:96) is not continuously differentiable, it is possible to approximate (cid:96) with additive error at most (cid:15)/ 2 , by its Moreau envelope (cid:101) (cid:96) ( x ) =
min y ∈X (cid:110) (cid:96) ( y )
+ G 2 2 (cid:15) (cid:107) x − y (cid:107) 2 (cid:111)
(see [15, Appendix A.1] for more details).
In particular, we can apply Nesterov’s accelerated gradient descent method [44] on an entropy-regularized version of our objective to solve the problem with complexity (cid:101) O (cid:0) N(cid:15) − 1 (cid:1) ; see
Appendix A.2 for more details.
Our contribution We propose algorithms that solve the problems (2) and (3) with complexity (cid:101) O (cid:0) N(cid:15)
− 2 / 3 + (cid:15) − 2 (cid:1) .
Compared to previous works, we obtain better convergence rates for DRO with general f -divergence when N (cid:29) 1 and for Group DRO when M (cid:29) N(cid:15) 4 / 3 .
When the losses have
O ( (cid:15) − 1 ) -Lipschitz gradient, we solve f -divergence DRO with complexity (cid:101) O (cid:0) N(cid:15) − 2 / 3 + √ N(cid:15) − 1 (cid:1) , and, under an even weaker mean-square smoothness assumption ( E j ∼ w i (cid:107)∇ (cid:96) j ( x ) − ∇ (cid:96)
j ( y ) (cid:107) 2 ≤
O ( (cid:15) − 2 ) (cid:107) x − y (cid:107) 2 for all x, y and i ), we solve Group DRO with complexity (cid:101) O (cid:0) N(cid:15) − 2 / 3 + N 3 / 4 (cid:15) − 1 (cid:1) .
Our complexity bounds are independent of the structure of f and { w i } , allowing us to consider arbitrarily f -divergence balls and support a large number of (potentially overlapping) groups.
Our rates are optimal up to logarithmic factors for the special case of minimizing max i ∈ [ N ] (cid:96) i ( x ) , which corresponds to Group DRO with N distinct groups and f -divergence DRO with f =
Our approach Our algorithms are based on a technique for acceleration with a ball optimization oracle , introduced by Carmon et al.
[13] and further developed in [15, 3].
Given a function F and a query point x , the ball optimization oracle returns an approximate minimizer of F inside a ball around x with radius r ; the works [13, 15, 3] show how to minimize F using (cid:101) O
( r − 2 / 3 ) oracle calls.
Our development consists of efﬁciently implementing ball oracles with radius r = (cid:101) O
( (cid:15) ) for the DRO problems (2) and (3), leveraging the small ball constraint to apply stochastic gradient estimators that would have exponential variance and/or cost without it.
Carmon et al.
[15] previously executed this strategy for minimizing the maximum loss, i.e., max q ∈ ∆ N (cid:80)
i q i (cid:96) i ( x ) , which is a special case of both Group DRO and f -divergence DRO.
However, the ball-oracle implementations of [15] do not directly apply to the DRO problems that we consider; our oracle implementations differ signiﬁcantly and intimately rely on the Group DRO and f -divergence problem structure.
We now brieﬂy review the main differences between our approach and [15], highlighting our key technical innovations along the way.
Since the Group DRO objective is max q ∈ ∆ M (cid:80) i q i L i ( x ) for L i ( x )
= (cid:80) j ∈
[ N ] w ij (cid:96) j
( x ) , one may naively apply the technique of [15] with L i replacing (cid:96) i .
However, every step of such a method would involve computing quantities of the form e L i ( x ) /(cid:15) (cid:48) (for some (cid:15) (cid:48)
= (cid:101) Θ( (cid:15) ) ), which can be up to N times more expensive than computing e (cid:96) j
( x ) /(cid:15) (cid:48) for a single j .
To avoid such expensive computation we use MLMC [9] to obtain an unbiased estimate of e L i ( x ) /(cid:15) with complexity O (1) and appropriately bounded variance.
In the weakly-smooth case we also adapt our estimator to facilitate variance reduction [33, 2].
For f -divergence we consider the well-known dual form [6, 50]
[Equation]
where ψ ∗ ( v ) = max t ≥ 0 { vt − ψ ( t ) } is the Fenchel dual of ψ .
Stochastic gradient methods applied directly on the dual formulation are notoriously unstable (see e.g., [42]).
This is due to the fact that the Fenchel dual ψ ∗ can be very badly behaved even for standard f -divergences.
We solve this problem by (a) considering a small ball, (b) entropy-regularizing ψ .
Our techniques rely on two technical observations: (i) log( ψ ∗ (cid:15) (cid:48) ( · )) is 1 /(cid:15)
(cid:48)
-Lipschitz for all convex ψ , and (ii) for 1-Lipschitz losses, y (cid:63) ( x ) =
argmin y ∈ R Υ
( x, y ) satisﬁes | y (cid:63) ( x ) − y (cid:63) ( x (cid:48) )
| ≤ (cid:107) x − x (cid:48) (cid:107) for all x, x (cid:48) .
To the best of our knowledge, these observations are new and potentially of independent interest.
Paper organization.
Section 2 provides notation and a concise summary of the ball acceleration framework (largely taken from prior work) on which we build our algorithms.
Sections 3 and present our main contributions in the Group and f -divergence DRO settings, respectively.
Finally,
Section 5 concludes with discussion on the limitations and possible extensions of our work.
1.1 Additional related work
MLMC estimators The multilevel Monte Carlo (MLMC) technique was introduced by Giles [26] and Heinrich [29] in order to reduce the computational cost of Monte Carlo estimation of integrals.
Blanchet and Glynn [9] extended this techniques to estimating functions of expectation and proposed several applications, including stochastic optimization [10].
In this work we use their estimator for two distinct purposes: (1) obtaining unbiased Moreau envelope gradient estimates for ball oracle acceleration as proposed by Asi et al.
[3], and (2) estimating the the exponential of an expectation for
Group DRO.
Levy et al.
[39] also rely on MLMC for DRO, but quite differently than we do: they directly estimate the DRO objective gradient via MLMC, while we estimate different quantities.
Other DRO methods Several additional works proposed algorithms with theoretical guarantees for f -divergence DRO.
Jin et al.
[32] considered non-convex and smooth losses.
Song et al.
[53] proposed an algorithm for linear models with complexity comparable to the “AGD on softmax” approach (Appendix A.2).
Namkoong and Duchi [42] proposed a primal-dual algorithm that is suitable for small uncertainty χ 2 sets (with size ρ (cid:28) 1 N ) and Curi et al.
[20] proposed a primal-dual algorithm specialized for CVaR.
Other works consider DRO with uncertainty sets deﬁned using the
Wasserstein distance [24, 23, 51, 34].
Another relevant line of works proposes reﬁnements for DRO that address some of the challenges in applying it to learning problems [60, 59, 55].
Preliminaries
Notation We write (cid:107)·(cid:107) for the Euclidean norm.
We denote by B r ( x 0 ) the Euclidean ball of radius r around x 0 .
We let ∆ n := { q ∈ R n ≥ 0 | 1 T q = 1 } denote the probability simplex in R n .
For the sequence z m , . .
, z n we use the shorthand z nm .
Using F as a generic placeholder (typically for a loss function (cid:96) i ), we make frequent use of the following assumption.
Assumption 1.
The function F : X → R is convex and and G-Lipschitz, i.e., for all x,
y ∈ X we have | F ( x ) − F ( y ) | ≤ G (cid:107) x
− y (cid:107) .
In addition, the domain X is a closed and convex set, and it has
Euclidean diameter at most R .
Throughout, N denotes the number of losses and, in Section 3, M denotes the number of groups.
We use (cid:15) for our target accuracy and r (cid:15) :
= (cid:15) (cid:48) /G
for the ball radius, where (cid:15) (cid:48)
= (cid:15)/ (2 log M ) for
Group-DRO (Section 3) and (cid:15) (cid:48)
= (cid:15)/ (2 log N ) for f -divergence DRO (Section 4).
Complexity model We measure an algorithm’s complexity by its expected number of (cid:96) i and ∇ (cid:96) i evaluations; bounds on expected evaluation number can be readily converted to more standard probability 1 bounds [see 3, Appendix A.3].
Moreover if X ⊂ R d , d = Ω(log N ) , 2 and the time to evaluate (cid:96) i and ∇ (cid:96) i is O ( d ) , the expected runtime of all the algorithms we consider is at most d times the evaluation complexity.
2.1 Ball oracle acceleration
We now brieﬂy summarize the complexity bounds given by the framework of [13, 15, 3] for accelerated minimization using queries to (inexact) ball optimization oracles, deﬁned as follows.
Deﬁnition 1.
An algorithm is a Ball Regularized Optimization Oracle of radius r ( r -BROO) for function F : X → R if for query point ¯ x ∈ X , regularization parameter λ > 0 and desired accuracy δ > 0 it returns O λ,δ (
¯ x ) ∈
X satisfying
[Equation]
Proposition 1.
Let F satisfy Assumption 1, let C F be the complexity of evaluating F exactly, and let C λ ( δ ) bound the complexity of an r -BROO query with δ, λ .
Assume that C λ ( δ ) is nonincreasing in λ and at most polynomial in 1 /δ .
For any (cid:15) > 0 , Algorithm 1 returns x such that
F ( x ) − min x (cid:63) ∈X F ( x (cid:63) ) ≤ (cid:15) with probability at least 12 .
For m (cid:15) =
O (cid:0) log GR 2 (cid:15)r (cid:1) and λ m =
O (cid:0) m 2 (cid:15) (cid:15) r 4 / 3 R 2 / 3 (cid:1) , the complexity of the algorithm is
[Equation]
Informally, the proposition shows that (cid:101) O
( R/r ) 2 / 3 ) BROO calls with λ = (cid:101) Ω( (cid:15)/ ( r 3 / 4 R 2 / 3 )) and accuracy δ
= (cid:101) O
( r ) sufﬁce to ﬁnd an (cid:15) -accurate solution.
As we show in the sequel, for C λ ( δ ) = (cid:101) O (cid:0) N + ( Gλδ ) 2 (cid:1) the resulting complexity bound is (cid:101) O (cid:0) N ( GR(cid:15) ) 2 / 3 + ( GR(cid:15) ) 2 (cid:1) .
The summation over j in bound (5) stems from the use of MLMC to de-bias the BROO output (i.e., make it exact in expectation): compared to the original proposal of Asi et al.
[3], our version of the procedure in
Appendix B slightly alters this MLMC scheme by de-biasing one accurate BROO call instead of averaging many inaccurate de-biased calls, improving our bounds by logarithmic factors.
Group DRO
In this section we develop our BROO implementations for the Group DRO objective (2).
In Section 3.
we describe an “exponentiated group-softmax” function that approximates L g-DRO with additive error at most (cid:15)/ 2 .
We then apply stochastic gradient methods on this function to obtain BROO implementations that yield improved rates for Group DRO via Proposition 1: we ﬁrst consider the non-smooth case in Section 3.2 and then the weakly-smooth case in Section 3.3.
3.1 Exponentiated group-softmax
Given a cheap and unbiased stochastic gradient estimator of ∇L g-DRO , we could use a variant of
SGD and minimize L g-DRO to (cid:15) -suboptimal solution using O ( (cid:15) − 2 ) steps.
However, obtaining an unbiased estimator is challenging due to the maximum operator in L g-DRO .
As a ﬁrst step we use entropy smoothing [7, 8, 5, 4] to replace the maximum in L g-DRO with the softmax operation.
More speciﬁcally, we use the trick from [15] and minimize the “exponentiated softmax” (that has the form of a weighted ﬁnite sum) within a small ball.
For target accuracy (cid:15) , regularization parameter λ ≥ 0 , center point ¯ x ∈ X and (cid:15) (cid:48)
= (cid:15)/ (2 log M ) > 0 , the (regularized) group-softmax function is
L smax ,(cid:15),
[Equation] [Equation]
=
[Equation]

[Equation]
2 where L i ( x ) = (cid:88)
j
[Equation]
( x
[Equation]
We will implement a BROO for L smax ,(cid:15) :
= L smax ,(cid:15), 0 , which is a uniform approximation of L g-DRO : |L g-DRO ( x ) − L smax ,(cid:15) ( x ) | ≤ (cid:15)/ 2 for all x ∈ X ; see Appendix C.1 for details.
The (regularized) exponentiated group-softmax is
[Equation]
In the following lemma we (easily) extend Carmon et al.
[15, Lemma 1] to exponentiated groupsoftmax, showing that Γ (cid:15),λ is well-behaved inside a ball of (appropriately small) radius r around ¯ x and facilitates minimizing L smax ,(cid:15),λ in that ball; see Appendix C.1 for the proof.
Lemma 1.
Let each (cid:96) i satisfy Assumption 1, and consider the restriction of L smax ,(cid:15),λ (6) and Γ (cid:15),λ (7) to B r (¯ x ) .
Then the functions have the same minimizer x (cid:63) ∈ B r (¯ x ) and, if λ ≤ O ( G/r ) and r ≤ O ( (cid:15) (cid:48) /G ) , then (a) Γ (cid:15),λ is Ω( λ ) -strongly convex, (b) each γ i is O ( G ) -Lipschitz and (c) for every x ∈ B r (¯ x ) we have L smax ,(cid:15),λ ( x ) − L smax ,(cid:15),λ ( x (cid:63) )
≤
O (Γ (cid:15),λ ( x ) − Γ (cid:15),λ ( x (cid:63) )) .
3.2 BROO implementation for Group DRO non-smooth losses
To motivate our BROO implementation, let us review how [15] use the exponentiated softmax in the special case of size-1 groups, i.e., L i = (cid:96) i , and explain the difﬁculty that their approach faces when the group structure is introduced.
The BROO implementation in [15] is based on SGD variant with the stochastic gradient estimator
ˆ g ( x )
=
e ( L i ( x ) −L i (¯ x )) /(cid:15) (cid:48)
∇L i ( x ) where i ∼ ¯ p i .
However, for
Group DRO where L i =
(cid:80) j ∈
[ N ] w ij (cid:96) j , the estimator
ˆ g ( x ) can be up to N times more expensive to compute.
Approximating ˆ g ( x ) by drawing j, j (cid:48) ∼ w i and taking e ( (cid:96) j
( x ) − (cid:96) j
(¯ x )) /(cid:15) (cid:48) ∇ (cid:96) j (cid:48)
( x ) will result in a biased estimator since E j ∼ w i e ( (cid:96)
j ( x ) − (cid:96) j
(¯ x )) /(cid:15) (cid:48) (cid:54)
=
e ( L i ( x ) −L i (¯ x )) /(cid:15) (cid:48) .
To address this challenge we propose a new gradient estimator based on the multilevel Monte Carlo (MLMC) method [9].
The MLMC unbiased estimator for γ i ( x ) =
(cid:15) (cid:48)
e ( L i ( x ) −L i (¯ x )) /(cid:15) (cid:48) , which we denote by (cid:99) M [ γ i ( x )] , is deﬁned as follows:
Draw J ∼ Geom (cid:16) 1 −
[Equation]
(cid:17) , S 1 , . .
. , S n iid ∼ w i and let (cid:99) M [ γ i ( x
[Equation]
x ; S
[Equation]
+
[Equation]
D
[Equation]
p J , where p j :=
P ( J = j ) =
(cid:0) 1 / √ 8 (cid:1) j (cid:16) 1 − 1 √ 8 (cid:17) and, for n ∈ 2 N , we deﬁne
[Equation]
and
[Equation]
With the MLMC estimator for γ i in hand, we estimate the gradient of Γ (cid:15),λ as follows:
Draw i ∼ p (¯ x ) , j ∼ w i and set ˆ g ( x ) = 1 (cid:15) (cid:48) (cid:99)
[Equation]
In the following lemma we summarize the important properties of the MLMC and gradient estimators; see Appendix C.2 for the proof.
Lemma 2.
Let each (cid:96) i satisfy Assumption 1, and let r ≤ (cid:15) (cid:48) G , λ ≤ Gr and x ∈ B r (¯ x ) .
Then (cid:99) M [ γ i ( x )] and ˆ g ( x ) are unbiased for γ i ( x ) and ∇ Γ (cid:15),λ ( x ) , respectively, and have bounded second moments:
E (cid:2) (cid:99) M [ γ i ( x )] (cid:3) 2 ≤ O (cid:16) G 4 (cid:107) x − ¯ x (cid:107) 4 (cid:15) (cid:48)+ (cid:15) (cid:48) 2 (cid:17) and E (cid:107) ˆ g ( x ) (cid:107) 2 ≤ O (cid:0) G 2 (cid:1) .
In addition, the complexity of computing (cid:99) M [ γ i ( x )] and ˆ g ( x ) is O (1) .
Due to Lemma 2 and since Γ (cid:15),λ is Ω( λ ) -strongly convex, we can use the Epoch-SGD algorithm of Hazan and Kale [28] with our gradient estimator (8).
This algorithm has rate of convergence
O (cid:0) G 2 / ( λT ) (cid:1) and our gradient estimator requires additional N function evaluations for precomputing the sampling probabilities { ¯ p i } .
We thus arrive at the following complexity bound.
Theorem 1.
Let each (cid:96) j satisfy Assumption 1, let (cid:15), δ, λ > 0 and let r (cid:15) =
(cid:15)/ (2 G log M ) .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ , EpochSGD [28,
Algorithm 1]) with the gradient estimator (8) outputs a valid r (cid:15) -BROO
response and has complexity
C λ ( δ ) =
O (cid:0) N + G 2 λ 2 δ 2 (cid:1) .
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L g-DRO (2) with probability at least 12 is
[Equation]
where H :=
M GR (cid:15)
[Equation]
We provide the proof for Theorem 1 in Appendix C.3; the ﬁnal complexity bound follows from straightforward calculations which we now brieﬂy outline.
According to Proposition 1, ﬁnding an (cid:15) 2 -suboptimal solution for L smax ,(cid:15) (and consequently an (cid:15) -suboptimal solution for L g-DRO ) involves (cid:101) O (cid:0) ( R/r (cid:15) ) 2 / 3 (cid:1) BROO calls with accuracy δ = (cid:101) Ω (cid:0) r (cid:15)
2 − J/ 2 (cid:1) and regularization strength λ ≥ λ m , where J = min { Geom ( 12 ) , m } .
We may therefore bound the complexity of each such call by m (cid:88)
[Equation]
where ( (cid:63) ) follows from substituting λ m = (cid:101) Ω (cid:0) (cid:15)r − 4 / 3 (cid:15)
R − 2 / 3 (cid:1) and m = (cid:101) O (1) .
Multiplying this bound by (cid:101) O (cid:0) ( R/r (cid:15) ) 2 / 3 (cid:1) yields (up to polylogarithmic factors)
the conclusion of Theorem 1.
3.3 Accelerated variance reduction for mean-square smooth losses
In this section we provide an algorithm with an improved rate of convergence under the following mean-square smoothness assumption.
Assumption 2.
For all
[Equation]
[Equation] [Equation]
and i
[Equation]
Note that assuming L -Lipschitz gradient for each (cid:96) i implies Assumption 2, but not the other way around.
To take advantage of Assumption
[Equation]
we ﬁrst rewrite the function Γ (cid:15),λ ( x ) in a way that is more amenable to variance reduction:
[Equation]
, where γ
[Equation]
x
[Equation]
x
[Equation]
and p
[Equation]
x
[Equation]
(Note that γ i ( x, ¯ x ) = γ i ( x ) ).
Given a reference point x (cid:48) , to compute a reduced-variance estimator of ∇ Γ (cid:15),λ ( x ) , we draw i ∼ p
i ( x (cid:48) ) and j ∼ w i , and set:
[Equation]
where ∇ (cid:96) λj ( x ) := ∇ (cid:96) j ( x ) + λ ( x − ¯ x ) and (cid:99) M [ γ i ( x, x (cid:48) )] is an MLMC estimator for γ i ( x, x (cid:48) ) deﬁned analogously to (cid:99) M [ γ i ( x )]
(see details in Appendix C.4).
The estimator (9) is not precisely standard
SVRG [33] since we use (cid:99) M [ γ i ( x, x (cid:48) )] as an estimator for γ i ( x, x (cid:48) ) .
Simple calculations show that E ˆ g x (cid:48) ( x ) = ∇ Γ (cid:15),λ ( x ) and the following lemma shows that ˆ g satisﬁes a type of variance bound conducive to variance-reduction schemes; see Appendix C.4 for the proof.
Lemma 3.
Let each (cid:96) j satisfy Assumptions 1 and 2.
For any λ ≤ Gr , r = (cid:15) (cid:48) G and x, x (cid:48) ∈ B r (¯ x ) , the variance of ˆ g x (cid:48)
( x ) is bounded by Var(ˆ g x (cid:48) ( x )) ≤ O (cid:16)(cid:0)
[Equation]
Accelerated variance reduction methods for convex functions typically require a stronger variance bound of the form Var(ˆ g x (cid:48)
( x )) ≤ 2 L ( F ( x (cid:48) )
− F ( x ) − (cid:104)∇ F ( x ) , x (cid:48) − x (cid:105) ) for every x [cf. 1, Lemma 2.4].
The guarantee of Lemma 3 is weaker, but still allows for certain accelerated rates via, e.g., the
Katyusha X algorithm [2].
With it, we obtain the following guarantee.
Theorem 2.
Let each (cid:96) j satisfy Assumptions 1 and 2.
Let (cid:15) > 0 , (cid:15) (cid:48) =
(cid:15)/ (2 log M ) and r (cid:15) =
(cid:15) (cid:48) /G .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ , KatyushaX s [2, Algorithm 2] with the gradient estimator (9) outputs a valid r (cid:15) -BROO
response and has complexity C λ ( δ )
=
O (cid:16)(cid:16) N
+ N
[Equation]
log (cid:0)
[Equation]
(cid:15) λδ 2 (cid:1)(cid:17) .
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L g-DRO (2) with probability at least 12 is
[Equation]
(cid:15) (cid:33) log 7 / 2 H (cid:33) where H :
= M GR (cid:15) .
We provide the proof of Theorem 2 in Appendix C.5.
For the special case of Group DRO with a single group satisfying Assumption 2 with L = Θ (cid:0) G 2 /(cid:15) (cid:1) , i.e. minimizing the average loss, we have the lower bound (cid:101) Ω (cid:0) N + N 3 / 4 GR(cid:15) (cid:1) [62] and for the case of N distinct groups, i.e. minimizing the maximal loss, we have the lower bound (cid:101) Ω (cid:0) N(cid:15)
− 2 / 3 (cid:1) [15].
This implies that in the weakly mean-square smooth setting the term scaling as N 3 / 4 (cid:15) − 1 and the term scaling as N(cid:15) − 2 / 3 are unimprovable.
DRO with f -divergence
In this section we develop our BROO implementation for the f -divergence objective (3).
In Section 4.
we reduce the original DRO problem to a regularized form using Lagrange multipliers.
Next, in
Section 4.2 we show that adding negative entropy regularization to the objective produces the stability properties necessary for efﬁcient ball optimization.
In Section 4.3 we describe a BROO implementation for the non-smooth case using a variant of Epoch-SGD [28], and in Section 4.4 we implement the BROO under a weak-smoothness assumption by carefully restarting an accelerated variance reduction method [1].
4.1 The dual problem
We ﬁrst note that (due to Slater’s condition), by Lagrange duality, the objective (3) is equivalent to
L f -
[Equation]
Writing ψ ( s ) :
= νN f ( Ns ) for ψ : R + → R , we therefore consider objectives of the form
[Equation]
where G is the Lipschitz constant of each loss (cid:96) i and for the last equality we use Lagrange duality, with ψ ∗ ( v ) :
= max t ∈ dom ( ψ ) { vt − ψ ( t ) } the Fenchel dual of ψ (for more details see Appendix D.1).
We show that under weak assumptions (introducing logarithmic dependence on bounds on f and the losses) we can solve the constrained problem (3) to accuracy (cid:15) by computing a polylogarithmic number of O ( (cid:15) )
-accurate minimizers of (10); see Appendix D.2 for details.
Since the complexity of solving (10) holds for any ν > 0 , and we have a lower bound for the required ν , for the remainder of this section we focus on minimizing L ψ for arbitrary convex ψ .
4.2 Stabilizing the gradient estimator
While minimizing (10) can be viewed as ERM (over x and y ), straightforward application of SGD does not solve it efﬁciently.
To see this, consider the standard gradient estimator formed by sampling i ∼ Unif ([ N ]) and taking ˆ g x = Nψ ∗(cid:48)
( (cid:96) i ( x ) − Gy ) ∇ (cid:96) i ( x ) and
ˆ g y
=
G (cid:0) 1 − Nψ ∗(cid:48) ( (cid:96) i ( x ) − Gy ) (cid:1) .
For general ψ , this estimator will have unbounded second moments, and therefore SGD using them would lack a convergence guarantee.
As an extreme example, consider ψ = 0 (corresponding to minimizing the maximum loss) whose conjugate function ψ ∗ ( v ) is 0 for v ≤ 0 and ∞ for v > 0 , leading to meaningless stochastic gradients.
We obtain bounded gradient estimates in two steps.
First, we ﬁnd a better distribution for i using a reference point ¯ x ∈ X with corresponding ¯ y =
argmin y ∈ R Υ(¯ x, y ) .
Namely, we note that the optimality condition for ¯ y implies that ψ ∗(cid:48)
( (cid:96) i (¯ x ) − G ¯ y ) is a pmf over [ N ] .
Therefore, we may sample i ∼ ψ ∗(cid:48) ( (cid:96) i (¯ x ) − G ¯ y ) and estimate the gradient of Υ at ( x, y ) using ˆ g x =
ρ i ( x, y ) ∇ (cid:96) i ( x ) and ˆ g y
=
G (cid:0) 1 − ρ i ( x, y ) (cid:1) , where ρ i ( x, y )
= ψ ∗(cid:48) ( (cid:96) i ( x ) − Gy ) ψ ∗(cid:48) ( (cid:96) i (¯ x )
− G ¯ y ) .
However, for general ψ (and ψ = 0 in particular), the ratio ρ i ( x, y ) can be unbounded even when x, y are arbitrarily close to ¯ x, ¯ y .
Our second step ensures that ρ i ( x, y ) is bounded around ¯ x, ¯ y by adding a small negative entropy term to ψ , deﬁning ψ
[Equation]
log q where (cid:15)
[Equation]
=
[Equation]
log N , (11) and
L ψ,(cid:15)
( x ) = min y ∈ R Υ (cid:15) ( x, y ) with Υ (cid:15) ( x, y
[Equation]
= (cid:88)
i ∈
[ N ]
[Equation]
∗ (cid:15)
[Equation]
(cid:96)
[Equation]
Due to our choice of (cid:15) (cid:48) , we have |L ψ ( x ) − L ψ,(cid:15)
( x ) | ≤ (cid:15)/ 2 for all x ∈ R d , and consequently an (cid:15)/ 2 -accurate minimizer of L ψ,(cid:15) is also an (cid:15) -accurate for L ψ (see Lemma 18 in
Appendix D.3).
When ψ =
we have ψ ∗ (cid:15) ( v ) =
e ( v − 1) /(cid:15) (cid:48) and therefore the corresponding ρ i ( x, y )
=
e ( (cid:96) i ( x ) − (cid:96) i (¯ x ) − G ( y − ¯ y )) /(cid:15) (cid:48) .The
following lemma, which might be of independent interest, shows that the same conclusion holds for any convex ψ .
Lemma 4.
For any convex ψ : R + → R and ψ (cid:15) deﬁned in (10) , log (cid:0) ψ ∗ (cid:15) (cid:48) ( · ) (cid:1) is 1 (cid:15) (cid:48) -Lipschitz.
See proof in Appendix D.4.
Thus, ψ ∗ (cid:15) (cid:48) ( v ) /ψ ∗ (cid:15) (cid:48)
(¯ v ) =
e log ψ ∗ (cid:15) (cid:48) ( v ) − log ψ ∗ (cid:15) (cid:48) (¯ v ) ≤ e ( v − ¯ v ) /(cid:15) (cid:48) and ρ i ( x, y ) ≤ e ( (cid:96) i ( x ) − (cid:96) i (¯ x ) − G ( y − ¯ y )) /(cid:15) (cid:48) continues to hold.
Therefore, if | y − ¯ y | ≤ (cid:15) (cid:48) /G
=
r (cid:15) and x ∈ B r (cid:15) (¯ x ) (so that | (cid:96) i ( x ) − (cid:96) i (¯ x ) | ≤ (cid:15) (cid:48)
if (cid:96) i satisﬁes Assumption 1), we have the bound ρ i ( x, y ) ≤ e 2 .
It remains to show that we may indeed restrict y to be within distance r (cid:15) from ¯ y .
To this end, we make the following observation which plays a key part in our analysis and might also be of independent interest (see proof in Appendix D.4).
Lemma 5.
For G > 0 , (cid:96) ( x ) =
( (cid:96) 1 ( x ) , . . .
, (cid:96) N ( x )) and y (cid:63) ( x ) = argmin y ∈ R Υ (cid:15) ( x, y ) , we have | y (cid:63) ( x ) − y (cid:63) ( x (cid:48) )
| ≤ 1 G (cid:107) (cid:96) ( x ) − (cid:96) ( x (cid:48) )
(cid:107) ∞ for all x, x (cid:48) ∈ X .
Moreover, if each (cid:96) i is G -Lipschitz, we have | y (cid:63) ( x ) − y (cid:63) ( x (cid:48) )
| ≤ (cid:107) x − x (cid:48) (cid:107) .
Lemma 5 implies that x (cid:63) , y (cid:63) =
argmin x ∈ B r(cid:15)
(¯ x ) ,y ∈ R Υ (cid:15) ( x, y ) satisfy | y (cid:63)
− ¯ y | ≤ (cid:107) x
(cid:63) − ¯ x (cid:107) ≤ r (cid:15) .
Therefore, when minimizing Υ (cid:15) (or any regularized version of it) inside the ball B r (cid:15) (¯ x ) , we may restrict y to [¯ y − r (cid:15) , ¯ y + r (cid:15) ] without loss of generality.
We also note that Lemma 5 holds for all values of (cid:15) and is therefore valid even without entropy regularization (as long as ψ ∗ is strongly convex y (cid:63) is unique, and if y (cid:63) is not unique then we can still choose y (cid:63) such that the bound of this lemma holds).
4.3 BROO implementation for f -divergence DRO with non-smooth losses
By the discussion above, to implement a BROO for L ψ,(cid:15)
( x ) (with radius r (cid:15) = (cid:15) (cid:48) /G , regularization λ , and query ¯ x ∈ X ) it sufﬁces to minimize Υ (cid:15),λ ( x, y ) :
= Υ (cid:15) ( x, y )
+ λ 2 (cid:107) x − ¯ x (cid:107) 2 over x ∈ B r (cid:15) (¯ x ) and y ∈ [¯ y − r (cid:15) , ¯ y + r (cid:15) ] , where ¯ y = argmin y ∈ R Υ (cid:15) (¯ x, y ) .
To that end we estimate the gradient of Υ (cid:15),λ ( x, y ) as follows.
Letting ¯ p i
= ψ ∗ (cid:15) (cid:48)
( (cid:96) i (¯ x ) −
G ¯ y )
(making ¯ p a pmf by optimality of ¯ y ), we sample i ∼ ¯ p and set
[Equation]
Lemma 4 implies the following bounds on our gradient estimator; see proof in Appendix D.5.
Lemma 6.
Let each (cid:96) i be G -Lipschitz, let ¯ x ∈ X and ¯ y = argmin y ∈ R Υ (cid:15),λ (¯ x, y ) .
Let r (cid:15) =
(cid:15) (cid:48) G , then for all x ∈ B r (cid:15) (¯ x ) and y ∈ [¯ y − r (cid:15) , ¯ y + r (cid:15) ] , the gradient estimators ˆ g x
and ˆ g y satisfy the following:
E i ∼ ¯ p
i
[ˆ g x ( x, y )] = ∇ x Υ (cid:15) ( x, y ) and E i ∼ ¯ p i
[ˆ g y ( x, y )]
= ∇ y Υ (cid:15) ( x, y ) .
E i ∼ ¯ p
i (cid:107) ˆ g x ( x, y ) (cid:107) 2 ≤ e 4 G 2 and E i ∼ ¯ p i
| ˆ g y ( x, y ) | 2 ≤ e 4 G 2 .
To implement the BROO using our gradient estimator we develop a variant of the Epoch-SGD algorithm of Hazan and Kale [28] (Algorithm 3 in Appendix D.5).
Similarly to Epoch-SGD, we apply standard SGD on Υ (cid:15),λ (with gradient estimator (13)) in “epochs” whose length doubles in every repetition.
Our algorithm differs slightly in how each epoch is initialized.
Standard Epoch-SGD initializes with the average of the previous epoch’s iterates, and strong convexity shows that the suboptimality and distance to the optimum shrink by a constant factor after every epoch.
However, since Υ (cid:15),λ is strongly convex only in x and not in y , we cannot directly use this scheme.
Instead, we set the initial y variable to be argmin y Υ (cid:15),λ ( x (cid:48) , y ) , where x (cid:48) is the initial x variable still deﬁned as the previous epoch’s average; this initialization has complexity N , but we only preform it a logarithmic number of times.
Using our initialization scheme and Lemma 5, we recover the original Epoch-SGD contraction argument, yielding the following complexity bound (see proof in Appendix D.5).
Theorem 3.
Let each (cid:96) i satisfy Assumption 1.
Let (cid:15), λ, δ > 0 , and r (cid:15) =
(cid:15)/ (2 G log N ) .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ < r (cid:15) / 2 , Algorithm 3 outputs a valid r (cid:15) -BROO
response for L ψ,(cid:15) and has complexity C λ ( δ )
=
O
[Equation]
G
[Equation]
N log (cid:0) r (cid:15) δ (cid:1)(cid:1) .
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L ψ (10) with probability at least 12 is
[Equation]
where H := N GR (cid:15) .
4.4 Accelerated variance reduction for smooth losses
In this section, we take advantage of the following smoothness assumption.
Assumption 3.
For every i ∈ [ N ] the loss (cid:96) i is L -smooth, i.e., has L -Lipschitz gradient.
Let us rewrite Υ (cid:15),λ in a form that is more amenable to variance reduction techniques:
[Equation]
i
[Equation]
[ N ] ¯ p i
υ
i ( x, y ) where υ i ( x, y ) :
= ψ ∗ (cid:15)
[Equation]
(cid:96)
[Equation]
x
[Equation]
p
[Equation]
and, as before ¯ p i =
ψ ∗ (cid:15) ( (cid:96) i (¯ x ) − G ¯ y ) for some ball center ¯ x ∈ X and ¯ y = argmin y ∈ R Υ (cid:15) (¯ x, y ) .
In the following lemma, we bound the smoothness of the functions
υ
i , deferring the proof to Appendix D.6.
Lemma 7.
For any i ∈ [ N ] , let (cid:96) i be G -Lipschitz and L -smooth, let r (cid:15) =
(cid:15) (cid:48) G and λ
= O (cid:0) Gr (cid:15) (cid:1) .
The restriction of υ i to x ∈ B r (cid:15) (¯ x ) and y ∈ [¯ y − r (cid:15) , ¯ y + r (cid:15) ] is O ( G ) -Lipschitz
and O (cid:0) L + G 2 (cid:15) (cid:48) (cid:1) -smooth.
Since Υ (cid:15),λ is a ﬁnite sum of smooth functions, we can obtain reduced-variance gradient estimates by the standard SVRG technique [33].
For any reference point x (cid:48) , y (cid:48) , the estimator is
[Equation]
where ∇ is with respect to the vector [ x, y ] .
Similar to non-smooth case, obtaining an efﬁcient
BROO implementation is complicated by the fact that Υ (cid:15),λ is strongly-convex in x but not in y .
Our solution is also similar: we propose a restart scheme and minimize over y exactly between restarts (Algorithm 4 in Appendix D.6), that gives the following complexity bound (see proof in
Appendix D.6).
Theorem 4.
Let each (cid:96) i satisfy Assumptions 1 and 3, let (cid:15), λ, δ > 0 , and r (cid:15)
= (cid:15) 2 G log N .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( Gr (cid:15) ) and accuracy δ , Algorithm 4 outputs a valid r (cid:15) -BROO
response for L ψ,(cid:15) and has complexity C λ ( δ ) =
[Equation]
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L ψ (10) with probability at least 12 is
[Equation]
where H := N GR (cid:15) .
Discussion
Limitations While our work indicates that the ball optimization approach offers signiﬁcant complexity gains for DRO, we note that turning the algorithms we propose into practical DRO methods faces several challenges.
A main challenge is the costly bisection procedure common to all MonteiroSvaiter-type acceleration schemes [41, 25, 13, 52].
Fortunately, very recently, two works [16, 36] (the former partially motivated by our paper) have shown how to remove the bisection from MonteiroSvaiter schemes, signiﬁcantly improving the practical potential of the methods we propose.
However, another practical limitation of our approach is the need to tune many parameters that are not known in advance, such as those relating to the ball radius r (cid:15) and smoothing level (cid:15) (cid:48) , as well as step sizes and number of iterations of oracle implementations; a more adaptive setting for these parameters is likely important.
Extensions First, it would be interesting to extend our approach to DRO objectives max q ∈U (cid:80) i ∈
[ N ] q i (cid:96) i ( x ) with uncertainty set U that is an arbitrary subset of the simplex.
While the subgradient method, the primal-dual method (Appendix A.1), and “AGD on softmax” (Appendix A.2) all apply to any U ⊆ ∆ N , our methods strongly rely on the structure of U induced by
Group- f -divergence DRO, and extending them to unstructured U ’s seems challenging.
Second, it would be interesting to generalize our results in the “opposite” direction of getting better complexity bounds for problems with additional structure.
For Group-DRO our bounds are essentially optimal when the number of groups M = Ω( N ) , but are suboptimal
when M =
O
We leave it as a question for further research if it is possible to obtain a stronger bound such as (cid:101) O (cid:0) N + M(cid:15) − 2 / 3 + (cid:15) − 2 (cid:1) , which recovers our result for M = N but improves on it for smaller values of M .
Taking CVaR at level α as a special case of f -divergence DRO, our bounds are optimal when α is close to 1 /N but suboptimal for larger value of α ; it would be interesting to obtain bounds such as (cid:101) O (cid:0) N +
α − 1 (cid:15) − 2 /
+
(cid:15) − 2 (cid:1)
A third possible extension of our research is DRO in the non-convex setting.
For this purpose, it might be possible to use the technique of Carmon et al.
[12] for turning accelerated convex optimization algorithms to improved-complexity methods for smooth non-convex optimization.
Acknowledgments
We thank Marc Teboulle for detailed and helpful feedback.
This work was supported by Israeli Science
Foundation (ISF) grant no. 2486/21 and the Len Blavatnik and the Blavatnik Family foundation.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Theorem 1, Theorem 2, Theorem 3 and Theorem 4.
(b) Did you describe the limitations of your work?
[Yes] See "Limitations" paragraph in Section 1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 1 Assumption 2 and Assumption 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Alternative algorithms for solving DRO problems A.1 Primal-dual stochastic mirror descent
In this section, we present a primal-dual method capable of solving all the DRO problems our paper considers, under an additional assumption of bounded losses: for every j and x we assume | (cid:96) j ( x ) | ≤ B (cid:96) .
Consider the primal-dual problem
[Equation]
where X ⊆ B R ( x 0 ) is a closed convex set as before and U is now an arbitrary closed convex subset of the simplex ∆ m and L i ( x )
= (cid:80) j ∈
[ N ] w ij (cid:96) j
( x ) are “group losses” with w i ∈ ∆ N for every i ∈ [ m ] .
This formulation subsumes both Group DRO (where m =
M and U = ∆ M ) and f -divergence DRO (where m = N , L i ( x ) =
(cid:96) i ( x ) , and U is an f -divergence ball).
As discussed in the introduction, several works have proposed primal-dual methods for DRO, but we could not ﬁnd in these works the precise rate we prove here (in Proposition 2 below) in its full generality.
Our proof is a straightforward specialization of the more general results of Carmon et al.
The particular algorithm we consider is primal-dual stochastic mirror descent, with distances generated by the squared Euclidean norm on X and entropy on U and gradient clipping for the U iterates, corresponding to the following recursion:
[Equation]
where η is a step-size parameter, Π [ − 1 , 1] m is the Euclidean projection to the unit box (i.e., entry-wise clipping to [ − 1 , 1] ), and ˆ g x and ˆ g q are unbiased estimators for ∇ x L pd and ∇ q L pd , respectively, given by
[Equation]
with e i ∈ R m being the i th standard basis vector in R m .
This method yields the following convergence guarantees.
Proposition 2.
Assume that each (cid:96) j convex and G -Lipschitz and satisﬁes | (cid:96) j ( x ) | ≤ B (cid:96) for every x ∈ X .
For T ∈ N let ¯ x T = 1 T (cid:80) Tt
=0 x t and ¯ q T = 1 T (cid:80) Tt =
q t , where { x t , q t } are the iterates deﬁned in (16) , with η = O (cid:16)
[Equation]
Then, for any (cid:15) > 0 , if T ≥ O (cid:16)
[Equation]
(cid:17) we have that
[Equation]
where L DRO ( x ) =
max
q ∈U
L pd ( x, q ) .
Proof.
The proposition is a direct corollary of a more general result by Carmon et al.
To show this, we rewrite the iterations (16) using “local norm setup” notation of [14].
In particular, let
U =
X
× U and for every z
=
( z x , z q ) ∈
U deﬁne the local norm of δ ∈ U ∗ at z as
[Equation]
In addition, we deﬁne the generating distance function
[Equation]
and write its associated Bregman divergence as V z ( z (cid:48) )
=
r ( z (cid:48) )
− r ( z ) − (cid:104)∇ r ( z ) , z (cid:48)
− z (cid:105) .
Next, we let Θ := max z,z (cid:48) ∈Z { r ( z ) − r
( z (cid:48) ) } and observe that since z x ∈ X ⊂ B R ( x 0 ) and z q ∈ U ⊂ ∆ m , then Θ = 2 log m .
Last, we deﬁne the function clip : Z ∗ → Z ∗ as follows:
[Equation]
where Π [ − 1 , 1] m denotes entry-wise clipping to [ − 1 , 1] .
By an argument directly analogous to [14,
Proposition 1], the quintuplet ( Z , (cid:107)·(cid:107) .
, r, Θ , clip) forms a valid local norm setup [14, Deﬁnition 1].
With this notation, the iterations (16) have the concise form
[Equation]
(18) where ˆ g ( z ) :
= (ˆ g x , − ˆ g q ) , with ˆ g x and ˆ g q as deﬁned in (17) above.
It is then straight-forward to verify that E ˆ g ( z ) =
( ∇ x L pd ( z ) , −∇ q L pd ( z )) for every z ∈ U , and that
E
[Equation]
for every z, w ∈ U .
Therefore, ˆ g is an L -local estimator [14, Deﬁnition 3] with L 2 =
G 2 R 2 / ( logm )+ mB 2 (cid:96) so that L 2 Θ = 2 G 2 R 2 + 2 B 2 (cid:96) m log m .
Proposition 2 now follows immediately from [14,
Proposition 2].
A.2 AGD on the softmax: complexity bound
In this appendix we brieﬂy develop the complexity guarantees of the “AGD on softmax” approach mentioned in the introduction.
While the idea is well known, we could not ﬁnd in the literature an analysis of the method for the general DRO setting (i.e., maximization over q in arbitrary subsets of the simplex), so we provide it here.
We wish to minimize, over x ∈ X ,
[Equation]
where each (cid:96) i is convex, G -Lipschitz and L -smooth
and U is an arbitrary closed convex subset of the simplex ∆ N ; note that this includes both Group DRO and f -divergence DRO as special cases.
We deﬁne the approximation
[Equation]
  with (cid:15) (cid:48)
= (cid:15) 2log N .
In addition, since (cid:80) i ∈ [ N ] q
i log q i ∈ [ − log N, 0] , for q ∈ ∆ N we have that
[Equation]
Thus for x satisfying (cid:101) L DRO ( x ) − min x ∈X (cid:101)
L DRO ( x ) ≤ (cid:15)/ 2 we have that L DRO ( x )
− min x (cid:63) ∈X L DRO ( x (cid:63) ) ≤ (cid:15) as well.
Next, we show that (cid:101) L DRO is (cid:101) O
(1 /(cid:15) )
-smooth when each (cid:96) i is O
(1 /(cid:15) )
-smooth.
For q ∈ U , the function Ψ( q ) =
(cid:80) i ∈
[ N ] (cid:15) (cid:48)
q i log( q i ) is (cid:15) (cid:48)
-strongly
convex w.r.t to the (cid:107)·(cid:107) 1 norm, therefore the conjugate function Ψ ∗ ( · ) is 1 (cid:15) (cid:48)
-smooth w.r.t to the dual norm (cid:107)·(cid:107) ∞ , such that
[Equation]
Input: BROO O λ,δ ( · ) , T max , initialization x 0 = v 0 and A 0 ≥
Parameters: Approximation parameters { δ k , β k , σ k } , stopping parameters A max and K max 1 for k = 0 , 1 , 2 , · · · do λ k +
=
λ
-B ISECTION
( x k , v k , A k ) a k +
=
[Equation]
k +1 (cid:112) 1 +
[Equation]
k
[Equation]
k and A k +
[Equation]
=
A k
[Equation]
a k +
[Equation] [Equation]
[Equation]
E
[Equation]
(cid:16)
[Equation] [Equation]
(cid:17)
[Equation]
Proj
[Equation]
(cid:0)
[Equation] [Equation]
g k +
[Equation]
(cid:1)
if A k +1 ≥ A max or k + 1 = K max then return x k +
In addition, let q (cid:63) ( (cid:96) ( x ))
= ∇ Ψ ∗ ( (cid:96) ( x ))
=
argmax q ∈U { (cid:96) ( x ) − Ψ
( q ) }
and note that (cid:101) L DRO ( x ) =
Ψ ∗ ( (cid:96) ( x )) .
Using this for every x,
y ∈ X we have (cid:13)(cid:13)(cid:13) ∇ (cid:101)
L DRO ( x ) − ∇ (cid:101)
L DRO ( y ) (cid:13)(cid:13)(cid:13) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88) i ∈
[ N ] ∇ (cid:96) i ( x
[Table]
( i ) follows since every (cid:96) i is G Lipschitz and L smooth, in addition for every (cid:96) i ( x )
∈ R we have that q (cid:63) ( (cid:96) ( x ))
∈ U ⊂ ∆ N , therefore (cid:80) i ∈
[ N ] q (cid:63)i ( (cid:96) ( x ))
= 1 , ( ii ) follows from the inequality in (19) and ( iii ) follows since each (cid:96) i is G -Lipschitz.
Since (cid:101)
L DRO is (cid:101)
L =
L + G 2 (cid:15) (cid:48) -smooth, Nesterov’s accelerated gradient descent [44] method is efﬁcient for minimizing it.
This method ﬁnds x such that (cid:101) L DRO ( x ) − min x (cid:101)
L DRO ( x ) ≤ (cid:15)/ 2 with
O (cid:16)(cid:113) (cid:101)
LR 2 /(cid:15) (cid:17)
=
(cid:101) O (cid:0) GR(cid:15) (cid:1) iterations when L =
O ( G 2 /(cid:15) ) .
Note that in every iteration we need to compute the full gradient of (cid:101) L DRO , which requires to evaluate each ∇ (cid:96) i and (cid:96) i .
Therefore, in the weakly smooth setting the complexity of this method is (cid:101) O ( NGR(cid:15) ) .
B Proof of Proposition
In this section we provide the proof for Proposition 1 that follows from the analysis of Algorithm 1; this section closely follows Asi et al.
[3], and we refer the readers to that paper for a more detailed exposition.
We begin with a short description of Algorithm 1.
This algorithm iteratively computes a λδ 2 2approximate minimizer of F λ ( x ) =
F ( x ) + λ 2 (cid:107) x − y (cid:107) 2 within a small ball of radius r around y .
To keep the ball constraint inactive it uses a bisection procedure that outputs the regularization strength value λ , such that for the minimizer ˆ x = argmin x ∈X F λ ( x ) with high probability we have (cid:107) ˆ x − y (cid:107) ≤ r .
It then computes a (nearly) unbiased gradient estimator of the Moreau envelope
M λ ( y ) =
min x ∈X F λ ( x ) and uses a momentum-like scheme to compute the next ball center.
Algorithm
Input: BROO O λ,δ ( · ) , query point y , regularization λ , bias β , and variance σ 2 .
1 Set T max = 2 G 2 λ 2 min { β 2 , 12 σ 2 } , T 0 = 14 G 2 log T max σ, and δ 0 =
G λ √ T2 x
=
O λ,δ 0 ( y k )
3 Sample J ∼ Geom (cid:0) 12 (cid:1) 4 if 2 J ≤ T max then δ J =
G λ √ 2 J T x J , x J − 1 =
O λ,δ J ( y ) , O λ,δ J − 1 ( y )
ˆ x = x 0 + 2 J
( x J − x J − 1 ) 8 else ˆ x = x10 return λ ( y − ˆ x )
Algorithm 1 is a variant of the accelerated proximal point method in [3, Algorithm 4], and we now describe the differences between the two.
The main difference is that the bias reduction scheme of Asi et al.
[3] averages (cid:101) O (cid:16) G 2 σ 2 + 1 (cid:17) calls to an estimator with accuracy δ (cid:48)
J = O (cid:0) G λ 2 J/ 2 (cid:1) where
J ∼ Geom (cid:0) 12 , T max (cid:1) and G is the Lipschitz constant of F .
In contrast, we use a single call to an estimator with higher accuracy δ J = (cid:101) O (cid:16) δ (cid:48)
J / (cid:112) G 2 /σ 2 (cid:17) ; cf.
the implementations of M OR G RAD E ST subroutine in each algorithm.
There are additional differences between the error tolerance settings of our algorithms, as described below.
The guarantees of [3, Proposition 2] require the following choice of approximation parameters:
[Equation]
60 a k which implies max k ≤ K max (cid:8)
[Equation]
+
[Equation]
(cid:9) ≤ (cid:15) 20 (where β k in our notation is δ k in the notation of [3]).
These parameters were chosen so that together with [3, Lemma 6] they give the following bound
E
[Equation]
which is a key component in the proof of [3, Proposition 2].
However, for improved efﬁciency we set different parameters:
[Equation]
To obtain the guarantees of [3, Proposition 2] for our implementation, in the following lemma we reprove [3, Lemma 6] with our parameters and show the same bound as in (20) (with a slightly different constant factor).
Lemma 8 (modiﬁcation of [3, Lemma 6]) .
Let F satisfy Assumption 1 with a minimizer x (cid:63) .
Let ϕ k
=
[Equation]
Deﬁne
[Equation]
x − y k (cid:107) 2 (cid:111) and assume that for each k we have (cid:107) ˆ x k − y k − 1 (cid:107) ≤ r and that one of the following must occur
λ k < 2 λ m = 2 (cid:15) r 4 / 3 R 2 / 3 log 2 (cid:16) GR 2 r(cid:15) (cid:17) , or 2.
(cid:107) ˆ x k − y k − 1 (cid:107) ≥ 34 r .
Then we have that
[Equation]
Proof.
Deﬁne the ﬁltration
[Equation]
where ζ i is the internal randomness of λ -B ISECTION
( x k , v k , A k ) and note that A k +1 , y k , ˆ x k +1 are deterministic when conditioned on x k , v k , A k , ζ k .
Following the proof of [3, Lemma 6], we deﬁne
[Equation]
and show it is a supermartingle adapted to ﬁlteration F k .
From [3, Lemma 5] we have
[Equation]
21) where
[Equation]
1 ϕ k +
+ a
[Equation]
k +1 σ 2 k +1 +
Ra k
+
β
k +1 .
Substituting the values of ϕ k +1 , σ 2 k +1 , β k +1 into the deﬁnition of µ k +1 gives
[Equation]
Recall that A k
+
=
a 2 k +1 λ
k +1 , therefore
[Equation]
If (cid:107) ˆ x k − y k − 1 (cid:107) ≥ 34 r , we have that
[Equation]
Else, if λ k < 2 λ m = 2 (cid:15) r 4 / 3 R 2 / 3 log 2 (cid:16) GR 2 r(cid:15) (cid:17) , we have µ
[Equation]
Now, note that λ m ≥ (cid:101) O (cid:0) (cid:15)R 2 (cid:1) ≥ 1 A max =
(cid:15) 9 R 2 and therefore a
[Equation]
From the deﬁnition of A k we have that A K max − 1 ≤ A max , therefore for every k ≤ K
[Equation]
Thus, when λ k < 2 λ m the bound on µ k +1 becomes
[Equation]
where the last inequality follows since A k ≥ 0 and λ k ≥ 0 .
Therefore, for every k ≤ K we have that
[Equation]
Noting that E | M k | < ∞ and substituting the bound on µ k +1 into (21) we get
[Equation]
Therefore M k is a supermartingle adapted to ﬁltration F k .
Since K is a stopping time adapted to ﬁltration F k , by the optional stopping theorem for supermartingles we have
[Equation]
For line 2 of Algorithm 1, we use the same λ -B ISECTION implementation of [15].
This implementation requires calling to a high-probability Ball Regularized Optimization Oracle (high-probability
BROO)
and we give the deﬁnition of it bellow.
Deﬁnition 2.
An algorithm is a probability 1 − p
Ball Regularized Optimization Oracle of radius r ( r -BROO) for function F : X → R if for query point ¯ x ∈ X ,probability p , regularization parameter λ > 0 and desired accuracy δ >
it returns O λ,δ (¯ x ) ∈
X that with probability at least 1 − p satisﬁes
[Equation]
In the following lemma we give the complexity guarantee for a high probability r -BROO.
Lemma 9.
Let C F be the complexity of evaluating F exactly, and C λ ( δ ) be an r -BROO implementation complexity.
Then the complexity of implementing a probability 1 − p r -BROO of Deﬁnitionis
[Equation]
Proof.
To obtain a high-probability r -BROO we run log 2 (cid:16) 1 p (cid:17) copies of r -BROO with query point ¯ x , regularization strength λ and accuracy δ/ √ 2 and take the best output, i.e., the output with the minimal value of F .
Applying Markov’s inequality to a single run of r -BROO with output x , gives
[Equation]
therefore with probability at least 1 − (cid:0) 12 (cid:1) log 2 ( 1 p )
= 1 − p , the best output x (cid:48) satisﬁes
[Equation]
Procedure require log(1 /p )
BROO calls and the same number of exact function evaluation (to choose the best BROO output), resulting in the claimed complexity bound log (cid:16) 1 p (cid:17)(cid:104) C λ (cid:16) δ √ 2 (cid:17) + C F (cid:105) .
To compute the gradient estimator in line 6 of Algorithm 1, we use Algorithm 2.
Our implementation is slightly different than [3] and in the following lemma we show it produces an estimator with the same bias and variance guarantees of [3].
Lemma 10.
Let F : X → R satisfy Assumption 1 and for query point
y ∈ X and regularization strength λ > 0 deﬁne x (cid:48)
= argmin x ∈X { F ( x ) + λ 2 (cid:107) x − y (cid:107) 2 } and g = λ ( y − x (cid:48) ) .
Then, for any bias and variance parameters β, σ > 0 Algorithm 2 outputs ˆ g = λ ( y − ˆ x ) satisfying (cid:107) E ˆ g − g (cid:107) ≤ β and E (cid:107) ˆ g − E ˆ g (cid:107) 2 ≤ σ 2 .
Proof.
First note that if x = O λ,δ (¯ x ) is the output of an r -BROO with accuracy δ and if x (cid:48) =
argmin x ∈ B r (¯ x ) F ( x ) , from Deﬁnition 1 and strong convexity (of F ( x ) + λ 2 (cid:107) x − ¯ x (cid:107) 2 ) we have: λ
[Equation]
giving
[Equation]
Let j max = (cid:98) log 2 T max (cid:99) , then from the deﬁnition of ˆ x in Algorithm 2 we have that
[Equation]
Therefore
[Equation]
with ( i ) following from Jensen inequality and ( ii ) following from the guarantee in (23).
To bound the variance of ˆ g note that
[Equation]
where the last inequality follows from (cid:107) a + b (cid:107) 2 ≤ 2 (cid:107) a (cid:107) 2 + 2 (cid:107) b (cid:107) 2 .
The deﬁnition of ˆ x gives
[Equation]
and from the guarantee in (23) we get
[Equation]
thus
[Equation]
In addition we have E (cid:107) x 0 − x (cid:48) (cid:107) 2 ≤ δ 20 =
G 2 λ 2 T 0 and substituting back we get
[Equation]
Combining the previous statements, we prove our main proposition.
Proposition 1.
Let F satisfy Assumption 1, let C F be the complexity of evaluating F exactly, and let C λ ( δ ) bound the complexity of an r -BROO query with δ, λ .
Assume that C λ ( δ ) is nonincreasing in λ and at most polynomial in 1 /δ .
For any (cid:15) > 0 , Algorithm 1 returns x such that
F ( x ) − min x (cid:63) ∈X F ( x (cid:63) ) ≤ (cid:15) with probability at least 12 .
For m (cid:15) =
O (cid:0) log GR 2 (cid:15)r (cid:1) and λ m =
O (cid:0) m 2 (cid:15) (cid:15) r 4 / 3 R 2 / 3 (cid:1) , the complexity of the algorithm is
[Equation]
Proof.
We divide the proof into a correctness argument and a complexity calculation
[Equation]
Correctness.
To prove the correctness of Proposition 1 we ﬁrst need to show that the guarantees of [3, Proposition 2] still hold for our implementation that includes different parameters ϕ k and σ 2 k :
[Equation]
and different implementation of lines 5 and 6.
Following Lemma 8, the guarantees of [3, Proposition 2] are still valid with our different choice of ϕ k and σ 2 k .
In addition, following Lemma 10, our implementation of line 6 is valid since it produces the same guarantees that the implementation in [3] gives.
Last, if the implementation of line 5 is valid it needs to satisfy
[Equation]
Note that x k +1 in our implementation is the output of r -BROO with accuracy δ k ≤ r √
[Equation]
, therefore by Deﬁnition 1 it satisﬁes
[Equation]
and for valid output of λ -B ISECTION we have
[Equation]
implying that line 5 is valid.
Now let p BROO be the probability that all calls to r -BROO result in a valid output.
Following [3, Proposition 2], for K max
= O (cid:16)(cid:0)
Rr (cid:1) 2 / 3 m (cid:15) (cid:17) with probability at least 1 − (cid:0) 1 − 23 (cid:1) − (1 − p BROO ) =
p BROO − 13 the algorithm outputs x that satisﬁes
[Equation]
Let p be the probability that a single BROO implementation produce invalid output and let K bisect-max be the maximal number of calls to high-probability r -BROO within line 2.
Then, p BROO ≥ 1 −
K max K bisect-max p and for
[Equation]
with probability at least 12 Algorithm 1 outputs (cid:15) 2 -suboptimal minimizer of F .
Complexity.
To bound the complexity of Algorithm 1 we ﬁrst bound the complexity of line 2 and the complexity of line 6 in the k -th iteration of Algorithm 1.
Note that, for x k +1 in line 5 we can use x 0 from Algorithm 2, and therefore the complexity of line 6 already includes the complexity of line 5.
Following [15, Proposition 2], λ -B ISECTION in line 2 requires m (cid:15)
=
O (cid:16) log (cid:16) GR 2 (cid:15)r (cid:17)(cid:17) calls to a high-probability r -BROO with accuracy δ = r 30 .
From Lemma 9 the complexity of a single call to a probability 1 − p r -BROO is O (cid:16) log (cid:16) 1 p (cid:17)
[ C λ ( r ) + C F ] (cid:17) .
We set p = 1 6 K max K bisect-max , and since
K max =
O (cid:16)(cid:0)
Rr (cid:1) 2 / 3 m (cid:15) (cid:17) and K bisect-max = m (cid:15) we get log (cid:16) 1 p (cid:17) =
O (cid:0) log (cid:0)
Rr m 2 (cid:15) (cid:1)(cid:1) ≤ m (cid:15) .
Therefore, the total complexity of λ -B ISECTION is O (cid:0) m 2 (cid:15)
[ C λ ( r ) + C F ] (cid:1) .
For the complexity of Line 6 note that Algorithm 2 calls to an r -BROO with accuracy δ J ≥ r
[Equation]
[Equation]
where J ∼ Geom (cid:0) 12 , j max (cid:1) and j max ≤ m (cid:15) .
Therefore, we can bound the complexity of
Algorithm 2 by
[Equation]
The complexity of the entire algorithm is at most K max times the complexity of a single iteration.
Using K max
= O (cid:16)(cid:0)
Rr (cid:1) 2 / 3 m (cid:15) (cid:17) , the total complexity becomes
[Equation]
C Group DRO
In this section we provide the proofs for the results of Section 3.
In Appendix C.1 we ﬁrst prove that the group-softmax is a uniform approximation of L g-DRO , then, through extension of [15], we show that we can approximate L g-DRO using the group-exponentiated softmax instead.
Next, in
Appendix C.2 we prove Lemma 2 and bound the moments of the MLMC and gradient estimators.
We then prove the complexity guarantees of Theorem 1 in Appendix C.3.
Last, in Appendix C.4 under the mean-square smoothness assumption, we provide the properties of the gradient estimator in (9) and in Appendix C.5 we prove the complexity guarantees of Theorem 2.
C.1 Exponentiated group-softmax
Recall the deﬁnition of the (regularized) group-softmax
L smax ,(cid:15),λ ( x ) :=
(cid:15) (cid:48) log   (cid:88) i ∈
[ M ] e L i ( x ) (cid:15) (cid:48) 
 + λ 2 (cid:107) x − ¯ x (cid:107) 2 where L i ( x ) = (cid:88)
j ∈ [ N ] w ij (cid:96) j
( x ) with L smax ,(cid:15), 0 ( x ) =
L smax ,(cid:15) ( x ) .
In addition, recall the deﬁnition of the (regularized) groupexponentiated softmax Γ
[Equation]
i ∈ [
[Equation]
] ¯ p i γ i ( x ) where γ i ( x ) = (cid:15) (cid:48)
e L i ( x ) −L i (¯ x )+ λ 2 (cid:107) x − ¯ x (cid:107) 2 (cid:15) (cid:48) and ¯ p
i =
e L i (¯ x ) (cid:15) (cid:48) (cid:80) i ∈
[ M ] e L i (¯ x ) (cid:15) (cid:48) .
Lemma 11.
Let L smax ,(cid:15) be the group-softmax deﬁned in eq.
(6) and L g-DRO be the Group DRO objective deﬁned in (2) .
Let (cid:15) > 0 and (cid:15) (cid:48)
= (cid:15)/ (2 log M ) > 0 .
Then for all x ∈ X we have that
[Equation]
Proof.
First note that
[Equation]
and
L smax ,(cid:15) ( x ) = (cid:15) (cid:48) log 
[Equation]
  .
In addition, for q ∈ ∆ M we have that (cid:80) i ∈
[ M ] q i log qi ∈ [ − log M, 0] .
Combining these facts gives
[Equation]
Lemma 1.
Let each (cid:96) i satisfy Assumption 1, and consider the restriction of L smax ,(cid:15),λ (6) and Γ (cid:15),λ (7) to B r (¯ x ) .
Then the functions have the same minimizer x (cid:63) ∈ B r (¯ x ) and, if λ ≤ O ( G/r ) and r ≤ O ( (cid:15) (cid:48) /G ) , then (a) Γ (cid:15),λ is Ω( λ ) -strongly convex, (b) each γ i is O ( G ) -Lipschitz and (c) for every x ∈ B r (¯ x ) we have L smax ,(cid:15),λ ( x ) − L smax ,(cid:15),λ ( x (cid:63) )
≤
O (Γ (cid:15),λ ( x ) − Γ (cid:15),λ ( x
[Table]
Proof.
This lemma is a simple extension of [15, Lemma 1], that considers the exponentiated-softmax: (cid:88)
[Table]
for some l 1 , . . .
, l N .
The only assumption that [15] have on l i (for the guarantees we state in Lemma 1) is that each l i is G -Lipschitz.
Note that each L i is G -Lipschitz since it is a weighted average of
G -Lipschitz functions.
Therefore, we can replace l i in [15, Lemma 1] with the group average L i and obtain Lemma 1.
C.2 MLMC estimator moment bounds
To make the MLMC estimator suitable for both Epoch-SGD and variance reduction methods, we rewrite its deﬁnition using more general notation.
Speciﬁcally, for every x, x (cid:48) ∈ X and S n 1 ∈ [ N ] n , let
[Equation]
so that (cid:98) γ ( x, ¯ x ; S n 1 )
= (cid:98) γ ( x ; S n 1 ) .
The MLMC estimator is
Draw J ∼ Geom
[Equation]
iid
[Equation]
i and let (cid:99)
[Equation]
where p j :=
P ( J = j ) =
(cid:0) 1 / √ 8 (cid:1) j (cid:16) 1 − 1 √ 8 (cid:17) and, for n ∈ 2 N we deﬁne
[Equation] [Equation]
Lemma 12.
Let each (cid:96) i satisfy Assumption 1, and let r ≤ (cid:15) (cid:48) G , λ ≤ Gr , (cid:107) x − x (cid:48) (cid:107) ≤ 2 r
and (cid:107) x
− ¯ x (cid:107) ≤ r .
For (cid:98) D n deﬁned in (25) we have E (cid:12)(cid:12)(cid:12)
[Equation]
x (cid:48) (cid:107)
[Equation]
2 (cid:15) (cid:48)(cid:19) .
Proof.
For abbreviation let
M
[Equation]
and
[Equation]
(cid:15) (cid:48)
[Equation]
where ˆ (cid:96) S j ( x ) =
(cid:96) S j ( x ) − (cid:96)
S j
( x (cid:48) ) .
We have the following bound on | M | :
[Equation]
with ( i ) following since each (cid:96) j is G -Lipschitz and λ ≤ G/r , ( ii ) follows since (cid:107) x − x (cid:48) (cid:107) ≤ 2 r
and (cid:107) x
− ¯ x (cid:107) ≤ r , and ( iii ) since r ≤ (cid:15) (cid:48) /G .
For | δ | we have the bound,
[Equation]
[Equation]
with the second inequality following since each (cid:96) j is G -Lipschitz and the last inequality since (cid:107) x − x (cid:48) (cid:107) ≤ 2 r and r ≤ (cid:15) (cid:48) /G .
We bound (cid:12)(cid:12)(cid:12) (cid:98) D n (cid:12)(cid:12)(cid:12) using the previous guarantees on | M | and | δ | : (cid:12)(cid:12)(cid:12) (cid:98) D n (cid:12)(cid:12)(cid:12)
[Equation]
(cid:15)
[Equation]
e
[Equation]
−
[Equation]
δ
[Equation]
e M
[Equation]
where ( i ) follows from (26) and ( ii ) from (27) and the inequality e x ≤ 1 + x + 2 x 2 for all x ≤with x = δ .
Therefore, we have that
[Equation]
Let Y i =
[Equation]
and note that the Lipschitz property of each (cid:96) j gives | Y i | ≤ 2 G (cid:107) x − x (cid:48) (cid:107) n(cid:15) (cid:48) , in addition, since the samples S 1 , . .
. , S n are i.i.d we have E ˆ (cid:96)
Si ( x ) n(cid:15)
[Equation]
and therefore
E Y i = 0 .
Thus, we can use Lemma 16 below, with Y i
=
ˆ (cid:96) Si
( x ) − ˆ (cid:96) Si + n/ 2 ( x ) n(cid:15) (cid:48) and c = 2 G (cid:107)
[Equation]
n(cid:15)
[Equation]
, to obtain the bound E [ δ ]
[Equation]
O (cid:18)
[Equation]
Therefore,
[Equation]
n 2 (cid:15) (cid:48)(cid:33) .
Lemma 2.
Let each (cid:96) i satisfy Assumption 1, and let r ≤ (cid:15) (cid:48) G , λ ≤ Gr and x ∈ B r (¯ x ) .
Then (cid:99) M [ γ i ( x )] and ˆ g ( x ) are unbiased for γ i ( x ) and ∇ Γ (cid:15),λ ( x ) , respectively, and have bounded second moments:
E (cid:2) (cid:99) M [ γ i ( x )] (cid:3) 2 ≤ O (cid:16) G 4 (cid:107) x − ¯ x (cid:107) 4 (cid:15) (cid:48)+ (cid:15) (cid:48) 2 (cid:17) and E (cid:107) ˆ g ( x ) (cid:107) 2 ≤ O (cid:0) G 2 (cid:1) .
In addition, the complexity of computing (cid:99) M [ γ i ( x )] and ˆ g ( x ) is O (1) .
Proof.
We ﬁrst prove the bias and moment bounds, and then address complexity.
Properties of the MLMC estimator.
We ﬁrst show that the MLMC estimator is unbiased.
For every n ∈ 2 N we have that E (cid:98) γ ( x ; S n/ 2) =
E (cid:98) γ ( x ; S nn/ 2+1 ) , therefore E (cid:98) D n = E (cid:98) γ ( x ; S n 1 ) − E (cid:98) γ ( x ; S n/ 2) and we get
E (cid:104) (cid:99) M
[ γ i ( x )] (cid:105) =
E (cid:98) γ ( x ; S 1 ) + ∞ (cid:88)
j = (cid:16) E (cid:98) γ ( x ; S 2 j 1 ) − E (cid:98) γ ( x ; S 2 j − 1) (cid:17) =
E (cid:98) γ
[Equation]
To bound the second moment of the estimator we use the inequality ( a + b ) 2 ≤ 2 a
+ 2 b 2 , yielding
[Equation]
Lemma 12 with x (cid:48)
=
¯ x gives E (cid:12) (cid:12)(cid:12) (cid:98) D n (cid:12) (cid:12)(cid:12) 2 ≤ O (cid:16) G 4 (cid:107) x − ¯ x (cid:107) 4 n 2 (cid:15) (cid:48)(cid:17) and substituting this bound into (29) while noting that E | (cid:98) γ ( x ; S 1 ) | 2 ≤ (cid:15) (cid:48)
2 e 3 gives
E (cid:12)(cid:12)(cid:12) (cid:99) M [ γ i ( x )] (cid:12)(cid:12)(cid:12) 2 ≤ O   (cid:15) (cid:48)
+ G 4 (cid:107) x − ¯ x (cid:107)(cid:15) (
cid:48)∞ (cid:88)
j = (cid:18) 1 − 1 √(cid:19) 2 1 .
j 2 2 j 

=
O (cid:32) G 4 (cid:107) x
− ¯ x (cid:107)(cid:15) (cid:48)+ (cid:15) (cid:48)(cid:33) .
[Equation]
Properties of the gradient estimator.
We use the fact that (cid:99) M [ γ i ( x )] is unbiased for γ i ( x ) (shown in eq.
(28) above) to argue that gradient estimator is also unbiased:
[Equation]
Next we bound the second moment of the gradient estimator
[Equation]
where ( i ) follows from (30), ( ii ) follows since each (cid:96) j is G -Lipschitz, λ ≤ Gr and (cid:107)
x − ¯ x (cid:107) ≤ r and ( iii ) since G 4 (cid:107) x − ¯ x
[Equation] [Equation]
≤ 1 .
Complexity of the MLMC and gradient estimators.
J ∼ Geom (1 − 1 √ 8 ) , therefore
[Equation]
Note that the estimator (cid:99) M [ γ i ( x )]
= (cid:98) γ ( x, S 1 ) + 1 P J (cid:98) D 2 J requires a single function evaluation for (cid:98) γ ( x, S 1 ) and 2 J function evaluations for the term (cid:98) D 2 J .
As a consequence the computation of (cid:99) M [ γ i ( x )] requires only O (1) function evaluations in expectation.
To compute ˆ g ( x ) we need to compute (cid:99) M [ γ i ( x )] and a single sub-gradient, hence, the complexity of computing ˆ g ( x ) is also O (1) in expectation.
C.3 Epoch-SGD BROO implementation
We state below the convergence rate of the Epoch-SGD algorithm.
Lemma 13 (Theorem 5, [28]) .
Let F : X → R be λ -strongly convex with an unbiased stochastic gradient estimator ˆ g satisfying E (cid:107) ˆ g ( x ) (cid:107) 2 ≤ O (cid:0) G 2 (cid:1) for all x ∈ X , and let x (cid:63) = argmin x ∈X F ( x ) .
Epoch-SGD ﬁnds an approximate minimizer x that satisﬁes
[Equation]
using T stochastic gradient queries.
Applying this lemma with F = Γ (cid:15),λ , x ∈ B r (cid:15) (¯ x ) and T
= O (cid:16) G 2 λ 2 δ 2 (cid:17) immediately gives the following guarantee on the BROO implementation complexity.
Theorem 1.
Let each (cid:96) j satisfy Assumption 1, let (cid:15), δ, λ > 0 and let r (cid:15) =
(cid:15)/ (2 G log M ) .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ , EpochSGD [28,
Algorithm 1]) with the gradient estimator (8) outputs a valid r (cid:15) -BROO
response and has complexity
C λ ( δ ) =
O (cid:0) N + G 2 λ 2 δ 2 (cid:1) .
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L g-DRO (2) with probability at least 12 is
[Equation]
where H :
= M
[Equation]
(cid:15) .
Proof.
We divide the proof into correctness and complexity arguments, addressing the BROO implementation and then the overall algorithm.
BROO implementation: correctness.
Following Lemma 1 we have that Γ (cid:15),λ is Ω( λ ) -strongly convex and Lemma 2 gives E (cid:107) ˆ g ( x ) (cid:107) 2 ≤ O (cid:0) G 2 (cid:1) .
Thus, we can directly apply Lemma 13 with
F = Γ (cid:15),λ , X =
B r (cid:15) (¯ x ) , the gradient estimator
ˆ g ( x ) deﬁned in (8) and T = 2 c
G 2 λ 2 δ 2 for a constant c > 0 for which Epoch-SGD outputs x that satisﬁes
[Equation]
Following Lemma 1 there is a value of c such that E L λ smax ,(cid:15) ( x ) − L λ smax ,
[Equation]
( x (cid:63) ) ≤ c ( E Γ (cid:15),λ ( x ) − Γ (cid:15),λ ( x (cid:63) )) and from (31) we obtain
[Equation]
Therefore, Epoch-SGD outputs a valid r (cid:15) -BROO
response for L smax ,(cid:15) .
BROO implementation: complexity.
For the BROO implementation we run Epoch-SGD with the gradient estimator
ˆ g ( x ) deﬁned in (8) and computation budget T =
O (cid:16) G 2 λ 2 δ 2 (cid:17) .
Therefore, we need to evaluate O (cid:16) G 2 λ 2 δ 2 (cid:17) stochastic gradient estimators with complexity O (1) , and our gradient estimator requires additional N functions evaluations for precomputing the sampling probabilities { ¯ p i } .
Thus, the total complexity of the BROO implementation is
[Equation] [Equation]
Minimizing
L g-DRO : correctness.
For any q ∈ ∆ M note that
L q ( x ) := (cid:80) i ∈
[ M ] q i L i ( x ) − (cid:15) (cid:48)
q i log q
i is G -Lipschitz, since L i is G -Lipschitz for all i ∈
[ M ] and therefore for all x ∈ X we have (cid:107)∇L q ( x ) (cid:107) =
(cid:13)(cid:13)(cid:13)(cid:80)
i ∈
[ M ] q i ∇L i
[Equation]
x ) (cid:13)(cid:13)(cid:13) ≤
G . Maximum operations
preserve the Lipschitz continuity and therefore L smax ,(cid:15) ( x ) = max q ∈ ∆ M L q ( x ) is also G -Lipschitz.
Thus, we can use Proposition 1 with F = L smax ,(cid:15) and obtain that the output ¯ x of Algorithmwith probability at least 12 will satisfy L smax ,(cid:15) (¯ x ) −
min x (cid:63) ∈X L smax ,(cid:15) ( x (cid:63) ) ≤ (cid:15)/ 2 .
In addition, from Lemma 11 for every x ∈ X we have that |L g-DRO ( x ) − L smax ,(cid:15) ( x ) | ≤ (cid:15)/ 2 .
Therefore, with probability at least
[Equation]
Minimizing L g-DRO : complexity.
The complexity of ﬁnding an (cid:15)/ 2 -suboptimal solution for L smax ,(cid:15) (and therefore an (cid:15) -suboptimal
solution for L g-DRO ) is bounded by Proposition 1 as:
[Equation]
where m (cid:15) =
O (cid:16) log (cid:16) GR 2 (cid:15)r (cid:15) (cid:17)(cid:17)
=
O (cid:0) log (cid:0) GR(cid:15) log M (cid:1)(cid:1) .
To obtain the total complexity we evaluate the complexity of running r (cid:15) -BROO with accuracy δ j = r 2 j/ 2 m 2 (cid:15) (for the MLMC implementation), and accuracy δ Bisection = r (cid:15) 30 (for the bisection procedure).
Using (32) and noting that λ m = (cid:15) r 4 / 3 (cid:15) R 2 / 3 m 2 (cid:15) we get the following BROO complexities:
C
λ m (cid:16)
[Equation]
C λ m (cid:0)
[Equation]
= O (cid:16)
[Equation]
Therefore
O   m (cid:15) m (cid:15) (cid:88)
[Equation]
 ≤ O (cid:32) m 2 (cid:15) (cid:32)(cid:18) GR (cid:15) (cid:19) 4 /+
N (cid:33)(cid:33) and
[Equation]
(cid:32)(cid:18) GR (cid:15) (cid:19)
[Equation]
(cid:15) (cid:33) .
Substituting the bounds into Proposition 1 with m (cid:15)
= log (cid:0) GR(cid:15) M (cid:1) and r (cid:15) =
[Equation]
2 G log M , the total complexity is
[Equation]
C.4 SVRG-like estimator properties
We ﬁrst give a deﬁnition of Γ (cid:15),λ that is more conducive to formulating variance reduction methods:
[Equation]
where γ i ( x, x
[Equation] [Equation]
and p i ( x (cid:48) ) :
= e
[Equation]
Therefore, the MLMC estimator of γ i ( x, x (cid:48) ) is
Draw J ∼ Geom (cid:16) 1 − 1 √ 8 (cid:17) , S 1 , . .
. , S n iid ∼ w i and let (cid:99) M [ γ i ( x, x (cid:48) )]
:= (cid:98) γ ( x, x (cid:48) ; S 1 ) + (cid:98) D 2 J p J with (cid:98) D n deﬁned in (25) and (cid:98) γ ( x, x (cid:48) ; S n 1 ) deﬁned in (24).
Lemma 14.
The SVRG-like estimator (9) is unbiased
[Equation]
Proof.
E
[Equation]
] (cid:17)
[Equation] [Equation]
[Equation]
(cid:48)
[Equation]
( i
[Equation]
= ∇ Γ
[Equation] [Equation]
(cid:88)
i
[Equation]
[ M ] (cid:88)
j
[Equation] [Equation]
with ( i ) following from the unbiased property of the MLMC estimator stated in Lemma 2 (that still holds for (cid:99) M [ γ i ( x, x (cid:48) )] ).
Lemma 3.
Let each (cid:96) j satisfy Assumptions 1 and 2.
For any λ ≤ G r , r = (cid:15) (cid:48) G and x, x (cid:48) ∈ B r (¯ x ) , the variance of ˆ g x (cid:48)
( x ) is bounded by Var(ˆ g x (cid:48) ( x )) ≤ O (cid:16)(cid:0)
[Equation]
Proof.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where ( i ) follows from the inequality E [ X − E X ]
2 = E
[ X 2 ] − [ E X ] 2 ≤ E
[ X 2 ] and ( ii ) from the inequality ( a + b ) 2 ≤ 2 a
+ 2 b 2 .
Next we bound separately each of the expectation terms.
Note that the ball constraint x ∈ B r (¯ x ) with r = (cid:15) (cid:48) G and λ ≤ Gr gives:
[Equation] [Equation]
therefore
[Equation]
with ( i ) following from the inequality ( a + b ) 2 ≤ 2 a
+
b
and ( ii ) from Assumption 2.
For the second expectation term we use the fact that each (cid:96) j is G -Lipschitz, λ ≤ Gr and (cid:107) x
− ¯ x (cid:107) ≤ r and thus (cid:107)∇ (cid:96)
j
[Equation]
x ) + λ ( x − ¯ x ) (cid:107) ≤ (cid:107)∇ (cid:96) j
( x )
[Equation]
+
[Equation]
λ ( x − ¯ x ) (cid:107) ≤ 2 G .
Therefore,
E
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
From the deﬁnition of (cid:99) M [ γ i ( x, x (cid:48) )] we get:
[Equation]
[Table]
with ( i ) following from Lemma
[Table]
( ii ) follows from the inequality e x − 1 ≤ x + 2 x 2 =
O ( x 2 ) for x ≤ 3 with x = (cid:96) S 1 ( x ) − (cid:96)
[Equation]
1 ( x (cid:48) )
(cid:15) (cid:48) ≤ 2 , ( iii ) follows since each (cid:96) j is G -Lipschitz and since e λ (cid:107) x − ¯ x (cid:107) 2 (cid:15) (cid:48)
=
O (1) and ( iv ) since G 2 (cid:107) x − x (cid:48) (cid:107) 2 (cid:15) (cid:48)≤ G 2 4 r 2 (cid:15) (cid:48)= 4 .
Therefore,
[Equation]
Finally we bound c x (cid:48) , ¯ x using the ball constraint x (cid:48)
∈ B r (¯ x ) and the fact that each L i is G -Lipschitz, therefore
[Equation]
Substituting back the bounds on each expectaion term and the bound on c x (cid:48) , ¯ x into (33) we get
Var(ˆ g ¯ x ( x )) ≤ O
[Equation]
C.5 Complexity of the reduced-variance BROO implementation
We ﬁrst state the complexity bounds of KatyushaX s [2]
Lemma 15 ([2, Theorems 1 and 4.3]) .
Let F be a λ -strongly convex function with minimizer x (cid:63) and let ˆ g x (cid:48)
( x ) be a stochastic gradient estimator satisfying the properties
[Equation]
ˆ g x
(cid:48)
[Equation]
) has evaluation complexity O (1) and preprocessing complexity O ( N ) , then KatyushaX s with the stochastic gradient estimator ˆ g x (cid:48) ﬁnds a point x satisfying E
[ F ( x ) − F ( x (cid:63) )] ≤ (cid:15) with complexity
O
[Equation]
Applying Lemma 15 with (cid:101) L = O (cid:16) L + G 2 (cid:15) (cid:17) and accuracy λδ 2 2 gives the following result.
Theorem 2.
Let each (cid:96) j satisfy Assumptions 1 and 2.
Let (cid:15) > 0 , (cid:15) (cid:48) =
(cid:15)/ (2 log M ) and r (cid:15) =
(cid:15) (cid:48) /G .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ , KatyushaX s [2, Algorithm 2] with the gradient estimator (9) outputs a valid r (cid:15) -BROO
response and has complexity C
[Equation]
δ
[Equation]
= O
[Equation]
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L g-DRO (2) with probability at least 12 is
[Equation]
where H :=
M GR (cid:15) .
Proof.
The proof is structured similarly to the proof of Theorem 1.
BROO implementation: correctness.
From Lemma 1 we have that Γ (cid:15),λ is Ω( λ ) -strongly convex, in addition, Lemma 3 and Lemma 14 show that the stochastic gradient estimator deﬁned in (9) is unbiased with Var(ˆ g x (cid:48)
( x )) ≤ (cid:101)
L 2 (cid:107) x − x (cid:48) (cid:107) 2 .
Thus, we can directly apply Lemma 15 with F = Γ (cid:15),λ and accuracy λδ 2 2 and obtain a valid BROO response.
BROO implementation: complexity.
We use KatyushaX s [2] for the BROO implementation.
Applying Lemma 15 with F = Γ (cid:15),λ , x 0 = ¯
x , (cid:101) L = O (cid:16) L + λ + G 2 (cid:15) (cid:48) (cid:17)
≤ O (cid:16) L + G 2 (cid:15) (cid:48) (cid:17) and accuracy λδ 2 2 the complexity of our implementation is
[Equation]
34) and note that Γ (cid:15),λ (¯ x ) − min x (cid:63) ∈ B r(cid:15)
(¯ x ) Γ (cid:15),λ ( x (cid:63) )
≤ Gr (cid:15) , since (from Lemma 1) Γ (cid:15),λ is O ( G ) -Lipschitz.
Minimizing L g-DRO : correctness.
Similarly to the proof of Theorem 1, combining the guarantees of Proposition 1 and Lemma 11, with probability at least 12 the output ¯ x of Algorithm 1 satisﬁes
L g-DRO (¯ x ) − min x (cid:63) ∈X L g-DRO ( x (cid:63) ) ≤ (cid:15) .
Minimizing L g-DRO : complexity.
The complexity of ﬁnding (cid:15)/ 2 -suboptimal solution for L smax ,(cid:15) and therefore an (cid:15) -suboptimal
solution for L g-DRO , is bounded by Proposition 1 as:
[Equation]
where m (cid:15) =
O (cid:0)
[Equation]
From the deﬁnitions of λ m and r (cid:15) we have
[Equation]
and
[Equation]
[Table]
[Equation]
[Table]
[Equation]
therefore
[Equation]
m
[Equation]
/ 3 (cid:15) (cid:15)m 2 (cid:15) 
[Equation]
log (cid:18)
[Equation]
(cid:15)
[Equation]
(cid:15) (cid:19)
[Equation]
O
[Equation]
(cid:32)
[Equation]
(cid:15)
[Equation]
(cid:114)
[Equation]
(cid:15) (cid:33) r 2 / 3 (cid:15) (cid:35)(cid:33) .
Similarly, we have that
O
[Equation]
(cid:18)(cid:18)
[Equation]
(cid:15) (cid:19)
[Equation]
(cid:19)
[Equation]
(cid:33) .
Substituting the bounds into Proposition 1 with m (cid:15)
= log (cid:0) GR(cid:15)
log M (cid:1) and r (cid:15) =
(cid:15) 2log M the total complexity is
O
[Equation]
N
[Equation]
(cid:19)
[Equation]
(cid:32) GR (cid:15)
[Equation]
(cid:114)
[Equation]
(cid:15) (cid:33) log 7 /
[Equation]
(cid:18) GR (cid:15) log M (cid:19)(cid:33) .
C.6 Helper lemmas
Lemma 16.
Let Y 1 , . . .
, Y n be a sequence of random i.i.d variables such that for every i ∈ [ n ] and a constant c >
we have that E
[ Y i ] = 0 and | Y
i | ≤ c with probability 1.
Then
[Equation]
Proof.
For i (cid:54) =
j we have that E
[ Y i Y j ] =
E [ Y i ] E
[ Y j ] = 0 , therefore
[Equation]
D DRO with f -divergence
In this section we provide the proofs for the results in Section 4.
In Appendix D.1 we provide the derivation of the dual formulation in (10).
In Appendix D.2 we show how to reduce the constrained problem (3) to a regularized problem of the form (10), then in Appendix D.3 we describe the properties of L ψ,(cid:15) , the approximation of (10).
In Appendix D.4 we provide the proofs for our main technical contribution and give guarantees on the stability of the gradient estimators.
Last, in Appendices D. and D.6 we give the complexity guarantees of our implementation in the non-smooth and slightly smooth cases.
D.1 Dual formulation of DRO with f -divergence
Here we give the derivation of the objective in (10), also considered in prior work [e.g., 42, 39, 32].
Recall the DRO with f -divergence objective:
[Equation]
We ﬁrst show the relation between L f -div and its regularized form (10).
Using the Lagrange multiplier ν for the constraint (cid:80) i ∈ [ N ] f ( Nq i ) N ≤ 1 and strong duality we get
[Equation]
where L ν · f ( x ) is the regularized form of L f
-div : writing ψ ( x ) =
νN f ( Nx ) , with slight abuse of notation we have
[Equation]
Adding a Lagrange multiplier y for the constraint that q ∈ ∆ N and using strong duality again gives
[Equation]
[Equation]
Finally, using ψ ∗ ( v ) := max t ∈ dom( ψ ) { vt − ψt } (the Fenchel conjugate of ψ ), we have
[Equation]
x
[Equation]
= min y ∈ R
[Equation]
[ N ] ψ ∗
[Equation]
x )
[Equation]
Gy
[Equation]
+ Gy (cid:41) , and note that ψ ∗ ( (cid:96) i ( x ) − Gy ) is equivalent to νf ∗ (cid:16) (cid:96) i ( x ) − Gy ν (cid:17) .
D.2 Minimizing the constrained objective using the regularized objective
In this section, we show that under the following Assumptions 4 and 5 we can reduce the constrained problem of minimizing (3) to the regularized problem of minimizing (10) by computing a polylogarithmic number of O ( (cid:15) )
-accurate minimizers of (10).
Assumption 4.
Each loss function (cid:96) i is bounded, i.e., (cid:96) i : X → [0 , B (cid:96) ] for every i ∈ [ N ] .
Assumption 5.
For any uncertainty set of the form U = { q ∈ ∆ N : D f ( q, p ) ≤ 1 } , the divergence function f is bounded, i.e., f : R + → [0 , B f ] for some B f ≥ 1 .
We note that the above assumptions are weak since the complexity of our approach only depends logarithmically on on B f B (cid:96) (cid:15) .
We ﬁrst cite a result on noisy one dimensional bisection, as given by a guarantees on the OneDimMinimizer algorithm in Cohen et al.
Lemma 17 (Lemma 33, Cohen et al.
let f : R → R be a B -Lipschitz convex function deﬁned on the interval [ (cid:96), u ] , and let G : R → R be an oracle such that |G ( y ) − f ( y ) | ≤ (cid:101) (cid:15) for all y. With
O (cid:16) log (cid:16) B ( u − (cid:96) ) (cid:101) (cid:15) (cid:17)(cid:17) calls to G , the algorithm OneDimMinimizer [18, Algorithm 8] outputs y (cid:48)
such that
[Equation]
We specialize Lemma 17 to our settings and provide the complexity guarantees of minimizing (3) to an (cid:15) -accurate solution using a noisy oracle G .
Proposition 3.
Let each (cid:96) i satisfy Assumption 4 and let f satisfy Assumption 5.
Deﬁne the function h
( ν ) :=
min x ∈X L ν · f ( x )+ ν with L ν · f deﬁned in (10) and let G be an oracle such that G ( ν ) ≥ h ( ν ) with probability 1 and G ( ν ) − h ( ν ) ≤ (cid:15) 5 with probability at least 12 .
Then applying OneDimMinimizer [18, Algorithm 8] on the interval [0 , B (cid:96) ] outputs ν (cid:48) that with probability at leat 99100 satisﬁes
[Equation]
using O (log( H ) log(log H )) calls to G , where H =
B f B (cid:96) (cid:15) .
Proof.
Let ˆ h q,x ( ν ) :
= (cid:80) i ∈
[ N ] q i (cid:96) i ( x ) − ν (cid:16) 1 N (cid:80) i ∈ [ N ] f ( Nq i ) − 1 (cid:17) and note that for any q and x the function ˆ h q,x is B f -Lipschitz, since it is linear in ν and (cid:12)(cid:12)(cid:12) 1 N (cid:80) i ∈ [ N ] f ( Nq i ) − 1 (cid:12)(cid:12)(cid:12) ≤ B f .
Minimization and maximization operations preserve the Lipschitz continuity and therefore the function h ( ν ) =
min x L ν · f ( x ) =
min x max q ˆ h q,x ( ν ) is also B f -Lipschitz continuous.
In addition for the q (cid:63) ∈ ∆ N that maximizes L ν · f ( x ) we have that (cid:88)
[Equation]
and rearranging gives
[Equation]
Therefore, for all ν > B (cid:96) we have h (cid:48)
( ν ) =
− 1 N (cid:80) i ∈
[ N ] f ( Nq (cid:63)i ) >
and therefore it sufﬁces to restrict h ( ν ) to [0 , B (cid:96) ] .
Next, to turn G into a high-probability oracle, we call it log 2 (cid:16) 100 log (cid:16) B f B (cid:96) (cid:15) (cid:17)(cid:17) times and choose the smallest output.
Therefore, with probability at least 1 − (cid:0) 12 (cid:1) log (cid:16) 100log (cid:16) BfB(cid:96)(cid:15) (cid:17)(cid:17)
= 1 − 1 / (cid:16) 100 log (cid:16) B f B (cid:96) (cid:15) (cid:17)(cid:17)
the result is within (cid:15) 5 of h ( ν ) .
Since h is B f -Lipschitz and deﬁned on [0 , B (cid:96) ] we can use Lemma 17 with (cid:96) = 0 , u = B f , (cid:101) (cid:15) =
(cid:15)/ 5 ,
B = B f and the high-probability version of G .
Therefore, using O (cid:16) log (cid:16) B f B (cid:96) (cid:15) (cid:17)(cid:17) calls to the high probability version of G and applying the union bound, we obtain that with probability at least
OneDimMinimizer outputs ν (cid:48) that satisﬁes
[Equation]
and therefore
[Equation]
Finally, in the following corollary we show that ﬁnding an (cid:15) -suboptimal solution for (3) requires a polylogarithmic number of O ( (cid:15) )
-accurate minimizers of (10) and a polylogarithmic number of evaluations of (10).
Corollary 5.
Let each (cid:96) i satisfy Assumption 4 and let f satisfy Assumption 5, then minimizing (3) to accuracy (cid:15) with probability at least 99100 requires O (log( H ) log(log H )) evaluations of (10) and O (log( H ) log(log H )) calls to an algorithm that with probability at least 12 returns an O ( (cid:15) ) - suboptimal point of (10) , where H =
B
f B (cid:96) (cid:15) .
Proof.
Note that L ν · f is deﬁned on (10).
Let (cid:101) G ( ν ) :
= L ν · f ( (cid:101) x ) + ν where (cid:101) x is the output of an algorithm that with probability at least 12 returns an (cid:15) 5 -suboptimal point of L ν · f and let h ( ν ) :
= min x ∈X L ν · f ( x ) + ν .
We have that (cid:101) G ( ν ) − h ( ν ) ≤ (cid:15) 5 with probability at least 12 , therefore, we can apply Proposition 3 with G = (cid:101) G , and obtain that with O (cid:16) log (cid:16) B f · B (cid:96) (cid:15) (cid:17) log (cid:16)
[Table]
to (cid:101) G
(i.e., to an algorithm that outputs (cid:15) 5 -suboptimal minimizer of L ν · f with probability at least 12 ),
OneDimMinimizer outputs ν
[Equation]
that satisﬁes with probability at least
[Equation]
Noting that L f -div ( x ) = min ν ≥
{L ν · f ( x ) + ν } we obtain
L f -div ( (cid:101) x ) − min x
L f -div ( x ) ≤ L ν (cid:48) ·
f ( (cid:101) x ) + ν (cid:48)
− min x L f -div ( x ) =
(cid:101) G ( ν (cid:48) )
− min x
L f -div ( x ) ≤ (cid:15).
Corollary 5 means that the complexity bounds for approximately minimizing the objective L ψ established by Theorems 3 and 4 also apply (with slightly larger logarithmic factors) to approximately minimizing the constrained f -divergence objective L f -div .
D.3 Properties of L ψ and L ψ,(cid:15)
Lemma 18.
For L ψ deﬁned in (10) and L ψ,(cid:15) deﬁned in (12) we have that
[Equation]
for all
[Equation]
∈
[Equation]
Proof.
Recall that (cid:15) (cid:48)
=
(cid:15) 2log N and for q ∈ ∆ N we have that (cid:80) i ∈
N q
i log q i ∈ [ − log N, 0] , therefore:
[Equation]
 
[Table]
D.4 Gradient estimator stability proofs
Lemma 4.
For any convex ψ : R + → R and ψ (cid:15) deﬁned in (10) , log (cid:0) ψ ∗ (cid:15) (cid:48) ( · ) (cid:1) is 1 (cid:15) (cid:48) -Lipschitz.
Note that from the deﬁnition of ψ ∗ (cid:15) (cid:48)
we have that ψ (cid:15) (cid:48) (0) →
−∞ , in addition since q ∈ R N + then ψ ∗ (cid:15) (cid:48) is non-negative and log (cid:0) ψ ∗ (cid:15) (cid:48) (cid:1) is well-deﬁned.
We now give the proof of Lemma 4.
Proof.
While we write the proof as though the function ψ is differentiable with derivative ψ (cid:48) , one may readily interpret ψ (cid:48) as an element in the subdifferential of ψ and the proof continues to hold.
Let φ (cid:15) =
(cid:15) (cid:48)
q log q and recall that ψ (cid:15) ( q ) =
ψ ( q ) + φ (cid:15) ( q ) .
Fix any two numbers v 1 , v 2 ∈ R and assume without loss of generality that v 2 > v 1 .
For i = 1 , 2 , let
[Equation] [Equation]
Note that by deﬁnition of the Fenchel dual (and strict convexity of ψ (cid:15) ), q i is the unique solution to
[Equation]
and moreover that q 2 ≥ q 1 since ψ ∗ (cid:15) is convex and therefore ψ ∗ (cid:15) (cid:48) is non-deceasing.
Similarly, p i is the unique solution to v i = φ (cid:48)
[Equation]
( p i ) and p 2 > p 1 .
Combining the two equalities yields
[Equation]
Rearranging, we ﬁnd that
[Equation]
where ψ (cid:48) ( q 2 ) − ψ
[Equation]
( q 1 ) ≥ 0 holds by convexity of ψ and q 2 ≥ q 1 .
Recalling that φ (cid:48)
[Equation]
( q
[Equation]
q , we have
[Equation]
where the last equality follows by substituting the deﬁnition of p i .
The proof is complete upon recalling that q i = ψ ∗ (cid:15) (cid:48) ( v i ) .
Lemma 5.
For G > 0 , (cid:96) ( x ) =
( (cid:96) 1 ( x ) , . . .
, (cid:96) N ( x )) and y (cid:63) ( x ) = argmin y ∈ R Υ (cid:15) ( x, y ) , we have | y (cid:63) ( x ) − y (cid:63) ( x (cid:48) )
| ≤ 1 G (cid:107)
[Equation]
( x ) − (cid:96) ( x
[Equation]
) (cid:107) ∞ for all x, x (cid:48) ∈ X .
Moreover, if each (cid:96) i is G -Lipschitz, we have | y (cid:63) ( x ) − y (cid:63) ( x (cid:48) )
| ≤ (cid:107) x − x (cid:48) (cid:107) .
Proof.
For x, x (cid:48) ∈
X w.l.o.g. assume that y (cid:63) ( x ) ≤ y (cid:63) ( x (cid:48) ) and observe that for every u ∈ X (cid:88)
[Equation]
= 1 .
Let (cid:101) (cid:96) i ( x ) =
(cid:96) i ( x ) + δ with δ := (cid:107) (cid:96)
[Equation]
x
[Equation]
) −
[Equation]
x )
[Equation]
∞ and (cid:101) y ( x ) :=
argmin y ∈ R (cid:110)(cid:80) i ∈
[ N ] ψ ∗
[Equation]
(cid:16)(cid:101) (cid:96) i ( x ) − Gy (cid:17) + Gy (cid:111) .
Then, according to (37) (cid:88)
i ∈
[ N ]
[Equation]
i
[Equation]
and due to the monotonicity of ψ ∗ (cid:15) (cid:48) and ( i ) we get
G (cid:101) y
[Equation]
x
[Equation]
= Gy (cid:63) ( x
[Equation]
By convexity, ψ ∗ (cid:15) (cid:48) is monotonically non decreasing, thus (cid:88)
i ∈
[Equation] [Equation]
i ∈
[Equation]
where ( i ) follows from noting that (cid:96) i ( x (cid:48) ) ≤ (cid:96) i ( x ) + max i ∈ [ N ] | (cid:96) i ( x (cid:48) ) − (cid:96) i ( x ) |
= (cid:101) (cid:96) i ( x ) .
Therefore,
Gy (cid:63) ( x (cid:48) ) ≤ G (cid:101) y
( x ) =
Gy (cid:63) ( x ) + δ giving
[Equation]
In addition, if each (cid:96) i is G -Lipschitz we have
[Equation]
D.5 Epoch-SGD BROO implementation
In this section we provide the analysis of our algorithm in the non-smooth case, which consists of combining our general BROO acceleration scheme (Algorithm 1) with a variant of Epoch-SGD [28] that we specialize in order to implement a BROO for Υ (cid:15),λ (Algorithm 3).
We organize this section as follows.
First, we prove Lemma 6 showing that our gradient estimators are unbiased with bounded second moment, and therefore can be used in Algorithm 3.
Then, in Proposition 4 we give the convergence rate of Algorithm 3.
Combining the previous statements with the guarantees of
Proposition 1 we prove Theorem 3.
For convenience, we restate the deﬁnitions of Υ (cid:15) and our stochastic estimators for ∇ x Υ (cid:15) ( x, y ) and ∇ y Υ (cid:15) ( x, y ) :
Υ
[Equation]
x, y
[Equation] [Equation]
i ∈
[ N ] ψ
[Equation]
, and
[Equation]
where ¯ p i =
ψ ∗ (cid:15) (cid:48) ( (cid:96) i (¯ x ) − G ¯ y ) .
Lemma 6.
Let each (cid:96) i be G -Lipschitz, let ¯ x ∈ X and ¯ y = argmin y ∈ R Υ (cid:15),λ (¯ x, y ) .
Let r (cid:15) =
(cid:15) (cid:48) G , then for all x ∈ B r (cid:15) (¯ x ) and y ∈ [¯ y − r (cid:15) , ¯ y + r (cid:15) ] , the gradient estimators ˆ g x
and ˆ g y satisfy the following:
E i ∼ ¯ p
i
[ˆ g x ( x, y )] = ∇ x Υ (cid:15) ( x, y ) and E i ∼ ¯ p i
[ˆ g y ( x, y )]
= ∇ y Υ (cid:15) ( x, y ) .
E i ∼ ¯ p
i (cid:107) ˆ g x ( x, y ) (cid:107) 2 ≤ e 4 G 2 and E i ∼ ¯ p i
| ˆ g y ( x, y ) | 2 ≤ e 4 G 2 .
Proof.
We ﬁrst show that the stochastic gradients ˆ g x , ˆ g y are unbiased
[Equation]
and
[Equation]
( x, y ) .
Next, we bound the second moment of the stochastic gradients.
For any i we have
[Equation]
where ( i ) follows from Lemma 4 and the fact that (cid:96) i is G -Lipschitz and ( ii ) uses G -Lipschitzness again together with x ∈ B r (cid:15) (¯ x ) and
y ∈
[¯ y − r (cid:15) , ¯ y + r (cid:15) ] to deduce that (cid:96) i ( x ) − Gy − ( (cid:96) i (¯ x ) − G ¯ y ) ≤ 2 Gr (cid:15) ≤ 2 (cid:15) (cid:48) .
Therefore, we have E (cid:107) ˆ g x ( x, y ) (cid:107) 2 ≤ e 4 G 2 as required.
The second moment bound on ˆ g y ( x, y ) follows similarly, since
[Equation]
Proposition 4.
Let (cid:15), λ > 0 , (cid:15) (cid:48) =
[Equation]
2log N and r (cid:15) =
(cid:15) (cid:48) G .
For any query point ¯ x let ¯ y = argmin y ∈ R Υ (cid:15),λ (¯ x, y ) and let x (cid:63) , y (cid:63) =
argmin x ∈ B r(cid:15)
(¯ x ) ,y ∈ [¯ y − r (cid:15) , ¯ y + r (cid:15) ]
Υ (cid:15),λ ( x, y ) .
For γ k = 1 8 λ 2 k ,
T ≥ 1 and threshold T threshold =
G 4 λ 2 (cid:15) (cid:48) 2 the output ( x, y ) of Algorithm 3 satisﬁes
[Equation]
Proof.
For convenience let x k = x (0) k and
y k
= y (0) k , and in addition let ¯ p i =
ψ (cid:48) (cid:15) ∗ ( (cid:96) i (¯ x ) − G ¯ y ) be the sampling probability from Algorithm 3.
We use induction to prove that
[Equation]
for all k .
We start with the base case (k=1).
[Equation]
where ( i ) follows from convexity of Υ (cid:15) and ( ii ) since due to optimality conditions we have that (cid:104)∇ y Υ
[Equation]
x 1 , y 1 ) , y (cid:63)
[Equation]
y
[Equation]
≥ 0 .
Therefore,
[Equation]
Input: The function Υ (cid:15) deﬁned in (12), ball center ¯ x , ball radius r (cid:15) , regularization parameter λ , smoothing parameter (cid:15) (cid:48) and iteration budget T .
Parameters: Initial step size γ 1 = 1 / (16 λ ) , epoch length T 1 = 128 and threshold
T threshold =
G 4 λ 2 (cid:15) (cid:48) 2 .
1 Initialize x (0)1 = ¯ x 2 Initialize y (0)1 = ¯ y = argmin y ∈ R Υ (cid:15) (¯ x, y )
Precomupte sampling probabilities ¯ p i =
ψ (cid:48) (cid:15) ∗ ( (cid:96) i (¯ x ) − G ¯ y ) 4 for k = 1 , . .
, (cid:100) log( T/ 128 + 1) (cid:101) do for t = 0 , 2 , · · · T k − 1 do
Sample i ∼ ¯ p i
Query stochastic gradients ˆ
[Table]
x (cid:16) x ( t ) k , y ( t )
[Table]
(cid:17) and ˆ
[Table]
(cid:16) x ( t ) k , y ( t ) k (cid:17) deﬁned in (13)
[Table]
=
[Table]
Update
[Equation]
[Table]
[Equation]
[Table]
[Figure]
Set x (0) k
+
=
1 T k (cid:80) t ∈ [ T k ] x ( t ) k
Set y (0) k
+
=
1 T k (cid:80) t ∈ [ T k ] y ( t ) k
Update T k +
= 2 T k
Update γ k +
= γ k / k ← k
+ if T k ≥ T threshold then
Recompute y (0) k +1 = argmin y ∈ R Υ (cid:15) ( x (0) k
+1 , y )
17 return x = x (0) k where the last inequality follows from Jensen’s inequality and Lemma 6.
This gives the base case of our induction.
Let V x ( x (cid:48) )
= 12 (cid:107) x − x (cid:48) (cid:107) 2 and V y
( y (cid:48) )
= 12 | y − y (cid:48) | 2 and suppose that there is a k such that E Υ (cid:15),λ ( x k , y k )
− Υ (cid:15),λ ( x (cid:63) , y (cid:63) ) ≤ e 4 G 2 λ 2 k .
Then, using the mirror descent regret bound [see, e.g., 3,
Lemma 3] for k + 1 we get
[Equation]
+
[Equation]
+ e 4 G2 λ 2 k + with ( i ) following from Lemma 6 and ( ii ) from the choice of T k = 64 · 2 k and γ k
= 1 λ 2 k +3 .
Next, note that the strong convexity of Υ (cid:15) ( x, y ) in x implies λ E V x k ( x (cid:63) ) ≤ e 4 G 2 λ 2 k since λ E V x k
( x (cid:63) ) ≤
E λ Υ (cid:15),λ ( x k , y k ) − E Υ (cid:15),λ ( x (cid:63) , y k )
≤ E λ Υ (cid:15),λ ( x k , y k ) − Υ (cid:15),λ ( x (cid:63) , y (cid:63) )
≤ e 4 G 2 λ 2 k by the induction hypothesis.
In addition from Lemma 5 we have
[Equation]
by the constraint on y .
We now bound E V y k
( y (cid:63) ) in each scenario T k ≤ T threshold =
G 4 λ 2 (cid:15) (cid:48) 2 or T k > T threshold .
If T k ≤ T threshold we have that | y k − y (cid:63) | ≤ (cid:15) (cid:48)
G ≤ G λ 2 k/ 2+2 and thus V y k ( y (cid:63) ) ≤ G 2 λ 2 2 k .
If T k > T threshold Algorithm 3 will recompute the optimal y k and using Lemma 5 we have
[Equation]
and as a result V
[Equation]
Therefore E V y k ( y (cid:63) ) ≤ e 4 G 2 λ 2 2 k and substituting back the bounds on E V y k
( y (cid:63) ) and E V x k ( x (cid:63) )
we obtain
[Equation] [Equation]
[Equation]
which completes the induction.
Let K be the iteration where the algorithm outputs x = x (0) K and let y =
y (0) K .
Noting that T
= O (2 K ) , we have
[Equation]
Theorem 3.
Let each (cid:96) i satisfy Assumption 1.
Let (cid:15), λ, δ > 0 , and r (cid:15) =
(cid:15)/ (2 G log N ) .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( G/r (cid:15) ) and accuracy δ < r (cid:15) / 2 , Algorithm 3 outputs a valid r (cid:15) -BROO
response for L ψ,(cid:15) and has complexity C λ ( δ ) =
O (cid:0) G 2 λ 2 δ 2 + N log (cid:0) r (cid:15) δ (cid:1)(cid:1) .
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L ψ (10) with probability at least 12 is
O (cid:32)
N
[Equation]
where H := N GR (cid:15) .
Proof.
We divide the proof into correctness arguments and complexity bounds.
BROO implementation: correctness.
We use Algorithm 3 with T =
O (cid:16) G 2 λ 2 δ 2 (cid:17) for the BROO implementation.
Applying Proposition 4 the output ( x, y ) of Algorithm 3 satisﬁes
[Equation]
Therefore, there is a constant c > 0 for which the output ( x, y ) of Algorithm 3 with T = cG 2 λ 2 δsatisﬁes E Υ (cid:15),λ ( x, y )
− Υ (cid:15),λ ( x (cid:63) , y
[Equation]
) ≤ λδ 2 2 , and since
[Equation]
Algorithm 3 returns a valid BROO response for L ψ,(cid:15) .
BROO implementation: complexity.
The total number of epochs that Algorithm 3 performs is
K = O (cid:16) log( G 2 δ 2 λ 2 ) (cid:17) .
In O (cid:16) log (cid:16) G 2 λ 2 δ 2 (cid:17) − log (cid:16) G 2 λ 2 r 2 (cid:15) (cid:17)(cid:17) =
O (cid:0) log (cid:0) r (cid:15)
[Equation]
(cid:1)(cid:1) epochs with T ≥ G
[Equation]
2 (cid:15) (cid:48)
[Equation]
=
G 2 r 2 (cid:15) λ 2 the algorithm performs O ( N ) function evaluations to recompute the optimal y .
In addition, the algorithm performs O (cid:16) G 2 λ 2 δ 2 (cid:17) stochastic gradient computations (each computation involves a single loss function (cid:96) i and a single sub-gradient ∇ (cid:96) i evaluation).
Therefore the total complexity of the BROO implementation is
[Equation]
Minimizing L ψ : correctness.
For any q ∈ ∆ N let L q ( x ) :
= (cid:80) i ∈
[ N ] q i (cid:96) i ( x ) − ψ (cid:15) ( q i ) and note that
L q is G -Lipschitz, since for all x ∈ X we have (cid:107)∇L q ( x ) (cid:107) =
(cid:13)(cid:13)(cid:13)(cid:80)
i ∈
[ N ] q i ∇ (cid:96) i ( x ) (cid:13)(cid:13)(cid:13)
≤ G .
Maximum operations preserve the Lipschitz continuity and therefore L ψ,(cid:15)
( x ) = max q ∈ ∆ N L q ( x ) is also GLipschitz.
Since L ψ,(cid:15) is G -Lipschitz, we can use Proposition 1 with F = L ψ,(cid:15) and obtain that with probability at least 12 Algorithm 1 outputs a point x such that L ψ,(cid:15)
( x ) − min x L ψ,(cid:15) ( x ) ≤ (cid:15)/ 2 .
In addition, from Lemma 18 we have
[Equation]
[Equation]
Minimizing L ψ : complexity.
We apply Proposition 1 with F = Υ (cid:15),λ and r (cid:15) =
(cid:15)/ (2 log N · G ) , thus the complexity of ﬁnding an (cid:15)/ 2 -suboptimal minimizer of L ψ,(cid:15) (and therefore an (cid:15) -suboptimal minimizer of L ψ ) is bounded as:
[Equation]
where m (cid:15) =
O (cid:16) log GR 2 (cid:15)r (cid:15) (cid:17)
=
O (cid:0) log GR(cid:15) log N (cid:1) and λ m =
O (cid:16) m 2 (cid:15) (cid:15) r 4 / 3 (cid:15) R 2 / 3 (cid:17) .
To obtain the total complexity bound, we evaluate the complexity of running r (cid:15) -BROO with accuracy δ j
= r
[Equation]
2 j/ 2 m 2 (cid:15) and δ = r (cid:15) 30 .
Using (39) we get
[Equation]
Thus
[Equation]
and
[Equation]
Substituting the bounds into Proposition
[Equation]
the total complexity is bounded as
O
[Equation]
D.6 Accelerated variance reduction BROO implementation
In this section we prove the complexity guarantees of our BROO implementation for (potentially only slightly) smooth losses.
We ﬁrst prove Lemma 7, showing that Υ (cid:15),λ (the approximation of our objective (10)) is a ﬁnite sum of smooth functions, and thus for the BROO implementation we can use a variance reduction method for a ﬁnite (weighted) sums.
Then, we give Deﬁnition 3 of a “valid accelerated variance reduction” (VR) method, and in Lemma 20 we prove the convergence rate of our
BROO implementation (Algorithm 4) which is simply a restart scheme utilizing any valid accelerated
VR method.
We then combine the guarantees of Lemma 20 and Proposition 1 to prove Theorem 4, our ﬁnal complexity guarantee in the smooth.
To begin, let us restate here the deﬁnition of Υ (cid:15),λ (that has the form of a weighted ﬁnite sum): Υ (cid:15),λ ( x, y )
= (cid:88) i ∈
[ N ] ¯ p i
υ
i ( x, y ) where υ i ( x, y ) :
= ψ ∗ (cid:15) (
[Equation]
i
[Equation]
x )
[Equation]
Gy
[Equation]
(40) and ¯ p i =
ψ ∗ (cid:15) (cid:48) ( (cid:96) i (¯ x ) − G ¯ y ) .
Lemma 7.
For any i ∈ [ N ] , let (cid:96) i be G -Lipschitz and L -smooth, let r (cid:15) =
(cid:15) (cid:48) G and λ
= O (cid:0) Gr (cid:15) (cid:1) .
The restriction of υ i to x ∈ B r (cid:15) (¯ x ) and y ∈ [¯ y − r (cid:15)
Proof.
To show the Lipschitz property we compute the gradient of υ i ( x, y ) with respect to x and y .
[Equation]
Similarly to the proof of Lemma 6, we have that
[Equation]
≤ e 2 and therefore (cid:107)∇ x υ i ( x, y ) (cid:107) ≤ e 2 G
= O ( G ) and |∇ y υ
i ( x, y )
| ≤ e 2 G
= O ( G ) , giving the ﬁrst statement.
To bound the smoothness of υ i , we compute the Hessian of υ i ( x, y ) .
[Equation]
Lemma 4 implies that, for all v , ψ
[Equation]
[Table]
≤ e 2 .
In addition
[Table]
each
[Table]
is L -smooth and G
[Table]
Lipschitz and λ
=
O
[Table]
Gr
[Table]
= O
[Table]
, therefore
[Table]
[Equation]
Applying Lemma 21 with h
=
υ
i we get that
[Equation]
proving that each υ i ( x, y ) is O (cid:16) L + G 2 (cid:15)
[Equation]
smooth
[Equation]
Deﬁnition 3.
For a given ball center ¯ x ∈ X , let z (cid:63) ∈ B r (cid:15) (¯ x ) × R minimize the function Υ (cid:15),λ : B r (cid:15) (¯ x ) × R
→
R , and let
¯ y = argmin y ∈ R Υ (cid:15),λ (¯ x, y ) .
Let V ARIANCE R EDUCTION be a procedure that takes in z ∈ X × R and complexity budget T , and outputs z (cid:48) ∈
X × R .
We say that
V ARIANCE R EDUCTION is a valid accelerated VR method if it has complexity T and satisﬁes the following: there a constant C such that for any α , input z , and
[Equation]
the output z (cid:48) of V ARIANCE R EDUCTION ( z ; T ) satisﬁes
[Equation]
Input:
[ N ] p i υ x, y ) deﬁned in (40), number of total restarts K , and an algorithm V ARIANCE R EDUCTION that takes in x,
y ∈
X × R and complexity budget T , and outputs x (cid:48) , y (cid:48)
∈
X × R .
Lemma 19.
Katyusha sf [38] is a valid accelerated VR method for some C
= O (1) .
Proof.
Immediate from [1, Theorem 4.1] and Lemma 7.
(We note that the theorem is stated for a ﬁnite sum with uniform weights, but the extension of the theorem and the method to non-uniform sampling is standard).
The following lemma shows that Algorithm 4, when coupled with any valid accelerated VR method yields a BROO implementation for
[Equation]
i.e., it outputs an approximate minimizer of
[Equation]
in B r (cid:15) (¯ x ) .
Lemma 20.
Let Assumptions 1 and 3 hold, and suppose Algorithm 4 uses a valid accelerated VR method with constant C (deﬁned above).
Then, for (cid:101) L
= L + G 2 /(cid:15) (cid:48) and T ≥ 2 C ( N + (cid:113) N (cid:101)
L/λ ) , for any K ≥
the output x of Algorithm 4 satisﬁes
[Equation]
Moreover, the complexity of Algorithm 4 is K
[Equation]
Proof.
Let z (cid:63) =
( x (cid:63) , y (cid:63) ) minimize Υ (cid:15),λ in B r (cid:15) (¯ x ) × R , so that x (cid:63) =
argmin x ∈ B r(cid:15)
(¯ x ) L ψ,(cid:15),λ ( x ) as well.
Note that for all of the outer loop iterations z k
=
( x k , y k ) we have, by the optimality of y k and
Lemma 5,
[Equation]
Moreover, the λ -strong convexity of L ψ,(cid:15),λ implies that
[Equation]
λ .
Furthermore, note that Υ (cid:15),λ ( z k ) =
min y ∈ R Υ (cid:15),λ ( x k , y )
=
L ψ,(cid:15),λ ( x k ) .
Recalling that by Lemmarestricting the domain of y to [¯ y − r (cid:15) , ¯ y + r (cid:15) ] does not change the optimal y , we conclude that for
V ARIANCE R EDUCTION to guarantee Υ (cid:15),λ ( x k +1 , y (cid:48) k +1 )
− Υ (cid:15),λ ( z (cid:63) )
≤ α it sufﬁces to choose
[Equation]
In particular, we see that T ≥ 2 C
[Equation]
sufﬁces for
[Equation]
, from which we conclude that
[Equation]
giving the claimed optimality bound.
Finally, the complexity of the method is clearly K ( T + N ) since we make K calls to V ARIANCE R EDUCTION with complexity T and K exact minimizations over y with complexity N .
To efﬁciently implement the BROO , we repeatedly apply an accelerated variance reduction scheme that does not require strong convexity, such as Katyusha ns [2], each time with complexity budget (cid:101)
O ( N + (cid:112) NL (cid:48) /λ ) , where L (cid:48)
=
L + G 2 /(cid:15) (cid:48) .
We start each repetition by the x variable output by the previous Katyusha ns call, and with y = argmin y ∈ R Υ (cid:15),λ ( y, x ) for that x .
Using Lemma 5 in lieu of strong-convexity in y , we show that error halves after each restart, and therefore a logarithmic number of restarts sufﬁces.
We arrive at the following complexity bound.
Theorem 4.
Let each (cid:96) i satisfy Assumptions 1 and 3, let (cid:15), λ, δ > 0 , and r (cid:15)
= (cid:15) 2 G log N .
For any query point ¯ x ∈ R d , regularization strength λ ≤ O ( Gr (cid:15) ) and accuracy δ , Algorithm 4 outputs a valid r (cid:15) -BROO
response for L ψ,(cid:15) and has complexity C λ ( δ ) =
O (cid:16)(cid:16)
[Equation]
Consequently, the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L ψ (10) with probability at least 12 is
[Equation]
where H := N GR (cid:15) .
Proof.
We ﬁrst show correctness and complexity of the BROO implementation and then argue the same points for the overall method.
BROO implementation: correctness and complexity.
Combining Lemmas 19 and 20 and setting K =
[Equation]
, we obain a valid BROO implementation with complexity
O
[Equation]
λδ(cid:33)  (41)
Furthermore, we note that Υ (cid:15) is
O ( G ) -Lipschitz
and therefore Υ (cid:15) ( x 0 , y 0 )
− argmin x ∈ B r(cid:15)
( x ) ,y ∈ R Υ (cid:15) ( x, y ) ≤
O ( Gr (cid:15) ) .
Minimizing L ψ : correctness.
Similarly to the proof of Theorem 3, we note that L ψ,(cid:15) is G -Lipschitz, and therefore we can apply Proposition 1 with F = L ψ,(cid:15) and obtain that with probability at least
Algorithm 1 outputs x such that L ψ,(cid:15) ( x ) − min x L ψ,(cid:15) ( x
[Equation]
≤
[Equation]
/ 2 and Lemma 18 gives
[Equation]
Minimizing L ψ : complexity.
Applying Proposition 1 with F = Υ (cid:15) and r (cid:15)
= (cid:15) 2 G log N , the complexity of ﬁnding an (cid:15) -suboptimal minimizer of L ψ is
O
[Equation]
42) where m (cid:15) =
O (cid:0) log GR 2 (cid:15)r (cid:15) (cid:1)
= log (cid:0) GR(cid:15) log N (cid:1) and λ m =
O (cid:0) m 2 (cid:15) (cid:15) r 4 / 3 R 2 / 3 (cid:1) .
Using similar calculations to the proof of Theorem 2 we obtain that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
with (cid:15)
[Equation]
λ m r 2 (cid:15) =
O (cid:16)(cid:0) GR(cid:15) (cid:1)
2 / 3 1 m 2 (cid:15) (cid:17) and
[Table]
[Equation]
O
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(cid:15) (cid:17) .
Therefore, m (cid:15)
[Equation]
and
O (cid:0)
[Equation]
+ N
[Equation]
m 3 (cid:15) (cid:1) ≤ O (cid:32)
m 4 (cid:15)
N
[Equation]
m 3 .
5 (cid:15) N 1 /(cid:32)
GR 1 /(cid:15)
+ (cid:114) LR 2 /(cid:15) (cid:33) r 2 / 3 (cid:15) (cid:33) .
Substituting the bounds into Proposition 1 with m (cid:15) = log (cid:0) GR(cid:15) log N (cid:1) and r (cid:15)
=
(cid:15) 2log
N the total complexity is
O
[Equation]
N
[Equation]
GR (cid:15) (cid:19)
[Equation]
/log
[Equation]
(cid:18) GR (cid:15) log N (cid:19)
+ N 1 /(cid:32)
GR (cid:15) + (cid:114) LR(cid:15) (cid:33) log 7 / 2 (cid:18) GR (cid:15) log N (cid:19)(cid:33) .
D.7 Helper lemmas
Lemma 21.
Let x ∈ R d , y ∈ R , then for any h : R d × R → R we have
[Equation]
Proof.
[Equation]
sup
[Equation]
with ( i ) following due to Hölder’s inequality and ( ii ) follows from the inequality 2 ab ≤ a 2 + b
[Equation]