Distributed Online and Bandit Convex Optimization
Abstract
We study the problems of distributed online and bandit convex optimization against an adaptive adversary.
Our goal is to minimize the average regret on M machines working in parallel over T rounds that can communicate R times intermittently.
Assuming the underlying cost functions are convex, our results show collaboration is not beneficial if the machines have access to the first-order gradient information at the queried points.
We show that in this setting, simple noncollaborative algorithms are min-max optimal, as opposed to the case for stochastic functions, where each machine samples the cost functions from a fixed distribution.
Next, we consider the more challenging setting of federated optimization with bandit (zeroth-order) feedback, where the machines can only access values of the cost functions at the queried points.
The key finding here is to identify the high-dimensional regime where collaboration is beneficial and may even lead to a linear speedup in the number of machines.
Our results are the first attempts towards bridging the gap between distributed online optimization against stochastic and adaptive adversaries.
1. Introduction
We consider the following distributed regret minimization problem on M machines with horizon T :
[Equation]
where f mt is a non-negative, convex cost function observed by machine m at time t and x mt is the model it plays.
This formulation captures distributed learning problems where the data is generated in real-time but isn’t stored, e.g., mobile keyboard prediction [11, 12] and self-driving vehicles [7, 20].
We want to solve this problem in the intermittent communication (IC) setting [29, 31] where the machines work in parallel and are allowed to communicate R times with K time steps in between communication rounds.
The IC setting captures the expensive nature of communication in collaborative learning, such as in cross-device federated learning [15, 17].
The IC setting has been widely studied over the past decade [1, 2, 4, 5, 23, 25, 27, 32–34].
Most existing works consider the “stochastic” setting where { f mt } ’s are sampled from a distribution specified in advance.
However, real-world applications may have distribution shifts, unmodeled perturbations, or even an adversarial sequence of cost functions, all of which violate the fixed distribution assumption.
To alleviate this issue, in this paper, we extend our understanding of distributed online optimization to “adaptive” adversaries that could potentially generate a worst-case sequence of cost functions.
Although some recent works have underlined the importance of the adaptive set- ting [3, 9, 10, 16, 18], our understanding of the optimal regret guarantees for problem (1) is still lacking.
We first show that, under usual assumptions, there is no benefit of collaboration if all the machines have access to the gradients, a.k.a.
first-order feedback for their cost functions.
Specifically, in this setting, running online gradient descent on each device without any communication is minmax optimal for problem (1).
Thus, we move to the harder setting of bandit convex optimization with two-point feedback.
We study a natural variant of F ED A VG equipped with a stochastic gradient estimator due to Shamir [22].
We show that collaboration reduces the variance of the stochastic gradient estimator and is thus beneficial for problems of high enough dimension.
We prove a linear speedup in the number of machines for high-dimensional problems, which mimics the stochastic setting [28, 31].
2. Setting
This section introduces notations, definitions, and assumptions used in our analysis.
Notation.
We denote the horizon by T = KR .
⪰ , ⪯ , ∼ = denote inequalities up to numerical constants.
We denote the average function by f t ( · ) :
= 1 M (cid:80) m ∈
[ M ] f mt ( · ) for all t ∈
[ T ] .
We use 1 A to denote the indicator function for the event A .
Our model space is denoted by X ⊆ R d .
We denote the expected averaged regret by Reg ( M, K, R ) in all the settings.
Function classes.
We consider two common [13, 21] function classes in this paper: (i) F G,B , the class of convex, differentiable, non-negative and G -Lipschitz functions, i.e., ∀ x, y ∈ X , | f
( x ) − f ( y ) |
≤ G ∥ x − y ∥ 2 , with bounded optima, i.e., ∥ x ⋆ ∥ 2 ≤ B, ∀ x ⋆ ∈ arg min x ∈X f ( x ) ; (ii)
F H,B , the class of convex, differentiable, non-negative and H -smooth functions, i.e., ∀ x, y ∈ X , ∥∇ f ( x ) − ∇ f ( y ) ∥ 2 ≤ H ∥ x − y ∥ 2 , with bounded optima.
F G,B includes linear cost functions, while F H,B consists of quadratic functions denoted by F G,Blin .
We also define F G,H,B := F G,B ∩ F H,B .
Adversary model.
Note that in the most general setting, each machine will face arbitrary functions from a class F at each time step.
Our algorithmic results are for this general model, which is usually referred to as an “adaptive” adversary.
We also consider a weaker “stochastic” adversary model to aid comparison.
More specifically, the adversary cannot adapt to the sequence of the models used by each machine but must fix a distribution in advance for each machine, i.e., ∀ m ∈
[ M ] , D m ∈ ∆( F ) such that at each time t ∈
[ T ] , f m t ∼ D m .
An example of this easier model is distributed stochastic optimization where f mt ( · ) :
= f ( · ; z mt ∼ D m ) ∈ F for f ( · ; · ) ∈ F .
Oracle model.
We consider two kinds of access to the cost functions in this paper.
Each machine m ∈
[ M ] for all time steps t ∈
[ T ] has access to one of the following: (i) gradient of f mt at a single point, a.k.a., first-order feedback; or (ii) function values of f mt at two different points, a.k.a., two-point bandit feedback.
We consider two more assumptions controlling how similar the cost functions look across machines and the average regret at the comparator [24]:
Assumption
[Equation]
[Equation] [Equation]
Assumption 2 ∀ x ⋆ ∈ arg min x ∈X (cid:80) t ∈ [ T ] f t ( x ) , 1 T (cid:80) t ∈
[ T ] f t ( x ⋆ ) ≤ F ⋆ .
For non-negative functions in F G,H,B , this implies 1 T (cid:80) t ∈
[ T ] ∥∇ f t ( x ⋆ ) ∥
22 ≤ HF ⋆ (c.f., Lemma 4.1 [24]).
Min-max regret.
We can finally define our problem class.
We use P M,K,R ( F ) :
= F ⊗ MKR to denote all selections of MKR functions from a class F .
We use the argument ζ , F ⋆ to further restrict this to selections that satisfy Assumptions 1 and 2 respectively.
Furthermore, with a slight abuse of notation, we use the superscript 1 to denote first-order feedback and (0 , 2) to denote twopoint zeroth-order feedback to the cost functions.
In this paper, we consider four problem classes:
P 1 M,K,R ( F G,D , ζ ) , P 0 , 2 M,K,R ( F G,B , ζ ) , P 1 M,K,R ( F H,B , ζ, F ⋆ ) , P 0 , 2 M,K,R ( F G,H,B , ζ, F ⋆ ) .
And we are interested in characterizing the min-max regret for these classes.
In particular, for a problem class
P , we want to characterize up to numerical constants the following quantity:
[Equation]
where A is a randomized algorithm producing models x mt ’s. For stochastic adversaries, the expectation is also taken over the randomness of sampling from the distributions D m ∈ ∆( F ) .
3. Collaboration doesn’t help with First-order Feedback
We first consider the class P 1 M,K,R ( F G,B , ζ ) .
Note the following bound is always true for any stream of functions and sequence of models:
[Equation]
This means we can upper bound regret in equation 1 by running online gradient descent (OGD) independently on each machine and not collaborating at all.
In other words:
[Equation]
The min-max rate for a single machine follows classical results using vanilla OGD (c.f., Theorem 3.1 in [13]).
But can collaborative algorithms beat this natural baseline?
No!
Consider the problem where the functions don’t vary across the machines but may change with time.
This problem satisfies Assumption 1 with ζ = 0 .
In this problem, the machines jointly see only T different functions but can make M first-order queries to the functions at each time step.
However, these additional queries are not useful as there is a known sample-complexity lower bound of GB/ √ T for P 11 ,K,R ( F G,B ) (c.f., Theorem 3.2 [13]) which holds for any number of first-order queries at each time step.
This implies that,
[Equation]
Combining equations (3) and (4), we conclude that R (cid:16) P 1 M,K,R ( F G,B , ζ ) (cid:17) ∼ = GB/ √ T .
Or in other words, there is no benefit of collaboration when the machines have first-order feedback.
We recall several interesting functions, such as quadratics, that don’t lie in F G,B but lie in
F H,B .
To understand the latter class we look at problems in P 1 M,K,R ( F H,B , ζ, F ⋆ ) .
In the single machine setting, we know that OGD incurs a regret of HB 2 /T + √ HF ⋆ B/ √ T (c.f., Theorem[24]).
This serves as the non-collaborative baseline.
Unfortunately, there is again a matching sample complexity lower bound for P 11 ,K,R ( F H,B , F ⋆ ) (c.f., Theorem 4 [28]).
Using a similar argument as before, we can obtain that,
[Equation]
which suggests that regret doesn’t improve with collaboration, either.
Thus, when the machines have first-order feedback for their own objectives, they do not benefit from collaboration.
The commonality between these problems is that even when the functions are the same across the machines, the hardest instances within the problem class do not benefit from the additional gradient accesses.
This is not surprising because linear functions are the hardest lipschitz and smooth functions in the adversarial online setting, and they are fully specified by their gradient.
This suggests that we should consider settings where machines have weaker oracles than first-order and may benefit through collaboration.
One such setting is with stochastic first-order oracles because, with additional stochastic gradients, the machines can reduce the variance of their gradient estimator.
This is one mechanism through which collaboration helps in the stochastic setting [28, 31], and we see next that it naturally arises in bandit convex optimization.
4. Online Local SGD Algorithm with Two-point Bandit Feedback
Algorithm 1: F ED OSGD ( η, δ ) with two-point bandit feedback
1 } do for m ∈
[ M ] in parallel do
Sample u mt ∼ Unif ( S d − 1 ) , i.e., a random unit vector Query function f mt at points ( x m, 1 t , x m, 2 t ) : = ( x mt + δu mt , x mt − δu mt ) Incur loss ( f mt ( x mt + δu mt ) + f mt ( x mt − δu mt ))
Compute stochastic gradient at point x mt as g mt = d ( f ( x mt + δu mt ) − f ( x mt − δu mt )) u mt 2 δ if ( t + 1) mod K = 0 then
Communicate to server: ( x mt − η · g mt ) On server x t +1 ← 1 M (cid:80) m ∈ [ M ] ( x mt − η · g mt ) Communicate to machine: x mt +1 ← x t +  else x mt +1 ← x [Equation]
mt
In this section, we study distributed bandit convex optimization with two-point feedback [6, 22], i.e., at each time step, the machines can query the value (and not the full gradient) of their cost functions at two different points.
We analyze the online variant of the F ED A VG or L OCALSGD algorithm, which is common in the stochastic setting.
We call the algorithm F ED OSGD and describe it in Algorithm 1.
In line 7, we use the stochastic gradient estimator, originally proposed by Shamir [22] and based on a similar estimator by Duchi et al.
For a smoothed version of the function ˆ f mt ( x ) :=
E u [ f mt ( x + δu )] , this estimator satisfies (c.f., Lemmas 3 and 5 [22]) for all t ∈
[ T ] , m ∈
[ M ] and x ∈ X ,
[Equation]
and
[Equation]
Equipped with this gradient estimator, we can prove the following guarantee for P 1 ,σM,K,R ( F G,B , ζ ) .
Theorem 1 Consider the problem class P 0 , 2 M,K,R ( F G,B , ζ ) .
With η = BG √ T · min (cid:110) 1 , [Equation] the queried points { x m,jt } T,M, 2 t,m,j =1 of Algorithm 1 satisfy:
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the choice of function queries.
When K = 1 , the above bound reduces to the first two terms, which are known to be tight for two-point bandit feedback [6, 13] (see Appendix A), making F ED OSGD optimal.
When K > 1 , we would like to compare our results to the non-collaborative baseline as we did in section 3.
Using the gradient estimator proposed by Shamir [22], the non-collaborative baseline gets regret O (cid:16) GB √ d/ √ KR (cid:17) .
Thus, as long as d ⪰ K 2 , F ED OSGD is better than the non-collaborative baseline.
Furthermore, if d ⪰ K 2 M 2 , then the second term in the upper bound dominates, and F ED OSGD gets a “linear speed-up” in the number of machines.
Unfortunately, the bound doesn’t improve with smaller ζ .
Note that the lipschitzness assumption is crucial to the two-point gradient estimator in algorithm 1.
While there are gradient estimators that don’t require lipschitzness or bounded gradients [8], they do require stronger assumptions such as bounded function values.
To avoid making these assumptions, we skip looking at the problems in P 0 , 2 M,K,R ( F H,B , ζ, F ⋆ ) and look at the problems in
P 0 , 2 M,K,R ( F G,H,B , ζ, F ⋆ ) .
Theorem
Consider the problem class P 0 , 2 M,K,R ( F G,H,B , ζ, F ⋆ ) .
With appropriate η (c.f., lemma 6), the queried points { x m,jt } T,M, 2 t,m,j =1 of Algorithm 1 satisfy:
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the choice of function queries.
The regret is also upper bounded as in theorem 1 for the corresponding step size.
The above result is a bit technical, so to interpret it, we consider the simpler class F G,Blin of linear functions with bounded gradients.
Linear functions are the “smoothest” Lipschitz functions as their smoothness constant H = 0 . We can get the following guarantee for this class:
Corollary 3 Consider the problem class P 0 , 2 M,K,R ( F G, 0 ,B lin , ζ, F ⋆ ) .
With appropriate η (c.f., lemma 6), the queried points { x m,jt } T,M, 2 t,m,j =1 of Algorithm 1 satisfy:
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the choice of function queries.
Unlike general Lipschitz functions, the last two terms are zero for linear functions when ζ =and the upper bound is smaller for smaller ζ .
In fact, when K = 1 or ζ = 0 , the upper bound is tight [6]. More generally, when K ≤ max(1 , G 2 ζ 2 d, G 2 d/ζ 2 M 2 ) then F ED OSGD is optimal.
Recall that in this setting, the non-collaborative baseline obtains a regret [24] of O ( GB √ d/ √ KR ) .
Thus, the benefit of collaboration through F ED OSGD again appears in high-dimensional problems in P 0 , 2 M,K,R ( F G,H,B , ζ, F ⋆ ) similar to what we discussed for P [Equation] , 2 M,K,R ( F G,B , ζ, F ⋆ ) .
5. Conclusion
In this paper, we show that, in the adaptive bandit setting, the benefit of collaboration is very similar to the stochastic setting, where the collaboration is useful when: (i)
There is stochasticity in the problem and (ii) The variance of the gradient estimators is “high” [31] and reduces with collaboration.
There are several open questions and directions: Does collaboration provably not help for the smaller class P 1 M,K,R ( F G,H,B , ζ, F ⋆ ) ?
This might require new proof techniques.
Is the final term tight in Theorems 1 and 2?
We don’t know any lower bounds in the intermittent communication setting against an adaptive adversary. Perhaps there is no gap between the stochastic and adaptive adversaries, and we can use existing techniques and online-to-batch conversion to provide a tight lower bound.
When K is large, but R is a fixed constant, the average regret of the non-collaborative baseline goes to zero, but our upper bounds for F ED OSGD don’t. It is unclear if our analysis is loose or if we need to modify the algorithm, for instance, add projection steps. 4.
How to obtain second-order methods in the distributed online setting, especially in the intermittent communication setting?
This only makes sense when the worst-case functions are not linear, which we might expect in the distributed setting [26].
For stochastic linear bandits, collaborative methods have been shown to attain optimal regret with very few rounds of communication [14].
What structures in the problem can we further exploit to reduce communication?
Acknowledgements
We thank the anonymous reviewers who helped us improve the writing of the paper.
This research was partly supported by NSF-BSF award 1718970, NSF TRIPOD IDEAL award, and the NSFSimons funded Collaboration on the Theoretical Foundations of Deep Learning.
Appendix A. Proof of Theorem
In this section and the next one, we consider access to a first-order stochastic oracle as an intermediate step before considering the zeroth-order oracle. Specifically, each machine has access to a stochastic gradient g mt of f mt at point x mt , such that it is unbiased and has bounded variance, i.e., for all x ∈ X , [Equation] In algorithm 1, we constructed a particular stochastic gradient estimator at x mt with σ 2 =
G 2 d .
We can define the corresponding problem class P 1 ,σM,K,R ( F G,B , ζ ) when the agents can access a stochastic first-order oracle.
We prove the following lemma about this problem class:
Lemma 4 Consider the problem class P 1 ,σM,K,R ( F G,B , ζ ) .
If we choose η = BG √ T · min (cid:26) 1 , G √ Mσ ,
[Equation]
, then the models { x mt } T,Mt,m =1 of Algorithm 1 satisfy the following guarantee:
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the stochastic gradients.
Proof Consider any time step t ∈
[ KR ] and define ghost iterate ¯ x t = 1 M (cid:80) m ∈
[ M ] x mt (which not might actually get computed).
If K = 1 , the machines calculate the stochastic gradient at the same point, ¯ x t .
Then using the update rule of Algorithm 1, we can get the following:
[Equation]
where E t is the expectation conditioned on the filtration at time t under which x mt ’s are measurable, and the last inequality is due to the convexity of each function.
Re-arranging this leads to
M (cid:88) m ∈
[ M ]
[Equation]
The last inequality comes from each function’s G -Lipschitzness.
For the last term in (6), we can upper bound it similar to lemma 8 in Woodworth et al.
[30] to get that
[Equation]
Plugging (7) into (6) and choosing a constant step-size η , and taking full expectation we get
[Equation]
Summing this over time t ∈ [ KR ] we get,
[Equation]
Finally choosing,
[Equation]
we can obtain,
[Equation]
Dividing by KR finishes the proof.
Remark 5 Note that when K = 1 , the upper bound in Lemma 4 reduces to the first two terms, both of which are known to be optimal due to lower bounds in the stochastic setting, i.e., against a stochastic online adversary [13, 19]. We now use this lemma to guarantee bandit two-point feedback oracles for the same function class. We recall that one can obtain a stochastic gradient for a “smoothed-version” ˆ f of a Lipschitz function f at any point x ∈ X , using two function value calls to f around the point x [6, 22]. With this lemma, we can prove Theorem 1.
Theorem 1 Consider the problem class P 0 , 2 M,K,R ( F G,B , ζ ) .
With η = BG √ T · min (cid:110) 1 , √ M √ d , 1 1 K> 1 √ Kd 1 /(cid:111) , the queried points { x m,jt } T,M, 2 t,m,j =1 of Algorithm 1 satisfy:
[Equation]
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the choice of function queries.
Proof First, we consider smoothed functions
[Equation]
for some δ > 0 and S d − 1 denoting the euclidean unit sphere.
Based on the gradient estimator in ( ?? ) proposed by Shamir [22] (which can be implemented with two-point bandit feedback) and Lemma 4, we can get the following regret guarantee (noting that σ ≤ c 1 √ dG for a numerical constant c 1 , c.f., [22]):
[Equation]
where the expectation is w.r.t.
the stochasticity in the stochastic gradient estimator.
To transform this into a regret guarantee for f we need to account for two things:
The difference between the smoothed function ˆ f and the original function f .
This is easy to handle because both these functions are pointwise close, i e., sup x ∈X | f ( x ) − ˆ f ( x ) | ≤ Gδ . 2.
The difference between the points ˆ x mt at which the stochastic gradient is computed for ˆ f mt and the actual points x m, 1 t and x m, 2 t on which we incur regret while making zeroth-order queries to f mt .
This is also easy to handle because due to the definition of the estimator in ??
, x m,
[Equation] , x m,
[Equation]
∈ B δ (ˆ x mt ) , where B δ ( x ) is the L 2 ball of radius δ around x .
In light of the last two observations, the average regret between the smoothed and original functions only differs by a factor of 2 Gδ , i.e.,
[Equation]
[Equation]
where the last inequality is due to the choice of δ such that δ ⪯ Dd 1 / 4 √ R (cid:16) 1 + d 1 / 4 √ MK (cid:17) .
Appendix B. Proof of Theorem
Similar to before, we start by looking at P 1 ,σM,K,R ( F G,H,B , ζ, F ⋆ ) .
We first prove the following Lemma:
Lemma 6 Consider the problem class P 1 ,σM,K,R ( F G,H,B , ζ, F ⋆ ) .
The models { x mt } T,Mt,m =1 of Algorithm 1 with appropriate η (specified in the proof) satisfy the following regret guarantee:
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t.
the stochastic gradients.
The models also satisfy the guarantee of lemma 4 with the same step-size.
Proof Consider any time step t ∈
[ KR ] and define ghost iterate ¯ x t = 1 M (cid:80) m ∈
[ M ] x mt (which not might actually get computed).
Then using the update rule of Algorithm
[Equation]
we can get:
[Equation]
[Equation]
where E t is the expectation taken with respect to the filtration at time t , and the last line comes from the convexity of each function.
Re-arranging this and taking expectation gives leads to
[Equation]
Bounding the blue term.
We consider two different ways to bound the term.
First note that similar to lemma 4 we can just use the following bound,
[Equation]
However, since we also have smoothness, we can use the self-bounding property (c.f., Lemma 4.
[24]) to get,
[Equation]
Bounding the red term.
We will bound the term in three different ways.
Similar to lemma 4, we can bound the term after taking expectation and then bounding the consensus term similar to Lemma 8 in Woodworth et al.
[30] as follows,
[Equation]
where τ ( t ) maps t to the last time step when communication happens.
Alternatively, we can use smoothness as follows after assuming η t ≤ 1 / 2 H , [Equation] [Equation] where we used lemma 8 from Woodworth et al.
[30] in the last inequality.
We can also use the lipschitzness and smoothness assumption together with a constant step size η < 1 / 2 H to obtain,
[Equation]
Combining everything.
After using a constant step-size η , summing (9) over time, we can use the upper bound of the red and blue terms in different ways. If we plug in (10) and (12) we recover the guarantee of lemma 4. This is not surprising because F G,H,B ⊆ F G,B . Combining the upper bounds in all other combinations assuming η < 12 H , we can show the following upper bound
[Equation]
where we used step size,
[Equation]
This finishes the proof
[Equation]
It is now straightforward to prove Theorem 2 similar to the proof for Theorem 1 by replacing σ 2 with G 2 d :
Theorem 2 Consider the problem class P 0 , 2 M,K,R ( F G,H,B , ζ, F ⋆ ) .
With appropriate η (c
[Equation]
f., lemma 6), the queried points { x m,jt } T,M, 2 t,m,j =1 of Algorithm 1 satisfy:
[Equation]
[Equation]
where x ⋆ ∈ arg min x ∈ R d (cid:80) t ∈ [ KR ] f t ( x ) , and the expectation is w.r.t. the choice of function queries.
The regret is also upper bounded as in theorem 1 for the corresponding step size.