Original: Between Stochastic and Adversarial Online Convex Optimization: Improved Regret Bounds via Smoothness
Revised: Between Stochastic and Adversarial Online Convex Optimization: Improved Regret Bounds via Smoothness

Original: Abstract
Revised: Abstract

Original: Stochastic and adversarial data are two widely studied settings in online learning.
Revised: Stochastic and adversarial data are two widely studied settings in online learning.

Original: But many optimization tasks are neither i.i.d.
Revised: But many optimization tasks are neither i.i.d.

Original: nor fully adversarial, which makes it of fundamental interest to get a better theoretical understanding of the world between these extremes.
Revised: nor fully adversarial, which makes it of fundamental interest to get a better theoretical understanding of the world between these extremes.

Original: In this work, we establish novel regret bounds for online convex optimization in a setting that interpolates between stochastic i.i.d.
Revised: In this work, we establish novel regret bounds for online convex optimization in a setting that interpolates between stochastic i.i.d.

Original: and fully adversarial losses.
Revised: and fully adversarial losses.

Original: By exploiting smoothness of the expected losses, these bounds replace dependence on the maximum gradient length by the variance of the gradients, which was previously known only for linear losses.
Revised: By exploiting smoothness of the expected losses, these bounds replace dependence on the maximum gradient length by the variance of the gradients, which was previously known only for linear losses.

Original: In addition, they weaken the i.i.d.
Revised: In addition, they weaken the i.i.d.

Original: assumption by allowing, for example, adversarially poisoned rounds, which were previously considered in the expert and bandit setting.
Revised: assumption by allowing, for example, adversarially poisoned rounds, which were previously considered in the expert and bandit setting.

Original: Our results extend this to the online convex optimization framework.
Revised: Our results extend this to the online convex optimization framework.

Original: In the fully i.i.d.
Revised: In the fully i.i.d.

Original: case, our bounds match the rates one would expect from results in stochastic acceleration, and in the fully adversarial case, they gracefully deteriorate to match the minimax regret.
Revised: case, our bounds match the rates one would expect from results in stochastic acceleration, and in the fully adversarial case, they gracefully deteriorate to match the minimax regret.

Original: We further provide lower bounds showing that our regret upper bounds are tight for all intermediate regimes in terms of the stochastic variance and the adversarial variation of the loss gradients.
Revised: We further provide lower bounds showing that our regret upper bounds are tight for all intermediate regimes in terms of the stochastic variance and the adversarial variation of the loss gradients.

Original:  Introduction
Revised:  Introduction

Original: Two of the main approaches for solving convex optimization problems under uncertain data are stochastic convex optimization (SCO) [21, 30] and online convex optimization (OCO) [37].
Revised: Two of the main approaches for solving convex optimization problems under uncertain data are stochastic convex optimization (SCO) [21, 30] and online convex optimization (OCO) [37].

Original: These two models are very different in their assumptions and goals, despite the fact that they share many techniques.
Revised: These two models are very different in their assumptions and goals, despite the fact that they share many techniques.

Original: In SCO it is assumed that the loss functions follow an independent, identically distributed (i.i.d.) process, and the goal is to minimize the excess risk , which is the optimization error under the expected loss.
Revised: In SCO it is assumed that the loss functions follow an independent, identically distributed (i.i.d.) process, and the goal is to minimize the excess risk , which is the optimization error under the expected loss.

Original: By contrast, in OCO the losses can be chosen adversarially and the goal is to minimize the cumulative regret , which is the difference between the cumulative incurred losses over rounds against the best ﬁxed strategy in hindsight.
Revised: By contrast, in OCO the losses can be chosen adversarially and the goal is to minimize the cumulative regret , which is the difference between the cumulative incurred losses over rounds against the best ﬁxed strategy in hindsight.

Original: Much less is known about what happens in between, in scenarios that interpolate between the i.i.d.
Revised: Much less is known about what happens in between, in scenarios that interpolate between the i.i.d.

Original: and adversarial settings.
Revised: and adversarial settings.

Original: This intermediate setting has drawn major attention in the recent years in the expert and bandit setting [14] [2] [36], however, as mentioned in [14], little is known for online convex optimization.
Revised: This intermediate setting has drawn major attention in the recent years in the expert and bandit setting [14] [2] [36], however, as mentioned in [14], little is known for online convex optimization.

Original: Our work studies this in a generalization of the OCO setting, in which nature chooses distributions for the data that may vary arbitrarily over time, and we provide regret bounds in terms of two quantities that measure how adversarial these distributions are.
Revised: Our work studies this in a generalization of the OCO setting, in which nature chooses distributions for the data that may vary arbitrarily over time, and we provide regret bounds in terms of two quantities that measure how adversarial these distributions are.

Original: The standard OCO setting corresponds to the case where the distributions are point-masses on adversarial data points.
Revised: The standard OCO setting corresponds to the case where the distributions are point-masses on adversarial data points.

Original: Main Contribution.
Revised: Main Contribution.

Original: Our main contribution is a new analysis of optimistic online algorithms [25, 26] that takes advantage of smoothness of the expected loss.
Revised: Our main contribution is a new analysis of optimistic online algorithms [25, 26] that takes advantage of smoothness of the expected loss.

Original: This analysis allows for a gradual interpolation between worst-case adversarial regret bounds and the best-known expected regret bounds in the stochastic case, and also provides quantiﬁable improvements for intermediate cases.
Revised: This analysis allows for a gradual interpolation between worst-case adversarial regret bounds and the best-known expected regret bounds in the stochastic case, and also provides quantiﬁable improvements for intermediate cases.

Original: To capture the full range between i.i.d.
Revised: To capture the full range between i.i.d.

Original: and fully adversarial settings, we consider a similar adversarial model as in [27], i.e., nature chooses distributions D t in iteration t , and the learner suffers loss f t ( x t , ξ t ) with ξ t ∼ D t .
Revised: and fully adversarial settings, we consider a similar adversarial model as in [27], i.e., nature chooses distributions D t in iteration t , and the learner suffers loss f t ( x t , ⇠ t ) with ⇠ t ⇠ D t .

Original: Importantly, we do not assume that the distributions D t are all the same, but they may vary adversarially over time.
Revised: Importantly, we do not assume that the distributions D t are all the same, but they may vary adversarially over time.

Original: To properly quantify the interpolation between the i.i.d.
Revised: To properly quantify the interpolation between the i.i.d.

Original: and fully adversarial settings in the regret bound, we introduce two parameters for the loss sequence.
Revised: and fully adversarial settings in the regret bound, we introduce two parameters for the loss sequence.

Original: Namely the cumulative variance , σ T , which captures the stochastic aspect of the learning task, i.e., the variance of the D t ; and the cumulative adversarial variation , Σ T , which captures the adversarial difﬁculties of the data, i.e., the difference between E ξ ∼ D t [ ∇ f ( · , ξ )] and E ξ ∼ D t − 1 [ ∇ f ( · , ξ )] .
Revised: Namely the cumulative variance ,  T , which captures the stochastic aspect of the learning task, i.e., the variance of the D t ; and the cumulative adversarial variation , ⌃ T , which captures the adversarial difﬁculties of the data, i.e., the difference between E ⇠ ⇠ D t [ r f ( · , ⇠ )] and E ⇠ ⇠ D t  1 [ r f ( · , ⇠ )] .

Original: With these two key quantities, our ﬁrst main result in Theorem 5 shows that the expected regret E [ R T ( u )] , that is the difference between the cumulative losses of the learner and a ﬁxed solution in hindsight, is bounded by
Revised: With these two key quantities, our ﬁrst main result in Theorem 5 shows that the expected regret E [ R T ( u )] , that is the difference between the cumulative losses of the learner and a ﬁxed solution in hindsight, is bounded by

Original: [Equation]
Revised: [Equation]

Original: where L is the smoothness constant of the expected functions F t =
Revised: where L is the smoothness constant of the expected functions F t =

Original: E ξ ∼ D t [ f ( · , ξ )] .
Revised: E ⇠ ⇠ D t [ f ( · , ⇠ )] .

Original: If, in addition, the functions F t are µ -strongly convex, then in Theorem 7 we obtain (below, κ :=
Revised: If, in addition, the functions F t are µ -strongly convex, then in Theorem 7 we obtain (below,  :=

Original: L/µ is the condition number)
Revised: L/µ is the condition number)

Original: [Equation]
Revised: [Equation]

Original: Both bounds are tight: we prove matching lower bounds in Theorems 6 and 8.
Revised: Both bounds are tight: we prove matching lower bounds in Theorems 6 and 8.

Original: In Section 3.
Revised: In Section 3.

Original: we show that our results match the known adversarial regret bounds as well as the best results in the i.i.d. case.
Revised: we show that our results match the known adversarial regret bounds as well as the best results in the i.i.d. case.

Original: For the latter, only the linear case was so far obtained directly via a regret analysis (see Sec.
Revised: For the latter, only the linear case was so far obtained directly via a regret analysis (see Sec.

Original: 5.2 in [25], and prior work [12, 4]).
Revised: 5.2 in [25], and prior work [12, 4]).

Original: Using optimistic mirror descent, they obtained the regret guarantee of R T ( u ) (cid:54) O 
Revised: Using optimistic mirror descent, they obtained the regret guarantee of R T ( u ) 6 O ( qP

Original: ( (cid:113)(cid:80) Tt [-=
 (cid:107)∇-] f ( x t , ξ t ) − m t (cid:107) 2 ) , where m t denotes an optimistic guess of the gradient that is chosen before round t .
Revised: Tt [-=
 (cid:107)∇-]=1 kr f ( x t , ⇠ t )  m t k 2 ) , where m t denotes an optimistic guess of the gradient that is chosen before round t .

Original: In the i.i.d.
Revised: In the i.i.d.

Original: case with the prediction m t = ∇ f ( x t − 1 , ξ t − 1 ) , this can be shown to imply that the expected regret is upper bounded by
Revised: case with the prediction m t = r f ( x t  1 , ⇠ t  1 ) , this can be shown to imply that the expected regret is upper bounded by

Original: E [ R T ( u )]
Revised: E [ R T ( u )]

Original: = O ( σ √ T + (cid:113)(cid:80)
Revised: = O (  p T + qP [Equation]

Original: Tt =1 E
Revised: [Equation] t

Original: [ (cid:107)∇ F t ( x t ) − ∇ F t ( x t − 1 ) (cid:107) 2 ]) , where σ denotes the variance of the stochastic gradients.
Revised: [Equation] x t )  [Equation] F t ( x t  1 ) k 2 ]) , where  denotes the variance of the stochastic gradients.

Original: This simpliﬁes to
Revised: This simpliﬁes to

Revised: E

Original: [Equation]
Revised: [Equation]

Original: for the i.i.d. case with linear functions ∇ F t , which is a special case of (1), because σ = ¯ σ T and ¯Σ T = 0 for i.i.d.
Revised: for the i.i.d. case with linear functions r F t , which is a special case of (1), because  = ¯  T and ¯ ⌃ T = 0 for i.i.d.

Original: losses, and ∇ F t ( x t ) 
Revised: losses, and r F t ( x t ) =

Original: = ∇ F t ( x t − 1 ) and L = 0 for linear functions.
Revised: r F t ( x t  1 ) and L = 0 for linear functions.

Original: It is not immediately obvious how to generalize this result to general convex functions with smoothness L > 0 , however.
Revised: It is not immediately obvious how to generalize this result to general convex functions with smoothness L > 0 , however.

Original: In this case, we can guess the appropriate regret bound based on known convergence results for stochastic accelerated gradient descent (SAGD) [17, 8]: if we knew in advance that the losses would be i.i.d.
Revised: In this case, we can guess the appropriate regret bound based on known convergence results for stochastic accelerated gradient descent (SAGD) [17, 8]: if we knew in advance that the losses would be i.i.d.

Original: and we did not care about computational efﬁciency, then we could run a new instance of SAGD for each round t .
Revised: and we did not care about computational efﬁciency, then we could run a new instance of SAGD for each round t .

Original: Summing the known rate for SAGD over t then gives E [ R T ( u )] (cid:54) O ( σ √ T + L ) (for more details see the batch-to-online conversion in Appendix D).
Revised: Summing the known rate for SAGD over t then gives E [ R T ( u )] 6 O (  p T + L ) (for more details see the batch-to-online conversion in Appendix D).

Original: This raises the question if a similar bound can also be obtained directly via a regret analysis, without assuming i.i.d.
Revised: This raises the question if a similar bound can also be obtained directly via a regret analysis, without assuming i.i.d.

Original: observations in advance.
Revised: observations in advance.

Original: This question is then answered by our result (1), which indeed reduces to this rate for general convex smooth i.i.d.
Revised: This question is then answered by our result (1), which indeed reduces to this rate for general convex smooth i.i.d.

Original: functions, matching the aforementioned bound up to constants.
Revised: functions, matching the aforementioned bound up to constants.

Original: We achieve this by using smoothness to bound E [ (cid:107)∇ F t ( x t ) − ∇ F t ( x t − 1 ) (cid:107) 2 ] (cid:54) L E [ (cid:107) x t − x t − 1 (cid:107) 2 ] , which can be canceled by a negative quadratic term that we obtain from an improved analysis of the regret.
Revised: We achieve this by using smoothness to bound E [ kr F t ( x t )  r F t ( x t  1 ) k 2 ] 6 L E [ k x t  x t  1 k 2 ] , which can be canceled by a negative quadratic term that we obtain from an improved analysis of the regret.

Original: The use of this negative term in the analysis dates back to [20], who used it to achieve an improved O (1 /T ) rate on the extra-gradient method.
Revised: The use of this negative term in the analysis dates back to [20], who used it to achieve an improved O (1 /T ) rate on the extra-gradient method.

Original: In addition to unifying the analysis of these two extreme cases and obtaining the best-known results via one algorithm (OFTRL (2) for convex functions and OFTRL on a surrogate loss (5) for strongly convex functions), our results give new insight for intermediate cases.
Revised: In addition to unifying the analysis of these two extreme cases and obtaining the best-known results via one algorithm (OFTRL (2) for convex functions and OFTRL on a surrogate loss (5) for strongly convex functions), our results give new insight for intermediate cases.

Original: Thus, as a second main contribution we shed light on a setting that is neither fully adversarial nor i.i.d.
Revised: Thus, as a second main contribution we shed light on a setting that is neither fully adversarial nor i.i.d.

Original: To illustrate this, we highlight some examples here, which received attention in the recent literature.
Revised: To illustrate this, we highlight some examples here, which received attention in the recent literature.

Original: Adversarial corruptions:
Revised: Adversarial corruptions:

Original: Consider i.i.d.
Revised: Consider i.i.d.

Original: functions with adversarial corruptions, as considered in the context of the expert and bandit settings in [14], [2].
Revised: functions with adversarial corruptions, as considered in the context of the expert and bandit settings in [14], [2].

Original: If the (cumulative) corruption level is bounded by a constant C , in [14] an expected regret bound of E [ R T ( u )]
Revised: If the (cumulative) corruption level is bounded by a constant C , in [14] an expected regret bound of E [ R T ( u )]

Original: =
Revised: =

Original: O
Revised: O

Original: ( R sT + (cid:112) CR sT ) was obtained, where R sT denotes the regret with respect to the uncorrupted data.
Revised: ( R sT + p CR sT ) was obtained, where R sT denotes the regret with respect to the uncorrupted data.

Original: In [14], the authors raised the question of whether it is possible to obtain regret bounds with a similar square-root dependence on the corruption level C for online convex optimization.
Revised: In [14], the authors raised the question of whether it is possible to obtain regret bounds with a similar square-root dependence on the corruption level C for online convex optimization.

Original: Indeed, for this intermediate model, we derive a regret bound
Revised: Indeed, for this intermediate model, we derive a regret bound

Original: [Equation]
Revised: [Equation]

Original: for the general convex case from our Theorem 5.
Revised: for the general convex case from our Theorem 5.

Original: We elaborate on this in Section 4.2.
Revised: We elaborate on this in Section 4.2.

Original: Random order models The random order model (ROM) dates back to [18] in combinatorial online learning.
Revised: The random order model (ROM) dates back to [18] in combinatorial online learning.

Original: It has drawn attention in the online convex optimization community as an elegant relaxation of the adversarial model [31, 6].
Revised: It has drawn attention in the online convex optimization community as an elegant relaxation of the adversarial model [31, 6].

Original: Complementary to the results in [31], we show that the dependence on G in the regret bound can be reduced to a dependence of σ , where σ denotes the variance of gradients in the uniform distribution over loss functions f 1 , . . .
Revised: Complementary to the results in [31], we show that the dependence on G in the regret bound can be reduced to a dependence of  , where  denotes the variance of gradients in the uniform distribution over loss functions f 1 , . . .

Original: f T .
Revised: f T .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) σ denotes a slightly weaker notion of variance (see Corollary 10).
Revised: where e  denotes a slightly weaker notion of variance (see Corollary 10).

Original: We derive these results from our main theorem under stronger assumptions than in [31], namely, we require individual convexity of the adversarial choices whereas [31] only assume cumulative convexity.
Revised: We derive these results from our main theorem under stronger assumptions than in [31], namely, we require individual convexity of the adversarial choices whereas [31] only assume cumulative convexity.

Original: However, we also obtain a better rate with σ instead of G as the leading factor, so the results are not directly comparable.
Revised: However, we also obtain a better rate with  instead of G as the leading factor, so the results are not directly comparable.

Original: We also consider a variant of the random order model, which we call the multiple pass random order model (multi-pass ROM) .
Revised: We also consider a variant of the random order model, which we call the multiple pass random order model (multi-pass ROM) .

Original: This is inspired by multiple shufﬂe SGD and can be considered another intermediate example between adversarial and stochastic data.
Revised: This is inspired by multiple shufﬂe SGD and can be considered another intermediate example between adversarial and stochastic data.

Original: We elaborate on both examples, i.e., the ROM and multi-pass ROM in Section 4.3.
Revised: We elaborate on both examples, i.e., the ROM and multi-pass ROM in Section 4.3.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: As mentioned in the previous section, our work is inspired by results in the gradual variation and in the stochastic approximation literature.
Revised: As mentioned in the previous section, our work is inspired by results in the gradual variation and in the stochastic approximation literature.

Original: The gradual variation literature dates back to [12], with later extensions by [4] and [25, 26].
Revised: The gradual variation literature dates back to [12], with later extensions by [4] and [25, 26].

Original: In addition to some technical relation to the aforementioned work, there is also a natural relation between our parameters ¯ σ T and ¯Σ T and variational parameters in [33], [12] or [4].
Revised: In addition to some technical relation to the aforementioned work, there is also a natural relation between our parameters ¯  T and ¯ ⌃ T and variational parameters in [33], [12] or [4].

Original: However, as we elaborate in Remark 4, there are some fundamental differences between these variational parameters and ¯ σ T , ¯Σ T , which prevent us from directly obtaining a smooth interpolation from these results.
Revised: However, as we elaborate in Remark 4, there are some fundamental differences between these variational parameters and ¯  T , ¯ ⌃ T , which prevent us from directly obtaining a smooth interpolation from these results.

Original: It is also interesting to note that there is a relation between ¯Σ T and the path length parameters considered in dynamic regret bounds [35, 34].
Revised: It is also interesting to note that there is a relation between ¯ ⌃ T and the path length parameters considered in dynamic regret bounds [35, 34].

Original: However, since their analysis targets a fundamentally different notion of regret, namely dynamic regret, the results are incomparable.
Revised: However, since their analysis targets a fundamentally different notion of regret, namely dynamic regret, the results are incomparable.

Original: With respect to the results, our ﬁndings are fundamentally different from the stochastic approximation literature, since we do not rely on the assumption that the data is following a distribution.
Revised: With respect to the results, our ﬁndings are fundamentally different from the stochastic approximation literature, since we do not rely on the assumption that the data is following a distribution.

Original: However, we were inspired by analysis techniques and the convergence thresholds set by this literature.
Revised: However, we were inspired by analysis techniques and the convergence thresholds set by this literature.

Original: Our approach of obtaining accelerated rates by negative terms arising from smoothness in a regret bound has previously been used in the context of variational inequalities and saddle-point problems.
Revised: Our approach of obtaining accelerated rates by negative terms arising from smoothness in a regret bound has previously been used in the context of variational inequalities and saddle-point problems.

Original: Using this idea, [20] obtained improved rates O (1 /T ) for the extra-gradient method.
Revised: Using this idea, [20] obtained improved rates O (1 /T ) for the extra-gradient method.

Original: More recently [17] showed that acceleration in stochastic convex optimization can beneﬁt from negative terms arising in optimistic FTRL via an anytime-online-to-batch conversion [5].
Revised: More recently [17] showed that acceleration in stochastic convex optimization can beneﬁt from negative terms arising in optimistic FTRL via an anytime-online-to-batch conversion [5].

Original: Although an important inspiration for our approach, the techniques of [17] do not directly carry over, because they evaluate gradients at the time-average of the algorithm’s iterates, making them much more stable than the last iterate, which comes up when controlling the regret.
Revised: Although an important inspiration for our approach, the techniques of [17] do not directly carry over, because they evaluate gradients at the time-average of the algorithm’s iterates, making them much more stable than the last iterate, which comes up when controlling the regret.

Original: Algorithms used both in SCO and OCO follow a vast literature on stochastic approximation methods, e.g. [28, 21, 24].
Revised: Algorithms used both in SCO and OCO follow a vast literature on stochastic approximation methods, e.g. [28, 21, 24].

Original: For this work, we are particularly interested in the more recent literature on acceleration in SCO [8, 15, 17].
Revised: For this work, we are particularly interested in the more recent literature on acceleration in SCO [8, 15, 17].

Original: In this research ﬁeld efﬁciency is traditionally measured in terms of excess risk.
Revised: In this research ﬁeld efﬁciency is traditionally measured in terms of excess risk.

Original: On the one hand, regret upper bounds can be converted into excess risk bounds, through the so-called online-to-batch conversions [3]; on the other hand, excess risk guarantees do not directly lead to regret bounds, and even if they do some key features of the rates might be lost.
Revised: On the one hand, regret upper bounds can be converted into excess risk bounds, through the so-called online-to-batch conversions [3]; on the other hand, excess risk guarantees do not directly lead to regret bounds, and even if they do some key features of the rates might be lost.

Original: These latter methods, known as batch-to-online conversions are discussed in Appendix D.
Revised: These latter methods, known as batch-to-online conversions are discussed in Appendix D.

Original: Outline In Section 2, after setting up notation and basic deﬁnitions, we introduce the stochastically extended adversarial model, a generalization of the standard adversarial model similar to the model used in smoothed analysis .
Revised: Outline In Section 2, after setting up notation and basic deﬁnitions, we introduce the stochastically extended adversarial model, a generalization of the standard adversarial model similar to the model used in smoothed analysis .

Original: Our main results can be found in Section 3.
Revised: Our main results can be found in Section 3.

Original: In Section 4 we illustrate our results by highlighting several special cases, such as the random order model and the adversarially corrupted stochastic model.
Revised: In Section 4 we illustrate our results by highlighting several special cases, such as the random order model and the adversarially corrupted stochastic model.

Original: Finally, in Section 5 we set our ﬁndings into a broader context and give perspective for future work.
Revised: Finally, in Section 5 we set our ﬁndings into a broader context and give perspective for future work.

Original:  Setting
Revised:  Setting

Original: We recall the online convex optimization (OCO) problem.
Revised: We recall the online convex optimization (OCO) problem.

Original: Here, we consider a sequence of convex functions f 1 , . . .
Revised: Here, we consider a sequence of convex functions f 1 , . . .

Original: f T deﬁned over a closed and bounded convex set X ⊆ R d , which become available to the learner sequentially.
Revised: f T deﬁned over a closed and bounded convex set X ✓ R d , which become available to the learner sequentially.

Original: In the standard adversarial model , the learner chooses x t ∈ X in round t , then function f t is revealed and the learner suffers loss f t ( x t ) .
Revised: In the standard adversarial model , the learner chooses x t 2 X in round t , then function f t is revealed and the learner suffers loss f t ( x t ) .

Original: The success of the learner is measured against all ﬁxed u ∈ X .
Revised: The success of the learner is measured against all ﬁxed u 2 X .

Original: Hence, the goal of the learner is to minimize the regret , that is, the difference between their cumulative loss(cid:80)
Revised: Hence, the goal of the learner is to minimize the regret , that is, the difference between their cumulative loss

Original:  Tt =1 f t ( x t ) and that of the best ﬁxed choice in hindsight, namely min u ∈ X(cid:80)
Revised: P Tt =1 f t ( x t ) and that of the best ﬁxed choice in hindsight, namely min u 2 X

Revised: P

Original: Tt =1 f t ( u ) .
Revised: Tt =1 f t ( u ) .

Original: Throughout the paper we use the notation [ T ] = { 1 , . .
Revised: Throughout the paper we use the notation [ T ] = { 1 , . .

Original: , T } .
Revised: , T } .

Original: We follow the convention that δ c denotes a Dirac measure at a point c , and (cid:107) · (cid:107) denotes the Euclidean norm.
Revised: We follow the convention that  c denotes a Dirac measure at a point c , and k · k denotes the Euclidean norm.

Original: 2.1 Stochastically Extended Adversarial Model
Revised: 2.1 Stochastically Extended Adversarial Model

Original: We extend the aforementioned adversarial model by letting nature choose a distribution D t from a set of distributions.
Revised: We extend the aforementioned adversarial model by letting nature choose a distribution D t from a set of distributions.

Original: Then the learner suffers loss f ( x t , ξ t ) where ξ t ∼ D t .
Revised: Then the learner suffers loss f ( x t , ⇠ t ) where ⇠ t ⇠ D t .

Original: Note that if the set of distributions is sufﬁciently rich, this model contains the standard adversarial model and the stochastic model as special cases (see Examples 1,2).
Revised: Note that if the set of distributions is sufﬁciently rich, this model contains the standard adversarial model and the stochastic model as special cases (see Examples 1,2).

Original: We introduce some notation to make this more precise.
Revised: We introduce some notation to make this more precise.

Original: Let X ⊂ R d be a closed convex set and Ξ a measurable space.
Revised: Let X ⇢ R d be a closed convex set and ⌅ a measurable space.

Original: Deﬁne f : X × Ξ → R and assume f ( · , ξ ) is convex over ξ ∈ Ξ .
Revised: Deﬁne f : X ⇥ ⌅ ! R and assume f ( · , ⇠ ) is convex over ⇠ 2 ⌅ .

Original: Suppose D is a set of probability distributions over Ξ .
Revised: Suppose D is a set of probability distributions over ⌅ .

Original: For any D ∈ D , we denote the gradient mean by ∇ F D ( x ) :=
Revised: For any D 2 D , we denote the gradient mean by r F D ( x ) :=

Original: E ξ ∼ D [ ∇ f ( x, ξ )] and the function mean by F D ( x ) :=
Revised: E ⇠ ⇠ D [ r f ( x, ⇠ )] and the function mean by F D ( x ) :=

Original: E ξ ∼ D [ f ( x, ξ )] .
Revised: E ⇠ ⇠ D [ f ( x, ⇠ )] .

Original: Furthermore, denote by σ 2 D an upper bound on the variance of the gradients
Revised: Furthermore, denote by  2 D an upper bound on the variance of the gradients

Original: [Equation]
Revised: [Equation]

Original: We introduce some shorthand notation when distributions are indexed by rounds.
Revised: We introduce some shorthand notation when distributions are indexed by rounds.

Original: Given t ∈ [ T ] , we write F t and σ 2 t instead of F D t ( x ) and σ 2 D t , respectively.
Revised: Given t 2 [ T ] , we write F t and  2 t instead of F D t ( x ) and  2 D t , respectively.

Original: Let us now introduce the stochastically extended adversary protocol .
Revised: Let us now introduce the stochastically extended adversary protocol .

Original: Deﬁnition 1 (Stochastically Extended Adversary (SEA)) .
Revised: Deﬁnition 1 (Stochastically Extended Adversary (SEA)) .

Original: In each round t , the learner chooses x t ∈ X , the SEA picks D t ∈ D .
Revised: In each round t , the learner chooses x t 2 X , the SEA picks D t 2 D .

Original: The learner and the SEA then both observe a sample ξ t ∼ D t , and the learner suffers loss f ( x t , ξ t ) .
Revised: The learner and the SEA then both observe a sample ⇠ t ⇠ D t , and the learner suffers loss f ( x t , ⇠ t ) .

Original: We note that the SEA model is closely related to the adversarial model considered in the context of smoothed analysis [27, 10, 32].
Revised: We note that the SEA model is closely related to the adversarial model considered in the context of smoothed analysis [27, 10, 32].

Original: However, in contrast to this line of work, we do not focus on distributions with sufﬁcient anti-concentration (c.f., Def. 1.1 in [10]).
Revised: However, in contrast to this line of work, we do not focus on distributions with sufﬁcient anti-concentration (c.f., Def. 1.1 in [10]).

Original: Indeed, this restriction would exclude, among others, the fully adversarial case as described below.
Revised: Indeed, this restriction would exclude, among others, the fully adversarial case as described below.

Original: Note also that we assume that the SEA is an adaptive adversary, which has access to the realizations ξ t , so the distribution D t +1 may depend on ξ 1 , . . . , ξ t .
Revised: Note also that we assume that the SEA is an adaptive adversary, which has access to the realizations ⇠ t , so the distribution D t +1 may depend on ⇠ 1 , . . . , ⇠ t .

Original: This assumption is not relevant for the fully adversarial or the i.i.d.
Revised: This assumption is not relevant for the fully adversarial or the i.i.d.

Original: settings.
Revised: settings.

Original: In the former, because there is no randomness, and in the latter, because there is no change in distribution.
Revised: In the former, because there is no randomness, and in the latter, because there is no change in distribution.

Original: However, it is important for some of the intermediate cases, and in particular for the random order model.
Revised: However, it is important for some of the intermediate cases, and in particular for the random order model.

Original: The SEA model contains several common settings from the literature as special cases.
Revised: The SEA model contains several common settings from the literature as special cases.

Original: To illustrate this, we list some examples.
Revised: To illustrate this, we list some examples.

Original: Adversarial Model: The SEA chooses a Dirac measure δ c t ∈ D in each round.
Revised: Adversarial Model: The SEA chooses a Dirac measure  c t 2 D in each round.

Original: Then for any ξ t ∼ δ c t , the SEA selects f ( · , ξ t ) , and the model reduces to an adversary selecting directly the functions f t ( · ) .
Revised: Then for any ⇠ t ⇠  c t , the SEA selects f ( · , ⇠ t ) , and the model reduces to an adversary selecting directly the functions f t ( · ) .

Original: Stochastic I.I.D. Model: The SEA chooses a ﬁxed D ∈ D and selects D t
Revised: Stochastic I.I.D. Model: The SEA chooses a ﬁxed D 2 D and selects D t

Original: =
Revised: =

Original: D at each round t .
Revised: D at each round t .

Original: Adversarially Corrupted I.I.D. Model: The adversary selects an i.i.d.
Revised: Adversarially Corrupted I.I.D. Model: The adversary selects an i.i.d.

Original: source D and perturbs the data with adversarial corruptions.
Revised: source D and perturbs the data with adversarial corruptions.

Original: This ﬁts in our framework as follows: given a corruption level C (cid:62) 0 , the SEA chooses distributions D t
Revised: This ﬁts in our framework as follows: given a corruption level C > 0 , the SEA chooses distributions D t

Original: = D ⊗ δ c t such that 
Revised: = D ⌦  c t such that P

Original: [Equation]
Revised: [Equation]

Original: Random Order Model (ROM): Among a ﬁxed family of losses F = ( f i , i ∈ [ n ]) , the SEA randomly picks functions in F via sampling without replacement, possibly performing multiple passes over the losses and reshufﬂing between the passes.
Revised: Random Order Model (ROM): Among a ﬁxed family of losses F = ( f i , i 2 [ n ]) , the SEA randomly picks functions in F via sampling without replacement, possibly performing multiple passes over the losses and reshufﬂing between the passes.

Original: Formally, deﬁne Ξ = [ n ] , and f ( x, ξ ) =
Revised: Formally, deﬁne ⌅ = [ n ] , and f ( x, ⇠ ) =

Original: f ξ for ξ ∈ Ξ .
Revised: f ⇠ for ⇠ 2 ⌅ .

Original: If t ∈ [ nk, n ( k + 1)] is in epoch k ∈ N , then the SEA chooses the distribution D t = Unif(Ξ \ { ξ s : s ∈ [ nk + 1 , n ( t − 1)] } ) , which is uniform on the functions that have not yet been sampled in epoch k .
Revised: If t 2 [ nk, n ( k + 1) ] is in epoch k 2 N , then the SEA chooses the distribution D t = Unif( ⌅ \ { ⇠ s : s 2 [ nk + 1 , n ( t  1)] } ) , which is uniform on the functions that have not yet been sampled in epoch k .

Original: To quantify the hardness of the loss sequence, we introduce the cumulative stochastic variance and adversarial variation ; we also deﬁne an average of these quantities.
Revised: To quantify the hardness of the loss sequence, we introduce the cumulative stochastic variance and adversarial variation ; we also deﬁne an average of these quantities.

Original: We denote by E the expectation taken with respect to the joint distribution of ( x 1 , ξ 1 , . . .
Revised: We denote by E the expectation taken with respect to the joint distribution of ( x 1 , ⇠ 1 , . . .

Original: , x T , ξ T ) .
Revised: , x T , ⇠ T ) .

Original: Note that the choice of the adversary D t can be random itself, as it depends on the past observations (of both the player’s actions and the realizations of the ξ t ’s).
Revised: Note that the choice of the adversary D t can be random itself, as it depends on the past observations (of both the player’s actions and the realizations of the ⇠ t ’s).

Original: In this case, σ t is also a random quantity.
Revised: In this case,  t is also a random quantity.

Original: Deﬁnition 2 (Cumulative Stochastic Variance and Cumulative Adversarial Variation) .
Revised: Deﬁnition 2 (Cumulative Stochastic Variance and Cumulative Adversarial Variation) .

Original: Suppose the
Revised: Suppose the

Original: SEA chooses distributions D 1 , . .
Revised: SEA chooses distributions D 1 , . .

Original: , D T .
Revised: , D T .

Original: Recall that σ 2 t is a shorthand for σ 2 D t .
Revised: Recall that  2 t is a shorthand for  2 D t .

Original: The cumulative stochastic variance and the cumulative adversarial variance are deﬁned as 
Revised: The cumulative stochastic variance and the cumulative adversarial variance are deﬁned as  (2)[1: T ] =

Original: [Equation]
Revised: E  T

Original:  and 
Revised: X t =  2 t  and ⌃ (2)[1: T ] = E  T X t = sup x 2 X  r F t ( x )

Original: [Equation]
Revised: [Equation]

Revised:  1 ( x )   .

Original: We also let ¯ σ T and ¯Σ T denote the square root of the average stochastic variance or adversarial variation, respectively; that is, ¯ σ 2 T =
Revised: We also let ¯  T and ¯ ⌃ T denote the square root of the average stochastic variance or adversarial variation, respectively; that is, ¯  2 T =

Original: σ (2)[1: T ] /T and ¯Σ 2 T = Σ (2)[1: T ] /T.
Revised:  (2)[1: T ] /T and ¯ ⌃ 2 T = ⌃ (2)[1: T ] /T.

Original: Note that in the special case when all f t are fully adversarial, ¯ σ T = 0 .
Revised: Note that in the special case when all f t are fully adversarial, ¯  T = 0 .

Original: On the contrary, in the stochastic case, i.e., if all for each round t , the distribution D t is equal to a ﬁxed (but arbitrarily chosen) D , then ¯Σ T = 0 .
Revised: On the contrary, in the stochastic case, i.e., if all for each round t , the distribution D t is equal to a ﬁxed (but arbitrarily chosen) D , then ¯ ⌃ T = 0 .

Original: In this case, ¯ σ T reduces to the common deﬁnition of the gradient variance upper bound in the SCO literature [7, 8].
Revised: In this case, ¯  T reduces to the common deﬁnition of the gradient variance upper bound in the SCO literature [7, 8].

Original: If however, the SEA chooses one distribution D i for the ﬁrst rounds and then switches to a different distribution D j , then ¯ σ T can only be upper bounded by max( σ i , σ j ) .
Revised: If however, the SEA chooses one distribution D i for the ﬁrst rounds and then switches to a different distribution D j , then ¯  T can only be upper bounded by max(  i ,  j ) .

Original: This upper bound can be pessimistic, however, for some results it gives a better intuition.
Revised: This upper bound can be pessimistic, however, for some results it gives a better intuition.

Original: For this purpose we also deﬁne the maximal stochastic variance and maximal adversarial variation .
Revised: For this purpose we also deﬁne the maximal stochastic variance and maximal adversarial variation .

Original: Deﬁnition 3 (Maximal Stochastic Variance and Maximal Adversarial Variation) .
Revised: Deﬁnition 3 (Maximal Stochastic Variance and Maximal Adversarial Variation) .

Original: Let σ 2max be an upper bound on all variances σ 2 t .
Revised: Let  2max be an upper bound on all variances  2 t .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Remark 4.
Revised: Remark 4.

Original: As we mentioned in the introduction, the cumulative stochastic variance and the adversarial variation have some similarities with parameters in gradual variation regret bounds.
Revised: As we mentioned in the introduction, the cumulative stochastic variance and the adversarial variation have some similarities with parameters in gradual variation regret bounds.

Original: For linear functions (cid:104) µ t , ·(cid:105) , the bounds in [12] involve the parameter Var T
Revised: For linear functions h µ t , · i , the bounds in [12] involve the parameter Var T

Original: = (cid:80) Tt =1 (cid:107) µ t − ¯ µ T (cid:107) 2 where ¯ µ T is the average of the gradients.
Revised: [Equation] the average of the gradients.

Original: For OCO with general convex functions, [4] provide upper bounds on the regret in terms of the L p -deviation D p = (cid:80)
Revised: For OCO with general convex functions, [4] provide upper bounds on the regret in terms of the L p -deviation D p = P

Original: Tt =1 sup x ∈ X (cid:107)∇ f t ( x ) − ∇ f t − 1 ( x ) (cid:107) 2 p .
Revised: Tt =1 sup x 2 X kr f t ( x )  r f t  1 ( x ) k 2 p .

Original: In Lemmasand 14 in Appendix A, we show that in the SEA framework, both of these types of bounds are generally worse than ours, and that the difference can be arbitrarily large.
Revised: In Lemmas 13 and 14 in Appendix A, we show that in the SEA framework, both of these types of bounds are generally worse than ours, and that the difference can be arbitrarily large.

Original: In [26] the regret is bounded in terms of (cid:80) Tt =1 (cid:107) g t − M t (cid:107) 2 .
Revised: In [26] the regret is bounded in terms of P Tt =1 k g t  M t k 2 .

Original: As mentioned in the introduction, unless the loss functions are linear or the learner has knowledge of the gradient mean,(cid:80) Tt =1 (cid:107) g t − M t
Revised: As mentioned in the introduction, unless the loss functions are linear or the learner has knowledge of the gradient mean,

Original: (cid:107)cannot directly be reduced to σ (2)[1: T ] or Σ (2)[1: T ] .
Revised: P Tt =1 k g t  M t k 2 cannot directly be reduced to  (2)[1: T ] or ⌃ (2)[1: T ] .

Original: 2.2 Assumptions
Revised: 2.2 Assumptions

Original: In our analysis, we will frequently use several of the following additional assumptions.
Revised: In our analysis, we will frequently use several of the following additional assumptions.

Original: Some of these were already mentioned in the introduction.
Revised: Some of these were already mentioned in the introduction.

Original: We keep them all together here, for the convenience of the reader and for clear reference.
Revised: We keep them all together here, for the convenience of the reader and for clear reference.

Original: For any D ∈ D :
Revised: For any D 2 D :

Original: (A0)
Revised: (A0)

Original: the adversary has access to independent samples ξ ∼ D .
Revised: the adversary has access to independent samples ⇠ ⇠ D .

Original: (A1) the function f ( · , ξ ) is convex, and gradients are bounded by G a.s. when ξ ∼ D .
Revised: (A1) the function f ( · , ⇠ ) is convex, and gradients are bounded by G a.s. when ⇠ ⇠ D .

Original: (A2) the expected function F D is L -smooth, i.e, ∇ F D is L -Lipschitz continuous.
Revised: (A2) the expected function F D is L -smooth, i.e, r F D is L -Lipschitz continuous.

Original: (A3) for any x ∈ X , the variance E ξ ∼ D (cid:107)∇ f ( x, ξ ) − ∇ F D ( x ) (cid:107) 2 ] is ﬁnite.
Revised: (A3) for any x 2 X , the variance E ⇠ ⇠ D [ kr f ( x, ⇠ )  r F D ( x ) k 2 ] is ﬁnite.

Original: (A4) the expected function F D ( · ) is µ -strongly convex.
Revised: (A4) the expected function F D ( · ) is µ -strongly convex.

Original: We assume that (A0) always holds.
Revised: We assume that (A0) always holds.

Original: Assumptions (A1) , (A2) and (A3) are standard in stochastic optimization, and are similar to common assumptions for online convex optimization.
Revised: Assumptions (A1) , (A2) and (A3) are standard in stochastic optimization, and are similar to common assumptions for online convex optimization.

Original: There, it is typically assumed that the adversarial samples f t ( · ) are convex (or even linear) and the gradient norms (cid:107)∇ f t ( · ) (cid:107) are bounded.
Revised: There, it is typically assumed that the adversarial samples f t ( · ) are convex (or even linear) and the gradient norms kr f t ( · ) k are bounded.

Original: Note that we only require gradient Lipschitz continuity and strong convexity to hold for the expected loss.
Revised: Note that we only require gradient Lipschitz continuity and strong convexity to hold for the expected loss.

Original:  Algorithms and Regret Bounds 3.1 Convex Smooth Functions
Revised:  Algorithms and Regret Bounds 3.1 Convex Smooth Functions

Original: We use Optimistic Follow-the-Regularised-Leader (OFTRL)
Revised: We use Optimistic Follow-the-Regularised-Leader (OFTRL)

Original: (see, e.g., [16, 25]) to minimize regret.
Revised: (see, e.g., [16, 25]) to minimize regret.

Original: Let ( η t ) t ∈ [ T ] be a non-decreasing and positive sequence of stepsizes, possibly tuned adaptively with the observations.
Revised: Let ( ⌘ t ) t 2 [ T ] be a non-decreasing and positive sequence of stepsizes, possibly tuned adaptively with the observations.

Original: At each step t , the learner makes an optimistic prediction M t ∈ R d and updates its iterates as
Revised: At each step t , the learner makes an optimistic prediction M t 2 R d and updates its iterates as

Original: [Equation]
Revised: [Equation]

Original: where we denoted by g t = ∇ f ( x t , ξ t ) the observed gradient at time t .
Revised: where we denoted by g t = r f ( x t , ⇠ t ) the observed gradient at time t .

Original: To state our results, we denote by E [ · ] the expectation with respect to the joint distribution of ( x 1 , ξ 1 , . . .
Revised: To state our results, we denote by E [ · ] the expectation with respect to the joint distribution of ( x 1 , ⇠ 1 , . . .

Original: , x T , ξ T ) .
Revised: , x T , ⇠ T ) .

Original: Our objective is to bound the average regret:
Revised: Our objective is to bound the average regret:

Original: [Equation]
Revised: [Equation]

Original: The following theorem, proved in Appendix B.1, is our main result for convex functions.
Revised: The following theorem, proved in Appendix B.1, is our main result for convex functions.

Original: Theorem 5.
Revised: Theorem 5.

Original: Fix a user-speciﬁed parameter ν > 0 .
Revised: Fix a user-speciﬁed parameter ⌫ > 0 .

Original: Under assumptions (A1) , (A2) , (A3) , OFTRL, with M t = g t − 1 and adaptive step-size η t =
Revised: Under assumptions (A1) , (A2) , (A3) , OFTRL, with M t = g t  1 and adaptive step-size ⌘ t =

Original: D 2 / ( ν + (cid:80) t − 1 s =1 η s (cid:107) g s − M s (cid:107) 2 ) , has regret
Revised: D 2 / ( ⌫ + P t  1 s =1 ⌘ s k g s  M s k 2 ) , has regret

Original: [Equation]
Revised: [Equation]

Original: The algorithm needs to know only D .
Revised: The algorithm needs to know only D .

Original: If G and L are also known, one can tune ν = LD 2 + D 2 G 2 to get
Revised: If G and L are also known, one can tune ⌫ = LD 2 + D 2 G 2 to get

Original: [Equation]
Revised: [Equation]

Original: Moreover, if only convexity of the individual losses holds (A1) , then tuning ν = 2 DG ensures the (deterministic) bound R T ( u ) (cid:54) 3 √ 2 DG √ T + 4 DG .
Revised: Moreover, if only convexity of the individual losses holds (A1) , then tuning ⌫ = 2 DG ensures the (deterministic) bound R T ( u ) 6 3 p 2 DG p T + 4 DG .

Original: Without prior knowledge of the smoothness parameter, the best the player can do is to tune ν according to a guessed value L 0 .
Revised: Without prior knowledge of the smoothness parameter, the best the player can do is to tune ⌫ according to a guessed value L 0 .

Original: This affects the constants in the bound by an additive term of order ( L 0 + L 2 /L 0 ) D 2 ; it would be interesting to determine if this is an inevitable price to pay for the lack of knowledge of L .
Revised: This affects the constants in the bound by an additive term of order ( L 0 + L 2 /L 0 ) D 2 ; it would be interesting to determine if this is an inevitable price to pay for the lack of knowledge of L .

Original: A similar discussion can be held for G .
Revised: A similar discussion can be held for G .

Original: Note that the worst-case regret bound of order DG √ T always holds every time OFTRL is used in this article, even without smoothness.
Revised: Note that the worst-case regret bound of order DG p T always holds every time OFTRL is used in this article, even without smoothness.

Original: To avoid distraction, we will not recall this fact in the applications.
Revised: To avoid distraction, we will not recall this fact in the applications.

Original: The algorithm and analysis dwell on two ideas: the adaptive tuning of the learning rate à la AdaHedge/AdaFTRL [19, 23] with optimism, together with the fact that we keep a negative Bregman divergence term in the analysis, which is crucial to obtain our bound.
Revised: The algorithm and analysis dwell on two ideas: the adaptive tuning of the learning rate à la AdaHedge/AdaFTRL [19, 23] with optimism, together with the fact that we keep a negative Bregman divergence term in the analysis, which is crucial to obtain our bound.

Original: Lower Bound.
Revised: Lower Bound.

Original: The upper bound in Theorem 5 is tight up to additive constants, as the following result shows.
Revised: The upper bound in Theorem 5 is tight up to additive constants, as the following result shows.

Original: Theorem 6.
Revised: Theorem 6.

Original: For any learning algorithm, and for any pair of positive numbers ( σ, Σ) there exists a function f : X × Ξ → R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) with ¯ σ T (cid:62) σ and ¯Σ T (cid:62) Σ such that
Revised: For any learning algorithm, and for any pair of positive numbers (  , ⌃ ) there exists a function f : X ⇥ ⌅ ! R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) with ¯  T >  and ¯ ⌃ T > ⌃ such that

Original: [Equation]
Revised: [Equation]

Original: The proof, in Appendix B.2 relies on a lower bound from stochastic optimization [1, 21] together with the fact that we can construct a sequence of convex and L -smooth loss functions such that ¯Σ T is in the order of the gradient norms G .
Revised: The proof, in Appendix B.2 relies on a lower bound from stochastic optimization [1, 21] together with the fact that we can construct a sequence of convex and L -smooth loss functions such that ¯ ⌃ T is in the order of the gradient norms G .

Original: Combining these insights with the lower bound Ω( DG √ T ) [23] gives the desired result.
Revised: Combining these insights with the lower bound ⌦ ( DG p T ) [23] gives the desired result.

Original: 3.2 Strongly Convex and Smooth Functions
Revised: 3.2 Strongly Convex and Smooth Functions

Original: Up to this point, we have only considered functions which satisfy the weaker set of assumptions (A1) , (A2) , (A3) .
Revised: Up to this point, we have only considered functions which satisfy the weaker set of assumptions (A1) , (A2) , (A3) .

Original: In this section, we show what improvements can be achieved if strong convexity also holds, that is, if (A4) is satisﬁed with some known parameter µ > 0 .
Revised: In this section, we show what improvements can be achieved if strong convexity also holds, that is, if (A4) is satisﬁed with some known parameter µ > 0 .

Original: For g t = ∇ f ( x t , ξ t ) , deﬁne the surrogate loss function
Revised: For g t = r f ( x t , ⇠ t ) , deﬁne the surrogate loss function

Original: [Equation]
Revised: [Equation]

Original: We use Optimistic Follow-the-Leader (OFTL) on the surrogate losses.
Revised: We use Optimistic Follow-the-Leader (OFTL) on the surrogate losses.

Original: For each step t , the learner makes an optimistic prediction of the next gradient M t ∈ R d and selects
Revised: For each step t , the learner makes an optimistic prediction of the next gradient M t 2 R d and selects

Original: [Equation]
Revised: [Equation]

Original: The next theorem is analogous to Theorem 5 for curved losses, and will be our main tool in establishing results for strongly convex losses; see Appendix B.3 for a proof.
Revised: The next theorem is analogous to Theorem 5 for curved losses, and will be our main tool in establishing results for strongly convex losses; see Appendix B.3 for a proof.

Original: Theorem 7.
Revised: Theorem 7.

Original: Under assumptions (A1) – (A4) , the expected regret of OFTL with M t = ∇ f ( x t − 1 , ξ t − 1 ) on surrogate loss functions (cid:96) t deﬁned in (4) is bounded as
Revised: Under assumptions (A1) – (A4) , the expected regret of OFTL with M t = r f ( x t  1 , ⇠ t  1 ) on surrogate loss functions ` t deﬁned in (4) is bounded as

Original: [Equation]
Revised: [Equation]

Original: Note that OFTL requires no tuning besides the strong convexity parameter used in the surrogate losses.
Revised: Note that OFTL requires no tuning besides the strong convexity parameter used in the surrogate losses.

Original: In particular, it is adaptive to the smoothness L .
Revised: In particular, it is adaptive to the smoothness L .

Original: Lower Bound
Revised: Lower Bound

Original: The bound in Theorem 7 is tight, as the next result, proved in Appendix B.4 shows.
Revised: The bound in Theorem 7 is tight, as the next result, proved in Appendix B.4 shows.

Original: Theorem 8.
Revised: Theorem 8.

Original: For any learning algorithm, and for any pair of positive numbers ( σ, Σ) there exists a function f : X × Ξ → R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) and (A4) with σ max (cid:62) σ and Σ max (cid:62) Σ such that
Revised: For any learning algorithm, and for any pair of positive numbers (  , ⌃ ) there exists a function f : X ⇥ ⌅ ! R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) and (A4) with  max >  and ⌃ max > ⌃ such that

Original: [Equation]
Revised: [Equation]

Original:  Implications
Revised:  Implications

Original: We derive consequences of our results from Section 3.
Revised: We derive consequences of our results from Section 3.

Original: Further examples can be found in Appendix E.
Revised: Further examples can be found in Appendix E.

Original: 4.1 Interpolating Known Results: Fully Adversarial and i.i.d. Data
Revised: 4.1 Interpolating Known Results: Fully Adversarial and i.i.d. Data

Original: A ﬁrst implication of our analysis is that we recover both the adversarial and i.i.d.
Revised: A ﬁrst implication of our analysis is that we recover both the adversarial and i.i.d.

Original: rates, via a single adaptive algorithm .
Revised: rates, via a single adaptive algorithm .

Original: Convex Case For adversarial data, σ t = 0 for all t , and Σ 2[1: T ] (cid:54) G √ 2 T .
Revised: Convex Case For adversarial data,  t = 0 for all t , and ⌃ 2[1: T ] 6 G p 2 T .

Original: Thus, Theoremguarantees a bound R T ( u ) (cid:54) O ( DG √ T ) , which is known to be the optimal rate up to the additive constants, cf.
Revised: Thus, Theoremguarantees a bound R T ( u ) 6 O ( DG p T ) , which is known to be the optimal rate up to the additive constants, cf.

Original: [37] (note that the expectation does not act on the regret in this case).
Revised: [37] (note that the expectation does not act on the regret in this case).

Original: Simultaneously, if the data is i.i.d.
Revised: Simultaneously, if the data is i.i.d.

Original: , then Theorem 5 guarantees that
Revised: , then Theorem 5 guarantees that

Original: [Equation]
Revised: [Equation]

Original: From standard online-to-batch conversion, this implies an excess risk for the related SCO problem of order O ( Dσ/ √ T + D ( LD + G ) /T ) , which matches the well-known result by [7] up to lower order terms.
Revised: From standard online-to-batch conversion, this implies an excess risk for the related SCO problem of order O ( D  / p T + D ( LD + G ) /T ) , which matches the well-known result by [7] up to lower order terms.

Original: On the other hand, using batch-to-online conversion (see Appendix D) with the best known accelerated convergence result in SCO, gives O ( Dσ √ T + LD 2 ) regret.
Revised: On the other hand, using batch-to-online conversion (see Appendix D) with the best known accelerated convergence result in SCO, gives O ( D  p T + LD 2 ) regret.

Original: Therefore, up to a constant, our result coincides with the best-known results from SCO.
Revised: Therefore, up to a constant, our result coincides with the best-known results from SCO.

Original: Note that also generalizes the improvement obtained for linear functions in the i.i.d.
Revised: Note that also generalizes the improvement obtained for linear functions in the i.i.d.

Original: setting in [25, Section 6.2].
Revised: setting in [25, Section 6.2].

Original: Strongly Convex Case
Revised: Strongly Convex Case

Original: The adaptive interpolation between i.i.d.
Revised: The adaptive interpolation between i.i.d.

Original: and adversarial rates also holds in the strongly convex case.
Revised: and adversarial rates also holds in the strongly convex case.

Original: For adversarial data, the bound of Theorem 7 is of order ( G 2 /µ ) log T , which is known to be the optimal worst-case rate, cf.
Revised: For adversarial data, the bound of Theorem 7 is of order ( G 2 /µ ) log T , which is known to be the optimal worst-case rate, cf.

Original: For i.i.d.
Revised: For i.i.d.

Original: data, the dependence on Gimproves to σ 2 , yielding a bound of order O (( σ 2 /µ ) log T + LD 2 κ log κ ) .
Revised: data, the dependence on Gimproves to  2 , yielding a bound of order O ((  2 /µ ) log T + LD 2  log  ) .

Original: This improvement is akin to improvements obtained by accelerated stochastic gradient descent in the context of stochastic optimization [8, 17].
Revised: This improvement is akin to improvements obtained by accelerated stochastic gradient descent in the context of stochastic optimization [8, 17].

Original: In fact, applying batch-to-online conversions and summing the optimization rates would yield a regret bound similar to ours; c.f. Appendix D.
Revised: In fact, applying batch-to-online conversions and summing the optimization rates would yield a regret bound similar to ours; c.f. Appendix D.

Original: 4.2 Adversarially Corrupted Stochastic Data
Revised: 4.2 Adversarially Corrupted Stochastic Data

Original: We consider a natural generalization to online convex optimization of the corruption model considered in the bandit literature [29, 36], also recently studied in [14] for prediction with expert advice.
Revised: We consider a natural generalization to online convex optimization of the corruption model considered in the bandit literature [29, 36], also recently studied in [14] for prediction with expert advice.

Original: There, the author obtains a regret bound that is the sum of the i.i.d.
Revised: There, the author obtains a regret bound that is the sum of the i.i.d.

Original: rate and of a term of order √ C where C is the total amount of perturbation.
Revised: rate and of a term of order p C where C is the total amount of perturbation.

Original: They then raise the open question of whether similar results could be obtained for general convex losses.
Revised: They then raise the open question of whether similar results could be obtained for general convex losses.

Original: We provide a positive answer to this question in this section, with the regret bound in Corollary 9.
Revised: We provide a positive answer to this question in this section, with the regret bound in Corollary 9.

Original: In this model, the generating process of the losses is decomposed as a combination of losses coming from i.i.d.
Revised: In this model, the generating process of the losses is decomposed as a combination of losses coming from i.i.d.

Original: data, with a small additive adversarial perturbation.
Revised: data, with a small additive adversarial perturbation.

Original: This ﬁts in the framework by setting ξ t = ( ξ iid ,t , c t ) ∼ D t =
Revised: This ﬁts in the framework by setting ⇠ t = ( ⇠ iid ,t , c t ) ⇠ D t =

Original: D ⊗ δ c t and
Revised: D ⌦  c t and

Original: [Equation]
Revised: [Equation]

Original: where c t is the adversarial part of the losses selected by the adversary, and ξ iid ,t ∼ D is a sequence of identically distributed random variables.
Revised: where c t is the adversarial part of the losses selected by the adversary, and ⇠ iid ,t ⇠ D is a sequence of identically distributed random variables.

Original: Note that, similarly to our inspirations [14, 29], and contrary to other corruption models for prediction with expert advice [2], we measure the regret against the perturbed data.
Revised: Note that, similarly to our inspirations [14, 29], and contrary to other corruption models for prediction with expert advice [2], we measure the regret against the perturbed data.

Original: Deﬁne F = E ξ ∼D [ h ( · , ξ )] , so that F t ( x ) = F ( x ) + c t ( x ) .
Revised: Deﬁne F = E ⇠ ⇠ D [ h ( · , ⇠ )] , so that F t ( x ) = F ( x ) + c t ( x ) .

Original: The amount of perturbation is measured by a parameter C > 0 bounding
Revised: The amount of perturbation is measured by a parameter C > 0 bounding

Original: [Equation]
Revised: [Equation]

Original: which is a natural measure of perturbation on the feedback used by the player (note that adding a constant to the perturbations does not change the regret).
Revised: which is a natural measure of perturbation on the feedback used by the player (note that adding a constant to the perturbations does not change the regret).

Original: In this case, the adversarial perturbation on the loss does not affect the variance and σ 2 D t =
Revised: In this case, the adversarial perturbation on the loss does not affect the variance and  2 D t =

Original: σ 2 D .
Revised:  2 D .

Original: The perturbation appears in the loss variation as for any t (cid:62) 2 , for any x ∈ X ,
Revised: The perturbation appears in the loss variation as for any t > 2 , for any x 2 X ,

Original: [Equation]
Revised: [Equation]

Original: Upon taking the supremum over x ∈ X and summing over t , we get (with the convention that c 0 ≡ 0 ),
Revised: Upon taking the supremum over x 2 X and summing over t , we get (with the convention that c 0 ⌘ 0 ),

Original: [Equation]
Revised: [Equation]

Original: Hence, Theorem 5 combined with the bounds on ¯ σ T and ¯Σ T yields the following regret guarantee.
Revised: Hence, Theorem 5 combined with the bounds on ¯  T and ¯ ⌃ T yields the following regret guarantee.

Original: Corollary 9.
Revised: Corollary 9.

Original: In the adversarially corrupted stochastic model, adaptive OFTRL enjoys the bound
Revised: In the adversarially corrupted stochastic model, adaptive OFTRL enjoys the bound

Original: [Equation]
Revised: [Equation]

Original: This regret bound is the sum of the i.i.d.
Revised: This regret bound is the sum of the i.i.d.

Original: rate for the unperturbed source with a term sublinear in the amount of perturbations C , achieved without prior knowledge of C .
Revised: rate for the unperturbed source with a term sublinear in the amount of perturbations C , achieved without prior knowledge of C .

Original: This provides an answer to a question by [14].
Revised: This provides an answer to a question by [14].

Original: The construction of the lower bound Theorem 6 also provides a partial lower bound for the corruption model, proving some worst-case optimality of our upper bound, see
Revised: The construction of the lower bound Theorem 6 also provides a partial lower bound for the corruption model, proving some worst-case optimality of our upper bound, see

Original: Appendix B.2.1.
Revised: Appendix B.2.1.

Original: An interesting open question that remains would be to extend these results to strongly convex losses.
Revised: An interesting open question that remains would be to extend these results to strongly convex losses.

Original: Regret against unperturbed losses Depending on modeling choices, one might consider the perturbations c t to be a true part of the data that deviate from an i.i.d.
Revised: Regret against unperturbed losses Depending on modeling choices, one might consider the perturbations c t to be a true part of the data that deviate from an i.i.d.

Original: model, or one may see them as adversarial noise which should be discarded.
Revised: model, or one may see them as adversarial noise which should be discarded.

Original: In the latter case, regret against the i.i.d.
Revised: In the latter case, regret against the i.i.d.

Original: part of the data is most relevant.
Revised: part of the data is most relevant.

Original: As noticed by [2, Observation 4] and [14, Remark 1], regret against the unperturbed losses and against the perturbed losses are closely related.
Revised: As noticed by [2, Observation 4] and [14, Remark 1], regret against the unperturbed losses and against the perturbed losses are closely related.

Original: Indeed, Corollary 9 implies the bound
Revised: Indeed, Corollary 9 implies the bound

Original: E (cid:34) T
Revised: E " T

Original: [Equation]
Revised: [Equation]

Original: 4.3 Random Order Models
Revised: 4.3 Random Order Models

Original: We apply our results from Section 3 to the Random Order model.
Revised: We apply our results from Section 3 to the Random Order model.

Original: The online ROM was introduced by [6] as a way of restricting the power of the adversary in OCO.
Revised: The online ROM was introduced by [6] as a way of restricting the power of the adversary in OCO.

Original: Note that the ROM is a popular model in combinatorial online optimization, e.g., in the context of proﬁt maximization [9].
Revised: Note that the ROM is a popular model in combinatorial online optimization, e.g., in the context of proﬁt maximization [9].

Original: Our results highlight that the rates in the ROM model, which is not i.i.d., are almost the same as the rates of the i.i.d. model obtained via sampling in the same set of losses with replacement.
Revised: Our results highlight that the rates in the ROM model, which is not i.i.d., are almost the same as the rates of the i.i.d. model obtained via sampling in the same set of losses with replacement.

Original: Corollary 10.
Revised: Corollary 10.

Original: In the single-pass ROM with convex and L -smooth losses ( f k ) k ∈ [ T ] , OFTRL (c.f. (2) ) enjoys the regret bound
Revised: In the single-pass ROM with convex and L -smooth losses ( f k ) k 2 [ T ] , OFTRL (c.f. (2) ) enjoys the regret bound

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (cid:101)

Original: [Equation] (cid:88) [Equation] We remark that σ 1 (cid:54) (cid:101) σ 1 (cid:54) min { 4 G 2 , Tσ 1 } .
Revised: We remark that  1e  1 6 min { 4 G 2 , T  1 } .

Original: Hence, the logarithm of the ratio log( (cid:101) σ 1 /σ 1 ) is always under control.
Revised: Hence, the logarithm of the ratio log( e  1 /  1 ) is always under control.

Original: The proof of Corollary 10 consists in controlling the adversarial variation and the cumulative variance thanks to the following lemma, proved in Appendix C.1.
Revised: The proof of Corollary 10 consists in controlling the adversarial variation and the cumulative variance thanks to the following lemma, proved in Appendix C.1.

Original: Lemma 11.
Revised: Lemma 11.

Original: In the single-pass ROM, we have Σ (2)[1: T ] (cid:54) 8 G 2 and ¯ σ (2)[1: T ] (cid:54)
Revised: In the single-pass ROM, we have ⌃ (2)[1: T ] 6 8 G 2 and ¯  (2)[1: T ] 6 T  21 log(2 e [Equation]

Original: Tσ 21 log(2 e 2 (cid:101) σ 21 /σ 21 ) .
Revised: [Equation]  21 ) .

Original: We would like to emphasize that our results are complementary to those of [6, 31].
Revised: We would like to emphasize that our results are complementary to those of [6, 31].

Original: The focus of these works is to relax the assumption that individual losses are convex, and to only require convexity of the average loss function, leading to very different technical challenges.
Revised: The focus of these works is to relax the assumption that individual losses are convex, and to only require convexity of the average loss function, leading to very different technical challenges.

Original: Inquiring if our results can also be achieved under the weaker assumptions of [31] would be an interesting direction for future work.
Revised: Inquiring if our results can also be achieved under the weaker assumptions of [31] would be an interesting direction for future work.

Original: We also consider the multi-pass ROM.
Revised: We also consider the multi-pass ROM.

Original: Let P ∈ N denote the number of passes.
Revised: Let P 2 N denote the number of passes.

Original: From Lemma 11 and
Revised: From Lemma 11 and

Original: Corollary 10 we directly obtain
Revised: Corollary 10 we directly obtain

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma 11 with Theorem 7 also gives the following corollary for strongly convex functions; see Appendix C.2 for a proof.
Revised: Combining Lemma 11 with Theorem 7 also gives the following corollary for strongly convex functions; see Appendix C.2 for a proof.

Original: Corollary 12.
Revised: Corollary 12.

Original: Under the same assumption as in Theorem 7, the expected regret of the ROM is bounded by
Revised: Under the same assumption as in Theorem 7, the expected regret of the ROM is bounded by

Original: [Equation]
Revised: [Equation]

Original: For multi-pass ROM with P passes, we obtain
Revised: For multi-pass ROM with P passes, we obtain

Original: [Equation]
Revised: [Equation]

Original:  Conclusion and future work
Revised:  Conclusion and future work

Original: As we showed, the exploitation of smoothness of the expected loss functions reduces the dependence of the regret bound on the maximal gradient norm to a dependence on the cumulative stochastic variance and the adversarial variation.
Revised: As we showed, the exploitation of smoothness of the expected loss functions reduces the dependence of the regret bound on the maximal gradient norm to a dependence on the cumulative stochastic variance and the adversarial variation.

Original: Furthermore, we took a step towards a deeper theoretical understanding of the practically relevant intermediate scenarios.
Revised: Furthermore, we took a step towards a deeper theoretical understanding of the practically relevant intermediate scenarios.

Original: Our approach also opens several interesting new research directions.
Revised: Our approach also opens several interesting new research directions.

Original: For instance, in the ROM, as mentioned in Section 4.3, an interesting question is whether a regret bound with dependence on σ instead of G can also be achieved with weaker assumptions as in [31].
Revised: For instance, in the ROM, as mentioned in Section 4.3, an interesting question is whether a regret bound with dependence on  instead of G can also be achieved with weaker assumptions as in [31].

Original: Another interesting question is whether it is possible to unify the analyses and algorithms for the strongly convex and convex case.
Revised: Another interesting question is whether it is possible to unify the analyses and algorithms for the strongly convex and convex case.

Original: So far our analyses of these cases were intrinsically different and the choice of the algorithm requires the knowledge of strong convexity constant µ .
Revised: So far our analyses of these cases were intrinsically different and the choice of the algorithm requires the knowledge of strong convexity constant µ .

Original: Since this knowledge might not be available, it is of practical interest to design an adaptive method that can automatically get the best rate without manually tuning for µ .
Revised: Since this knowledge might not be available, it is of practical interest to design an adaptive method that can automatically get the best rate without manually tuning for µ .

Original: Furthermore, it would be interesting to investigate if similar or better improvements of the regret bound can be achieved with different optimistic predictions M t .
Revised: Furthermore, it would be interesting to investigate if similar or better improvements of the regret bound can be achieved with different optimistic predictions M t .

Original: Our matching lower bounds show that there are instances where M t =
Revised: Our matching lower bounds show that there are instances where M t =

Original: g t − 1 captures the right adversarial behavior.
Revised: g t  1 captures the right adversarial behavior.

Original: However, there are cases where other measures of regularity would be more relevant.
Revised: However, there are cases where other measures of regularity would be more relevant.

Original: One such example would be an adversary oscillating between two distributions.
Revised: One such example would be an adversary oscillating between two distributions.

Original: Interestingly, using M t = g t − 2 would then recover optimal bounds.
Revised: Interestingly, using M t = g t  2 would then recover optimal bounds.

Original: An interesting future research direction would be to investigate what can be achieved when using linear combinations of past gradients as optimistic predictions.
Revised: An interesting future research direction would be to investigate what can be achieved when using linear combinations of past gradients as optimistic predictions.

Original: Social impact: this work is theoretical and does not give rise to any direct societal concerns.
Revised: Social impact: this work is theoretical and does not give rise to any direct societal concerns.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Sachs, Hadiji and Van Erven were supported by the Netherlands Organization for Scientiﬁc Research (NWO) under grant number VI.Vidi.192.095.
Revised: Sachs, Hadiji, and Van Erven were supported by the Netherlands Organization for Scientiﬁc Research (NWO) under grant number VI.Vidi.192.095.

Original: Guzmán’s research is partially supported by the INRIA
Revised: Guzmán’s research is partially supported by the INRIA

Original: Associate Teams project, FONDECYT 1210362 grant and ANID Anillo ACT210005 grant, and
Revised: Associate Teams project, FONDECYT 1210362 grant and ANID Anillo ACT210005 grant, and

Original: National Center for Artiﬁcial Intelligence CENIA FB210017, Basal ANID.
Revised: National Center for Artiﬁcial Intelligence CENIA FB210017, Basal ANID.

Original: Part of this work was done while CG was at the University of Twente.[-A Proofs of Section Note that Var T = (cid:80) Tt =
-]
Revised: Part of this work was done while CG was at the University of Twente.[-A Proofs of Section Note that Var T = (cid:80) Tt =
-]

Original: (cid:107)∇ f t ( x ) − µ ∗ T (cid:107) 2 can be understood as an empirical approximation of σ (2)[1: T ] .

Original: The following lemma shows the relation of Var T to parameters σ (2)[1: T ] and Σ (2)[1: T ] .

Original: Lemma 13.

Original: Deﬁne Var T = (cid:80)

Original: [-Tt =
-]
Revised: [-Tt =
-]

Original: (cid:107)∇ f t ( x ) − µ ∗ T (cid:107) 2 with µ ∗ T = 1 T (cid:80) Tt =1 ∇ f t ( x ) .

Original: In expectation with respect to distributions D 1 , . . .

Original: D T ,

Original: [Equation]

Original: Furthermore, there exists distributions such that E [Var T ] is arbitrarily larger than σ (2)[1: T ] + Σ (2)[1: T ] .

Original: Proof.

Original: Since the distribution mean minimizes the least squares error, E [Var T ] (cid:62) σ (2)[1: T ] always holds.

Original: Using the same argument, we have

Original: [Equation]

Original: Consider X =

Original: [ − 1 , 1] and f ( x, ξ ) = ξx .

Original: Now suppose the SEA chooses truncated normal distribution with mean − 2 for the ﬁrst T/ 2 rounds, then truncated normal distribution with mean 2 for the remaining rounds.

Original: Assume in both cases the variance is σ 2 and truncation is in the range [ − G, G ] for

Original: G > 0 .

Original: Hence, for sufﬁciently large T , µ T = 0 and Var T =

Original: [-(cid:80) Tt =1 (cid:107) f t (cid:107) 
-]
Revised: [-(cid:80) Tt =1 (cid:107) f t (cid:107) 
-]

Original: ∝ TG 2 .

Original: However, σ (2)[1: T ] is equal to Tσ 2 which can be considerably smaller than TG 2 .

Original: The price for the distribution switch is captured with a small constant overhead by Σ (2)[1: T ] (cid:54) 2 G 2 .

Original: Thus, σ (2)[1: T ] + Σ (2)[1: T ] ∝ Tσ 2 + 2 G 2 .

Original: Lemma 14.

Original: Deﬁne D p = (cid:80)

Original: Tt =1 sup x ∈ X

Original: (cid:107)∇ f t ( x ) − ∇ f t − 1 ( x ) (cid:107) 2 p .

Original: In the SEA framework,

Original: [Equation]

Original: Furthermore, there exist instances such that E [ D 2 ] (cid:29) σ (2)[1: T ] + Σ (2)[1: T ] .

Original: Proof.

Original: We shall in fact prove that E [ D 2 ] (cid:62) max( σ (2)[1: T ] , Σ (2)[1: T ] ) , which directly implies the ﬁrst part of the statement.

Original: Fix a time t (cid:62) 2 and let G t − 1 denote the σ -algebra generated by ( x 1 , ξ 1 , . . .

Original: , ξ t − 1 , x t ) .

Original: Then for any x ∈ X , the variable F t ( x ) =

Original: E [ f t ( x ) |G t − 1 ] is G t − 1 -measurable, and we have

Original: E

Original: [Equation]

Original: sup

Original: [Equation]

Original: since f t − 1 = f ( · , ξ t − 1 ) is G t − 1 -measurable, and ∇ F t ( x ) = E (cid:2) ∇ f t ( x ) | G t − 1 (cid:3) .

Original: Therefore, by conditioning on G t − 1 at time step t , and applying the tower rule, we obtain

Original: [Equation]

Original: The lower bound by Σ (2)[1: T ] holds by a direct application of Jensen’s inequality, and by swapping suprema and expectations.

Original: For the second part of the lemma, consider the d -dimensional euclidean ball X

Original: = B d (1) ⊂ R d , and Ξ =

Original: [ d ] .

Original: Deﬁne

Original: [Equation]

Original: Consider a fully stochastic (i.i.d.)

Original: SEA picking ξ t ∈

Original: [ d ] uniformly at random at every time step.

Original: Then Σ (2)[1: T ] = 0 .

Original: We shall now see that σ (2)[1: T ] (cid:54) T/d .

Original: Indeed, for any x ∈ X and t ∈ [ T ] , then F t does not depend on t and its value is

Original: [Equation]

Original: which is a convex and smooth function.

Original: We can upper bound the variance, as for any x ∈ X ,

Original: [Equation]

Original: Therefore, after the taking the supremum over x ∈ X , we see that σ 2 t (cid:54) 1 /d .

Original: On the other hand, for any I, J ∈ [ d ] , we have

Original: [Equation]

Original: The maximum in the ball of this difference is reached at x = √ 2 / 2( e I + e J ) and

Original: [Equation]

Original: Therefore, if I and J are independent and uniformly distributed over [ d ] , then

Original: [Equation]

Original: Summarizing the above inequalities, we have built an example in which

Original: [Equation]

Original: In particular, the expectation of the variation D 2 can be arbitrarily larger than the cumulative variance, and our bounds are then tighter than those obtained via a direct application of known results.

Original: B Proofs of SectionB.1 Proof of Theorem

Original: To prove Theorem 5, we need the following well-known result from the literature.

Original: Lemma 15.

Original: Suppose f t ( · ) are convex for all t ∈

Original: [ T ] and ψ t ( · ) = 2 η t (cid:107) · (cid:107) 2 .

Original: Further, let g t ∈

Original: ∂f t ( x t ) and assume η t (cid:62) η t +1 .

Original: Then the regret for OFTRL is bounded by

Original: [Equation]

Original: Proof.

Original: Denote F t ( x ) :

Original: =

Original: ψ t ( x ) + (cid:80) t − 1 s =1 f s ( x ) .

Original: Note that F t is 2 η t -strongly convex.

Original: Thus, Thm 7.

Original: in [22] gives

Original: [Equation]

Original: Since ψ t ( x ) − ψ t +1 ( x ) (cid:54) 0 and ψ T +1 ( u ) (cid:54) D 2 η T , this gives

Original: [Equation]

Original: Theorem 5.

Original: Fix a user-speciﬁed parameter ν > 0 .

Original: Under assumptions (A1) , (A2) , (A3) , OFTRL, with M t = g t − 1 and adaptive step-size η t =

Original: D 2 / ( ν + (cid:80) t − 1 s =1 η s (cid:107) g s − M s (cid:107) 2 ) , has regret

Original: [Equation]

Original: The algorithm needs to know only D .

Original: If G and L are also known, one can tune ν = LD 2 + D 2 G 2 to get

Original: [Equation]

Original: Moreover, if only convexity of the individual losses holds (A1) , then tuning ν = 2 DG ensures the (deterministic) bound R T ( u ) (cid:54) 3 √ 2 DG √ T + 4 DG .

Original: Proof of Theorem 5.

Original: Write g t = ∇ f ( x t , ξ t ) and denote by E the expectation with respect to all the randomness.

Original: Using the Optimistic FTRL bound from Lemma 15,

Original: [Equation]

Original: where the last inequality is obtained by separating the negative norm term in two parts, and keeping half of it in the regret bound.

Original: Let us plug in the value η t ,

Original: [Equation]

Original: and use the fact that η t (cid:54) D 2 /C to further upper bound the deterministic regret by

Original: [Equation]

Original: To bound the second term above, we ﬁrst compute

Original: [Equation]

Original: t =

Original: [Equation]

Original: Since η t (cid:54) D 2 / ( (cid:80) t − 1 s =1 η s (cid:107) g s − M s (cid:107) 2 ) (cid:54) 2 D

Original: T (cid:88) t

Original: [-= (cid:107) g t − M t (cid:107) 
-]
Revised: [-= (cid:107) g t − M t (cid:107) 
-]

Original: + (cid:32)(cid:80)

Original: Tt =1 η 2 t (cid:107) g

Original: t − M t (cid:107)(cid:80)

Original: Tt =1 η t (cid:107) g t − M t (cid:107)(cid:33) T (cid:88) t

Original: = η t (cid:107) g

Original: t − M t (cid:107) 2 .

Original: Now we use the fact X 2 (cid:54) 2 A + BX implies X (cid:54) √ 2 A + B for A, B > 0 .

Original: Hence

Original: T (cid:88) t =

Original: η t (cid:107) g

Original: t − M t (cid:107) 2 (cid:54) D (cid:118)(cid:117)(cid:117)(cid:116)

Original: T (cid:88) t

Original: [-= (cid:107) g t − M t (cid:107) 
-]
Revised: [-= (cid:107) g t − M t (cid:107) 
-]

Original: + (cid:80) Tt =1 η 2 t (cid:107) g t − M t

Original: (cid:107)(cid:80)

Original: Tt =1 η t (cid:107) g t − M t (cid:107) 2 .

Original: Next we use η t (cid:54) D 2 /ν

Original: and (cid:107)

Original: g t − M t (cid:107) 2 (cid:54) 4 G 2 to bound the last term.

Original: The sum satisﬁes (cid:80) T t =1 η 2 t (cid:107) g t − M t (cid:107) 4 (cid:54) (4 G 2 D 2 /ν ) (cid:80) Tt =1 η t (cid:107)

Original: [-g t − M t (cid:107) 
-]
Revised: [-g t − M t (cid:107) 
-]

Original: and we can further bound the term above (cid:54) D (cid:118)(cid:117)(cid:117)(cid:116) 2 T (cid:88) t

Original: [-= (cid:107) g t − M t (cid:107) 
-]
Revised: [-= (cid:107) g t − M t (cid:107) 
-]

Original: + 4 D 2 G 2 ν .

Original: All in all, we have

Original: [Equation]

Original: Note that we have not used any assumption on the expected F t ’s, and in particular not the smoothness.

Original: Therefore, even if the expected losses are not smooth

Original: [Equation]

Original: our analysis already entails that if (cid:107) M t (cid:107) (cid:54) G

Original: [Equation]

Original: proving the ﬁnal claim of the statement.

Original: Let us now proceed with the proof of the ﬁner results.

Original: We use the value of M t , together with the fact that (by convexity of a (cid:55)→ (cid:107) a (cid:107) 2 ), for any t (cid:62) 2 ,

Original: [Equation]

Original: Therefore, using the inequality √ a + b (cid:54) √ a + √ b , as well as (cid:107) g 1 (cid:107) (cid:54) G and reorganizing the terms,

Original: [Equation]

Original: The sum of the variations of x t ’s can be canceled thanks to the negative term in (9), as

Original: [Equation]

Original: After replacing these bounds in (9)

Original: [Equation]

Original: we have obtained the regret bound

Original: T (cid:88)

Original: [Equation]

Original: We will then take expectations in the inequality above.

Original: To bound the right-hand side

Original: [Equation]

Original: let us denote by

Original: G t =

Original: σ ( x 1 , ξ 1 , . . .

Original: , x t − 1 , ξ t − 1 , x t ) , then ξ t is distributed according to D t given G t

Original: [Equation]

Original: and since x t is

Original: G t -measurable, therefore

Original: [Equation]

Original: Therefore, by the tower rule,

Original: [Equation]

Original: [Equation]

Original: The ﬁnal result follows from taking expectations in (11), applying Jensen’s inequality, incorporating (12) and using the deﬁnitions of ¯ σ T and ¯Σ T .

Original: B.2 Proof of Theorem

Original: Theorem 6.

Original: For any learning algorithm, and for any pair of positive numbers ( σ, Σ) there exists a function f : X × Ξ → R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) with ¯ σ T (cid:62) σ and ¯Σ T (cid:62) Σ such that

Original: [Equation]

Original: Proof of Theorem 6.

Original: Suppose we are given two parameters ˆ σ T and ˆΣ T , we show that there exists a sequence of distributions D 1 , . . .

Original: D T such that the expected regret is at least Ω

Original: ( D (ˆ σ T + ˆΣ T ) √ T ) .

Original: Let 1 (cid:54) a < b be constants such that a (cid:62) 12 b .

Original: Since for any closed convex set, there exist an afﬁne transformation which maps it to the interval [ a, b ] , we assume without loss of generality that X

Original: = [ a, b ] .

Original: Suppose f : X × Ξ

Original: → R and let z ( σ ) , z (Σ) ∈ R .

Original: Assume the gradients have the form ∇ f ( x, ξ )

Original: =

Original: z ( σ ) or ∇ f ( x, ξ )

Original: =

Original: z (Σ) .

Original: Assume SEA chooses each case with probability 1 / 2 .

Original: The idea is to construct two sequences { z ( σ ) t } t ∈ [ T ] and { z (Σ) t } t ∈ [ T ] such that these sequences have at least Ω( D ˆ σ T √ T ) and Ω

Original: ( D ˆ Σ T √ T ) expected regret, respectively.

Original: Therefore, let x t denote the learners choice in round t and deﬁne linearised regret with respect to { z ( σ ) t } t ∈ [ T ] and { z (Σ) t } t ∈

Original: [ T ] .

Original: [Equation]

Original: [Equation]

Original: and

Original: [Equation]

Original: min

Original: [Equation]

Original: t ∈

Original: [Equation]

Original: (cid:68)

Original: [Equation]

Original: Case R Σ T : Let G = ˆΣ T .

Original: Deﬁne z : X → R , z ( x ) = 14 b Gx 2 .

Original: Then z is G -Lipschitz, smooth and z (cid:48)

Original: ( x ) ∈ [ 12 G, G ] for any x ∈ X .

Original: Let { ε t } t ∈ [ T ] be an i.i.d.

Original: sequence of Rademacher random variables, that is, P [ ε t = 1] = P [ ε t = − 1] = 1 / 2 .

Original: The sequence { z (Σ) t } t ∈ [ T ] is deﬁned as

Original: [Equation]

Original: if t even

Original: [Equation]

Original: if

Original: [Equation]

Original: odd .

Original: Using that E [ ε ] = 0 together with the deﬁnition of z (Σ) t gives

Original: [Equation]

Original: Now use that for a linear function l ( x ) , max x ∈ [ a,b ] l ( x ) = max x ∈{ a,b } l ( x ) =

Original: l ( a + b )+ | l ( a − b ) |.

Original: [Equation]

Original: t = ε t ( a

Original: [Equation]

Original: b ) 

Original: 

Original: [Equation]

Original: G 8 E ε ∼ Rad   (cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)

Original: T/ 2 (cid:88) t =

Original: Gε t ( a − b ) (cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12) 

Original: 

Original: [Equation]

Original: [Equation]

Original: (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) .

Original: Where we have used E [ ε ] = 0 again.

Original: Now we use that by deﬁnition D = sup x,y ∈ X (cid:107) x − y (cid:107) .

Original: [Equation]

Original: In the last step, we have used the Khintchine inequality.

Original: Now note that G 2 (cid:62) 12 sup x ∈ X (cid:107) z (cid:48)

Original: ( x ) (cid:107) 2 = 12 sup x ∈ X (cid:107) z

Original: (cid:48) ( x ) − 0 (cid:107) 2 .

Original: Due to the deﬁnition of the sequence { z (Σ) t } t ∈ [ T ] , if (cid:107)∇ f ( x, ξ t ) (cid:107) (cid:54) = 0 , then (cid:13)(cid:13) ∇ f ( x, ξ t − 1 ) (cid:13)(cid:13) = 0 .

Original: Thus, 12 sup x ∈ X (cid:107) z (cid:48) ( x ) − 0 (cid:107) 2 = 12 sup x ∈ X

Original: (cid:13)(cid:13) ∇ f ( x, ξ t ) − ∇ f ( x, ξ t − 1 ) (cid:13)(cid:13) 2 for any t ∈

Original: [ T ] .

Original: Thus, √ G 2 T = (cid:113) T/ (2 T ) (cid:80) Tt =

Original: 1 sup x ∈ X (cid:107)∇ f ( x, ξ t ) − ∇ f ( x, ξ t − 1 ) (cid:107)

Original: 2 = ¯Σ T (cid:112) T/ 2 .

Original: Setting the value G = ˆΣ T completes this part of the proof.

Original: Case R σT :

Original: We will show this part by contradiction.

Original: Suppose that D is a distribution such that the variance of the gradients σ is equal to ˆ σ T .

Original: Suppose the SEA picks this distribution every round and assume for contradiction that E [ R σ T ] (cid:54) o

Original: ( Dσ √ T ) .

Original: Using online-to-batch conversion gives a convergence bound of order o ( Dσ/ √ T ) which contradicts well-known lower bounds from stochastic optimization (c.f., [1, 21] Section 5).

Original: B.2.1 Lower Bound for Adversarially Corrupted Stochastic Data

Original: A careful look at the proof of Theorem 6 shows that the same construction applies to the corrupted data model, providing some worst-case optimality of our bounds.

Original: Proposition 16.

Original: For any algorithm, for any C > 0 , and σ > 0 there exists an instance of the

Original: Adversarially Corrupted model with variance σ , corruption level C , and gradients of norm G = C/T such that

Original: [Equation]

Original: Proof.

Original: Case I.

Original: If √ GC (cid:62) σ √ T .

Original: The ﬁrst construction from the proof of Theorem 6 guarantees the existence of a sequence of linear losses with gradient G against which the regret is Ω( DG √ T )

Original: =

Original: Ω( D √ GC ) .

Original: This sequence can be seen as adversarial perturbations on identically stochastic data.

Original: Case II.

Original: If σ √ T (cid:62) √ GC .

Original: Then the stochastic lower bound in the second part of the proof of

Original: Theorem 6 provides a lower bound on the expected regret of Ω( Dσ √ T ) .

Original: B.3 Proof of Theorem

Original: We ﬁrst need a well-known result for OFTL for strongly convex loss functions.

Original: Lemma 17.

Original: Suppose f t ( · ) are µ -strongly convex for all t ∈

Original: [ T ] and ψ t ( · ) = 0 .

Original: Further, let m t :

Original: X → R denote the optimistic prediction, and g t ∈

Original: ∂f t ( x t ) , M t ∈ ∂m t ( x t ) .

Original: Then the regret for OFTRL is bounded by

Original: [Equation]

Original: [Table]

Original: This is a well-known result and can be found in the literature, e.g., [22].

Original: We include a short proof for completeness.

Original: Proof.

Original: Let ¯ F t ( x ) = (cid:80) t − 1 s =1 f t ( x ) and G t ∈

Original: ∂

Original: ¯ F t +1 ( x t ) .

Original: Note that ¯ F t is [( t − 1) µ ] -strongly convex.

Original: From standard analysis (see, e.g., [22] Lem. 7.1) we obtain

Original: T (cid:88)

Original: [Equation]

Original: Due to convexity, G t ∈ ∂ ¯ F t +1 ( x t ) =

Original: ∂ ¯ F t ( x t ) ∩ ∂f t ( x t ) and due to update operation 0 ∈

Original: ∂ ¯ F t ( x t ) ∩ ∂m t ( x t ) .

Original: Thus, there exist g t ∈ ∂f t ( x t ) and M t ∈ ∂m t ( x t ) such that G t =

Original: g t − M t , which completes the proof.

Original: Theorem 7.

Original: Under assumptions (A1) – (A4) , the expected regret of OFTL with M t = ∇ f ( x t − 1 , ξ t − 1 ) on surrogate loss functions (cid:96) t deﬁned in (4) is bounded as

Original: E

Original: [Equation]

Original: Proof of Theorem 7.

Original: Thanks to the strong convexity assumption (A4)

Original: [Equation]

Original: Taking expectation and using the deﬁnition of (cid:96) t gives

Original: [Equation]

Original: Now each function (cid:96) t is µ -strongly convex, and ∇ (cid:96) t ( x t ) =

Original: g t .

Original: Thus we can apply Lemma

Original: T

Original: [Equation]

Original: where we used the inequality (cid:104) a, b (cid:105) (cid:54) 12 c (cid:107) a (cid:107) 2 + c 2 (cid:107) b (cid:107) 2 .

Original: Once again, keeping the negative norm term is crucial.

Original: Indeed, using the convexity of x (cid:55)→ (cid:107) x (cid:107) 2 and the smoothness assumption on ∇ F t , we get that for t (cid:62)

Original: [Equation]

Original: (cid:54) 4 (cid:13)(cid:13)

Original: g t − ∇ F t ( x t ) (cid:13)(cid:13) 2 + 4 L 2 (cid:107) x t − x t − 1 (cid:107)+ 4 (cid:13)(cid:13) ∇ F t ( x t − 1 ) − ∇ F t − 1 ( x t − 1 ) (cid:13)(cid:13) 2 + 4 (cid:13)(cid:13) ∇ F t − 1 ( x t − 1 ) − g t − 1 (cid:13)(cid:13) 2 .

Original: So that, upper bounding the ﬁrst term

Original: [Equation]

Original: x

Original: [Equation]

Original: −

Original: [Equation]

Original: u )

Original: [Equation]

Original: GD we get

Original: [Equation]

Original: The indices can be simpliﬁed by noting that,

Original: [Equation]

Original: To recover

Original: [Equation]

Original: Deﬁne the condition number κ = L/µ .

Original: Then, for t (cid:62) 16 κ , we have 4 L 2 µt − µt 4 (cid:54) 0 .

Original: Therefore the second term can be bounded independently of T

Original: [Equation]

Original: Combining all bounds, and incorporating the deﬁnition of σ max and Σ max ,

Original: [Equation]

Original: B.4 Proof of Theorem

Original: Theorem 8.

Original: For any learning algorithm, and for any pair of positive numbers ( σ, Σ) there exists a function f : X × Ξ → R and a sequence of distributions satisfying assumptions (A1) , (A2) , (A3) and (A4) with σ max (cid:62) σ and Σ max (cid:62) Σ such that

Original: [Equation]

Original: Proof of Theorem 8.

Original: Let ˆ σ max , ˆΣ max be given parameters and set G = max(ˆ σ max , ˆΣ max / 2) .

Original: We want to show that there exists sequence of distributions D 1 , . . .

Original: D T such that

Original: [Equation]

Original: max and Σ max = ˆΣ max .

Original: E [ R T ( u )] (cid:62) c

Original: 1 µ (Σ 2max + σ 2max ) log T for some constant c > 0 .

Original: F 1 , . .

Original: . , F T are µ -strongly convex.

Original: Consider the iterations up to T − 3 .

Original: From Corollary 20 in [13] we obtain an Ω( 1 µ G 2 log( T − 3)) lower bound on the expected regret.

Original: Thus, there exist a realization ξ 1 , . . .

Original: , ξ T − 3 and corresponding µ -strongly convex functions f ( · , ξ 1 ) , . . .

Original: , f ( · , ξ T − 3 ) such that with respect to this realization,

Original: R T ( u ) (cid:62) Ω( 1 µ G 2 log( T − 3)) .

Original: We now let δ 1 , . . .

Original: δ T − 3 be the Dirac measure corresponding to this realization.

Original: Then, σ max = 0 and max t ∈ [ T − 3] sup x ∈ X (cid:107)∇ F t ( x ) − ∇ F t − 1 ( x ) (cid:107) (cid:54) 2 G .

Original: But we do not necessarily have that σ max

Original: = ˆ σ max and max t ∈ [ T − 3] sup x ∈ X (cid:107)∇ F t ( x ) − ∇ F t − 1 ( x ) (cid:107) =

Original: ˆΣ max .

Original: To guarantee this we want to choose D T − 2 , D T − 1 , D T , such that

Original: sup x ∈ X (cid:107)∇ F T − 2 ( x ) − ∇ F T − 1 ( x ) (cid:107) =

Original: ˆΣ max and (cid:107)∇ F T − i ( x ) (cid:107) (cid:54) G for i = 1 , 2 . 2.

Original: σ T =

Original: ˆ σ max and (cid:107)∇ F T ( x ) (cid:107) (cid:54) G

Original: [Equation]

Original: To satisfy Condition 1, let D T − 2 , D T − 1 be Dirac measures, such that ∇ F T − 2 ( x )

Original: =

Original: −∇ F T − 1 ( x ) and (cid:107)∇ F T − 2 ( x ) (cid:107) = 12 ˆΣ max .

Original: Then, by deﬁnition of G , we know that (cid:107)∇ F T − 2 ( x ) (cid:107) (cid:54) G and sup

Original: [Equation]

Original: F

Original: [Equation]

Original: x )

Original: [Equation]

Original: ( x ) (cid:107) = ˆΣ max .

Original: Condition 2 can be satisﬁed by setting D T to be any distribution with sufﬁcient variance.

Original: This gives

Original: [Equation]

Original: Now it remains to show that the last term is negligible.

Original: Indeed, from the upper bound, we know

Original: [Equation]

Original: Hence, for any T (cid:62) 10 , we get

Original: [Equation]

Original: (cid:0) Σ 2 max + σ 2 max (cid:1) log( T ) which together with (14) completes the proof.

Original: C Missing Proofs of Section

Original: We ﬁrst show the following general property of the variance for the ROM.

Original: This proposition will useful for showing the claims of this section.

Original: Proposition 18.

Original: For any t ∈ [ T ] , the variance of the ROM with respect to D t satisﬁes

Original: [Equation]

Original: for any x ∈ X .

Original: Proof.

Original: For any x ∈ X , since ∇ F t ( x ) =

Original: E ξ ∼D t [ ∇ f ( x, ξ )] , we have

Original: [Equation]

Original: Now, let T t ⊆

Original: [ T ] denote a subset of indices of gradients which remain to be selected in round t , and let k t ∈ T t − 1 \ T t be the index selected at round t .

Original: For any x ∈ X

Original: [Equation]

Original: which is the claimed result.

Original: C.1 Proof of Lemma

Original: Note that in any case, ˜ σ 1 (cid:54)

Original: Tσ 1 , and therefore log(˜ σ 1 /σ 1 ) (cid:54) log( T ) .

Original: Thus Lemma 11 directly yields σ (2)[1: T ] (cid:54) σ 1 T log( T ) .

Original: This means in particular that the rate of OFTRL in the ROM is never more than a factor √ log T worse than the i.i.d.

Original: sampling with replacement rate of σ 1 √ T ; the next bound can often be much tighter.

Original: Lemma 11.

Original: In the single-pass ROM, we have Σ (2)[1: T ] (cid:54) 8 G 2 and ¯ σ (2)[1: T ] (cid:54)

Original: Tσ 21 log(2 e 2 (cid:101) σ 21 /σ 21 ) .

Original: Proof of Lemma 11.

Original: Let us begin with the adversarial variation.

Original: We will show that deterministically (that is, for any order in which the losses are selected), for any x ∈ X ,

Original: [Equation]

Original: With the same notation as in Proposition 18, recall that we denote by T t ⊆

Original: [ T ] the support of D t and k t =

Original: T t − 1 \ T t .

Original: We have | T t | =

Original: T − t + 1 and for any x ∈ X ,

Original: [Equation]

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: x

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: ( x

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: Thus, after maximising over x ∈ X , and taking expectations (note that the inequality holds almost surely) and summing over rounds t ∈

Original: [ T ] ,

Original: [Equation]

Original: Variance.

Original: From Proposition

Original: [Equation]

Original: we know that

Original: [Equation]

Original: Moreover, one can see that

Original: [Equation]

Original: Let us introduce a threshold time step τ ∈ [ T ] , of which we will set the value later.

Original: We upper bound E [ σ 2 t ] by (15) for the rounds before τ and by (16) for the other rounds:

Original: [Equation]

Original: Now using standard bounds on the harmonic series,

Original: [Equation]

Original: Therefore for any τ ∈ [ T − 1] , we get

Original: [Equation]

Original: We now conclude by setting the appropriate value for τ .

Original: If Tσ 2 1 / (cid:101) σ 2 1 (cid:54) 2 , then log T (cid:54) log(2 (cid:101) σ 2 1 /σ 2 1 ) , and taking τ =

Original: T gives a bound of Tσ 21 (1 + log T ) (cid:54)

Original: Tσ 21 (1 + log(2 (cid:101) σ 21 /σ 21 )) , which is (better than) the claimed result.

Original: Otherwise, we take τ = T − (cid:98)

Original: Tσ 21 / ˜ σ 21 (cid:99) , then

Original: [Equation]

Original: and the argument of the logarithm can be bounded as

Original: [Equation]

Original: where we used the fact that Tσ 21 / (cid:101) σ 21 > 2 .

Original: This yields the ﬁnal bound

Original: [Equation]

Original: C.2 Proof of Corollary

Original: Corollary 12.

Original: Under the same assumption as in Theorem 7, the expected regret of the ROM is bounded by

Original: [Equation]

Original: For multi-pass ROM with P passes

Original: [Equation]

Original: we obtain

Original: [Equation]

Original: Proof of Corollary 12.

Original: Single-pass ROM:

Original: From Theorem 7 we obtain (c.f. (13))

Original: [Equation]

Original: By Lemma

Original: [Equation]

Original: we have

Original: [Equation]

Original: Furthermore, recall that by Proposition 18 E [ σ 2 t ] (cid:54) T/

Original: ( T − t + 1) σ 21 .

Original: [Equation]

Original: Indeed, using a standard bound on the harmonic series,

Original: T

Original: [Equation]

Original: Combining these bounds gives the ﬁrst part of the corollary.

Original: Multi-pass ROM:

Original: The critical term to upper bound, is the differences of the means whenever a pass ends and a new pass starts.

Original: Thus, for P ∈ N passes, we need to control sup x ∈ X (cid:13) (cid:13) ∇ F t ( x ) − ∇ F t − 1 ( x ) (cid:13) (cid:13) 2 for t = jn + 1 , with j ∈ [ P ] .

Original: Inside the i -th pass, for k ∈ [ n ] we bound the k -th variation by

Original: [Equation]

Original: and we bound it by G 2 between the passes, so that

Original: [Equation]

Original: D Batch-to-online Conversion

Original: Consider the stochastic optimization problem min x ∈ X E ξ ∼ D [ f ( x, ξ )] and let x ∗ denote a minimiser for this problem.

Original: Further

Original: [Equation]

Original: let A be any ﬁrst order stochastic optimization method with convergence guarantee E ξ ∼ D

Original: [ f ( x t , ξ ) − f ( x ∗ , ξ )] (cid:54) c ( t ) .

Original: As input A takes an initial iterate x 1 and a sequence of i.i

Original: [Equation]

Original: d. samples { f ( · , ξ s ) } s ∈ [ t ] .

Original: We let A ( x 1 , { f ( · , ξ s ) } s ∈

Original: [ t ] ) denote the output x t +1 of the stochastic optimization algorithm with respect to the given input.

Original: Now consider an OCO with f ( · , ξ 1 ) , . . .

Original: f ( · , ξ T ) and ξ 1 , . . .

Original: ξ T are sampled i.i.d.

Original: from a distribution.

Original: Input: Stochastic ﬁrst order method A for t = 1 , 2 , . . .

Original: [-T do play x t and suffer loss f ( x t , ξ t ) ; restart A and set x t +
-]
Revised: [-T do play x t and suffer loss f ( x t , ξ t ) ; restart A and set x t +
-]

Original: =

Original: A ( x 1 , { f ( · , ξ s ) } s ∈

Original: [ t ] ) end

Original: This batch-to-online conversion trivially achieves (cid:80)

Original: Tt =1 c ( t ) expected regret.

Original: However, with this conversion, some aspects of the stochastic convergence bound are lost.

Original: Consider for instance a convergence rate c ( t ) =

Original: O ( LD 2 /t + Dσ/ √ t ) , from the the ﬁrst-order stochastic approximation method in [7] and the accelerated version c ( t ) =

Original: O ( LD 2 /t 2 + Dσ/ √ t ) [8, 17].

Original: In both cases, the functions are assumed to satisfy (A1) - (A3) .

Original: Batch-to-online conversion yields

Original: [Equation]

Original: and

Original: [Equation]

Original: The beneﬁts of acceleration can be seen in the lower-order terms.

Original: Now using standard online-tobatch [3] conversion gives the convergence bounds

Original: [Equation]

Original: In the case of accelerated stochastic approximation, the beneﬁts of acceleration are inevitably lost through batch-to-online and online-to-batch conversion.

Original: E Additional Examples for Intermediate Cases

Original: We provide regret bounds for intermediate cases not discussed in the main body of the paper, namely the cases when the adversary selects slowly shifting distributions and when the adversary switches rarely between distributions.

Original: Distribution shift:

Original: In this example, the SEA picks D t and D t − 1 , such that ∇ F t ( x ) is close to the mean of the previous distribution gradient ∇ F t − 1 ( x ) .

Original: We shall consider two kinds of distribution shifts.

Original: Firstly, when the means are close on average, that is, when (1 /T )

Original: (cid:80) Tt =

Original: 1 sup x ∈ X (cid:107)∇ F t ( x ) − ∇ F t − 1 ( x ) (cid:107) 2 (cid:54) ε , secondly, when this holds for each iteration t , i.e., sup x ∈ X (cid:107)∇ F t ( x ) − ∇ F t − 1 ( x ) (cid:107) 2 (cid:54) ε .

Original: We refer to the former as the average distribution shift case, and to the latter as the bounded distribution shift case.

Original: For strongly convex functions, Theorem 7 directly yields the regret bound

Original: [Equation]

Original: For the considerably weaker assumption of an average distribution shift, we have

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: µTε.

Original: To obtain the ﬁrst inequality, we have used the Cauchy-Schwarz inequality together with the fact that √ a + b (cid:54) √ a + √ b , and the second inequality follows directly from the deﬁnition of the averaged distribution shift.

Original: Now suppose ε (cid:54) 1 /T , then we obtain the following regret bound in case of average distribution shift.

Original: [Equation]

Original: Since Σ (2)[1: T ] (cid:54) Tε for the average distribution shift, for convex and smooth functions, Theorementails that

Original: [Equation]

Original: Distribution switch:

Original: SEA switches c times between distributions D 1 , . . .

Original: , D c ∈ D .

Original: These switches can happen at any round and the learner does not know when a switch occurs.

Original: In this case, we can upper bound Σ (2)[1: T ] (cid:54) Σ 2max c .

Original: Thus, for strongly convex functions Theorem 7 directly yields

Original: [Equation]

Original: And for convex smooth functions Theorem 5 gives

Original: [Equation]