A Regret-Variance Trade-Off in Online Learning
Abstract
We consider prediction with expert advice for strongly convex and bounded losses, and investigate trade-offs between regret and “variance” (i.e., squared difference of learner’s predictions and best expert predictions).
With K experts, the Exponentially Weighted Average (EWA) algorithm is known to achieve O (log K ) regret.
We prove that a variant of EWA either achieves a negative regret (i.e., the algorithm outperforms the best expert), or guarantees a O (log K ) bound on both variance and regret.
Building on this result, we show several examples of how variance of predictions can be exploited in learning.
In the online to batch analysis, we show that a large empirical variance allows to stop the online to batch conversion early and outperform the risk of the best predictor in the class.
We also recover the optimal rate of model selection aggregation when we do not consider early stopping.
In online prediction with corrupted losses, we show that the effect of corruption on the regret can be compensated by a large variance.
In online selective sampling, we design an algorithm that samples less when the variance is large, while guaranteeing the optimal regret bound in expectation.
In online learning with abstention, we use a similar term as the variance to derive the first high-probability
O (log K ) regret bound in this setting.
Finally, we extend our results to the setting of online linear regression.
Introduction
In the online learning protocol, the learner interacts with an unknown environment in a sequence of rounds.
In each round t = 1 , 2 , . . .
, T the learner makes a prediction (cid:98) y t ∈ [ − 12 M, 12 M ] and suffers loss ℓ t ( (cid:98) y t ) , where ℓ 1 , ℓ 2 . . .
is a sequence of differentiable loss functions unknown to the learner.
The goal is to achieve small regret R T , defined by R T =
(cid:80) Tt =1 (cid:0) ℓ t ( (cid:98) y t ) − ℓ t ( y ⋆t ) (cid:1) where y ⋆t are the predictions of some reference forecaster.
We consider two special cases of online learning.
In prediction with expert advice, the learner receives the predictions y t (1) , . . .
, y K ( t ) ∈ [ − 12 M, 12 M ] of K experts at the beginning of each round t .
The learner then predicts with a convex combination (cid:98)
y t =
(cid:80) K i =
1 p t ( i ) y t
( i ) of the experts’ predictions, where p t = (cid:0) p t (1) , . . .
, p t ( K ) (cid:1) is a probability distribution over experts updated after each round.
The goal in this setting is to have small regret with respect to the predictions y ⋆t
=
y t ( i ⋆ ) of any fixed expert i ⋆ .
In online linear regression, the learner has access to a feature vector x t ∈ R d at each round.
This is used to compute predictions (cid:98) y t =
⟨ w t , x t ⟩ , where w t ∈ R d is a parameter vector to be updated at the end of the round.
The goal is to have small regret with respect to the predictions y ⋆t = ⟨ u , x t ⟩ of any fixed linear forecaster u ∈ R d such that ∥ u ∥ 2 ≤ D and | y ⋆t | ≤ 12 M for all t .
In both settings, we assume that the losses ℓ t are µ -strongly convex, which is to say that there exists µ > 0 such that for all y, x ∈ [ − 12 M, 12 M ] ,
[Equation]
where we write ℓ ′ t to denote the derivative of ℓ t .
Without loss of generality, throughout the paper we assume that µ ≤ 2 .
A typical example of a strongly convex loss is the squared loss ℓ t ( y ) =
( y − y t ) 2 with µ = 2 .
While standard online learning algorithms, such as, for example, Exponentially Weighted Average (EWA) [Vovk, 1990, Littlestone and Warmuth, 1994], are directly applied to the (mixable) losses ℓ t , we take a different approach.
Our algorithms provide bounds on the linearized regret (cid:101) R T of the form (cid:101) R
[Equation]
for some η ∈ (0 , H ] to be chosen freely by the learner and where H, C T , B T ≥ 0 are problem-specific parameters.
In the expert setting, several algorithms provide such a guarantee: Squint [Koolen and Van Erven, 2015], Adapt-ML-Prod [Gaillard et al., 2014], BOA [Wintenberger, 2017], Squint+C and Squint+L [Mhammedi et al., 2019].
In online linear regression, the MetaGrad algorithm [Van Erven et al., 2021] and its variants by Mhammedi et al.
[2019], Wang et al.
[2020], Chen et al.
[2021] satisfy bounds like (2).
Although in both settings the aforementioned algorithms achieve the optimal tuning of η in (2) without any preliminary information, our applications do not require this tuning property.
As a consequence, the algorithms we derive in Section 3 are simpler and have slightly better guarantees.
More importantly, unlike the aforementioned algorithms, we focus on exploiting the curvature of the loss.
In particular, by combining equations (1) and (2), we obtain the following lemma, which features the central inequality of our work.
Lemma 1. Consider a sequence ℓ 1 , . .
. , ℓ T of µ -strongly convex differentiable losses.
Suppose that predictions (cid:98) y t guarantee the second-order bound (2) .
Then
[Equation]
Next, we show a prototypical example of the bounds we derive in the rest of this work.
Example 1.
Consider the expert setting with the squared loss, ℓ t ( y ) =
( y − y t ) 2 , where y, y t ∈ [ − 1 , 1] for all t ≥ 1 .
Then the predictions (cid:98)
y t of our algorithm satisfy
[Equation]
Even though (3) seems relatively inconsequential, we exploit the negative variance term in several applications.
A straightforward implication is that (3) recovers the usual constant regret bound
R T =
O (log( K )) .
More interestingly, however: if we attain the worst-case performance O (log( K )) , then the variance is small and bounded by O (log( K )) , if the variance is large enough, then the regret becomes negative.
For the role of negative variance terms in the analysis of statistical and online learning, we refer to Section 2.
Remark.
Negative quadratic terms similar to the one in Lemma 1 appear in online convex optimization, for example in the analysis of the online gradient descent [Hazan, 2016, Theorem 3.3].
However, as far as we are aware, in online convex optimization the algorithms are not tuned to obtain a negative term in the regret bounds.
In this paper we argue that for online regression, and especially for prediction with expert advice, setting η < µ/ 2 in Lemma 1 and thus obtaining a negative term can be a powerful tool, which plays a central role in all of our applications.
In prediction with expert advice the standard approach to exploit the curvature of the loss is through mixability rather than through the negative quadratic terms.
In particular, we show that the regret bound in (3) cannot be achieved by EWA with a fixed learning rate, despite the fact that constant regret is achievable by this algorithm via mixability.
Proposition 1 (Informal) .
Consider the setup of online prediction with expert advice and bounded strongly convex losses.
The bound (3) cannot be achieved by the standard EWA algorithm.
For the sake of presentation, we defer the formal description of this result to Section 3.1.
The proof of Proposition 1 is motivated by a result of Audibert [2007] on the sub-optimality of online to batch converted EWA in deviation.
Contributions and Outline.
In Section 4, we show our first three applications of Lemma 1 in the framework of statistical learning.
In the expert setting, we show that online to batch conversion may be stopped early if the empirical variance of our predictions is sufficiently large.
In particular, in high-variance regimes we may stop early because the excess risk bound is negative with high probability, and when the variance is small, we recover the optimal excess risk bound up to a log log T factor.
By exploiting the negative variance term again, we show an optimal high-probability excess risk bound for online to batch conversion of algorithms that satisfy (2).
The optimal high-probability excess risk bound (called the optimal rate of model selection aggregation) was previously known to be achieved by several estimators appearing in [Audibert, 2007, Lecué and Mendelson, 2009, Lecué and Rigollet, 2014] whose analyses are specific to the statistical learning setup.
Our result can also be seen as a simplification and strengthening of a result by Wintenberger [2017].
We also show a high-probability excess risk bound for online to batch conversion of an online regression algorithm in the bounded setup, which is to say that both the feature vectors and derivatives of the losses are bounded.
It was previously shown by Mourtada et al.
[2021] that online to batch converted versions of the optimal Vovk-Azoury-Warmuth forecaster [Vovk, 2001, Azoury and Warmuth, 2001] have constant excess risk with constant probability.
In Section 5, we use the negative variance term to counteract corrupted feedback in online learning.
Our result complements previous results where losses are assumed to be stochastic and corrupted by an adversary—see, e.g., Lykouris et al.
[2018], Zimmert and Seldin [2021], Amir et al.
Ito [2021].
In Section 6 we consider the selective sampling setting [Atlas et al., 1989], where the learner’s goal is to control regret while saving on the number of time the current loss is observed.
We show that the optimal bound can be recovered while only observing a fraction of all losses if losses are observed proportionally to the cumulative variance.
Finally, in Section 7 we discuss how our ideas are not limited to online learning with strongly convex losses, but may also be applied to online learning with abstention and to online multiclass classification.
All bounds in the main text with suppressed constants have detailed statements in the appendix.
Before discussing our applications, we introduce some notation and discuss the related work.
The algorithms that we use to derive most of our results are introduced in Section 3.
Notation.
We use the standard O ( · ) to suppress constants and use (cid:101) O ( · ) to suppress constants and log( T ) -factors.
We use log( · ) to denote the logarithm with base e .
The symbol
[ A ] denotes the indicator of the event A .
For an integer K we denote [ K ] = { 1 , . . .
, K } .
We use p ( i ) ∝ g
( i ) to denote p ( i ) =
g ( i ) (cid:46)(cid:16)(cid:80) Ki
=1 g ( i ′ ) (cid:17) and ∂ y ℓ ( y, Y ) to denote the derivative of ℓ with respect to y .
The symbol I denotes the identity matrix whose dimensions are clear from the context.
For a set of random variables Y, X 1 , . . .
, X t − 1 we write E t − 1 [ Y ] =
E [ Y | X 1 , . . .
, X t − 1 ] .
Related work
Online learning with losses with curvature.
For a thorough introduction to online learning, we refer the reader to [Cesa-Bianchi and Lugosi, 2006, Hazan, 2016, Orabona, 2019].
Strongly convex losses are a special case of mixable losses.
With mixable losses, EWA on a finite set of experts achieves O (log( K )) regret—see, e.g., [Cesa-Bianchi and Lugosi, 2006, Mhammedi and Williamson, 2018].
Mhammedi and Williamson [2018] observe that—in some cases—regret may be negative for mixable losses, but they do not explore this topic further.
The role of the negative terms due to curvature in online and statistical learning.
Although bounds of the form (3) are not explicitly present in the literature, the appearance of negative terms is common when proving fast rates in statistical learning with squared loss.
In particular, the empirical star algorithm of Audibert [2007]—as well as other aggregation algorithms [Lecué and
Mendelson, 2009, Lecué and Rigollet, 2014, Wintenberger, 2017]—exploit the curvature of the loss
Input η ∈ (0 , 12 ] , M > 0 ;
Initialize γ = ηM 2 , p 1 ( i ) =
1 K for all i ; for t = 1 , . . .
, T do
Receive expert predictions y t (1) , . .
, y t ( K )
Predict (cid:98) y
t
=
(cid:80)
Ki =
1 p t ( i ) y t
( i )
Receive g t and κ t
Set (cid:101) ℓ t ( i ) =
γ ( y t ( i ) − (cid:98) y t ) g t + κ t − 1 ( γ ( y t ( i ) − (cid:98) y t ) g t )
Set p t +1 ( i ) ∝ exp( − κ t (cid:80) ts =1 (cid:101) ℓ s ( i )) through the negative term which compensates the variance term. We also refer to some more recent statistical learning results whose analysis is based on the same idea [Mendelson, 2019, Bousquet and Zhivotovskiy, 2021, Puchkin and Zhivotovskiy, 2022, Saad and Blanchard, 2021] . Similarly, in the context of online learning the negative quadratic term appears in Rakhlin and Sridharan [2014], where the so-called sequential offset Rademacher complexity is studied.
Van Erven et al. [2021] also obtain a bound that is very similar to the bound in Lemma 1 (specifically in the proof of their Theorem 1), but they choose η to match the negative term in their bound. Importantly, in these papers the role of the negative term is only to get the fast rate by compensating the variance term.
In contrast, in our case the negative variance terms also appear in the final regret bound and play their role in applications.
Suboptimality of EWA for prediction with expert advice.
In the setup of prediction with expert advice, the classical Exponentially Weighted Average (EWA) algorithm
[Vovk, 1990, Littlestone and
Warmuth, 1994] is known to give a constant regret in the case of strongly convex losses.
However, despite being optimal in this setting, this algorithm has several known drawbacks.
In the case of general losses, EWA does not deliver the second-order bound (2).
As a result, unlike more advanced algorithms such as Squint [Koolen and Van Erven, 2015], EWA with a fixed learning rate (and even a decreasing learning rate) cannot adapt to certain benign stochastic environments where, for example, the Bernstein assumption holds [Mourtada and Gaïffas, 2019].
The second source of suboptimality comes from online to batch conversions in the strongly convex case.
Although in the statistical setting
EWA performs optimally in expectation, it does not do so with high probability
[Audibert, 2007].
As a matter of fact, it will be clear from our analysis that this is related to the fact that EWA does not satisfy a bound of the form (3) (see Theorem 1).
This problem is one of the motivations behind the work of Wintenberger [2017].
Exploiting negative regret.
The possibility of getting a negative excess risk has been recently explicitly exploited by Puchkin and Zhivotovskiy [2022, see their Algorithm 3.3 and Theorem 3.4] in the setup of active learning with abstentions.
In the context of online to batch conversion of online learning algorithms, a similar idea is exploited in Section 4.1.
Moreover, our selective sampling results in Section 6 are of the same flavor.
Our Algorithms
In the following, g 1 , g 2 , . . . are real numbers in a bounded interval.
However, in most applications we use g t = ℓ ′ t ( (cid:98) y t ) .
The algorithms in this section are simplified versions of algorithms in the literature.
Namely, in Section 3.1 we present a simplified version of Squint [Koolen and Van Erven, 2015] and in
Section 3.2 we present a simplified version of MetaGrad [Van Erven et al., 2021].
The simplifications lie in the tuning of the learning rate.
Squint and MetaGrad optimize the learning rate online at a small cost, whereas in our applications we only need a fixed learning rate that is known in advance, in which case we do not pay the small cost for optimizing the learning rate.
The proofs of the results in this section are postponed to Appendix A.
Algorithm Input η > 0 , σ > 0 , G > 0 , Z > 0 ; Initialize γ = ηG 2 , w 1 = 0 , and Σ − 1 1 = 1 σ I ; for t = 1 , . . . , T do
Receive x t
Set W t = (cid:84) ts =1 { w : |⟨ w , x s ⟩| ≤ Z }
Set w t = argmin w ∈W t ( w − (cid:101) w t ) ⊤ Σ − 1 t ( w − (cid:101) w t )
Predict (cid:98) y t = ⟨ w t , x t ⟩
Receive g t and κ t
Set z t = γ x t g t
Set Σ − 1 t +1 =
κ t 2 z t z ⊤ t + Σ − 1 t
Set (cid:101) w t
{++
+}
=
w t − z t Σ t +
3.1 A Simple Algorithm for Prediction with Expert Advice
Algorithm 1 is a simplified version of Squint [Koolen and Van Erven, 2015].
The κ t parameter is relevant only to the selective sampling setting, where it is used to control the range of loss estimates.
In all other settings we set κ t = 1 for all t .
Note that in round t both κ t and κ t − 1 are used to update the algorithm.
This is due to a technicality in the analysis of the algorithm, where a κ t − 1 (cid:101) ℓ t ( i ) term appears and we want to use the inequality x − x 2 ≤ log(1 + x ) for | x | ≤ 12 (specifically in equation (7)).
The regret bound of Algorithm 1 can be found in Lemma 2.
Lemma 2.
For all g 1 , . . .
, g T ∈ [ − M, M ] and κ 0 =
κ 1 ≥ κ 2 · · · ≥ κ T such that κ t ∈ (0 , 1] , the predictions (cid:98)
y t of Algorithm 1 run with input M and η ∈ (0 , 12 ] satisfy
T (cid:88) t
= ( (cid:98) y t − y t
( i ))
g t ≤ M 2 log( K ) κ
T η + η
T (cid:88)
[-+
 =-][Equation]
provided max i | y t ( i ) − y ⋆t | ≤ M for all t ≥ 1 .
As an immediate corollary of Lemma 2 and Lemma 1 we have the following regret bound.
Corollary 1.
Fix an arbitrary sequence ℓ 1 , . .
. , ℓ T of µ -strongly convex differentiable losses such that max t | ℓ ′ t | ≤ M .
Provided that max i | y ⋆t − y t ( i ) | ≤ M for all t ≥ 1 , the predictions (cid:98)
y t of
Algorithm 1 run with inputs M and η ∈ [0 , 12 ) , κ t = 1 for all t , and feedback g t = ℓ ′ t ( (cid:98) y t ) , satisfy
[Equation]
As an example, let us consider the squared loss, which is 2 -strongly convex.
In the setup of Example 1, since | ℓ ′ t ( (cid:98) y t ) | = 2 | ( (cid:98) y t − y t ) | ≤ 4 , Algorithm 1 with g t = ℓ ′ t ( (cid:98) y t ) and
η = 12 gives us the regret bound claimed in Example 1, namely R T ≤ 32 log( K ) − 1 2 (cid:80) T t =1 ( (cid:98)
y t − y ⋆ t ) 2 .
Our next result is a formal version of Proposition 1 saying that the above regret bound cannot be achieved by the standard EWA algorithm.
For standard notation and explicit details on this algorithm we refer to Appendix E.
Theorem 1.
Consider the squared loss and two experts y t (1) = 0 and y t (2)
=
for all t ≥ 1 . Let (cid:98) y EWA t be the EWA predictions.
There is a sequence y 1 , y 2 , . . .
such that y t ∈ [0 , 1] , t ≥ 1 and, for large enough T , the regret of EWA with η = 12 satisfies − 12 log T ≤ R T ≤ 2 log 2 and, at the same time,
T (cid:88) [Equation] [Equation] −
[Equation] [Equation]
3.2 A Simple Algorithm for Online Regression
In the following we use y t ( u ) = ⟨ u , x t ⟩ .
We prove a regret bound for Algorithm 2, which is a simplified version of MetaGrad [Van Erven et al., 2021].
The role of the parameter Z in the algorithm is to ensure the predictions (cid:98) y t are bounded, which will be important in the statistical learning setting in Section 4.
Similarly to Algorithm 1, the κ t parameter is only used in the selective sampling setting.
Lemma 3.
For all g 1 , . . .
, g T ∈ R and κ 1 ≥ · · · ≥ κ T ∈ (0 , 1] , the predictions (cid:98)
y t of Algorithmrun with inputs η > 0 , σ = D 2 , G ≥ max t | g t | , and Z >[-
-]
T (cid:88) t =
[Equation]
−
[Equation]
u )) [Equation] for any x 1 , . . .
, x T ∈ R d , and for any u ∈ W T ≡ (cid:84) Tt =1 { w : |⟨ w , x t ⟩| ≤ Z } such that ∥ u ∥ 2 ≤ D .
Example 2.
Consider the setup of Example 1 and suppose that max t ∥ x t ∥ 2 , ∥ u ∥ 2 ≤ 1 and M = 1 .
We have that | ℓ ′ t ( (cid:98) y t )
| = | 2( (cid:98)
y t − y t ) | ≤ 4 and thus, by Lemma 3, an appropriately tuned Algorithmwith g t = ℓ ′ t
( (cid:98) y t ) satisfies (2) with C T = 8 + 8 d log (cid:0) 1 + η 2 Td (cid:1) and B T = 0 .
Thus, by Lemma 1, setting η = 12 gives us
[Equation]
Statistical Learning
We discuss an application of our general results in the context of statistical learning where we are interested in the generalization of estimators to unseen samples.
A tool often used in converting online learning algorithms to the statistical learning setting is online to batch conversion [Cesa-Bianchi et al., 2004].
Let us recall the setup.
Assume that we are given a family F of real-valued functions defined on the instance space X .
We observe T i.i.d.
observations ( X t , Y t )
Tt =1 distributed according to some unknown distribution P on X × R .
Given the loss function ℓ : R 2 → R , define the risk R ( f ) of f : X → R as
R ( f ) =
E ℓ ( f ( X ) , Y ) , where the expectation is taken with respect to the joint distribution of X and Y .
We are interested in bounding the excess risk R ( (cid:98) f ) − inf f ∈F R ( f ) , where (cid:98) f is constructed based on the sample ( X t , Y t )
Tt =1 .
Assume that there is a sequence of predictors (cid:98)
f 1 , . .
, (cid:98) f T trained in an online manner using ( X t , Y t ) Tt =1 (that is, (cid:98) f
k depends on ( X t , Y t )
k − 1 t =1 ) such that almost surely (cid:80)
Tt =1 (cid:16) ℓ ( (cid:98) f t ( X t ) , Y t ) − ℓ ( f ⋆ ( X t ) , Y t ) (cid:17) ≤ R T , where R T is non-random.
In this case, a standard online to batch conversion approach gives an in-expectation excess risk bound E R (cid:16) 1 T (cid:80) Tt
=1 f t (cid:17) − inf f ∈F R ( f ) ≤ R T T , for any loss convex in its first argument and where the expectation is taken with respect to the learning sample ( X t , Y t )
Tt =1 .
However, getting a high-probability version of this result is a known challenge if one wants to get the fast rate O (cid:0) 1 T (cid:1) .
A standard way of proving a high-probability result is to apply Freedman’s inequality for martingales [Kakade and Tewari, 2008] that leads to a variance term scaling as O (cid:16) 1 √ T (cid:17) .
For example, Audibert [2007] showed that this is the case if one wants to prove a high-probability excess risk bounds based on EWA.
A way to handle the variance term in Freedman’s inequality is by exploiting the Bernstein assumption as in Kakade and Tewari [2008].
Unfortunately, this assumption is not necessarily satisfied by the stochastic environments we are considering.
The main idea in this section is to use the negative term from
Lemma 1 to cancel out this variance term appearing due to Freedman’s inequality 1 .
We use the following notation when applying the online algorithms in the statistical setting: ℓ t ( · ) = ℓ ( · , Y t ) and y t ( f ) = f
( X t ) .
4.1 Statistical Learning: Model Selection Aggregation
In this section, we discuss the application of our results to the model selection (MS) aggregation.
This setup was introduced by Nemirovski [2000] and further studied by Tsybakov [2003] and by Audibert [2007], Lecué and Mendelson [2009], Lecué and Rigollet [2014], Wintenberger [2017], Mourtada et al.
[2021] among other works.
We introduce some notation.
Assume that we are given a finite dictionary F = { f 1 , . . .
, f K } of real-valued functions defined on the instance space X .
We observe
T i.i.d.
observations ( X t , Y t )
Tt =1 distributed according to some unknown distribution P on X × R .
Input T , M , η , stopping threshold S ;
Initialize S = 0 , provide η , and M as input for Algorithm 1 ; while S < T and S > µ 8 min f ∈F (cid:80)
St =1 ( (cid:98)
y t − f ( X t )) 2 do Receive X t and send f 1 ( X t ) , . . , f K ( X t ) as expert predictions to Algorithm Receive p t and (cid:98) y t =
(cid:80)
Ki =
1 p t ( i ) f
i ( X t ) from Algorithm
Predict (cid:98) y t and receive ℓ t Send g t = ℓ ′ t
( (cid:98) y t ) and κ t = 1 to Algorithm
Set S = S
+
Output (cid:98) f = 1 S (cid:80)
St =1 (cid:80) Ki
=1 p t ( i ) f
i ;
Given the loss function ℓ : R 2 → R , define the risk R ( f ) of f : X → R as R ( f ) = E ℓ ( f ( X ) , Y ) , where the expectation is taken with respect to the joint distribution of X and Y .
In the model selection aggregation, one is interested in constructing an estimator (cid:98) f
based on the random sample ( X t , Y t )
Tt = such that, with probability at least 1 − δ ,
[Equation]
under appropriate boundedness and curvature assumptions on the loss function ℓ .
Analogously to
Tsybakov [2003], the bound of the form (4) will be called the optimal rate of aggregation .
We make use of a variant of online to batch conversion [Cesa-Bianchi et al., 2004] where we stop the procedure early if the empirical variance of predictions is sufficiently large.
We sketch the idea.
Let
S be the number of samples we have used before we terminated the procedure.
We use Algorithmas our aggregation procedure and use (cid:98) f
= 1 S (cid:80)
St =1 (cid:80) Ki
=1 p t ( i ) f i .
By Jensen’s inequality we have
R ( (cid:98) f ) ≤ 1 S (cid:80)
St =1 E t − 1 (cid:104) ℓ t (cid:16)(cid:80)
Ki =1 p t ( i ) f
i ( X t ) (cid:17)(cid:105) .
To motivate stopping early, observe that if the empirical variance in Lemma 1 is sufficiently large, we may conclude that the excess risk is negative and we have outperformed the best f ∈ F .
The result can be found in Theorem 2 below, whose proof is implied by Theorem 8 in Appendix B.
Theorem 2.
Suppose that for all f ∈ F | f ( X ) | ≤ 12 almost surely, that | ∂ y ℓ
y, Y )
| ≤ 1 almost surely for all y such that | y | ≤ 12 , and that ℓ is µ -strongly convex in its first argument.
Then, with probability at least 1 − δ , Algorithm 3 with input parameters T , S
=
O (cid:16) log( K )+log(log
( T ) /δ ))
µ (cid:17)
[Equation]
η = µ 8 , and M = 1 satisfies
[Equation]
where S is the number of steps of Algorithm 3.
When Algorithm 3 terminates at step S =
T , we recover the optimal high probability bound for model selection aggregation (4) up to an additive log log T term.
However, when S = T our bound tells us slightly more, because we know that for all t ′ < T , min f ∈F (cid:80) t ′ t =1 ( (cid:98)
y t − f ( X t ))
= O (log K + log log T ) , which means that on the sequence ( X t , Y t )
Tt =1 our predictions (cid:98) y t are essentially following the prediction of the currently best expert at each round.
In the special case where we are solely interested in the best possible performance of the online to batch conversion of Algorithm 1, we can remove the log log T term appearing in the previous bound.
We remark that, apart from the work of Wintenberger [2017], no known analysis based on the online to batch conversion achieved the optimal rate of aggregation (4).
We also believe that our analysis is simpler than for previously known algorithms.
The result can be found in Theorem 3 below, whose result is implied by Theorem 9 in Appendix B.
Theorem 3.
Suppose that for all f ∈ F | f ( X ) | ≤ 12 almost surely, that | ∂ y ℓ
y, Y )
| ≤ 1 almost surely for all y such that | y | ≤ 12 , and that ℓ is µ -strongly convex in its first argument.
Then, with probability at least 1 − δ , Algorithm 3 with input parameters T , η = µ 4 , S = ∞ , and M = 1 , guarantees
[Equation]
4.2 Statistical Learning: Regression
We consider the statistical learning setting where one has access to T i.i.d.
samples of pairs ( X t , Y t ) ∈ R d × R .
In this case, we consider F ⊆ { x
(cid:55)→ ⟨ w , x ⟩ : w ∈ R d } .
For w ∈ R d we define the risk as
R ( w ) =
E [ ℓ ( ⟨ w , X ⟩ , Y )] .
We assume that ℓ is µ -strongly convex in its first argument.
To the best of our knowledge there are no known high probability excess risk bounds in regression based on online to batch conversions with convergence rate O (cid:16) d log( T ) T (cid:17) .
We provide such a result in the bounded setup where the feature vectors, derivatives of the losses, and the norm of the reference vector are bounded.
Similarly to before, for a result that holds with high probability, one needs to control the cumulative variance of our prediction.
For standard online learning algorithms the control of the variance may prove troublesome.
For example, Mourtada et al.
[2021] showed that a version of
Vovk-Azoury-Warmuth forecaster [Vovk, 2001, Azoury and Warmuth, 2001] may have a O (1) excess risk bound with constant probability, whereas in expectation the Vovk-Azoury-Warmuth forecaster guarantees a O (cid:16) d log( T ) T (cid:17) excess risk bound.
Instead, we leverage the negative empirical variance of
Lemma 1 to control the variance of the online to batch conversion, leading to the following excess risk bound, whose result is implied by Theorem 10 in Appendix B.
Theorem 4.
Suppose that ∥ X ∥ 2 ≤ 1 and sup y ∈ [ − 1 , 1] | ∂ y ℓ ( y, Y ) | ≤ 1 almost surely, ∥ w ∥ 2 ≤ 1 , and that ℓ is µ -strongly convex in its first argument.
Then, with probability at least 1 − δ ,
[Equation]
(cid:32) [Equation] T T (cid:88) t = w t (cid:33) − R ( w ) =
O (cid:18) d log( T ) + log(1 /δ )
µT (cid:19) , where w t are given by Algorithm 2 with η = µ 4 , σ = 1 , Z = 1 , G = 1 , κ t = 1 , and feedback g t = ℓ ′ t ( ⟨ w t , X t ⟩ ) for t = 1 , . . .
, T .
Corrupted Feedback
In this section, we study a setting where the loss derivatives ℓ ′ t ( (cid:98) y t ) observed by the learner at each round t may be adversarially corrupted by unknown additive constants c t , and we are interested in the best possible dependence on c 1 , . .
, c T in the regret bound.
To better explain our setting, we start with the following example.
Example 3.
In the online regression setting, suppose that ℓ t ( (cid:98) y t ) =
( (cid:98) y t − y t ) 2 for all t , but the learner observes corrupted outcomes y t − c t / 2 .
Hence, the squared loss derivative computed by the [-learneris 
 (-]learner is 2( (cid:98) y t − y t + c t / 2) =
ℓ ′ t ( (cid:98) y t ) + c t , which can be handled by the algorithms developed in this section.
Several variants of this setting have been studied in prior work, see for example [Lykouris et al., 2018, Amir et al., 2020, Zimmert and Seldin, 2021, Ito, 2021] and the references therein.
The main difference between our setting and these previous settings is that we assume our losses to be strongly convex and the environment is not necessarily stochastic.
Although the results in this section are rather straightforward corollaries of our bounds, we believe that it is instructive to provide some explicit results.
All proofs of the results in this section are postponed to Appendix C.
Our first result shows the performance of Algorithm 1 in the setup with corrupted gradients.
The proof follows from observing that ℓ ′ t ( (cid:98) y t )( (cid:98) y t − y ⋆t ) =
( ℓ ′ t ( (cid:98) y t ) + c t )( (cid:98) y t − y ⋆t ) − c t ( (cid:98)
y t − y ⋆t ) and that for any λ > 0 , the inequality | c t ( (cid:98)
y t − y ⋆t )
| ≤ c 2 t λ + λ 4 ( (cid:98) y t − y ⋆t ) 2 holds.
The λ 4 ( (cid:98) y t − y ⋆t ) 2 term can be compensated for by the negative µ 2 ( (cid:98)
y t − y ⋆t )
2 appearing in Lemma 1, leading to a (cid:80)
Tt =1 c 2 t additive term in the regret bound.
In particular, our result implies that as long as (cid:80) Tt =1 c 2 t is of order
O (log K ) , the same regret bound as if the losses were not corrupted.
The formal statement can be found in Theorem 5 below.
Theorem 5.
Fix an arbitrary sequence ℓ 1 , . .
. , ℓ T of µ -strongly convex differentiable losses and corruptions c 1 , . .
. , c T ∈ R .
Then the predictions (cid:98) y t of Algorithm 1 run with inputs M ≥ max t | ℓ ′ t ( (cid:98) y t ) + c t | , η = µ 4 , feedback g t = ℓ ′ t ( (cid:98) y t ) + c t , and κ t = 1 satisfy
R T ≤ 8 M 2 log( K ) µ + (cid:88) T t = c 2 t µ − µ (cid:88) T t =1 ( (cid:98) y t − y t ( i ⋆ ))provided max i max t | (cid:98)
y t − y t ( i ) | ≤ M .
Next we prove an analog of Theorem 5 in the online regression setup.
Theorem 6.
Fix an arbitrary sequence ℓ 1 , . .
. , ℓ T of µ -strongly convex differentiable losses and corruptions c 1 , . .
. , c T ∈ R .
Then the predictions (cid:98) y t of Algorithm 2 run with inputs η = µ 8 , σ = D 2 ,
G ≥ max t | ℓ ′ t ( (cid:98) y t )
+ c t | , Z > 0 , feedback g t = ℓ ′ t ( (cid:98) y t ) + c t , and κ t = 1 , satisfy
R T ≤ 4 dGµ log (cid:18) 1 + TD 2 µ 2 max t ∥ x t ∥2 d (cid:19)
+ 4 Gµ +
T (cid:88) t = c 2 t µ − µ
T (cid:88) t
= ( (cid:98) y t − y ⋆t ) 2 , for any x 1 , . . .
, x T ∈ R d , and for any u ∈ W T ≡ (cid:84) Tt =1 { w : |⟨ w , x t ⟩| ≤ Z } such that ∥ u ∥ 2 ≤ D and y ⋆t = ⟨ u , x t ⟩ for all t ≥ 1 .
Selective Sampling
We consider a variant of the selective sampling setting—see, e.g., [Atlas et al., 1989, Freund et al., 1997, Cesa-Bianchi et al., 2003, 2006, Orabona and Cesa-Bianchi, 2011]—where the learner has access to the expert predictions (or, equivalently, to feature vectors), but can observe its own loss only upon request.
The goal is to trade off the number of loss requests with regret guarantees.
We show that if the variance is high, with only a fraction of all losses requested we obtain the same guarantee (in expectation and up to constants) as when all losses are requested.
Let o t = 1 with probability q t and o t = 0 with probability 1 − q t .
In each round, if o t = 1 the loss ℓ t at round t is requested, and we use the loss estimator o t q t − 1 ℓ t to update.
Note that this is not the importance weighted estimator, as E t − 1 [ o t q t − 1 ] =
q t q t − 1 .
The reason for choosing this particular loss estimator is that we have better control of the range of the loss which allows us to tune κ t in
Algorithm 1 accordingly.
The probability of requesting a loss is
[Equation]
where β > 0 is chosen by the learner.
Our result for the selective sampling setting can be found in
Theorem 7 below, whose statement is implied by Theorem 11 in Appendix D. Theorem 7 implies the following: if β = O ( µ − 3 / 2 log( K )) , then with only an expected number (cid:80) Tt =
1 q t of loss requests, we obtain (up to constants) the same regret guarantee as we would have obtained if we had requested all losses.
With this particular choice of β , q t < 1 as soon the bound in Lemma 1 becomes negative.
In other words, when the variance is high we only need a fraction of the losses to recover the worst-case optimal regret bound (in expectation).
A similar result can be obtained in the regression setting, see
Appendix D.1.
Theorem 7.
Fix an arbitrary sequence ℓ 1 , . .
ℓ T of µ -strongly convex differentiable losses.
Provided max [Equation] y t − y t ( i ) | ≤ 1 , the predictions (cid:98)
y t of Algorithm 1 run with inputs M = 1 ≥ max t | ℓ ′ t ( (cid:98) y t ) | , η = µ 4 , feedback g t =
o t q t − 1 ℓ ′ t
( (cid:98) y t ) , and κ t =
q t satisfy
E (cid:34) T (cid:88) t = ( ℓ t ( (cid:98) y t ) − ℓ t ( y ⋆t ))
(cid:35) =
O (cid:18) log( K ) µ
+ log( K )µ 3 β(cid:19) .
Further Extensions
In Appendix G we present another application of Lemma 1.
Namely, we show that we may restart
Algorithm 1 for free whenever the regret becomes negative.
This gives us a regret bound where we compete with a new expert after each restart, which is a stronger notion of regret than when we compete with a fixed expert in all rounds.
The ideas of Lemma 1 also naturally extend beyond online learning with strongly convex losses.
We discuss one such extension to the online prediction with expert advice setting.
The online prediction with abstention setting was introduced by Neu and Zhivotovskiy [2020] and proceeds as follows.
In each round t = 1 , . . .
, T the learner receives expert predictions y t ( i ) ∈ {− 1 , 1 } , i = 1 , . . .
, K and the learner can then either predict (cid:101)
y t ∈ {− 1 , 1 } or abstain from prediction.
If the learner predicts with (cid:101) y t , the learner suffers the binary loss ℓ t ( (cid:101) y t ) = [-
-]1 [ (cid:101) y t ̸ = y t ] , where y t ∈ {− 1 , 1 } .
If the learners abstains from prediction, the learner suffers abstention cost ρ ∈ [0 , 12 ) .
Let a t = 1 for prediction and a t = 0 for abstention.
The total loss of the learner is therefore equal to
[Equation]
Assume that the prediction strategy is random is a sense that a t are Bernoulli random variables whose means might depend on previous observations.
The work of Neu and Zhivotovskiy [2020] shows that there is a randomized prediction strategy such that for any data generating mechanism it holds that
[Equation]
independently of T , where the expectation is taken with respect to the randomness of a t ; here y ⋆t =
y t ( i ⋆ ) and i ⋆ = argmin
i (cid:80) Tt =1
[ y t ( i ) ̸
= y t ] .
Although it was shown that the randomization is necessary to achieve the regret bound (6), it is unclear if the same regret bound can be achieved with high probability with respect to the randomization of the algorithm.
In Appendix F, we answer this question using the techniques we developed and provide a randomized algorithm such that, with probability at least 1 − δ ,
[Equation]
In Appendix F we prove Lemma 7, which is the analog of Lemma 1 for the abstention setting.
The equivalent of the negative term in Lemma 1 is used to compensate for the variance of the highprobability statement, which allows us to recover the above bound.
This also implies that our other applications of Lemma 1 can be exported to the online learning with abstention setting.
Acknowledgments and Disclosure of Funding
Dirk van der Hoeven and Nicolò Cesa-Bianchi gratefully acknowledge partial support from the MIUR
PRIN grant Algorithms, Games, and Digital Markets (ALGADIMAR) and the EU Horizon
ICT-48 research and innovation action under grant agreement 951847, project ELISE (European
Learning and Intelligent Systems Excellence).
A significant part of this work was done while Nikita
Zhivotovskiy was at ETH, Zürich.
During this period, Nikita Zhivotovskiy was funded in part by
ETH Foundations of Data Science (ETH-FDS).