Precise Regret Bounds for Log-loss via a Truncated Bayesian Algorithm
Abstract
We study sequential general online regression, known also as sequential probability assignments, under logarithmic loss when compared against a broad class of experts.
We obtain tight, often matching, lower and upper bounds for sequential minimax regret, which is defined as the excess loss incurred by the predictor over the best expert in the class.
After proving a general upper bound we consider some specific classes of experts from Lipschitz class to bounded Hessian class and derive matching lower and upper bounds with provably optimal constants.
Our bounds work for a wide range of values of the data dimension and the number of rounds.
To derive lower bounds, we use tools from information theory (e.g., Shtarkov sum), and for upper bounds we resort to new “smooth truncated covering” of the class of experts.
This allows us to find constructive proofs by applying a simple and novel truncated Bayesian algorithm.
Our proofs are substantially simpler than the existing ones and yet provide tighter (and often optimal) bounds.
Introduction
In online learning and sequential probability assignments arising in information theory, portfolio optimization and machine learning, the training algorithm consumes d dimensional data in rounds t ∈ { 1 , 2 , . . .
, T } and predicts the label ˆ y t based on data received and labels observed so far.
After prediction, the true label y t is revealed and the loss ℓ ( y t , ˆ y t ) is incurred.
The (pointwise) regret is defined as the (excess) loss incurred by the algorithm over a class of experts, also called the hypothesis class.
More precisely, in each round t ≥ 1 the learner obtains a d dimensional input/ feature vector x t ∈ R d .
In addition to x t the learner may use the past observations ( x r , y r ) , r < t to make a prediction ˆ y t of true label.
Therefore, the prediction can be written as ˆ y t = ϕ t
( y t − 1 , x t ) , where y t − 1 represents the labels in the past t − 1 rounds, x t represents the input vectors in t rounds, and ϕ t represents the strategy of the learner to obtain its prediction based on the past and current observations.
Once a prediction is made, nature reveals the true label y t and the learner incurs loss ℓ : ˆ Y × Y → R , where ˆ Y and Y are the prediction and true label domains respectively.
Hereafter, we assume throughout ˆ Y =
[0 , 1] , Y = { 0 , 1 } with logarithmic loss
[Equation]
In regret analysis, we are interested in comparing the accumulated loss of the learner with that of the best strategy within a predefined class of predictors (experts) denoted by H .
More precisely, H is a collection of predicting functions h :
R d
(cid:55)→ ˆ Y with input being x t at time t .
Therefore, given a learner ϕ t , t > 0 and ( y t , x t ) Tt =1 after T rounds the pointwise regret is defined as
[Equation]
where ˆ y t = ϕ t ( y t − 1 , x t ) .
Observe that the first term above represents the accumulated loss incurred by the learning algorithm, while the second summation deals with the best prediction within H .
We highlight two useful perspectives on analyzing the regret next.
Fixed Design: This point of view studies the minimal regret for the worst realization of the label with the feature vector x T known in advance.
Suppose that the learner has a fixed strategy ϕ t , t > 0 .
Then, the fixed design minimax regret for a given x T is defined as
[Equation]
Further, the fixed design maximal minimax regret is given by:
[Equation]
Sequential Design: In this paper we mostly focus on the sequential or agnostic regret in which the optimization on regret is performed at each time t without knowing in advance x T or y T .
Then the sequential (maximal) minimax regret is given by [24]:
[Equation]
In [37] it is shown that r aT ( H ) ≥ r ∗ T ( H ) for all H .
We will use r ∗ T ( H ) as our tool to derive lower bounds for r aT ( H ) .
Our main goal is to gain insights into the growth of sequential regret r aT ( H ) for various classes H and to show how the structure of H , as well as the relationship between d and T impact the precise growth of the regret.
To see this more clearly, we briefly review regret in universal source coding.
Regrets in Information Theory.
In universal compression, the dependence between regret and the reference class was intensively studied [10, 21, 27, 28, 32, 38, 39].
Here, there is no feature vector x t , and the dimension d = 1 .
A sequence y T is generated by a source P that belongs to a class of sources S , which can be viewed as the reference class H in online learning.
The minimax regret for the logarithmic loss is given by [9, 31, 10]:
[Equation]
where Q is the universal probability assignment approximating the unknown P .
The main question is how the structure of S impacts the growth of the minimax regret.
Let m denote the alphabet size (in online learning, we only consider m = 2 ).
It is known [10, 21, 27, 28, 32, 38, 39] that for Markov sources of order r , regret grows as m r ( m − 1) / 2 log T for fixed m
[27, 21, 28, 33], while in [33] minimax regret was analyzed for all ranges of m and T .
For non-Markovian sources, the growth is super logarithmic.
For example, for renewal sources of order r the regret is Θ( T r/ ( r +1) )
[7] and the precise constant in front of the leading term is know for r = 1 [11].
However, it should be pointed out that [6, 1] studied the general classes of densities smoothly parameterized by a d -dimensional data to obtain general results for minimax regret that can be phrased as an online regret.
Main Contributions.
Our main results are summarized in Table 1.
One of the main contributions of this paper is the concept of a global sequential covering used to prove constructively general upper bounds on regret (Theorem 1).
We establish Theorem 1 via a novel smooth truncation approach enabling us to find tight upper bounds that subsume the state-of-the-art results (e.g., [23, 3]) obtained non-algorithmically.
In fact, Algorithm 2 developed in this paper achieves these bounds.
Moreover,
Theorem 1 provides optimal constants that are crucial to derive the best bounds in special cases discussed next.
For general Lipschitz parametric class H , in Theorem 2, we derive the upper bound d log( T/d ) + O ( d ) for T > d .
In Theorem 3, we show that the leading constant 1 (in front of d log( T/d ) ) is optimal for T ≫ d log( T ) .
Furthermore, we obtain the best constant for the leading term d 2 log( T/d ) when the Hessian of log f is bounded for any function f ∈ H (see Theorem 4).
Then, we show in Theorem 5 that the constant 12 in our bound is optimal for functions of the form f ( ⟨ w , x ⟩ ) , where w ∈ R d is the parameter of the function, ⟨ w , x ⟩ is the inner product in R d , w and x are in a general ℓ s -norm unite ball, and T ≫ d ( s +2)
/s .
This result recovers all the lower bounds in [29] obtained for logistic regression (however, the technique of [29] works for other functions with bounded second derivatives, like the probit function).
Lastly, when d ≥ T , for a linear function of the form |⟨ w , x ⟩| we show that the growth is at least Ω( T s/ ( s +1) ) under ℓ s ball and
at most ˜ O
( T 2 / 3 ) for ℓ 2 ball (see Theorem 6 and Example 2).
The main technique used in our paper (smooth truncation) is novel with other potential applications (e.g., average minimax regret).
Instead of the conventional approach for truncating only the values close to { 0 , 1 } , we truncate all values in [0 , 1] in a smooth way (see Lemma 4).
This allows us to obtain an upper bound via a simple truncated Bayesian algorithm.
Our proofs are substantially simpler (cf.
[3]) yet provide tighter and often optimal bounds.
In summary, our main contributions are: (i) constructive proofs through a new smooth truncated
Bayesian algorithm; (ii) the novel application of global sequential covering in the context of logarithmic loss; (iii) lower and upper bounds with optimal leading constants; and (iv) novel informationtheoretic techniques for the lower bounds.
[Table]
General
[Table]
inf
[Table]
General Lipschitz f
Any r aT ( H f ) ≤ d log (cid:0) Td + 1 (cid:1) + O ( d )
Theoremunder ℓ s ball
T ≫ d log T r aT ( H f ) ≥ d log (cid:0) Td (cid:1) − O
( d log log T ) Theorem
Bounded Hessian
Any r aT ( H f ) ≤ d 2 log (cid:18) T d +(cid:19) + O ( d )
Theorem 4 of log f under ℓ 2 ball f ( ⟨ w , x ⟩ ) with f ′ (0) ̸
=
T ≫ d ( s +2)
/s r aT ( H f ) ≥ d 2 log (cid:0)
T
[Equation] [Equation]
/s
[Equation]
− O ( d )
Theorem 5 under ℓ s ball |⟨ w , x ⟩| under ℓ s ball
[Equation]
Theorem|⟨ w , x ⟩| under ℓ 2 ball
[Equation]
Example
Related Work
In this paper we study sequential minimax regret for general online regression with logarithmic loss using tools of information theory, in particular universal source coding (lower bounds)
[1, 10, 18, 21, 26, 27, 28, 38] and sequential covering (upper bounds).
Most of the existing works in online regression deals with logistic regression.
We first mention the work of [13], who studied pointwise regret of logistic regression for the proper setting.
Unlike improper learning, studied in this paper, where feature x t at time t is also available to the learner, [13] showed that pointwise regret is Θ( T 1 / 3 ) for d = 1 and O ( √ T ) for d > 1 .
Furthermore, [17] demonstrates that regret for logistic regression grows as O ( d log T/d ) .
This was further generalized in [12].
These results were strengthened in [29], which also provides matching lower bounds.
Precise asymptotics for the fixed design minimax regret were recently presented in [14, 15].
Regret bounds under logarithmic loss for general expert class H was first investigated by Vovk under the framework of mixable losses [16, 34].
In particular, Vovk showed that for finite class H , the regret growth is log |H| via the aggregating algorithm (i.e., the Bayesian algorithm that we will discuss below).
We refer the reader to [5, Chapter 3.5, 3.6] and the references therein for more results on this topic.
Cesa-Bianchi and Lugosi [5] were the first to investigate log-loss under general (infinite) expert class H [5, Chapter 9.10, 9.11], where they derived a general upper bound using the concept of covering number and a two-stage prediction scheme.
In particular, Cesa-Bianchi and Lugosi showed that for Lipschitz parametric classes with values bounded away from { 0 , 1 } , one can achieve a regret bound of the form d/ 2 log( T/d ) .
When the values are close to { 0 , 1 } , they used a hard truncation approach, which gives a sub-optimal bound of the form 3 / 2 d log( T/d ) (i.e., this bound is not explicitly shown in [5] but can be derived using their approach).
Moreover, the approach of [5] only works for fixed design regret (or simulatable in their context).
In [23], the authors extended the result of [5, Chapter 9.10] to the sequential case via the machinery of sequential covering that was established in [22].
However, [23] also used the same hard truncation as in [5] resulting in suboptimal upper bounds.
In [3], the authors obtained an upper bound similar to the upper bound presented in
Theorem 1 via the observation that the log function is self-concordant.
In particular, this allows them to resolve the tight bounds for non-parametric Lipschitz functions that map [0 , 1] s → [0 , 1] .
However, their bounds are proved non-constructively , i.e., the proof does not provide an algorithm that achieves such bounds.
In [4], the authors used a similar idea of smoothing for controlling the unboundedness of log-loss, however, they are assumed that the features x T are presented i.i.d. .
More importantly, the results in [4] only holds for the average case regret.
Problem Formulation and Preliminaries
We denote X as the input feature space and H as the concept class, which is a set of functions mapping
X → [0 , 1] .
We use an auxiliary set W to index H .
We say that a function g is sequential if it maps
X ∗ → [0 , 1] , where X ∗ is set of all finite sequences with elements in X .
We denote G as a class of sequential functions.
If T is a time horizon, then for any t ∈ [ T ] , we write x t = { x 1 , · · · , x t } and y t = { y 1 , · · · , y t } .
We use standard asymptotic notation f ( t ) =
O ( g ( t ))
if there exists a constant
C such that f ( t ) ≤ Cg ( t ) for sufficient large t ≥ 0 , and f ( t ) ≪ g ( t ) if lim sup t →∞ f ( t ) /g
( t ) = 0 .
We assume the log function log( x ) is the nature logarithm to the base of e .
The main objective of this paper is to study the growth of the sequential minimax regret r aT ( H ) for a large class of experts H .
We accomplish this goal using two different techniques.
For the lower bound, we precisely estimate the fixed design minimax regret r ∗ T ( H| x T ) using the Shtarkov sum [31], discussed next.
For the upper bound, we construct a global cover set G of H and design a new (truncated)
Bayesian algorithm to find precise bounds with constants that are provably optimal.
Lower Bounds.
We investigate the lower bound of adversarial regret r aT ( H ) by considering its corresponding fixed design minimax regret r ∗ T ( H| x T ) and r ∗ T ( H ) =
max x T r ∗ T ( H| x T ) .
We are able to do this due to the recent result [37], which we quote next.
Lemma 1 (Wu et al., 2022) .
Let H be any general hypothesis class and ℓ be any loss function.
Then
[Equation]
and the inequality is strict for certain H , and loss function ℓ .
We establish precise growth of r ∗ T ( H ) by estimating the Shtarkov sum that was intensively analyzed in information theory [31, 10] and recently applied in online learning [30, 14].
For the logarithmic loss, the Shtarkov sum (conditioned on x T ) is defined as follows
[Equation]
where P h ( y T | x T ) =
(cid:81)
Tt =1 h ( x t )
y t (1 − h ( x t ))
1 − y t
and we interpret h ( x t ) =
P ( y t = 1 | x t ) .
The regret can be expressed in terms of the Shtarkov sum (see [14, Equation (6)] or [5, Theorem 9.1]) as
[Equation]
It is known that the leading term in the Shtarkov sum for parametric classes H is often independent of x T [29, 12, 14, 15].
Therefore, the Shtarkov sum often gives the leading growth of r ∗ T ( H| x T ) independent of x T , which also suggests the leading growth of the agnostic regret r aT ( H ) .
Upper Bounds.
We now discuss our constructive approach to upper bounds.
In the next section, we present our Smooth truncated Bayesian Algorithm (Algorithm 2) that provides a constructive and often achievable upper bound.
Here we focus on some, mostly known, preliminaries.
Let G ⊂ [0 , 1] X ∗ be any reference class.
Let W be an index set of G and µ be an arbitrary finite measure over W .
The standard Bayesian predictor with prior µ is presented in Algorithm 1.
Based on this algorithm, we have the following two lemmas that are used to establish most of the upper bounds in this paper.
See e.g., [19, Lemma 3] or [5, Chapter 3.3] for proofs.
Input : Reference class G := { g w : w ∈ W} with index set W and prior µ over W 1: Set p w ( y 0 | x 0 )
= 1 for all w ∈ W .
2: for t = 1 , · · · , T do 3:
Receive feature vector x t 4:
Make prediction using the following equation:
[Equation]
Receive label y t 6:
For all w ∈ W , update: p w ( y t | x t ) =
e − ℓ ( g w ( x t ) ,y t )
p w ( y t − 1 | x t − 1 ) .
7: end for
Lemma 2.
Let G be a collection of functions
g w : X ∗ → [0 , 1] , w ∈ W .
Let ˆ y t be the Bayesian prediction rule as in Step 4 of Algorithm 1 with prior µ .
Then, for any x T and y T we have
[Equation]
µ
[Equation]
where p w ( y T | x T ) =
e − (cid:80) Tt =1 ℓ ( g w ( x t ) ,y t ) and ℓ is the log-loss as in equation (1).
The following lemma bounds the regret under log-loss of finite classes, which is well known.
Lemma 3.
For any finite class of experts G , we have r aT ( G ) ≤ log |G| .
Main Results
We start with a concept of covering set called the global sequential cover that was used implicitly in [24, Section 6.1] for the Lipschitz losses and dated back to the ideas in [2].
Definition 1 (Global sequential covering) .
For any H ⊂ [0 , 1] X , we say that class of sequential functions G ⊂ [0 , 1] X ∗ is a global sequential α -covering of H at scale T if for any x T ∈ X T and h ∈ H , there exists g ∈ G
such that ∀ t ∈
[ T ] ,
[Equation]
Throughout we assume that 0 ≤ α ≤ 1 .
Note that the global sequential covering is different from the (local) sequential covering used in [3] (originally from [24]), since our covering function does not depend on the underlying trees as in [24] 2 .
This is crucial to apply our covering set directly in an algorithmic way (see Algorithm 2).
Particularly, it enables us to establish our lower and upper bounds for Lipschitz classes of functions with the optimal constants on the leading term.
We further improve these results for Lipschitz class with bounded Hessian.
Finally, we study cases when the data dimension d grows faster than T by bounding the covering size through the sequential fat-shattering number.
In particular, we prove matching (up to poly log T factor) upper and lower bounds for the generalized linear functions.
General Results.
We are now in the position to state our first main general finding.
Theorem 1.
If for any α > 0 there exists a global sequential α -covering set G α of H , then
[Equation]
and this bound is achievable by Algorithm 2.
We should point out that Theorem 1 also improves the results of [3] by obtaining better constants in front of both αT and log |G
α | (i.e., from (4 , 4) to (2 , 1) ).
The proof is based on the following key lemma that is established in Appendix A.
Input : Reference class G with index set W and prior µ over W , and truncation parameter α 1: Let p w ( y 0 | x 0 ) =
1 for all w ∈ W 2: for t = 1 , · · · , T do 3:
Receive feature x t 4:
For all w ∈ W , set
[Equation]
Make prediction
[Equation]
Receive label y t 7:
For all w ∈ W , update: p w ( y t | x t ) =
e − ℓ (˜ g w ( x t ) ,y t )
p w ( y t − 1 | x t − 1 ) .
8: end for
Lemma 4.
Suppose H has a global sequential α -covering set G for some α ∈ [0 , 1] .
Then, there exists a truncated set ˜ G of G with | ˜
G| = |G| such that for all x T , y T and h ∈ H there exists a ˜ g ∈ ˜
G satisfying
[Equation] [Equation]
where p h ( y T | x T ) =
T (cid:89) t
= h ( x t )
y t (1 − h ( x t ))
1 − y t and p ˜ g
( y T | x T ) =
T (cid:89) t = ˜ g ( x t ) y t (1 − ˜ g ( x t ))
1 − y t .
Proof of Theorem 1.
We show that for any 0 ≤ α ≤ 1 if an α -covering set G α exists, then one can achieve the claimed bound for such an α .
To do so, we run the Smooth truncated Bayesian Algorithm (Algorithm 2) on G α with uniform prior and truncation parameter α .
We denote by ˜
G
α the truncated class of G α as in Lemma 4 (same as the step 4 of Algorithm 2).
We now fix x T , y T .
By Lemma(with G being ˜ G α ), we have
[Equation]
|
[Equation]
| , the last equality follows from |G α | = | ˜
G α | .
Since (cid:80) Tt =1 ℓ ( f ( x t ) , y t ) =
− log p
f ( y T | x T ) for any function f , then by Lemma 4 we conclude that
[Equation]
The result follows by combining the inequalities and noticing that log(1+ x ) ≤ x for all x ≥ − 1 .
We further note that for any constant c 1 , c 2 for which the bound r aT ( H ) ≤ c 1 αT + c 2 log |G
α | holds universally we must have c 1 ≥ 2 and c 2 ≥ 1 .
Therefore, our bounds are optimal with respect to the constants 3 .
To see this, we let X =
[ T ] and define g to be the function that maps every t ∈ [ T ] to 12 .
Let H be the class of functions that maps to [1 / 2 − α, 1 / 2 + α ] .
Clearly, H is α -covered by g .
By noting that the maximum probability is (1 / 2 + α ) T = (1 + 2 α ) T (1 / 2) T , we compute the Shtarkov sum (5) to get:
[Equation]
where ∼ holds when α is sufficiently small.
This implies that we must have c 1 ≥ 2 .
The fact that c 2 ≥ 1 is due to the fact that mixability constant of log-loss is 1 , which also follows from Theorembelow.
Lipschitz Parametric Class.
We now consider a Lipschitz parametric function class.
Given a function f : W × X → [0 , 1] , define the following class
[Equation]
where w ∈ W is often a vector in R d .
We will assume that f ( w , x ) is L -Lipschitz on w for every x , where L ∈ R + .
More formally, ∀ w 1 , w 2 ∈ W and x ∈ X we have
[Equation]
where || · || is some norm on W .
For example, if we take W ⊂ R d then the norm can be ℓ 1 , ℓ 2 or ℓ ∞ norm.
For any specific norm || · || , we write B ( R ) for the ball under such norm with radius R in W .
In particular, we denote by B ds ( R ) the ball in R d of radius R under ℓ s norm centered at the origin.
Theorem 2.
Let f : B ds ( R ) × R d → [0 , 1] be a L -Lipschitz function under ℓ s norm.
Then
[Equation]
Proof.
By L -Lipschitz condition, to find an α -covering in the sense of Definition 1, we only need to find a covering of B ds ( R ) with radius α/L .
By standard result (see e.g. Lemma 5.7 and Example 5.
of [35]) we know that the covering size is upper bounded by
[Equation]
By Theorem 1, we find
[Equation]
Taking α = d/T , we conclude
[Equation]
This completes the proof for T ≥ d .
The upper bound T is achieved by predicting 12 every time.
Example 1.
For logistic function f ( w , x ) =
(1 + e −⟨ w , x ⟩ ) − 1 , and w ∈ B d 2 ( R ) with x ∈ B d 2 (1) our result recovers those of [12], but with a better leading constant (the bound in [12] has a constant 5 ).
Note that, the result in [3] also provides a sub-optimal constant c ∼ 4 .
Moreover, our bounds have a logarithmic dependency on Lipschitz constant L .
The question arises whether the factor in front of log T can be improved to d/ 2 instead of d as discussed in some recent papers [29, 14, 15].
In Theorem 3 below, we show that, in general, it cannot unless we further strengthen our assumption (see Theorem 4).
For the ease of presentation, we only consider the parameters restricted to ℓ 2 norm.
The proof can be found in Appendix B.
Theorem 3.
For any d, T, R, L such that T ≫ d log( RLT ) , there exists L -Lipschitz function f : B d 2 ( R ) × R d → [0 , 1] such that
[Equation]
Lipschitz Class with Bounded Hessian.
As we have demonstrated in Theorem 3 the leading constant 1 of the regret for Lipschitz parametric classes can not be improved in general.
We now show that for some special function f one can improve the constant to 12 , as already noticed in [29, 14, 15].
For any function f : R d × R d → [0 , 1] , we say the Hessian of log f is uniformly bounded on X ⊂ R d , if there exists constant C such that for any w ∈ R d and x ∈ X and y ∈ { 0 , 1 } we have
[Equation]
where ∇ 2 w is the Hessian at w .
The proof of the next theorem can be found in Appendix C.
Theorem 4.
Let f : R d × R d → [0 , 1] be a function such that the Hessian of log f is uniformly bounded by C on X .
Let
[Equation]
be such a class of f restricted to some compact set W ⊂ R d .
Then
[Equation]
where W ∗ = { w + u | w ∈ W , u ∈ B d 2 ( (cid:112) d/CT ) } , Vol ( · ) is volume under Lebesgue measure.
In particular, for W = B
d 2 ( R ) , we have
[Equation]
Note that, Theorem 4 subsumes the results of [17, 29] 4 , where the authors considered functions of form f ( ⟨ w , x ⟩ ) and requires that the second derivative of log f is bounded, see also [5, Chapter 11.10].
However, the KL-divergence-based argument of [17] can not be used directly in the setup of
Theorem 4 since we do not assume the function f has a linear structure.
Our main proof technique of
Theorem 4 is a direct application of Lemma 2 and an estimation of the integrals via Taylor expansion; see Appendix C for more details on the proof.
Finally, we complete this part with the following lower bound for generalized linear functions under unit ℓ s balls.
See Appendix D for proof.
Theorem 5.
Let f : R → [0 , 1] be an arbitrary function such that there exists c 1 , c 2 ∈ (0 , 1) and for all r > 0 we have [ c 1 − c 2 d − r ,
c
1 + c 2 d − r ] ⊂ f
([ − d − r , d − r ]) for sufficiently large d .
Let
[Equation]
where s > 0 .
Then
[Equation]
where O hides some absolute constant that is independent of d, T .
Note that for the logistic function f ( x ) =
(1 + e − x ) − 1 Theorem 5 holds with c 1 = 12 and c 2 = 15 .
Therefore,
If s = 1 , then
[Equation]
If s = 2 , then
[Equation]
If s = ∞ , then
[Equation]
This recovers all the lower bounds from [29].
We note that a simple sufficient condition for Theoremto hold is to require f ′ (0) ̸
= 0 if f ( x ) is differentiable.
Large Growth.
We now present some results for large d growing even faster than T .
We will show that the size of global sequential covering (Definition 1) of a class H can be bounded by the sequential fat-shattering number of H in a similar fashion as in [24].
We first introduce the notion of sequential fat-shattering number as in [24].
We denote { 0 , 1 } d ∗ to be the set of all binary sequences of length less than or equal to d .
A binary tree of depth d with labels in X is defined to be a map τ : { 0 , 1 } d ∗ → X .
For any function class
H ⊂ [0 , 1] X , we say H
α -fat shatters tree τ if there exists [0 , 1] -value tree s : { 0 , 1 } d ∗ → [0 , 1] such that for any binary sequence ϵ d 1 ∈ { 0 , 1 } d ∗ there exist h ∈ H such that for all t ∈ [ d ] :
If ϵ t = 0 , then h ( τ ( ϵ t − 1 1 )) ≤ s ( ϵ t − 1 1 ) − α ; 2.
If ϵ t = 1 , then h ( τ ( ϵ t − 1 1 )) ≥ s ( ϵ t − 1 1 ) + α.
Definition 2.
The sequential α -fat shattering number of H is defined to be the maximum number d ( α ) such that H α -fat shatters a tree τ of depth d :
= d ( α ) .
In the lemma below, we present an upper bound for the cardinality of the global covering with algorithmically constructed cover set G α , see e.g., [24, Section 6.1].
We provide a proof in Appendix E for completeness.
Lemma 5.
Let H ⊂ X be any class and d ( α ) be the sequential α -fat shattering number of H .
Then there exists a global sequential α -covering set G α of H as in Definition 1 such that
[Equation]
Example 2.
By [24] we know that the sequential α -fat shattering number of linear functions f ( w , x ) = |⟨
w , x ⟩| with w , x ∈ B d 2 (1) is of order ˜ O
( α − 2 ) where in ˜
O we hide a polylog factor.
Lemma 5 implies that the global sequential α -covering number is upper bounded by
[Equation]
By Theorem 1, we have
[Equation]
by taking α = T − 1 / 3 .
This bound is independent of the data dimension d .
Remark 1.
Observe that for any class H with sequential fat-shattering number of order α − s one can achieve a regret upper bound of order ˜ O
( T s/s +1 ) by Theorem 1.
We refer to [24, 25] for the estimations of sequential fat-shattering number of a variety of classes.
Finally, we present the following general lower bound.
See Appendix F for proof.
Theorem 6.
For any s ≥ 1 , we define
[Equation]
We can view the vectors in D s as functions mapping [ T ] → [0 , 1] .
Then
[Equation]
To see why Theorem 6 implies a lower bound for f ( w , x ) = |⟨ w , x ⟩| with d ≥ T , as in Example 2, we take w , x ∈ B T 2 (1) (i.e., with d = T ) and define x t = e t with e t being the standard base of R T that takes value 1 at position t and zeros otherwise.
Note that the functions of H f with f ( w , x ) = |⟨ w , x ⟩| restricted on x T is exactly D 2 .
Then
[Equation]
and this is a matching lower bound of Example 2.
Note that, it is proved in [23] that for function f ( w , x ) = ⟨ w , x ⟩ +1, one can achieve the regret of form ˜ O ( √ T ) 5 .
Example 2 implies that the generalized linear functions of form f ( ⟨ w , x ⟩ ) can have different regrets with polynomial gap even with a simple shift on the value (though they have the same covering number).
It is therefore an interesting open problem to investigate a tighter complexity measure (instead of a covering number as in Definition 1) that captures this phenomenon.
Conclusion
In this paper, we presented best known lower and upper bounds on sequential online regret for a large class of experts.
We accomplish this by designing a new smooth truncated Bayesian algorithm, together with the concept of global sequential covering, that achieves these upper bounds.
For the lower bound, we use a novel information-theoretic approach based on the Shtarkov sum.
We expect that these techniques can be generalized to a broader set of problems, e.g., when the features x T is present stochastically.
We leave these to the future investigations.
Acknowledgments
This work was partially supported by the NSF Center for Science of Information (CSoI)
Grant
CCF-0939370, by NSF Grants CCF-2006440, CCF-2007238, and CCF- 2211423, and in addition by
Google Research Grant and by Rolls Royce.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Lemma
We construct the set ˜ G as in Algorithm 2.
For any g ∈ G we define a smooth truncated function ˜
g such that for any x t ∈ X ∗
[Equation]
We introduce the following short-hand notation, for any function f we define
[Equation]
For any x T , y T and h ∈ H , let g ∈ G be a α -covering of h and ˜
g be the truncated function as defined above.
For any t , we consider two cases.
Case 1 : If y t = 1 , we have:
[Equation] [Equation]
[Equation] [Equation]
Case 2 : If y t = 0 , we have
[Equation] [Equation]
Now, combining the two cases, we have
[Equation] [Equation]
This completes the proof of Lemma 4.
B Proof of Theorem
We need the following two lemmas, where the proofs are straightforward.
Lemma 6.
Let P be a finite class of distributions over the same domain X .
Denote
[Equation]
be the Shtarkov sum.
Then for any estimation rule Φ : X → P we have
[Equation]
Lemma 7.
For any M and T ≫ log M , there exist M vectors v 1 , v 2 , · · · , v M ∈ { 0 , 1 } T such that for any i ̸ =
j ∈ [ M ] we have
[Equation]
Now we are in the position to prove Theorem 3.
Let x 1 , · · · , x T ∈ R
d be any distinct points.
We will construct a L -Lipschitz function f ( w , x ) such that the regret restricted only on x T is large.
To do so, we consider a maximum packing M of the parameter space B d 2 ( R ) of radius α/L > 0 (where α is to be determined latter).
Standard volume argument (see Chapter 5 of [35]) yields that
[Equation]
Now, we will define a L -Lipschitz functions f ( w , x ) only on w ∈ M and x ∈ { x 1 , · · · , x T } .
By
Lemma 7 (assume for now the conditions are satisfied), we can find | M | binary vectors V ⊂ { 0 , 1 } T such that any pair of the vectors has Hamming distance lower bounded by T/ 4 .
For each of the vector v ∈ V , we define a vector u ∈ [0 , 1] T in the following way, for all t ∈
[ T ]
If v [ t ] = 0 then set u [ t ] = 0 ; 2.
If v [ t ] = 1 then set u [ t ] = α .
Denote by U be the set of all such vectors u .
Note that | U | = | M | .
For any w ∈ M , we can associate a unique u ∈ U such that for all t ∈
[ T ] f
[Equation]
w ,
[Equation]
=
u [ t ] .
We now show that f is indeed L -Lipschitz restricted on M for all x t ∈ { x 1 , · · · , x T } .
This is because for any w 1 ̸ = w 2 ∈ M
we have | f ( w 1 , x t ) − f ( w 2 , x t ) | ≤ α by definition of U and || w 1 − w
|| 2 ≥ α/L since M is a packing.
We now view the vectors in u ∈ U as a product of Bernoulli distributions with each coordinate t independently sampled from Bern( u [ t ]) .
We show that the sources in U are identifiable.
To see this, we note that for any distinct pairs u 1 , u 2 ∈ U , there exist a set I ∈
[ T ] such that u 1 and u 2 differ on
I and | I | ≥ T/ 4 .
This further implies that there exist a set J ⊂ I with | J | ≥ T/ 8 such that u 1 takes all 0 on J and u 2 takes all α on J (or vice versa).
We can then distinguish u 1 , u 2 by checking if the samples on J are all 0 s or not.
The probability of making error is upper bounded by
[Equation]
Since there are only | M | 2 such pairs, we have the probability of wrongly identifying the source upper bounded by
[Equation]
Taking α = 16 d log( RLT ) T , the error probability is upper bounded by
[Equation]
for sufficient large d, T , where we have use the fact that | M | ≤ (
[Equation]
log
[Equation]
RLT
[Equation]
) d .
Note that we only showed a lower bound on | M | before, but this is not a problem since we can always remove some points from M to make the upper bound holds as well.
By Lemma 6, we know that the Shtarkov sum of sources in U is lower bounded by | M | / 2 .
Therefore, we have
[Equation]
Now, we have to extend the function to the whole set
B d 2 ( R ) and keep the L -Lipschitz property.
This follows from a classical result in real analysis (see [20, Theorem 1]) by defining for all w ∈ B d 2 ( R ) and x t ∈ { x 1 , · · · , x T }
[Equation]
For the x ̸∈ { x 1 , · · · , x T } , we can simply let f ( w , x ) =
for all w .
Finally, we need to check that the condition of Lemma 7 holds for our choice of α , this is satisfied by our assumption T ≫ d log( RLT ) .
C Proof of Theorem
To make the proof more transparent, we only prove the case for W =
B d 2 ( R ) since the proof for other compact W follows similar path.
Note that, for W = B d 2 ( R ) , we have W ∗ =
B d 2 ( R + (cid:112) d/CT ) .
The proof resembles that of [12] but running the Bayesian predictor (Algorithm 1) over W ∗ instead of W with G being H f and µ being Lebesgue measure.
Let x T , y T and ˆ y T be the feature, label and predictions of the Bayesian predictor respectively.
By Lemma
[Equation]
where µ is the Lebesgue measure and
[Equation]
We now write h t ( w ) def = log f ( w , x t )
y t (1 − f ( w , x t )) 1 − y t to simplify notation.
It is easy to see that ℓ ( f ( w , x t ) , y t ) =
− h t ( w ) .
Let w ∗ be the point in B d 2 ( R ) that maximizes
[Equation]
( w ) .
Let u = ∇ h ( w ∗ ) be the gradient of h at w ∗ .
By Taylor theorem, we have for any w ∈ B d 2 ( R + (cid:112) d/CT ) h ( w
[Equation]
where w ′ is a convex combination of w and w ∗ and u τ is the transpose of u .
Now, the key observation is that for any point w such that u τ ( w − w ∗ ) ≥ 0 we have
[Equation]
where the last inequality follows from our assumption about the bounded Hessian of log f .
Let B be the half ball of radius (cid:112) d/CT centered at w ∗ such that for all w ∈ B we have u T ( w − w ∗ ) ≥ 0 .
By (26), for all w ∈ B
[Equation]
Note that B ⊂ B d
( R + (cid:112) d/CT ) .
Then using above observations we arrive at
[Equation]
This completes the proof of Theorem
[Equation]
Remark 2.
When compared to the technique in [40], Theorem 4 does not assume that the gradient critical point of the loss is zero (e.g., the minimum may occur on the boundary).
This is why we need to restrict to the half ball B in order to discard the linear term of Taylor expansion in Equation (27).
Moreover, in the proof we work directly on the continuous space instead of a discretized cover, giving an efficient algorithm provided the posterior is efficiently samplable (by e.g., assuming some log-concavity of f as in [12]).
D Proof of Theorem
We start with the following technical lemma.
Lemma 8.
The following inequality holds, for r > 0 :
[Equation]
where P ( y | w ) =
w k (1 − w ) T/d − k with k being the number of 1 s in y .
Proof.
By Stirling approximation, for all k ∈ [ T/d ] , there exists a constant C ∈ R + such that
[Equation]
Since P ( y | w ) achieves maximum at w = k ∗ d/T , we have
[Equation]
Therefore, for each k in the above summation, we have that
[Equation]
Therefore, the LHS of (35) is lower bounded by
[Equation]
for sufficient large d .
Now we are ready to prove Theorem 5.
We choose a particular x T : We split the x T into d blocks each with length of T/d .
With that, the i th part of the inputs and the outputs are denoted by x ( i ) =
( x ( T/d ) ∗ ( i − 1)+1 , · · · , x ( T/d ) ∗ i ) and y ( i ) =
( y ( T/d ) ∗ ( i − 1)+1 , · · · , y ( T/d ) ∗ i ) , respectively.
We define for any x t in the i th block x ( i ) equals e i the standard d base of R d with 1 in position i and 0 s otherwise.
Note that, with these choice of x t s, we have ⟨ w , x t ⟩ = w i , where w i is the i th coordinate of w and x t ∈ x ( i ) .
We will lower bound r ∗ T ( H f | x T ) , which will automatically give a lower bound on r aT ( H f ) .
We only need to compute the following Shtarkov sum
[Equation]
(36) where
P f
( y ( i ) | w i ) =
f ( w i ) k
i (1 − f ( w i ))
T/d − k i with k
i being the number of 1 s in y ( i ) .
We observe
S T ( H f | x T ) ≥ (cid:88)
[Equation]
=
[Equation]
sup
[Equation]
 d where P ( y | w ) is as in Lemma 8 and the last inequality holds since [ c 1 − c 2 d − 1 /s , c
1 + c 2 d − 1 /s ] ⊂ f
([ d − 1 /s , d − 1 /s ]) by the assumption.
Now, Lemma 8 implies that
[Equation]
where c is some absolute constant that is independent of d, T .
We conclude
[Equation]
which completes the proof.
E Proof of Lemma
We first introduce a discretized notion of fat-shattering number, which can be viewed as a misspecified
Littlestone dimension [2, 8], see also [25].
For any α > 0 , we can choose K ≤ ⌈ 1 / 2 α ⌉ points z 1 < z 2 · · · < z K in the interval [0 , 1] such that any point in [0 , 1] is α close to some z k and z k +
− z k
= 2 α for all k ∈
[ K ] .
Now, we define a discretized class H ′ for the [0 , 1] -valued class H in the following way.
For any h ∈ H , we define function h ′ ∈ H ′ such that for any x ∈ X we have
[Equation]
where we break ties arbitrarily.
Input : Hypothesis class H with functions map X
→
[ K ] 1: Let H ∗
=
H 2: for t = 1 , · · · , T do 3:
Receive feature x t 4:
For k ∈ [ K ] , let
H ∗ ( x t ,k ) def = { h ∈ H ∗ |
h ( x t ) =
k } 5:
Make prediction
[Equation]
(where we break ties arbitrarily and deal with empty classes as in Definition 3) 6:
Receive label y t 7:
If | ˆ y t − y t | ≥ 2 , set
H ∗ =
H ∗ ( x t ,y t ) 8:
If | ˆ y t − y t | < 2 , set
H ∗ =
H ∗ 9: end for
We now view the functions in H ′ as functions map X
→
[ K ] (i.e., we view each z k as its index k ).
For any discretized class H ′ , we define the discretized 1 -shattering as follows.
For any X -valued tree τ of depth d , we say H ′ 1 -shatters
τ , if there exists [ K ] -valued tree s : { 0 , 1 } d ∗ → [ K ] such that for any ϵ d 1 ∈ { 0 , 1 } d ∗ there exist h ′ ∈ H ′ such that for all t ∈ [ d ] :
If ϵ t = 0 , then h ′ ( τ ( ϵ t − 1 1 )) ≤ s ( ϵ t − 1 1 ) − 1 .
if ϵ t = 1 , then h ′ ( τ ( ϵ t − 1 1 )) ≥ s ( ϵ t − 1 1 ) + 1 .
Definition 3.
The discretized 1 -shattering number of a discretized class H ′ is defined to be the maximum number d such that H ′ 1 -shatters some tree τ of depth d .
This number is denoted as
FAT 1 ( H ′ ) .
If no such tree exists, we define the 1 -shattering number to be 0 if H ′ is non-empty and − 1 if H ′ is empty.
The proof of Lemma 5 follows from the following three lemmas.
Lemma 9.
The discretized 1 -shattering number of H ′ is upper bounded by the α -fat shattering number of H where H ′ is the discretized class of H at scale α .
Proof.
Let τ be the tree of depth d that is shattered by H with a [ K ] -valued tee s .
We define a [0 , 1] -valued tree s ′ as follows for any ϵ t 1 ∈ { 0 , 1 } d ∗ ,
[Equation]
We now show that the τ and s ′ are the desired pair that is α -shattered by H .
This follows from the fact that for any z k and z l with k ̸ = l if some y ∈ [0 , 1] is closer to z l , then
[Equation]
as easy to see.
For any discretized class H ′ , we say a class G of functions map X ∗ →
[ K ] 1 -covers H ′ if for any x 1 , · · · , x T ∈ X and h ′ ∈ H ′
there exists g ∈ G such that for all t ∈
[ T ]
[Equation]
The following result is crucial for our following analysis, which is an analogy of Lemma 12 of [2] (see also [8, 24]).
Lemma 10.
Suppose the discretized 1 -shattering number of H ′ is upper bounded by d , then there exists a 1 -covering set G of H ′ such that
[Equation]
Proof.
We now describe an algorithm that is similar to the SOA algorithm of [2], which we will call it M-SOA (Algorithm 3) 7 .
The algorithm goes as follows: it maintains a running hypothesis class
H ∗ , initially equals H ′ .
Let ( x t , y t ) be the sample label pair received at round t .
We will denote by
H ∗ ( x t ,y t ) the functions in H ∗ that is consistent with ( x t , y t ) , i.e., for all h ∈ H ∗ ( x t ,y t ) we have h ( x t ) =
y t .
At time step t , the algorithm M-SOA will predict k ∈
[ K ] such that FAT 1 ( H ∗ ( x t ,k ) ) is maximum, where we denote by FAT 1 ( H ∗ ( x t ,k ) )
the discretised 1-shattering number of H ∗ ( x t ,k ) and break ties arbitrarily.
After receiving the true label y t , the M-SOA algorithm will do the following.
If | ˆ y t − y t | ≥ 2 , then it sets H ∗
=
H ∗ ( x t ,y t ) .
Else, it remains on the same H ∗ .
We then continue the prediction procedure for the next time step with the new H ∗ .
We say the algorithm M-SOA makes an error at time step t if | ˆ y t − y t | ≥ 2 where ˆ y t is the prediction given by M-SOA at time step t .
We claim that the M-SOA will make at most d errors if the samples ( x T , y T ) is consistent with some h ∈ H ′ .
To see this, we prove by induction on d and T (the base case for d = 0 or T = 0 is easy to check).
Suppose we have observed x 1 at the first step.
We show that there can not be two element k 1 , k 2 ∈
[ K ] such that | k 1 − k 2 | ≥ 2 and both H ′ ( x 1 ,k 1 ) and H ′ ( x 1 ,k 2 ) has discretized 1 -shattering number ≥ d .
Otherwise, we can concatenate the shattering tree of H ′ ( x 1 ,k 1 ) and H ′ ( x 1 ,k 2 ) with the root labeled by x 1 to form a depth d + 1 shattering tree of H ′ (with s ( ϕ ) being any number ∈ ( k 1 , k 2 ) ).
This is a contradiction, since the discretized 1 -shattering number of H ′ is upper bounded by d .
This shows that either we will make no error at the first step or the discretized 1 -shattering number decreased by at least 1 on the remaining consistent class of functions (after y 1 has been revealed).
For the first case, by induction hypothesis for T − 1 we have the number of errors is at most d .
For the second case, we also have the number of errors upper bounded by d − 1 + 1 = d .
We now follow the idea from the proof of Lemma 12 of [2] to construct a covering set G .
For any subset I ⊂ [ T ] of size | I | ≤ d and { k t } t ∈ I ∈ [ K ] | I | , we define a function g by running our M-SOA algorithm by changing steps 7 − 8 as follows.
At each time step t ∈ [ I ] , we update H ∗ =
H ∗ ( x t ,k t ) .
Otherwise, for any t ̸∈ I , we remain on the same H ∗ .
The values of g for each x t is given by the output of M-SOA at time step t .
Since the M-SOA will make at most d errors if the sample-label pairs ( x T , y T ) are consistent with some function in H ′ , we know that any h ∈ H ′ is 1 -covered by the function generated by running
M-SOA with some I and { k t } t ∈ I in the above fashion.
To complete we observe that by a simple counting argument the number of such pairs I and { k t } t ∈ I is at most d (cid:88) t
= (cid:18) T t (cid:19)
K t which completes the proof.
Finally, we need the following lemma that relates 1 -covering of H ′ with global sequential α -covering of H .
Lemma 11.
Suppose there exist a 1 -covering set G of H ′ , then there exists a global 3 α -covering G ′ of H such that |G| = |G ′
| , where H ′ is the discretised class of H at scale α .
Proof.
For any g ∈ G , we define a function g ′ such that for all x t we have
[Equation]
The claim follows from the fact that any y that is closest to z k satisfies | y − z k | ≤ α and if some z 1 -covers z
k then we have | z − z k | ≤ 2 α , by triangle inequality
[Equation]
as needed.
The proof of Lemma 5 follows from Lemma 9, Lemma 10, and Lemma 11.
F Proof of Theorem
It is sufficient to compute the Shtarkov sum as in (5).
For any y T ∈ { 0 , 1 } T with k 1 s, we claim that
[Equation]
where
[Equation]
To see this, we use a perturbation argument.
Denote I be the positions in y T that takes value 1 such that | I | =
k .
For any p such that p ( y T ) is maximum, we must have p j = 0 for all j ̸∈ I .
Suppose otherwise, we then can move some probability mass on p j to some p i < 1 with i ∈
I , which will increase the value of p ( y T ) , thus a contradiction.
Now, we need to show that
[Equation]
this follows easily by AM-GM (i.e., arithmetic mean vs geometric mean) inequality since (cid:80) i ∈ I
p s i ≤ 1 and it takes equality when p i =
k 1 /s for all i ∈ I .
Now, the Shtarkov sum can be written as
[Equation]
To find a lower bound, we only need to estimate the maximum term in the summation.
We have
[Equation]
where the last inequality follows by taking k = 1 e T s/s +1 , and we also use the fact that
[Equation]
Therefore, we have
[Equation]
which completes the proof.