Efficient Methods for Non-stationary Online Learning
Abstract
Non-stationary online learning has drawn much attention in recent years.
In particular, dynamic regret and adaptive regret are proposed as two principled performance measures for online convex optimization in non-stationary environments.
To optimize them, a two-layer online ensemble is usually deployed due to the inherent uncertainty of the non-stationarity, in which a group of base-learners are maintained and a meta-algorithm is employed to track the best one on the fly.
However, the two-layer structure raises the concern about the computational complexity — those methods typically maintain O (log T ) base-learners simultaneously for a T -round online game and thus perform multiple projections onto the feasible domain per round, which becomes the computational bottleneck when the domain is complicated.
In this paper, we present efficient methods for optimizing dynamic regret and adaptive regret, which reduce the number of projections per round from O (log T ) to 1 .
Moreover, our obtained algorithms require only one gradient query and one function evaluation at each round.
Our technique hinges on the reduction mechanism developed in parameter-free online learning and requires non-trivial twists on non-stationary online methods.
Empirical studies verify our theoretical findings.
Introduction
Classic online learning minimizes the static regret, which benchmarks the online learner’s performance against the best fixed decision in hindsight.
In many real-world online applications, however, the environments are non-stationary [Zhou, 2022] and static regret becomes less attractive since it would be unrealistic to assume the existence of a single decision behaved satisfactorily over time.
To address the limitation, in recent years, researchers have studied more strengthened performance measures to facilitate online algorithms with the capability of handling non-stationarity.
In particular, dynamic regret [Zinkevich, 2003; Zhang et al., 2018a] and adaptive regret [Hazan and Seshadhri, 2009; Daniely et al., 2015] are proposed as two principled metrics to guide the algorithm design.
We focus on the online convex optimization (OCO) setting [Hazan, 2016].
OCO can be deemed as a game between the learner and the environments.
At each round t ∈ [ T ] , the learner submits her decision x t ∈ X from a convex feasible domain X ⊆ R d and simultaneously environments choose a convex function f t : X (cid:55)→ R , and subsequently the learner suffers an instantaneous loss f t ( x t ) .
1.1 Dynamic Regret and Adaptive Regret
Dynamic regret is proposed by Zinkevich [2003] to compare the online learner’s performance against a sequence of any feasible comparators u 1 , . . .
, u T ∈ X .
Formally, it is defined as
[Equation]
Dynamic regret minimization enables the learner to track changing comparators.
A favorable dynamic regret bound should scale with a certain non-stationarity measure dependent on the comparators such as the path length P T = (cid:80) t =2 ∥
u t − u t − 1 ∥ 2 .
Notably, the classic static regret can be treated as a special case of dynamic regret by specifying the comparators as the best fixed decision in hindsight.
Adaptive regret is proposed by Hazan and Seshadhri [2009] and further strengthened by Daniely et al.
[2015], which measures the regret over any interval I =
[ r, s ] ⊆
[ T ] with a length of τ = |
I | , i.e.,
[Equation]
Since the minimizers of different intervals can be different, adaptive regret minimization also ensures the capability of competing with changing comparators.
A desired adaptive regret bound should be as close as the minimax static regret of this interval.
Algorithms with adaptive regret matching static regret of this interval up to logarithmic terms in T are referred to strongly adaptive [Daniely et al., 2015].
Moreover, adaptive regret includes static regret when choosing I =
[ T ] .
It is worth noting that the relationship between dynamic regret and adaptive regret for OCO is generally unclear [Zhang, 2020, Section 5], even though a black-box reduction from dynamic regret to adaptive regret has been proven for the simper expert setting (i.e., online linear optimization over simplex)
[Luo and Schapire, 2015, Theorem 4].
Hence, the two measures are separately developed and many algorithms have been proposed, including algorithms for dynamic regret [Zinkevich, 2003;
Hall and Willett, 2013; Zhang et al., 2018a; Zhao et al., 2020, 2021b,a; Baby and Wang, 2021;
Zhao et al., 2022a] and the ones for adaptive regret [Hazan and Seshadhri, 2009; Daniely et al., 2015; Jun et al., 2017; Zhang et al., 2018b, 2019].
Note that there are also studies [Zhang et al., 2020; Cutkosky, 2020] optimizing both measures simultaneously by an even strengthened metric (cid:80) st = r f t ( x t ) − (cid:80)
st = r f t ( u t ) over any interval [ r, s ] ⊆
[ T ] , hence called “interval dynamic regret”.
1.2 Two-layer Structure and Projection Complexity Issue
The fundamental challenge of optimizing these two non-stationary regret measures is the uncertainty of the environmental non-stationarity.
Concretely, to ensure the robustness to the unknown environments, dynamic regret aims to compete with any feasible comparator sequence, while adaptive regret examines the local performance over any intervals.
The unknown comparators or unknown intervals bring considerable uncertainty to online optimization.
To address the issue, a two-layer structure is usually deployed to optimize the measures, where a set of base-learners are maintained to handle the different possibilities of online environments and a meta-algorithm is employed to combine them all and track the unknown best one.
Such a framework successfully achieves many state-of-the-art results, including the O ( (cid:112) T (1 + P T )) dynamic regret [Zhang et al., 2018a] and the O ( (cid:112) ( F T + P T )(1 + P T )) small-loss dynamic regret for smooth functions [Zhao et al., 2020], where P T = (cid:80) Tt =
∥
u t − u t − 1 ∥ 2 is the path length and F T =
(cid:80) Tt =1 f t ( u t ) is the cumulative loss of comparators; as well as the O ( (cid:112) | I | log T ) adaptive regret
[Jun et al., 2017] and the
O ( √ F I log F
I log F T ) small-loss adaptive regret for smooth functions [Zhang et al., 2019] for any interval I =
[ r, s ] ⊆
[ T ] , where F I = min x ∈X (cid:80) st = r f t ( x ) and F T = min x ∈X (cid:80) Tt =1 f t ( x ) .
Besides, an O ( (cid:112) | I | (log T + P I ))
interval dynamic regret is also achieved by a two-layer (or even three-layer) structure [Zhang et al., 2020],
where P I = (cid:80) st = r ∥
u t − u t − 1 ∥ 2 is the path length over the interval.
The two-layer methods have demonstrated great effectiveness in tackling non-stationary online environments, whereas the gain is at the price of heavier computations than the methods for minimizing static regret.
While it is believed that additional computations are necessary for more robustness, we are wondering whether it is possible to pay for a “minimal” computation overhead for adapting to the non-stationarity.
To this end, we focus on the popular first-order online methods and aim to streamline unnecessary computations while retaining the same regret guarantees.
Arguably, the most computationally expensive step of each round is the projection onto the convex feasible domain, namely, the projection operation Π X [ y ] =
arg min x ∈X ∥ x − y ∥ 2 for a convex set X ⊆ R d .
Typical two-layer non-stationary online algorithms require maintaining N =
O (log T ) base-learners simultaneously to cover the possibility of unknown environments.
Define the projection complexity of online methods as the number of projections onto the feasible domain per round.
Then, those non-stationary methods suffer an O (log T ) projection complexity, whereas standard online methods for static regret minimization require only one projection per round such as online gradient descent [Zinkevich, 2003].
1.3 Our Contributions and Techniques
In this paper, we design a generic mechanism to reduce the projection complexity of many existing non-stationary methods from O (log T ) to 1 without sacrificing the regret optimality , hence matching the projection complexity of stationary methods.
Our reduction is inspired by the recent advance in parameter-free online learning [Cutkosky and Orabona, 2018; Mhammedi et al., 2019].
The idea is simple: we reduce the original problem learned in the feasible domain X to an alternative one learned in a surrogate domain Y ⊇ X such that the projection onto it is much cheaper, e.g., simply choosing
Y as a properly scaled Euclidean ball; and moreover, a carefully designed surrogate loss is necessary for the alternative problem to retain the regret optimality.
We reveal that a necessary condition for our reduction mechanism to deploy and reduce the projection complexity is that the non-stationary online algorithm shall query the function gradient once and evaluate the function value once per round .
Several algorithms for the worst-case dynamic regret or adaptive regret already satisfy the requirements, so we can immediately deploy the reduction and obtain their efficient counterparts with the same regret guarantees and 1 projection complexity.
However, many non-stationary algorithms, particularly those designed for small-loss bounds, do not satisfy the requirement.
Hence, we require non-trivial efforts to make them compatible.
Due to this, we have developed a series of algorithms that achieve worst-case/small-loss dynamic regret and adaptive regret with one projection per round (actually, with one gradient query and one function evaluation per round as well).
Despite that the reduction mechanism of this paper has been studied in parameter-free online learning, applying it to non-stationary online learning requires new ideas and non-trivial modifications.
Here we highlight the technical innovation.
The main challenge comes from the reduction condition mentioned earlier — as the surrogate loss involves the projection operation, our reduction requires the algorithm query one gradient and evaluate one function value at each round.
However, many nonstationary algorithms do not satisfy the requirement, which is to be contrasted to the parameter-free algorithms such as MetaGrad [van Erven and Koolen, 2016; Mhammedi et al., 2019] that naturally satisfy the condition.
For example, the SACS algorithm [Zhang et al., 2019] enjoys the best known small-loss adaptive regret, yet the method requires N gradient queries and N +1 function evaluations at each round, where N = O (log T ) is the number of base-learners.
Thus, we have to dig into the algorithm and modify it to fit our reduction.
First, we replace their meta-algorithm with Adapt-MLProd [Gaillard et al., 2014], an expert-tracking algorithm with a second-order regret with excess losses to accommodate the linearized loss that is used to ensure one gradient query per round.
Second, we introduce a sequence of time-varying thresholds to adaptively determine the problem-dependent geometric covers in contrast to a fixed threshold used in their method.
In particular, we register the cumulative loss of the final decisions rather than the base-learner’s one to compare it with the changing thresholds, which renders the design of one function value evaluation per round and also turns out to be crucial for achieving an improved small-loss bound that can recover the best known worst-case adaptive adaptive regret (by contrast, SACS cannot obtain optimal worst-case adaptive regret).
To summarize, our final algorithm only requires one projection/gradient query/function evaluation at each round, substantially improving the efficiency of SACS algorithm that requires N projections/gradient queries/function evaluations per round.
1.4 Assumptions
We list several standard assumptions used in OCO [Shalev-Shwartz, 2012; Hazan, 2016].
Notably, not all assumptions are always required.
We will explicitly state the requirements in the theorem.
Assumption 1 (bounded gradient) .
The norm of the gradients of online functions over the domain X is bounded by G , i.e., ∥∇ f t ( x ) ∥ 2 ≤ G , for all x ∈ X and t ∈ [ T ] .
Assumption 2 (bounded domain) .
The domain X ⊆ R d contains the origin 0 , and the diameter of the domain X is at most D , i.e., ∥ x − x ′ ∥ 2 ≤ D for any x , x ′ ∈ X .
Assumption 3 (non-negativity and smoothness) .
All the online functions are non-negative and
L -smooth, i.e., for any x , x ′ ∈ X and t ∈ [ T ] , ∥∇ f t ( x ) − ∇ f t (
[Equation]
Organization.
The rest is structured as follows.
Section 2 presents the reduction mechanism and illustrates its application to dynamic regret minimization.
Section 3 provides efficient methods for optimizing adaptive regret.
Section 4 reports the experiments.
Section 5 concludes the paper and makes discussions.
All the proofs and omitted details for algorithms are deferred to the appendices.
The Reduction Mechanism and Dynamic Regret Minimization
We start from the dynamic regret minimization.
First, we briefly review existing methods in Section 2.1, and then present our reduction mechanism and illustrate how to apply it to reducing the projection complexity of dynamic regret methods in Section 2.2.
2.1 A Brief Review of Dynamic Regret Minimization
Zhang et al.
[2018a] propose a two-layer online algorithm called Ader with an O ( (cid:112) T (1 + P T )) dynamic regret, which is proven to be minimax optimal for convex functions.
Ader maintains a group of base-learners, each performing online gradient descent (OGD)
[Zinkevich, 2003] with a customized step size specified by the pool H = { η 1 , . . .
, η N } , and then uses a meta-algorithm to combine them all.
Denoted by B 1 , . . .
, B N the N base-learners.
For each i ∈ [ N ] , B i updates by
[Equation]
where η i ∈ H is the associated step size and Π X [ · ] denotes the projection onto the feasible domain X with Π X
[ y ] =
arg min x ∈X ∥ y − x ∥ 2 .
Notably, all the base-learners share the same gradient ∇ f t ( x t ) rather than using their individual one ∇ f t ( x t,i ) .
This is because Ader optimizes the linearized loss ℓ t ( x ) =
⟨∇ f t ( x t ) , x ⟩ , which enjoys the benign property of ∇ ℓ t ( x t,i ) =
∇ f t ( x t ) for all i ∈ [ N ] .
Furthermore, the meta-algorithm evaluates each base-learner by ℓ t ( x t,i ) =
⟨∇ f t ( x t ) , x t,i ⟩ and updates the weight vector p t +1 ∈
∆
N by the Hedge algorithm
[Freund and Schapire, 1997], namely,
[Equation]
where ε > 0 is the learning rate of the meta-algorithm.
The final prediction is obtained by x t +1 =
(cid:80)
Ni =1 p t +1 ,i x t +1 ,i .
The learner submits the prediction x t +1 and then receives the loss f t +1 ( x t +1 ) and the gradient ∇ f t +1 ( x t +1 ) as the feedback of this round.
Under a suitable configuration of the step size pool H with N =
O (log T ) and learning rate ε =
Θ( (cid:112) (ln N ) /T ) , Ader enjoys an
O ( (cid:112) T (1 + P T ))
dynamic regret [Zhang et al., 2018a, Theorem 4].
For convex and smooth functions, Zhao et al.
[2021b] demonstrate that a similar two-layer structure can attain an O ( (cid:112) ( F T + P T )(1 + P T )) small-loss dynamic regret under a suitable setting of the step size pool H and time-varying learning rates of meta-algorithm { ε t } Tt =1 , where F T = (cid:80) Tt =1 f t ( u t ) is the cumulative loss of the comparators.
This bound safeguards the minimax rate in the worst case, while can be much smaller than O ( (cid:112) T (1 + P T )) bound in benign environments.
2.2 The Reduction Mechanism for Reducing Projection Complexity
As demonstrated in the update (3), all the base-learners require projecting the intermediate solution onto the domain X to ensure the feasibility.
As a result, O (log T ) projections are required at each round, which is generally time-consuming particularly when the domain X is complicated.
We present a generic reduction mechanism for reducing the projection complexity and apply it to dynamic regret methods.
Our reduction builds upon the seminal work [Cutkosky and Orabona, 2018] and a further refined result [Cutkosky, 2020], who propose a black-box reduction from constrained online learning to the unconstrained setting (or another constrained problem with a larger domain) .
Reduction mechanism.
Given an algorithm for non-stationary online learning Algo whose projection complexity is O (log T ) , our reduction mechanism builds on it to yield an algorithm
Efficient-Algo with 1 projection onto X per round and retaining the same order of regret.
The central idea is to replace expensive projections onto the original domain X with other much cheaper projections.
To this end, we introduce a surrogate domain Y defined as the minimum Euclidean ball containing the feasible domain X , i.e., Y = { x | ∥ x
∥ 2 ≤ D } ⊇ X .
Then, the reduced algorithm
Algo works on Y whose projection can be realized by a simple rescaling.
More importantly, to avoid regret degeneration, it is necessary to carefully construct the surrogate loss g t : Y (cid:55)→ R as
[Equation]
[Equation]
For each i ∈ [ N ] the base learner B i produces the local decision by
[Equation]
[Table]
algorithm updates weight by p
[Table]
[Table] [Table]
[Equation]
[Table]
(cid:80)
[Table]
[Equation]
[Table]
[Equation]
N
[Equation]
[Table]
Compute
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
10: end where S X ( y ) =
inf x ∈X ∥
y − x ∥ 2 is the distance function to X and v t
= ( y t − x t ) / ∥
y t − x t ∥ 2 is the vector indicating the projection direction.
The main protocol of our reduction is presented as follows.
The input includes original functions { f t } Tt =1 , the feasible domain X , and the reduced algorithm Algo
Our reduction enjoys the regret safeness due to the following benign properties of surrogate loss.
Theorem 1 (Theorem 2 of Cutkosky [2020]) .
The surrogate loss g t : Y (cid:55)→ R defined in (5) is convex.
Moreover, we have ∥∇ g t ( y t ) ∥ 2 ≤ ∥∇ f t ( x t ) ∥ 2 and for any u t ∈ X
[Equation]
The theorem shows the convexity of the surrogate loss g t ( y ) and we thus have f t ( x t ) − f t ( u t ) ≤ ⟨∇ g t ( y t ) , y t − u t ⟩ , which implies that it suffices to optimize the linearized upper bound, i.e., to optimize function ℓ t ( y ) =
⟨∇ g t ( y t ) , y ⟩ .
The following lemma specifies the gradient calculation.
Lemma 1.
For any y ∈ Y , ∇ g t ( y ) = ∇ f t ( x t ) when ⟨∇ f t ( x t ) , v t ⟩ ≥ 0 ; and ∇ g t ( y )
=
∇ f t ( x t ) − ⟨∇ f t ( x t ) , v t ⟩· ( y − Π X
[ y ]) / ∥
y − Π X
[ y ] ∥ 2 when ⟨∇ f t ( x t ) , v t ⟩ < 0 .
Here v t = ( y t − x t ) / ∥
y t − x t ∥ 2 .
In particular, ∇ g t ( y t ) = ∇ f t ( x t ) − ⟨∇ f t ( x t ) , v t ⟩ · v t when ⟨∇ f t ( x t ) , v t ⟩ < 0 .
Reduction requirements.
An important necessary condition for the reduction is to require the reduced algorithm satisfying one gradient query and one function evaluation at each round.
Indeed, the reduction essentially updates according to the surrogate loss { g t } Tt =1 .
Note that the definition of surrogate loss involves the distance function S X ( y ) , see Eq.
Thus, each evaluation of g t ( y ) leads to one projection onto X due to the calculation of S X ( y ) .
Similarly, each gradient query of ∇ g t ( y ) also contributes to one projection, see Lemma 1 for details.
To summarize, we can use the reduction to ensure a 1 projection complexity, only when the reduced algorithm satisfies the requirements of one gradient query and one function evaluation per round.
Below, we demonstrate the usage of our reduction mechanism for two methods of dynamic regret minimization that satisfy the conditions, including the worst-case method [Zhang et al., 2018a] and the small-loss method [Zhao et al., 2021b].
Application to dynamic regret minimization.
Algorithm 1 summarizes the main procedures of our efficient methods for optimizing dynamic regret, which is an instance of the reduction mechanism by picking Algo as Ader [Zhang et al., 2018a].
More specifically, Lines 6 – 8 are essentially performing
Ader algorithm using the surrogate loss { g t } Tt =1 over the surrogate domain Y .
Note that the base update in Line 6 is essentially performing OGD with projection onto Y , a scaled Euclidean ball, and thus the projection admits a simple closed form.
The overall algorithm requires projecting onto X only once per round, see Line 9.
Our method provably retains the same dynamic regret.
Theorem 2.
Set the step size pool as H = (cid:8) η i =
i −
( D/G ) (cid:112) 5 / (2 T ) | i ∈
[ N ] (cid:9) with
N = ⌈ 2 − 1 log 2 (1 + 2 T/ 5) ⌉
+ 1 and the learning rate as ε = (cid:112) (ln N ) / (1 + G 2 D 2 T ) .
Under
Assumptions 1 and 2, our algorithm requires one projection onto X per round and enjoys
[Equation]
For smooth and non-negative functions, the Sword++ algorithm
[Zhao et al., 2021b] achieves an
O ( (cid:112) ( F T + P T )(1 + P T )) small-loss dynamic regret, which requires one gradient and one function value per iteration.
However, notice that the surrogate loss g t ( · ) in Eq.
(5) is neither smooth nor non-negative, which hinders the application of our reduction to their method.
Fortunately, owing to the benign property of ∥∇ g t ( y t ) ∥ 2 ≤ ∥∇ f t ( x t ) ∥ 2 (see Theorem 1), we can still deploy the reduction via an improved analysis and obtain a projection-efficient algorithm with the same small-loss bound.
Theorem 3.
Set the step size pool as H = (cid:8) η i =
i − 1 (cid:112) 5 D 2 / (1 + 8 LGDT )
| i ∈
[ N ] (cid:9) with
N = ⌈ 2 − 1 log 2 ((5 D 2 + 2 D 2 T )(1 + 8 LGDT ) / (5 D 2 ))
⌉ + 1 and the learning rate of the metaalgorithm as ε t = (cid:113) (ln N ) / (1 + D 2 (cid:80) t − 1 s =1 ∥∇ g s ( y s ) ∥ 22 ) .
Under Assumptions 1, 2, and 3, our algorithm requires one projection onto X per round and enjoys the following dynamic regret:
[Equation]
where F T = (cid:80) Tt =1 f t ( u t ) is the cumulative loss of the comparators.
Adaptive Regret Minimization
In this section, we present our efficient methods to minimize adaptive regret.
First, we briefly review existing methods in Section 3.1, and then present our efficent methods to reducing the projection complexity of adaptive regret methods in Section 3.2.
3.1 A Brief Review of Adaptive Regret Minimization
Adaptive regret minimization ensures the online learner to be competitive with a fixed decision over every contiguous interval.
For the worst-case bound, the best known result is the O ( (cid:112) | I | log T )
adaptive regret bound achieved by the CBCE algorithm
[Jun et al., 2017].
CBCE algorithm requires multiple gradients at each round.
Wang et al.
[2018] improve CBCE by using the linearized loss to make it requiring one gradient per iteration and retaining the same adaptive regret.
Moreover, the improved CBCE algorithm only evaluates the function value once per iteration.
Therefore, we can directly apply our reduction and obtain a projection-efficient variation with the same adaptive regret.
More details can be found in Appendix C.1.
Now, we focus on the more challenging case of small-loss adaptive regret.
The best known result is the O ( √ F I log F I log F T ) bound for any interval
I =
[ r, s ] ⊆
[ T ] obtained by the SACS algorithm [Zhang et al., 2019], where F I = min x ∈X (cid:80) st = r f t ( x ) and F T = min x ∈X (cid:80) Tt =1 f t ( x ) .
However, SACS does not satisfy our reduction requirements, because it requires N gradient queries (i.e., ∇ f t ( x t,i ) for i ∈ [ N ] ) and N + 1 function evaluations (i.e., f t ( x t,i ) for i ∈
[ N ] , and f t ( x t ) ) at round t ∈
[ T ] , where N denotes the number of active base-learners and x t,i denotes local decision returned by the i -th base-learner.
To address so, we have to modify the algorithm to fit our purpose.
In the following, we first sketch the SACS algorithm and then present our modifications.
In fact, to optimize the adaptive regret, an online algorithm usually consists of the three components:
(i) base-algorithm: an online algorithm that can attain low (static) regret in a given interval; (ii) scheduling: a set of intervals and each one is associated with a base-learner who aims to minimize the static regret over the interval (from starting time to ending time); (iii) meta-algorithm: a combining algorithm that can track the best base-learner on the fly.
The specific configurations of the SACS algorithm is as follows.
First, SACS uses scale-free online gradient descent (SOGD)
[Orabona and Pál, 2018] as the base-algorithm, which ensures a small-loss regret in a given interval.
Second, SACS employs AdaNormalHedge [Luo and Schapire, 2015] as the meta-algorithm, which supports the sleeping expert setup and also enjoys a small-loss regret.
Finally, SACS designs a clever strategy of problem-dependent geometric covers to determine the set of intervals such that the number of active base-learners also depends on the small-loss quantity.
As a result, SACS can achieve a fully problem-dependent adaptive regret of order O ( √ F I log F I log F T ) , scaling with the cumulative loss of comparators.
However, SACS also suffers from an O (log T ) projection complexity in the worst case due to a two-layer structure; and moreover, it can be observed that SACS only attains an O ( (cid:112) | I | log | I | log T ) bound in the worst case, which exhibits an (cid:112) log | I | gap compared with the best known result of O ( (cid:112) | I | log T )
[Jun et al., 2017].
Below, we present an efficient algorithm for small-loss adaptive regret, which resolves the above two issues simultaneously.
3.2 Efficient Algorithms for Adaptive Regret
As multiple gradient queries and function evaluations are involved in all the three components of
SACS, we have to make plenty of modifications to achieve an algorithm with small-loss adaptive regret yet requiring only one gradient query and function evaluation per round.
With such an algorithm on hand, we can then deploy our reduction to achieve an efficient method with 1 projection complexity.
Below we present the details.
By the reduction mechanism, it is noticeable that we only need to consider the input online functions as surrogate loss { g t } Tt =1 , where g t is defined in Eq.
Base-algorithm.
We use SOGD with a linearized surrogate loss ⟨∇ g t ( y t ) , y ⟩ over the surrogate domain Y .
Denote by A t the set of active base-learners’ indices, then the base-learner B i updates by
[Equation]
9) with η t,i =
D/ (cid:113) ( δ + (cid:80) ts
=
τ i ∥∇ g s ( y s ) ∥ 22 ) , where τ i denotes the starting time of the base-learner
i ∈
A t .
The projection onto Y can be easily calculated by a simple rescaling if needed.
Notably, owing to the convexity of the surrogate loss g t , we can use the same gradient ∇ g t ( y t ) for all the base-learners at each round, ensuring one gradient query of ∇ f t ( x t ) at each round.
Geometric Covers.
The covers consist of a set of intervals that specify the alive time of baselearners.
To achieve a small-loss adaptive regret, SACS [Zhang et al., 2019] employs a clever covering construction called problem-dependent geometric covers (PGC) — instead of initiating a base-learner at each round t like earlier algorithms [Daniely et al., 2015; Jun et al., 2017], SACS adds a new base-learner only when the cumulative loss exceeds a pre-defined threshold .
As a result, the number of active base-learners relates to the small-loss quantity such that the overall algorithm achieves a fully problem-dependent adaptive regret.
Notably, to determine the threshold, SACS monitors the cumulative loss of the latest base-learner f t ( x t,i † ) with i † being the latest base-learner’s index, but clearly this will introduction an additional function evaluation beyond f t ( x t ) at each round.
To avoid the limitation, instead of using a fixed threshold to decide the initiations of base-learners, we design a sequence of time-varying thresholds to adaptively start a new base-learner according to amount of cumulative loss of final decisions (e.g., f t ( x t ) ), bypassing the requirement of additional function evaluation.
This realizes the condition of one function evaluation per round.
Also, the new design of thresholds mechanism is important to ensure that the overall small-loss bound can simultaneously recover the best known worst-case guarantee, which SACS fails to achieve [Zhang et al., 2019].
Let C 1 , C 2 , C 3 , . . .
denote the sequence of thresholds, and they will be determined by a threshold generating function G ( · ) :
N (cid:55)→ R + that will be specified later.
Our problem-dependent geometric covers are set as follows.
We initialize the setting by s 1 = 1 .
We set s 2 as the round when the cumulative loss of the overall algorithm (namely, (cid:80) ts =1 f s ( x s ) ) exceeds the threshold C 1 and then initialize a new instance of SOGD starting at this round.
The process is repeated until the end of online game.
We thus generate a sequence of points s 1 , s 2 , . . .
, referred to as the markers .
See
Input: threshold generating function G ( · ) : N (cid:55)→ R + .
1: Initialize total intervals m = 1 , marker s 1 = 1 , threshold C 1 = G (1) ; let x 1 be any point in X .
2: for t = 1 to T do 3:
Receive the gradient information ∇ f t ( x t ) .
Construct the surrogate loss g t : Y (cid:55)→ R according to Eq.
Compute the (sub-)gradient ∇ g t ( y t ) according to Lemma 1.
Compute L t =
L t − 1 + f t ( x t ) % constructing Problem-dependent Geometric Covers(PGC) 7:
if L t > C m then 8:
Set L t = 0 , remove base-learners B k whose ending point e k = m + 1 .
Set m ← m + 1 , s m ← t , C m =
G ( m ) .
Initialize a new base-learner with ending point e m =
j satisfying [ m, j − 1] ∈ C , where
C = (cid:83) k ∈ N ∪{ 0 } C k and C k = (cid:8)
[ i · 2 k , ( i + 1) · 2 k − 1] | i is odd (cid:9) for all k ∈ N ∪ { 0 } .
Set γ m =
ln(1 + 2 m ) , w t,m = 1 , η t,m = min { 1 / 2 , √ γ m } for the meta-algorithm.
12: end if 13:
Send ∇ g t ( y t ) to all base-learners and obtain local predictions y t
+1 ,i for i ∈
A t .
Meta-algorithm updates weight p t +1 ∈ ∆ | A t +1 | according to Eq.
(11), Eq.
(12), and Eq.
Compute y t +1 =
(cid:80)
i
A
1 p
,i y
,i .
Submit x
= Π
▷ The only projection onto feasible domain X per round.
17: end for the condition in Line 7, registration of markers in Line 9, and the overall updates in Lines 7 – 11 of
Algorithm 2.
Those markers specify the starting time (and the ending time) of base-learners and thus construct the PGC as
[Equation]
is
[Equation]
It is noteworthy to emphasize that PGC is constructed by the language of “marker”, whose exact time stamp is unknown ahead of time but is only determined according to the learner’s performance on the fly.
Moreover, the notation C in Lines 10 of Algorithm 2 is defined based on the registered indexes of markers, and there is a one-one correspondence from the interval in C to that in PGC (cid:101) C .
More concretely, an interval [ i · 2 k , ( i +1) · 2 k − 1] ∈ C will be mapped into the interval [ s i · 2 k , s ( i +1) · 2 k − 1] in PGC, managing the alive time of base-learners in a geometric manner with respect to the subscripts.
Meta-algorithm.
SACS uses the AdaNormalHedge
[Luo and Schapire, 2015] as the meta-algorithm, however, this is not suitable for our propose.
To ensure one projection per iteration, we cannot use multiple function values, i.e., { g t ( y t,i ) } Ni =1 , for meta-algorithm to evaluate the loss.
Instead, we can only use the linearized loss value, namely, {⟨∇ g t ( y t ) , y t,i ⟩} Ni =1 in the weight update of metaalgorithm.
The small-loss regret bound in the meta-algorithm of SACS crucially relies on the original function values, which is unfortunately inaccessible in our case.
Technically, when fed with linearized loss, it is hard to establish a squared gradient-norm bound and then convert it to the small loss due to the first-order regret bound of AdaNormalHedge.
Based on this crucial technical observation, we propose to use the Adapt-ML-Prod algorithm
[Gaillard et al., 2014] as the meta-algorithm in our method.
The key advantage is that it enjoys a second-order regret and also supports the sleeping expert setup.
Adapt-ML-Prod maintains multiple learning rates η t +1 and an intermediate weight vector w t +1 , which are updated by the following rule.
For any active base-learner i ∈
A t +1 , η t +1 ,i = min
[Equation]
where γ i = ln(1 + 2 i ) is a certain scaling factor and the feedback loss is constructed as for i
∈
A t (cid:98) ℓ t = ⟨∇ g t ( y t ) , y t ⟩ / (2 GD ) , and ℓ t,i = ⟨∇
g t ( y t ) , y t,i ⟩ / (2 GD ) .
The final weight vector p t +1 ∈ ∆ | A t +1 | is obtained by
[Equation]
Notably, the meta update only uses one gradient at round t , namely, ∇ g t ( y t ) .
Finally, we compute y t
+1 =
(cid:80)
i ∈
A t +1 p t +1 ,
i y t +1 ,i as the overall prediction in the surrogate domain
Y and calculate x t +1 =
Π X
[ y t +1 ] to ensure the feasibility.
This is the only projection onto X at each round.
Algorithm 2 summarizes the main procedures of our efficient methods for small-loss adaptive regret.
Albeit with a similar two-layer structure as SACS, our algorithm exhibits salient differences in base-algorithm, meta-algorithm, and geometric covers.
As a benefit, we can successfully deploy our reduction mechanism and make the overall algorithm project onto the feasible domain X once per round, see Line 16.
Our method retains the same small-loss adaptive regret as [Zhang et al., 2019].
Theorem 4.
Under Assumptions 1–3, setting the threshold generating function G ( m ) = Θ(log m ) whose explicit form is in Eq.
(46) of Appendix C, Algorithm 2 requires only one projection onto X per round and enjoys the small-loss adaptive regret:
[Equation]
for any interval I =
[ r, s ] ⊆
[ T ] , where F I = min x ∈X (cid:80) st = r f t ( x ) and F T = min x ∈X (cid:80) Tt =1 f t ( x ) .
Remark 1.
Note that the O ( √ F I log F I log F T ) small-loss bound of Zhang et al.
[2019] becomes
O ( (cid:112) | I | log | I | log T ) in the worst case, looser than the O ( (cid:112) | I | log T ) bound [Jun et al., 2017] by a factor of log | I | .
We show that this limitation can be actually avoided by the new design of thresholds mechanism and a refined analysis.
More discussions can be found in Appendix C.3.
Indeed, our result in (14) can strictly match the best known problem-independent result in the worst case.
Experiment
In this section, we provide empirical studies to evaluate our proposed methods.
General Setup.
We conduct experiments on the synthetic data.
We consider the following online regression problem.
Let T denote the number of total rounds.
At each round t ∈ [ T ] the learner outputs the model parameter w t ∈ W ⊆
R d and simultaneously receives a data sample ( x t , y t ) with x t ∈ X ⊆
R d being the feature and y t ∈ R being the corresponding label.
The learner can then evaluate her model by the online loss f t ( w t )
= 12 ( x ⊤ t w t − y t ) 2 which uses a square loss to evaluate the difference between the predictive value x ⊤ t w t and the ground-truth label y t , and then use the feedback information to update the model.
In the simulations, we set T = 20000 , the domain diameter as D = 6 , and the dimension of the domain as d = 8 .
The feasible domain W is set as an ellipsoid
W = (cid:8) w ∈ R d | w ⊤ Ew ≤ λ min ( E ) · ( D/ 2) 2 (cid:9) , where E is a certain diagonal matrix and λ min ( E ) denotes its minimum eigenvalue.
Then, a projection onto W requires solving a convex programming that is generally expensive.
In the experiment, we use scipy.optimize.
NonlinearConstraint to solve it to perform the projection onto the feasible domain.
To simulate the non-stationary online environments, we control the way to generate the date samples { ( x t , y t ) }
Tt =1 .
Specifically, for t ∈ [ T ] , the feature x t is randomly sampled in an Euclidean ball with a diameter D same as the feasible domain of model parameters; and the corresponding label is set as y t
= x ⊤ t w ∗ t + ε t , where ε t is the random noise drawn from [0 , 0 . 1] and w ∗ t is the underlying ground-truth model from the feasible domain W generated according to a certain strategy specified below.
For dynamic regret minimization, we simulate piecewise-stationary model drifts, as dynamic regret will be linear in T and thus vacuous when the model drift happens every round due to a linear path length measure.
Concretely, we split the time horizon evenly into 25 stages and restrict the underlying model parameter w ∗ t to be stationary within a stage.
For adaptive regret minimization, we simulate gradually evolving model drifts, where the underlying model parameter w ∗ t +1 is generated based on the last-round model parameter w ∗ t with an additional random walk in the feasible domain
W .
The step size of random walk is set to be proportional to D/T to ensure a smooth model change.
Contenders.
For both dynamic regret and adaptive regret minimization, we directly work on the small-loss online methods.
We choose the Sword algorithm
[Zhao et al., 2021b] as the contender of
[Figure]
our efficient method for dynamic regret (Algorithm 1) and choose the SACS algorithm
[Zhang et al., 2019] as the contender of our efficient method for adaptive regret (Algorithm 2).
Results.
We repeat the experiments for five times with different random seeds and report the results (mean and standard deviation) in Figure 1.
We use a machine with a single CPU (Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz) and 32GB main memory to conduct the experiments.
We plot both cumulative loss and running time (in seconds) for all the methods.
We first examine the performance of dynamic regret minimization, see Figure 1(a) for cumulative loss and see Figure 1(b) for running time.
The empirical results show that our method has a comparable performance to Sword without much sacrifice of cumulative loss, while our method can achieve about 10 times speedup due to the improved projection complexity.
Second, as shown in Figure 1(c) and Figure 1(d), a similar performance enhancement also appears in adaptive regret minimization, though the speedup is slightly smaller due to the fact that fewer learners are required to maintain for adaptive regret.
To summarize, the empirical results show the effectiveness of our methods in retaining the regret performance and also the efficiency in terms of the running time due to the reduced projection complexity.
Conclusion
In this paper, we design a generic reduction mechanism that can reduce the projection complexity of two-layer methods for non-stationary online learning, hence approaching a clearer resolution of necessary computational overhead for robustness to non-stationarity.
Building on the reduction mechanism, we develop a series of online algorithms for optimizing dynamic regret and adaptive regret.
All the algorithms retain the best known regret guarantees, and more importantly, require one projection onto the feasible domain per iteration.
It is further worth mentioning that, due to the requirement of our reduction, all our algorithms only need one gradient query and one function evaluation at each round as well, which can be appealing in situations with limited feedback.
Our reduction can also be applied to other settings to achieve light project complexity, for example, dynamic regret of OCO with memory [Zhao et al., 2022b], OCO with switching cost [Zhang et al., 2021], and related applications such as online non-stochastic control [Hazan et al., 2020].
Moreover, it is possible to derive similar efficient algorithms for minimizing the interval dynamic regret, an even stringent measure for non-stationary online convex optimization.
There is one important open question left on another type of problem-dependent bound that scales with gradient variation [Chiang et al., 2012], which plays an important role in establishing fast convergence in zero-sum games [Syrgkanis et al., 2015; Zhang et al., 2022].
Although Zhao et al.
[2021b] have devised a two-layer method that enjoys a gradient-variation dynamic regret and requires one gradient per iteration, it is quite challenging to incorporate the optimistic online learning into our reduction mechanism due to the constrained feasible domain and the complicated two-layer structure.
Finally, it would be greatly important to further understand the minimal computational overhead in response to the robustness to non-stationarity, in particular, some information-theoretic arguments would be highly interesting.
Acknowledgements
This research was supported by NSFC (61921006, 62206125), JiangsuSF (BK20220776), National
Postdoctoral Program for Innovative Talent, and Collaborative Innovation Center of Novel Software Technology and Industrialization.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See discussions in Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Omitted Details for Reduction Mechanism
In this section, we provide the proofs of Theorem 1 and Lemma 1.
A.1 Properties of Distance Function
Before presenting the proofs, we here collect two useful lemmas regarding the distance function used in the surrogate loss, which will be useful in the following proofs.
The proofs of the two lemmas can be found in the seminal paper of Cutkosky and Orabona [2018].
Lemma 2 (Proposition 1 of Cutkosky and Orabona [2018]) .
The distance function S X ( y ) =
inf x ∈X ∥
y − x ∥ 2 is convex and 1 -Lipschitz for any closed convex feasible domain X ⊆ R d .
Lemma 3 (Theorem 4 of Cutkosky and Orabona [2018]) .
Let X ⊆
R
d a closed convex set.
Given y ∈ R d and y / ∈ X .
Let x =
Π X
[ y ] .
Then we have { y − x ∥
y − x ∥ 2 } = ∂S X ( y ) .
A.2 Proof of Theorem
Theorem 1 is originally due to Cutkosky [2020], and for self-containedness we restate their proof using our notations.
Proof.
When ⟨∇ f t ( x t ) , v t ⟩ ≥ 0 , by the definition of the surrogate loss defined in Eq.
(5), we have g t ( y ) =
⟨∇ f t ( x t ) , y ⟩ , which is linear in y and thus convex (in fact linear in y ).
It is clear that ∥∇ g t ( y t ) ∥ 2 = ∥∇ f t ( x t ) ∥ 2 and thus satisfies the claimed inequality of gradient norms in the statement.
Moreover, the inequality (6) holds evidently due to the linear surrogate loss in this case.
Let us focus on the case when ⟨∇ f t ( x t ) , v t ⟩ < 0 .
First, it can be verified that the surrogate loss g t ( y ) =
⟨∇ f t ( x t ) , y ⟩ − ⟨∇ f t ( x t ) , v t ⟩ · S X ( y ) is convex due to the convexity of S X ( y ) shown in
Lemma 2 and the condition of ⟨∇ f t ( x t ) , v t ⟩ < 0 in this case.
Next, the gradient of g t ( · ) at the y t point can be calculated according to Lemma 1 as,
[Equation]
where v t =
( y t − x t ) /
∥
y t − x t ∥ 2 .
Notice that ∥ v t ∥ 2 = 1 and ∇ g t ( y t ) is an orthogonal projection of ∇ f t ( x t ) onto the subspace perpendicular to the vector v t , so we have ∥∇ g t ( y t ) ∥ 2 ≤ ∥∇ f t ( x t ) ∥ 2 .
Finally, we proceed to prove the inequality (6) in this case.
Since the comparator u t ∈ X is in the feasible domain, we have S X ( u t ) =
∥
u t − u t ∥ 2 = 0 and get
[Equation]
where the last inequality holds owing to the convexity of the surrogate loss proven earlier.
Combining the two cases finishes the proof.
A.3 Proof of Lemma
Lemma 1 is originally due to Cutkosky and Orabona [2018], and for self-containedness we restate their proof using our notations.
Proof.
With a slight abuse of notations, for simplicity we use the notation ∇ g t ( y ) to denote the (sub-)gradient of surrogate function g t ( · ) at point y , no matter whether the function is differentiable.
When ⟨∇ f t ( x t ) , v t ⟩ ≥ 0 , the surrogate loss is g t ( y ) =
⟨∇ f t ( x t ) , y ⟩ by definition in Eq.
Therefore, the gradient simply becomes ∇ g t ( y t ) =
∇ f t ( x t ) .
When ⟨∇ f t ( x t ) , v t ⟩ < 0 , the surrogate loss becomes g t ( y ) =
⟨∇ f t ( x t ) , y ⟩−⟨∇ f t ( x t ) , v t ⟩· S X ( y ) according to definition in Eq.
By Lemma 3, the gradient ∇ g t ( y ) can be calculated by
[Equation]
where the computation needs the projection onto domain X .
In particular, for y t , we have
[Equation]
This ends the proof.
B Omitted Details for Dynamic Regret Minimization
In this section, we provide the proofs for the theorems presented in Section 2.
Specifically, we first prove the worst-case bound (Theorem 2) and then work on the small-loss bound (Theorem 3).
B.1 Proof of Theorem
Proof.
Notice that Zhang et al.
[2018a] propose the improved Ader algorithm (see Algorithmand Algorithm 4 in their paper), which uses the linearized loss as the input to make the online algorithm requiring one gradient and one function evaluation per iteration.
So the algorithm satisfies the requirements of our reduction mechanism, and our algorithm can be regarded as the improved
Ader equipped with the projection-efficient reduction.
As a consequence, we can directly obtain the same dynamic regret guarantee and ensure 1 projection complexity at the same time, by following the same proof of the improved Ader as well as the reduction guarantee (Theorem 1).
B.2 Proof of Theorem
Proof.
By the reduction guarantee shown in Theorem 1, we have the following result that decomposes the dynamic regret into the two terms.
[Equation]
≤
[Equation]
base
[Equation]
regret
[Equation]
15) where in (15) the first term is called meta-regret as it measures the regret overhead of the metaalgorithm to track the unknown best base-learner, and the second term is called the base-regret to denote the dynamic regret of the base-learner i .
Note that the above decomposition holds for any base-learner index i ∈ [ N ] .
Upper bound of meta-regret.
As the meta-algorithm can be regarded as a FTRL with time-varying learning rates and a negative entropy regularizer, we apply Lemma 10 to obtain an upper bound for the meta-regret.
Indeed, by choosing ℓ t,i = ⟨∇
g t ( y t ) , y t,i ⟩ in Lemma 10, we can achieve that
T
[Equation] [Equation]
[Equation]
where the first inequality holds because we have ∥ ℓ t ∥ 2 ∞
=
max i ∈
[ N ] ( ⟨∇ g t ( y t ) , y t,i ⟩ ) 2 ≤
D 2 ∥∇ g t ( y t ) ∥ 22 by Cauchy-Schwarz inequality and ∥∇ g t ( y t ) ∥ 2 ≤ ∥∇ f t ( x t ) ∥ 2 ≤ G (see Theorem 1), the second inequality makes use of √ a + b ≤ √ a + √ b , the third inequality holds by the property of the surrogate loss (also via Theorem 1), and the last inequality is due to the self-bounding properties of smooth functions (see Lemma 12).
Note that O (ln N )
=
O (log log T ) can be treated as a constant following previous studies [Luo and Schapire, 2015; Gaillard et al., 2014]
Upper bound of base-regret.
According to Lemma 7 and noticing that the comparator sequence u 1 , . . .
, u T ∈ X ⊆ Y and the diameter of Y equals to 2 D by definition, with slight modifications, we have the following dynamic regret bound.
[Equation]
where the second inequality is due to the property of the surrogate loss (see Theorem 1) and the last one is due to the self-bounding property of smooth functions (see Lemma 12).
Note that the property of ∥∇ g t ( y t ) ∥ 2 ≤ ∥∇ f t ( x t ) ∥ 2 (see Theorem 1) plays an important role in the above analysis.
Although the surrogate functions { g t } Tt =1 are not guaranteed to be smooth and non-negative, we can upper bound its gradient norm by that defined over the original functions { g t } Tt =1 , which are indeed smooth and non-negative.
We thus can utilize the self-bounding properties to establish a small-loss bound for the meta-regret and base-regret.
Upper bound of dynamic regret.
Plugging the above upper bounds of meta-regret and base-regret together, we achieve
[Equation]
which holds for any base-learner’s index i ∈
[ N ] .
Next, we specify the base-algorithm E i compared with.
Indeed, we aim at choosing the one with step size closest to the (near-)optimal step size η ∗ =
(cid:113) 5 D 2 +2 DP T 1+8 LF x T , where we denote by F x T = (cid:80)
Tt =1 f t ( x t ) the cumulative loss of the decisions.
By Assumption 1 and Assumption 2, we have
F x T ∈ [0 , GDT ] and then the possible minimum optimal and maximum step size are
[Equation]
The construction of step size pool is by discretizing the interval [ η min , η max ] with intervals with exponentially increasing length.
The step size of each base-learner is designed to be monotonically increasing with respect to the index.
Consequently, it is evident to verify that there exists an index i ∗ ∈
[ N ] such that
[Equation]
As the upper bounds of meta-regret and base-regret hold for any compared base-learner, we can choose the index as i ∗ in particular.
Then the second and the third terms in the inequality (19) satisfy
[Equation] [Equation]
[Equation]
Substituting inequality (21) into inequality (19), we have,
T (cid:88) t =
[Equation]
where the last inequality holds by Lemma 17.
Hence, we complete the proof of Theorem 3.
C Omitted Details for Adaptive Regret Minimization
In this section, we present omitted details for minimizing the worst-case and small-loss adaptive regret.
First, in Appendix C.
we describe the efficient algorithm for attaining the worst-case adaptive regret, which provably enjoys the same guarantee as the prior best known work and meanwhile requires one projection only per iteration.
Next, we focus on the proof of the main theorem for small-loss adaptive regret, i.e., Theorem 4.
To this end, Appendix C.2 provides three key lemmas,
Appendix C.3 presents the proof of Theorem C.3, and Appendix C.4 – Appendix C.6 give the proofs of three key lemmas.
C.1 Results for the Worst-Case Adaptive Regret
In this section, we present results for the worst-case adaptive regret omitted in the main text.
The best known result is the O ( (cid:112) | I | log T ) worst-case adaptive regret attained by the CBCE algorithm
[Jun et al., 2017], which is achieved by the coin-betting framework with the sleeping expert mechanism.
However, CBCE requires multiple gradients at each round.
Wang et al.
[2018] improve CBCE by using the linearized loss to make the algorithm requiring one gradient per iteration and retaining the same adaptive regret.
Moreover, the improved CBCE algorithm only evaluates the function value once per iteration, and the algorithm is presented in [Wang et al., 2018, Algorithm 4 and Algorithm 5] .
Therefore, we can simply feed the surrogate loss g t constructed in (5) to the improved CBCE of Wang et al.
[2018], and the obtained algorithm can ensure the same order of adaptive regret and also require only one projection onto the feasible domain per iteration.
C.2 Key Lemmas
In this part, we present three key lemmas for proving the small-loss adaptive regret, namely, Theorem 4.
We then prove Theorem 4 based on these lemmas in Appendix C.3.
Finally, we present the proofs for those lemmas in the following several subsections.
The first lemma gives the upper bound of meta-regret of our efficient method for small-loss adaptive regret, which heavily relies on the structure of the problem-dependent geometric covers.
Lemma 4.
Under Assumptions 2 and 3, for any interval I =
[ i, j ] ∈ (cid:101) C in the geometric covers defined in Eq.
(10) on which we suppose m -th base-learner is active, Algorithm 2 ensures
[Equation]
 , which holds for any time stamp t ∈ [ i, j ] .
Combining above lemma and the analysis of the base-regret upper bound, we can obtain the following adaptive regret for any interval of the problem-dependent geometric covers.
Lemma 5.
Under Assumptions 1, 2, and 3, for any interval [ i, j ] ∈ (cid:101) C in the geometric covers defined in Eq.
(10) , on which we assume m -th expert-algorithm is active, Algorithm 2 ensures
[Equation]
which holds for any time stamp t ∈
[ i, j ] and any comparator u ∈ X .
The above two lemmas rely on the unknown variable of m , and the following lemma presents an upper bound for m in terms of the small-loss quantity F t .
Lemma 6.
Under Assumptions 1, 2, and 3, for any interval [ i, j ] ∈ (cid:101) C and any t ∈
[ i, j ] , the variable m specified in Lemma 4 and Lemma 5 can be bounded by
[Equation]
This immediately implies that Algorithm 2 ensures
[Equation]
where F [ a,b ] = min u ∈X (cid:80) bτ = a f τ ( u ) denotes the cumulative loss of the comparator within the interval [ a, b ] ⊆
[ T ] .
C.3 Proof of Theorem
Proof.
Recall that Theorem 4 exhibits an O (cid:0) min {√ F I log F
I log F T , (cid:112) | I | log T } (cid:1) adaptive regret for any interval
I =
[ r, s ] ⊆
[ T ] , where F I = min x ∈X (cid:80) st = r f t ( x ) and F T = min x ∈X (cid:80) Tt =1 f t ( x ) .
The bound consists of two parts, including a small-loss bound of O ( √ F I log F
I log F T ) and a worst-case bound of O ( (cid:112) | I | log T ) .
Below, we present the proofs of the two bounds respectively.
Before showing the proofs, we emphasize again that our result strictly improves the small-loss bound of Zhang et al.
[2019], who give an O ( √ F I log F
I log F T ) bound that becomes O ( (cid:112) | I | log | I | log T ) in the worst case and thus is looser than the O ( (cid:112) | I | log T ) problem-independent bound
[Jun et al., 2017] by a factor of log | I | .
Our regret guarantee consists of the small-loss bound and another worst-case bound acting as a safety guarantee for the worst case.
Indeed, in the worst-case situation, our bound becomes O ( (cid:112) | I | log T ) and strictly match the best known worst-case result [Jun et al., 2017].
We note that our improvement is owing to a refined analysis in the proof as well as our careful algorithm design that only uses one function evaluations to adaptively determine the geometric covers, which is to be contrasted to SACS [Zhang et al., 2019] that uses the latest base-learner’s decision to determine the covers.
Small-loss regret bound.
Let s p be the smallest marker that larger than r , and let s q be the largest marker that is not large than s , then we have
[Equation]
and s
[Equation]
s < s q +1 .
We bound the regret over the interval [ r, s p − 1] as,
[Equation]
The last inequality is because of the construction rule of marker and the fact that f t ( x t ) ≤ GD for any t ∈
[ T ] by Assumptions 1 – 3.
By Lemma 11, we can find v consecutive intervals
[Equation] [Equation]
(24) such that i 1 = p, i v ≤ q <
i
[Equation]
+1 , and v ≤ ⌈ log 2 ( q − p + 2) ⌉ .
Notice that,
[Equation]
For the neat presentation, we define,
[Equation]
where we use the notations µ ( t ) = ln(1+(1+ln(1+ t ))
/ (2 e ))
=
O (log log t ) that can be essentially regarded as a constant and C 1 = G (1) , where G ( · ) is the threshold generating function defined in (46).
For intervals I 1 to I v , by Lemma 5, s
[Equation]
p
[Equation]
f
[Equation]
k = (cid:16)
[Equation]
Combining (23) and (26), the adaptive regret on any interval i =
[ r, s ] will be
[Equation]
Next
[Equation]
we show that v and C p − 1 are of order
O (log F [ r,s ] ) and O (log F T ) respectively.
By the definition of the time-varying threshold (see the threshold generating function Eq.
and Lemma 6, the threshold can be bounded as,
[Equation]
which is of order O (log F T ) .
With the same argument as (48)
[Equation]
it can be shown that between markers s p and s q , for any u ′ ∈ X ,
[Equation] [Equation]
which suggests
[Equation]
We thus have
[Equation]
(cid:19)(cid:25) =
O (log F [ r,s ] ) .
Combining the upper bounds of C p − 1 and v as well as the adaptive regret bound in (27) yields s
[Equation]
where the last step is true as we follow the same convention in [Zhang et al., 2019] to treat the log F I log F T as the non-leading term.
Hence, we finish the proof of the small-loss adaptive regret.
Worst-case regret bound.
The above proof aims at obtaining small-loss type regret bound, and one of the key steps is to use Cauchy-Schwarz inequality to bound (25), which results in an additional
O ( (cid:112) log F
[ r,s ] ) term.
Next, we show that actually this extra term can be avoided by the new design thresholds mechanism and thus asymptotically achieve the same worst-case adaptive regret as the best known result [Jun et al., 2017].
From (43) in Lemma 5, we have that for any interval I =
[ i, j ] in problem-dependent covers defined in (10), the adaptive regret is at most
[Equation]
where we use the notation F x [ a,b ] = (cid:80) bt = a f t ( x t ) to denote the cumulative loss of the returned decisions within the interval [ a, b ] ⊆
[ T ] .
Then, we can use Lemma 6 to upper bound m ≤ O ( T ) as only the worst-case behavior matters now.
Moreover, for the consecutive intervals defined in (24), we have the following facts:
[Equation]
[ v ] , and |
[Equation]
The first relationship between consecutive foot-indexes of time markers will be used to show that thresholds will not grow too fast during an interval in the geometric covers, which can verified by the construction of cover defined in (10).
The second inequality indicates that the times the cumulative loss of algorithm exceeds thresholds in an interval decreases exponentially from I v to I 1 , and this can be verified in the proof of [Zhang et al., 2019, Lemma 11].
For the interval I k with k ∈ [ v − 1] in (24)
[Equation]
our algorithm’s cumulative loss within the interval is upper bounded by
[Equation]
We then split a given interval [ r, s ] into three parts to analyze, namely, the consecutive v − 1 intervals
I 1 to I v − 1 , interval [ r, s p − 1]
[Equation]
and [ s i v , s ] , where notably the last two intervals are not fully covered by any interval in geometric covers.
For intervals I 1 to I v − 1 , we have
[Equation]
O (cid:32)(cid:114) log T ·
[Equation]
i v
[Equation]
· | i v − i
[Equation]
|
[Equation]
− a +
[Equation]
≤ O (cid:32) v log
[Equation]
+ (cid:112) log T · C i v − 1 · + ∞ (cid:88)
[Equation] [Equation]
| i v − i
[Equation]
− 1 |
[Equation]
(cid:33) ≤ O (cid:16) v log T + (cid:112) log T · C i v − 1 · | i v − i v − 1 | (cid:17) , where the second inequality is due to the monotonically increasing property of thresholds, the third inequality is by (28), and the last inequality is by the summation of geometric sequence.
By the setting of time-varying thresholds as specified in Eq.
(46), we know C
i v − 1 =
O (log( i v )) .
Furthermore, | i v − i v − 1 | represents the number of markers that our algorithm generates during the interval I v − 1 , so it can be upper bounded as
[Equation]
because the total loss of the algorithm during | I | is at most GD | I | and we use the smallest threshold
C i v − 1 during the interval I v − 1 to calculate the maximum number of possible markers.
Due to the relationship of 2 i v − 1 ≥ i v by (28), we then get
[Equation]
where the last inequality is due to the fact that v is of order O
(log F [ r,s ] )
=
O (log | I | ) .
Remind that the variable v appears in our analysis by Lemma 11, which is independent of the worst-case analysis.
Now we proceed to upper bound the regret over intervals [ r, s p − 1] and [ s i v , s ] .
By a similar analysis used early, we have
[Equation]
and s
[Equation]
=
[Equation]
iv
[Equation]
Combining everything together achieves
[Equation]
log T + log | I | log T
[Equation]
= O
[Equation]
log
[Equation]
= O ( (cid:112) | I | log T ) .
The last step holds by considering the following cases.
• When the interval length is | I | =
Θ( T α ) with α ∈ (0 , 1] .
Then,
[Equation]
• When the interval length is | I | = Θ(log β T ) , and note that β ∈ [1 ,
+ ∞ ) as | I | = Ω(log T ) is the minimum order to ensure the adaptive regret to be non-trivial.
Then,
[Equation] [Equation]
where the second equality is true for β > 1 and otherwise we can treat O (log log T ) as a constant following previous studies
[Luo and Schapire, 2015; Gaillard et al., 2014].
Hence we finish the proof for the worst-case adaptive regret bound.
Combining both small-loss bound and the worst-case safety guarantee, we complete the proof of Theorem 4.
Discussions about the analysis of worst-case bound.
We point out that the same analysis for worst-case adaptive bound cannot be applied to SACS [Zhang et al., 2019] directly, because the new designed time-varying thresholds mechanism plays an important role.
Indeed, SACS monitors the cumulative loss of the new-added base-learner and switches to the another one immediately after a new marker is registered, which causes intermittent performance monitoring for the remaining markers the base-learner may go through.
Thus, we cannot employ a similar argument (like Eq.
(28) in our analysis) to bound the cumulative loss by the summation of thresholds for adaptive regret of
SACS, as the performance of base-learner on the whole active interval is absent.
Finally, we would like to emphasize the necessity of monitoring the final decisions’ cumulative loss.
To meet the one projection requirement, our base-algorithm is updated over the surrogate loss and the provided gradient information is about the final decision only.
Thus the thresholds are set on the cumulative loss of final decisions to exploit the limited available information.
C.4 Proof of Lemma
Proof.
First we introduce some useful variables to help us prove the adaptivity of Adapt-ML-Prod under sleeping-expert setting.
Similar to the proof technique proposed in [Daniely et al., 2015], for any interval [ i, j ] ∈ (cid:101) C in the geometric covers defined in (10), on which we suppose m -th base-learner is active, we define the following pseudo-weight for the m -th base-learner,
[Equation]
In addition, we use (cid:102) W t = (cid:80) k ∈
[ T ] (cid:101) w t,k to denote the summation of pseudo-weights for all possible base-learners up to time t .
As for the problem-dependent geometric covers, in the worst case there are at most T base-learners generated, we use [ T ] to denote the indexes for all the base-learners.
Notice that the pseudo-weight (cid:101) w t is defined as 0 for asleep base-learners till time t , so we can include all possible ones safely in the definition even though they are not generated in practical implementations of the algorithm.
In the following, we use the classic potential argument [Gaillard et al., 2014] by showing both lower bound and upper bound of ln (cid:102) W t +1 to establish relationships between certain concerned quantities.
Lower bound of ln (cid:102) W t +1 .
We claim that for t ∈ [ i, j ] it holds that
[Equation]
We prove the above inequality by induction on t .
When t =
i , by definition,
[Equation]
where the inequality is because of ln(1 + x ) ≥ x − x 2 , ∀ x ≥ − 1 / 2 .
Suppose the statement holds for ln (cid:101) w t,m , then we proceed to check the situation for t + 1 round as follows.
Indeed,
[Equation]
Then, as (cid:101) w t +1 ,m is positive for any m -th base-learner, we have ln (cid:102) W t +1 ≥ ln (cid:101) w t +1 ,m .
Combining (30) obtains the desired lower bound of ln (cid:102) W t +1 .
Upper bound of ln (cid:102) W t +1 .
By the construction of the geometric covers as specified in Eq.
(10), we know that there will be at most 2 m base-learners initialized for the m -th base-learner active on interval [ i, j ] till her end.
This is because m -th base-learner is initialized when m -th marker is recorded, and she will expire before the moment when 2 m -th marker is recorded, as demonstrated by the construction of cover defined in Eq.
Owing to this property, we have (cid:102)
[Equation]
as others’ pseudo-weight equals to 0 by definition.
So we can upper bound (cid:102) W t +1 as,
[Equation]
where with a slight abuse of notations, we denote by [ i k , j k ] ∈ (cid:101) C the active time for k -th base-learner.
For the second term in (31), we have
[Equation]
where the first equality holds by the definition of (cid:101)
w t
+1 ,k , the second inequality is by the updating rule of (cid:101)
w t
+1 ,k and Lemma (18), and the second term in the last equality equals to 0 due to the weight update rule in (13) and the fact of (cid:101) w t,k = w t,k for any t ∈
[ i k , j k ] .
Combining (31), (32) and by induction, we obtain the following upper bound:
[Equation]
where we denote α ∧ β = min { α, β } .
We now turn to analyze the third term in (32).
Indeed, Gaillard et al.
[2014] have analyzed it under the static regret measure.
For the sake of completeness, we present the proof with our notations.
For any k ∈ [2 m ] , for any τ ∈ [ i k , t ∧ j k ] , the relationship between η τ,k and η τ +1 ,k can be considered as three cases,
• η τ,k = η
τ +1 ,k
= 1 / 2 , • η τ +1 ,k = (cid:113) γ k / (1 + (cid:80) τu =
i k r 2 u,k ) <
η τ,k = 12 , • η τ +1 ,k ≤ η τ,k < 1 / 2 .
In all cases, the ratio η τ,k /η τ +1 ,k − 1 is at most
[Equation]
[Equation]
where the second inequality uses √ 1 + x ≤ 1 + x/
and the third inequality follows from Lemmawith the choice of f ( x ) = 1 /x .
Substituting (34) into (33), we get
[Equation]
Further taking the logarithm over the above inequality gives the upper bound of ln (cid:102) W t +1 .
Upper bound of meta-regret.
Now, we can lower bound and upper bound ln (cid:102) W t +1 by (30) and (35), with arrangement, which yields the upper bound of scaled meta-regret.
Concretely, t (cid:88) τ
[Equation]
where we denote µ ( t )
=
ln(1 + (1 + ln(1 + t )) / (2 e )) .The
second inequality is by Lemma 14 and choose f ( x ) = 1 / √ x .
The last equality is by the choice of γ m = ln(1 + 2 m ) .
As for the third inequality, there are two cases to be considered:
• when (cid:113) 1 + (cid:80) tτ =
i r 2 τ,m > 2 √ γ m , we have that (36) is at most
[Equation]
• when (cid:113) 1 + (cid:80) tτ =
i r 2 τ,m ≤ 2 √ γ m , we have that η t +1 ,m = 1 /
and (36) is at most
[Equation]
Taking both cases into account implies the third inequality.
Finally
[Equation]
we end the proof by evaluating the meta-regret in terms of the surrogate loss.
t (cid:88)
[Equation]
2 m ) + 6 GDµ ( t ) ≤ (cid:18)
[Equation]
(cid:112)
[Equation]
(cid:19)(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88)
τ
[Equation]
i
[Equation]
( y τ
[Equation] [Equation]
18 GD ln(1 + 2 m ) + 6 GDµ ( t )
[Equation]
2 D (cid:18)
[Equation]
(cid:19)(cid:118)(cid:117)(cid:117)(cid:116) t (cid:88)
[Equation]
i
[Equation] [Equation]
[Equation]
6 GDµ ( t ) (38)
[Equation]
D
[Equation]
(cid:112)
[Equation]
(cid:19)(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88)
[Equation]
i
[Equation] [Equation]
t
[Equation]
(39) where the second inequality is true because 1 ≤ (cid:112) ln(1 + 2 m ) ≤ ln(1 + 2 m ) holds for any m ≥ 1 , the third inequality is by Cauchy-Schwarz inequality, the forth inequality is by Theorem 1 and the last inequality is due to the self-bounded property of smooth functions (see Lemma 12).
C.5 Proof of Lemma
Proof.
Similar to the proof of dynamic regret (see Theorem 3), we start the proof by decomposing the interval regret into meta-regret and base-regret in terms of the surrogate loss by Theorem 1, t (cid:88)
[Equation]
t
[Equation]
where our analysis will be performed by tracking the m -th base-learner, whose corresponding active interval is exactly the analyzed one.
Our analysis is satisfied to any interval since there is always a base-learner active on it by the algorithm design.
Upper bound of base-regret.
Since the base-algorithm (SOGD) guarantees anytime regret, direct application of Lemma 8 with the assumption of surrogate domain Y can upper bound the base-regret, t
[Equation]
where we skip some steps for transforming ∥∇ g τ ( y τ ) ∥ 22 into 4 Lf τ ( x τ ) .
The similar arguments can be found in the proof of Theorem 3.
Upper bound of meta-regret.
By Lemma 4 (see Eq.
(39) for a detailed form), we can upper bound the meta-regret as
[Equation]
where we denote µ ( t )
=
ln(1 + (1 + ln(1 + t )) / (2 e )) , which is of order O (log log t ) and can be treated as a constant.
Upper bound of interval regret.
Substituting (41), (42) into (40) obtains the interval regret t (cid:88)
τ
= i f
[Equation] [Equation]
− t (cid:88) τ
= i f τ ( u ) ≤ O   (cid:118)(cid:117)(cid:117)(cid:116) log m · t (cid:88) τ
= i f τ ( x τ ) + log m 
 , (43) which is related to the returned decisions of our algorithm.
Further, by applying Lemma 17 to the preceding inequality, we can substitute (cid:80) tτ =
i f τ ( x τ ) into decision-independent factor (cid:80) tτ =
i f τ ( u ) , t (cid:88) τ
= i f τ ( x τ ) − t (cid:88) τ
= i f τ ( u ) ≤ 4 D √ L (cid:18)(cid:112) ln(1 + 2 m ) + µ ( t ) (cid:112) ln(1 + 2 m ) +(cid:19)(cid:118)(cid:117)(cid:117)(cid:116) t (cid:88) τ
= i f τ ( u ) + 18 GD ln(1 + 2 m ) + 6 GDµ ( t ) + 4 D √ δ + 18 GD ln(1 + 2 m ) + 6 GDµ ( t ) + 4 D √ δ + 16 D 2 L (cid:18)(cid:112) ln(1 + 2 m ) + µ ( t ) (cid:112) ln(1 + 2 m )
+(cid:19)≤ 4 D √ L (cid:18)(cid:112) ln(1 + 2 m )
+ µ ( t ) (cid:112) ln(1 + 2 m )
+(cid:19)(cid:118)(cid:117)(cid:117)(cid:116) t (cid:88)
τ
= i f τ ( u ) + 27 GD ln(1 + 2 m ) + 9 GDµ ( t ) + 6 D √ δ + 24 D 2 L (cid:18)(cid:112) ln(1 + 2 m )
+ µ ( t ) (cid:112) ln(1 + 2 m )
+(cid:19)≤
[Equation] [Equation]
The second inequality makes use of √ a + b ≤ √ a + √ b and √ ab ≤ ( a 2 + b 2 ) / 2 .
The last inequality holds by ( a + b + c ) 2 ≤ 3( a 2 + b 2 + c 2 ) .
Finally, with a slight abuse of notations, we show that actually the interval regret can be related to the best offline cumulative loss, i.e., the quantity F [ i,t ] = min u ∈X (cid:80) tτ =
i f τ ( u ) .
Notice that, the above arguments hold for any u ∈ X , so we can simply choose u ∗ ∈ arg min u ′ ∈X (cid:80) tτ =
i f τ ( u ′ ) as the comparator in the above arguments, and for any ˚ u ∈ X , t (cid:88)
[Equation]
This ends the proof.
C.6 Proof of Lemma
Proof.
We claim that the cumulative loss of our algorithm within every interval in the geometric covers (cid:101) C as defined in Eq.
(10) can be upper bounded by the cumulative loss of any comparator and the threshold set for the interval.
To see this, we denote by [ i k , j k ] ∈ (cid:101)
C the active interval for the k -th base-learner.
By Lemma 5, for any t ′ ∈ [ i k , j k ] we have
[Equation]
(45) where G ( k ) is the threshold generating function defined by
G ( k ) =
(54 GD +168 D 2 L ) ln(1+2 k )
+168 D 2 Lµ 2 ( T )+18 GDµ ( T )+6 D √
δ +672 D 2 L, (46) with µ ( T ) =
ln(1 + 1+ln(1+ T ) 2 e ) .
This is true by applying ab ≤ a 2 / 4 + b 2 and ( a + b + c ) 2 ≤ 3( a 2 + b 2 + c 2 ) to split (cid:80) f t ( u ) outside the root in (44).
We denote s k the k -th marker made by the algorithm and it is known that i k
= s k
(but s k +1 − 1 ≤ j k because the base-learner may survive several markers) by the cover mechanism.
According to our algorithm design, we set the threshold for interval [ s k , s k +1 − 1] as
[Equation]
By the construction rule of the markers, we have,
[Equation]
For the k -th base-learner, her ending time j k should be equal or larger than s k +1 − 1 , so we can use (45) to evaluate the lower bound of the cumulative loss of the comparator by setting t ′ =
s k
+1 − 1 .
Indeed, we have
[Equation]
It is worth emphasizing that, after making each marker, the cover will initialize a new base-learner and hence the evaluation as shown above can be made between every two consecutive markers thanks to the streaming initialized new learners.
Next, we proceed to upper bound m given in the lemma.
As the m -th base-learner is active on interval [ i, j ] , we have the following result on the interval from marker s 1 to s m ,
[Equation]
The first inequality is because of (47).
The second inequality holds since C a is increasing with respect to its index, see the threshold generating function in Eq.
Therefore, rearranging the above inequality shows that the quantity m satisfies the following inequality for any comparator u ∈ X ,
[Equation]
where the last inequality makes use of the nonnegative assumption on loss function.
In particular, we choose the comparator to be the best offline decision optimizing the cumulative loss within this interval and thus achieve
[Equation]
where we denote F [1 ,t ]
= min u ∈X (cid:80) tτ
=
1 f τ ( u ) .
Now combining Lemma 5 and (48), once given time t , we can upper bound the number of baselearners by the cumulative loss of comparators,
[Equation]
which ends the proof of Lemma 6.
D Useful Lemmas
This section collects some lemmas useful for the proofs.
D.1 OGD and Dynamic Regret
This part provides the dynamic regret of online gradient descent (OGD)
[Zinkevich, 2003] and scale-free online gradient descent (SOGD)
[Orabona and Pál, 2018] from the view of online mirror descent (OMD), which is a common and powerful online learning framework.
Following the analysis in [Zhao et al., 2021b], we can directly obtain dynamic regret of OGD and SOGD [Orabona and Pál, 2018] in a unified view owing to the versatility of OMD.
Online mirror descent algorithm updates according to the following rule:
[Equation]
where η t > 0 is the time-varying step size, h t ( · ) : x (cid:55)→ R is the convex loss function, and
D ψ ( · , · ) is the Bregman divergence induced by the regularizer function ψ ( · ) defined as D ψ ( x , y )
= ψ ( x ) − ψ ( y ) − ⟨∇ ψ ( y ) , x − y ⟩ .
OMD following dynamic regret guarantee [Zhao et al., 2021b].
Theorem 5 (Theorem 1 of Zhao et al.
[2021b]) .
Suppose that the regularizer ψ : X (cid:55)→ R is 1-strongly convex with respect to the norm ∥ · ∥ .
The dynamic regret of Optimistic Mirror Descent (OMD) whose update rule specified in (49) is bounded as follows:
[Equation]
which holds for any comparator sequence u 1 , .
u T ∈ X .
Choosing ψ ( x ) = 12 ∥ x ∥ 22 will lead to the update form of online gradient descent used as base learners in our algorithm:
[Equation]
where the Bregman divergence becomes D ψ ( x , x t )
= 12 ∥ x − x t ∥ 22 w.r.t.
the choice of regularizer.
We proceed to show the dynamic regret of online gradient descent (OGD),
Lemma 7.
Under Assumption 2, by choosing static step size η t = η > 0 , Online Gradient Descent defined in equation (50) satisfies:
[Equation]
Proof.
Applying Theorem 5 with the choices of ψ ( x ) = 12 ∥ x ∥ 22 and fixed step size η t = η > 0 gives:
[Equation]
where the last inequality is due to:
[Equation]
∥ u t − 1 − u t ∥ 2 .
D.2 Self-Confident Tuning
Orabona and Pál [2018] have analyzed the regret bound of SOGD.
For completeness, we here provide the regret analysis under the OMD framework.
Indeed, SOGD can be treated as OMD with a self-confident learning rate.
Thus, we have the following lemma.
Lemma 8.
Under assumption 1 and 2, the OMD algorithm defined in equation (49) with the choices of regularizer ψ ( x ) =
∥ x ∥ 22 and the time-varying learning rate defined as:
[Equation]
for some δ > 0 , has the following guarantee:
[Equation]
where u ∈ X can be any comparator.
Proof.
We start the proof with the application of Theorem 5.
For any fixed comparator u ∈ X ,
T
[Equation]
≤ D2 η
[Equation]
+
D
[Equation]
T (cid:88) t =
[Equation]
(cid:18)
[Equation]
η t − η t
[Equation]
(cid:19)
[Equation]
T (cid:88) t =
[Equation]
t ∥∇ f t
[Equation]
t
[Equation]
Applying Lemma 13 to the second term of (51) and by the definition of η T , the regret bound becomes
T (cid:88) t =
[Equation]
( x
[Equation]
−
[Equation]
(cid:88)
[Equation]
which completes the proof.
To bound the meta-regret of our dynamic methods, we introduce the FTRL lemma [Orabona, 2019,
Corollary 7.8] under the time-varying learning rate.
Lemma 9 (FTRL Lemma) .
Suppose that the regularizer function ψ : X (cid:55)→ R is α -strongly convex with respect to the norm ∥ · ∥ .
Let f t be a sequence of convex loss functions and ψ t ( x ) =
η t ( ψ ( x ) − min x ′ ∈X ψ ( x ′ )) , where η t +1 ≤ η t , t = 1 , . . .
, T .
Then the decision sequence x t generated by
[Equation]
satisfies the following regret upper bound,
[Equation]
for any u ∈ X .
Based on the preceding lemma, we can derive the regret upper bound for Hedge algorithm with a self-confident learning rate.
Lemma 10.
Consider the prediction with expert advice setting with N experts and the linear loss f t ( x ) = ⟨ ℓ t , x ⟩ , where ℓ t ∈ R d .
Then the self-confident tuning Hedge, whose initial decision is p 1 = 1 /N · 1 and update rules are
[Equation]
ε t +1 =
[Equation]
ensures the following regret guarantee: for any i ∈ [ N ]
T (cid:88)
[Equation]
Proof.
It is easy to verify that, the mentioned self-confident tuning Hedge can be treated as a special case of the time-varying FTRL algorithm by choosing ψ ( p ) =
(cid:80)
Ns =
p s ln p s , which is 1 -strongly convex with respect to ∥ · ∥ 1 , and ψ t ( p )
= 1 ε t ψ ( p ) .
Thus, by Lemma 9, we have
T (cid:88)
[Equation] [Equation]
where the first inequality chooses u as the one-hot vector with all entries being 0 except the i -th one as 1 , and second inequality is by Lemma 15.
D.3 Facts on Geometric Covers
Lemma 11 (Lemma 11 of Zhang et al.
Let [ s p , s q ] ⊆
[ T ] be an arbitrary interval that starts from a marker s p and ends at another marker s q .
Then, we can find a sequence of consecutive intervals
[Equation]
such that
[Equation]
D.4 Technical Lemmas
In this section, we will present several technical lemmas used in our proof.
Lemma 12 (Lemma 3.1 of Srebro et al.
For an L -smooth and nonnegative function f :
[Equation]
Lemma 13 (Lemma 3.5 of Auer et al.
Let l 1 , . . .
, l T be non-negative real numbers.
Then:
[Equation]
Lemma 14 (Lemma 14 of Gaillard et al.
Let a 0 > 0 and a 1 , . .
. , a m ∈ [0 , 1] be real numbers and let f : (0 , + ∞ ) (cid:55)→ [0 , + ∞ ) be a non-increasing function.
Then
[Equation]
u
[Equation]
du
[Equation]
Lemma 15 (Lemma 4.8 of Pogodin and Lattimore [2019]) .
Let a 1 , a 2 , . .
. , a T
[Equation]
Lemma 16 (Lemma 5 of Shalev-Shwartz [2007]) .
For any x, y, a ∈ R + that satisfies x − y ≤ √ ax ,
[Equation]
Based on Lemma 16, we can achieve the following result.
Lemma 17.
For any x, y, a, b ∈ R + that satisfies x − y ≤ √ ax + b ,
[Equation]
Lemma 18 (Lemma 13 of Gaillard et al.
For all x > 0 and all α ≥ 1 , we have
[Equation]