Adaptive Oracle-Efﬁcient Online Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The classical algorithms for online learning and decision-making have the beneﬁtof achieving the optimal performance guarantees, but suffer from computationalcomplexity limitations when implemented at scale.
More recent sophisticated techniques, which we refer to as oracle-efﬁcient methods, address this problem by dispatching to an ofﬂine optimization oracle that can search through an exponentiallylarge (or even inﬁnite) space of decisions and select that which performed thebest on any dataset.
But despite the beneﬁts of computational feasibility, mostoracle-efﬁcient algorithms exhibit one major limitation: while performing well inworst-case settings, they do not adapt well to friendly environments.
In this paperwe consider two such friendly scenarios, (a) “small-loss” problems and (b) IID data.
We provide a new framework for designing follow-the-perturbed-leader algorithmsthat are oracle-efﬁcient and adapt well to the small-loss environment, under aparticular condition which we call approximability (which is spiritually relatedto sufﬁcient conditions provided in (Dudík et al., 2020)).
We identify a series ofreal-world settings, including online auctions and transductive online classiﬁcationproblems, for which approximability holds.
We also extend the algorithm to an IID data setting and establish a “best-of-both-worlds” bound in the oracle-efﬁcientsetting.
Introduction
Online learning is a fundamental paradigm for modeling sequential decision making problems (CesaBianchi & Lugosi, 2006; Shalev-Shwartz, 2011; Hazan, 2016).
Online learning is usually formulatedas a zero-sum game between a learner and an adversary.
In each round t = 1 , . . .
, T , the learnerﬁrst picks an action x t from a (ﬁnite) set X
= { x (1) , . . .
, x ( K ) } with cardinality equal to K .
Inthe meantime, an adversary reveals its action y t 2 Y .
As a consequence, the learner observes 24 y t , and suffers a loss f ( x t , y t ) , where f : X ⇥ Y 7!
The goal is to minimize the regret ,which is deﬁned as the difference between the cumulative loss of the learner
P Tt =1 f ( x t , y t ) , and thecumulative loss of the best action in hindsight L ⇤ T = min x 2 X
P
Tt =1 f ( x, y t ) .
A wide variety of algorithms have been proposed for the goal of minimizing worst-case regret (withoutany consideration of computational complexity per iteration); see (Cesa-Bianchi & Lugosi, 2006; -Shwartz, 2011; Hazan, 2016) for representative surveys of this literature.
These algorithmsall obtain a worst-case regret bound of the order O ( p T log K ) , which is known to be minimaxoptimal (Cesa-Bianchi & Lugosi, 2006).
Over the last two decades, sophisticated adaptive algorithmshave been designed that additionally enjoy problem-dependent performance guarantees, which canautomatically lead to better results in friendly environments.
One of the most important example forthis kind of guarantees is the so-called “small-loss” bound (Hutter & Poland, 2005; Cesa-Bianchi& Lugosi, 2006; Van Erven et al., 2014).
Such a bound depends on the best cumulative loss inhindsight (i.e. L ⇤ T ) instead of the total number of rounds ( T ).
Thus, this bound is much tighter thanthe worst-case bound, especially when the best decision performs well in the sense of incurring a very small loss.
Another example is the “best-of-both-worlds” bound (Van Erven et al., 2014), whichresults in an even tighter regret bound for independent and identically distributed (IID) loss functions.
However, all of these algorithms applied out-of-the-box suffer a linear dependence on the numberof decisions K .
This is prohibitively expensive, especially in problems such as network routing(Awerbuch & Kleinberg, 2008) and combinatorial market design (Cesa-Bianchi et al., 2014), wherethe cardinality of the decision set grows exponentially with the natural expression of the problem.
Although several efﬁcient algorithms do exist when the loss functions have certain special structure,such as linearity (Kalai & Vempala, 2005), or convexity (Zinkevich, 2003), these results can not beeasily extended to the more general settings of interest.
Notice that the efﬁciency of the above specialized methods is usually made possible by assumingthat the corresponding ofﬂine optimization problem (i.e., minimizing the (averaged) loss) can besolved efﬁciently.
This observation motivates the oracle-efﬁcient online learning problem (Hazan& Koren, 2016).
In this setting, the learner has access to a black-box ofﬂine oracle, which, givena real-weighted dataset S =
{ ( w ( j ) , y ( j ) ) } nj =1 , can efﬁciently return the solution to the followingproblem:
[Equation]
The goal is to design oracle-efﬁcient algorithms which can query the ofﬂine-oracle O (1) times eachround.
Concrete examples of such an oracle include algorithms for empirical risk minimization(Bishop, 2007), data-driven market design (Nisan & Ronen, 2007), and dynamic programming(Bertsekas, 2019).
As pointed out by Hazan & Koren (2016), the design of oracle-efﬁcient algorithms is extremelychallenging and such an algorithm does not exist in the worst case.
Nevertheless, recent work(Daskalakis & Syrgkanis, 2016; Syrgkanis et al., 2016; Dudík et al., 2020) has introduced a seriesof algorithms which are oracle-efﬁcient when certain sufﬁcient conditions are met.
Among them,the state-of-the-art method is the generalized-follow-the-perturbed-leader algorithm (GFTPL, Dudíket al., 2020), which is a variant of the classical follow-the-perturbed-leader (FTPL) algorithm (Kalai& Vempala, 2005).
Similar to FTPL, GFTPL perturbs the cumulative loss of each decision byadding a random variable, and chooses the decision with the smallest perturbed loss as x t .
However,the vanilla FTPL perturbs each decision independently, which requires to generate K independentrandom variables in total.
Moreover, the oracle in (1) can not be applied here since the oracle can nothandle the perturbation term.
To address these limitations, GFTPL only generates a low-dimensionalnoise vector in the beginning, and constructs K dependent perturbations based on the multiplicationbetween the noise vector and a perturbation translation matrix (PTM).
In this way, the computationalcomplexity for the noise generation is largely reduced.
On the other hand, oracle-efﬁciency can beachieved by setting the elements in the PTM as carefully designed synthetic losses.
Dudík et al.(2020) show that a worst-case optimal regret bound can be obtained when the PTM is admissible ,i.e., every two rows are signiﬁcantly distinct.
This serves as a sufﬁcient condition for achievingoracle-efﬁciency.
While these results form a solid foundation for general worst-case oracle-efﬁcient online learning, itremains unclear whether problem-dependent , or data-adaptive bounds are achievable in conjunctionwith oracle-efﬁciency.
In other words, the design of a generally applicable oracle-efﬁcient andadaptive online learning algorithm has remained open.
In this paper, we provide an afﬁrmative answerto this problem and make the following contributions.
• We propose a variant of the GFTPL algorithm (Dudík et al., 2020), and derive a newsufﬁcient condition for ensuring oracle-efﬁciency while achieving the small-loss bound.
Our
key observation is that while the admissibility condition of the PTM in GFTPL successfully
stabilizes the algorithm (by ensuring that P [ x t 6 = x t +1 ] is small), it is too weak to adapt to a
small loss.
We address this challenge via a new condition for PTM, called approximability.
Critically, this condition ensures a stronger stability measure, i.e., the ratio of P [ x t =
x ( i ) ] and P [ x t +1 =
x ( i ) ] is upper-bounded by a universal constant for any i {+
+}
[ K ] .
We obtain the small-loss bound by equipping GFTPL with an approximable PTM, a data-
dependent step-size and Laplace distribution for the perturbation noise.
Our new condition
of approximability
is simple and interpretable, and can be easily veriﬁed for an arbitraryPTM.
It shares both similarities and differences from the admissibility condition proposed in Dudík et al.
We demonstrate this through several examples where one of thesufﬁcient conditions holds, but not the other.
• We identiﬁed a series of real-world applications where approximable PTMs exist, for
example: (a) a series of online auctions problems (Dudík et al., 2020); (b) problems witha small adversary’s action space |Y| (Daskalakis & Syrgkanis, 2016); and (c) transductiveonline classiﬁcation (Syrgkanis et al., 2016; Dudík et al., 2020).
This is the ﬁrst-time thatthe small-loss bound is obtained in these applications.
To achieve this, we introduce novelPTMs and analysis for showing the approximability condition on these PTMs.
• We achieve the “best-of-both-worlds” bound, which enjoys even tighter results when thedata is IID or the number of leader changes is small.
The main idea is to combine
our
proposed algorithm with vanilla FTL leveraging ideas from a meta-algorithm introducedin Van Erven et al.
Related Work
Our work contributes to two classes of problems: oracle-efﬁcient online learning and adaptive onlinelearning.
In this section, we brieﬂy review the related work in these areas.
2.1 Oracle-efﬁcient online learning
For oracle-efﬁcient online learning, the pioneer work of Hazan & Koren (2016) points out thatoracle-efﬁcient methods do not exist when dealing with general hostile adversaries, which impliesthat additional assumptions on the problem structure have to be made.
Daskalakis & Syrgkanis(2016) consider the setting in which the cardinality of the adversary’s action set Y is ﬁnite and small,and propose to add a series of “fake” losses to the learning history based on random samples from
Y .
They prove that for this setting an O ( |Y| p T ) bound can be obtained.
Syrgkanis et al.
(2016)study the contextual combinatorial online learning problem, where each action is associated witha binary vector.
They also assume the loss function set contains all linear functions as a sub-class.
For this problem, Syrgkanis et al.
(2016) construct a set of synthetic losses for perturbation basedon randomly-selected contexts, and achieve worst-case optimal bounds when all the contextualinformation can be obtained beforehand, or when there exists a small set of contexts that can telleach decision apart.
Dudík et al.
(2020) focus on the general non-contextual setting, and proposethe generalized FTPL algorithm.
This algorithm generates a small number of random variables atthe beginning, and then perturbs the learning history via the linear combinations of these randomvariables.
The algorithm can be implemented efﬁciently by setting the combination parameters ascarefully designed loss values.
Niazadeh et al.
(2021) consider a more complicated combinatorialsetting where the ofﬂine problem is NP-hard, but a robust approximation oracle exists.
For thiscase, they propose an online algorithm based on the approximation oracle, and prove that it has lowapproximate regret, which is a measure weaker than regret.
Note that the aforementioned methodssuffer from the worst-case bounds, and cannot beneﬁt from nice structures in data.
Recently, severalconcurrent work (Block et al., 2022; Haghtalab et al., 2022a) investigate how to obtain tighter boundsoracle-efﬁciently when the distribution of data is close to the uniform distribution (Rakhlin et al.
,2011; Haghtalab et al., 2022b).
However, their results do not improve the dependence on T .
In this paper, we mainly focus on the so-called the learning with expert advice setting (Cesa-Bianchi& Lugosi, 2006), where the action set is discrete, and the loss can be highly non-convex.
On the otherhand, efﬁcient algorithms can be obtained when the loss functions have certain properties, such aslinearity (Kalai & Vempala, 2005; Hutter & Poland, 2005; Awerbuch & Kleinberg, 2008), convexity(Zinkevich, 2003; Hazan et al., 2007) or submodularity (Hazan & Kale, 2012).
Finally, we note that,in this paper we mainly focus on the full-information setting, where the learner can observe the wholeloss function after the action is submitted.
Meanwhile, online learning with oracle has also beenwidely studied in the bandit setting (Langford & Zhang, 2008; Dudik et al., 2011; Agarwal et al.
,2014; Foster et al., 2018; Foster & Rakhlin, 2020).
2.2 Adaptive online learning
In this paper, we focus on designing oracle-efﬁcient algorithms with problem-dependent regretguarantees.
Note that this kind of bound can be achieved by many inefﬁcient algorithms in general, such as Hedge and its variants (Cesa-Bianchi & Lugosi, 2006; De Rooij et al., 2014; Luo & Schapire,2015), and follow-the-perturbed-leader (Kalai & Vempala, 2005; Van Erven et al., 2014) or followthe-regularized-leader (Orabona, 2019) type of methods.
Small-loss can also be obtained efﬁcientlywhen the loss functions are simply linear (Hutter & Poland, 2005; Syrgkanis et al., 2016).
On theother hand, in online convex optimization, i.e., when the decision set and the loss functions areconvex, small-loss bound can be obtained when the loss functions are smooth (Srebro et al., 2010;
Orabona et al., 2012; Wang et al., 2020).
However, these algorithms heavily rely on the specialstructure of the loss functions.
In this paper, we take the ﬁrst step to extend these methods to supportthe complicated and non-convex problems which appear in real-world applications.
Apart from the small-loss, there exist other types of problem-dependent bounds, such as secondorder bound (Cesa-Bianchi et al., 2005; Gaillard et al., 2014), or quantile bound (Chaudhuri et al.
,2009; Koolen & Erven, 2015), but how to obtain these more reﬁned theoretical guarantees in theoracle-efﬁcient setting remains an open problem.
GFTPL with Small-Loss Bound
In this section, forgoing computational complexity, we provide a new FTPL type of algorithmthat enjoys the small-loss bound.
We then show that the proposed algorithm can be implementedefﬁciently by the ofﬂine oracle in Section 4.
Before diving into the details, we ﬁrst brieﬂy recall thedeﬁnition of online learning and regret.
Preliminaries.
The online decision problem we consider can be described as follows.
In each round t , a learner picks an action x t 2 X =
[ x (1) , . . .
, x ( K ) ] .
After observing the adversary’s decision y t 2 Y , the learner suffers a loss f ( x t , y t ) where the loss function f : X ⇥ Y 7! [0 , 1] is known.
The regret is deﬁned as
[Equation]
where L ⇤ T = min k 2 [ K ]
P
Tj =1 f ( x ( k ) , y j ) is the cumulative loss of the best action in hindsight, and theexpectation is taken with respect to the potentially randomized strategy of the learner.
Our proposed= algorithm follows the framework of GFTPL (Dudík et al., 2020).
To help understanding, we ﬁrst give a brief introduction to the intuition behind this method.
Speciﬁcally, in each round t , GFTPL picks x t by solving the following optimization problem:
[Equation]
where ↵ is a N -dimensional noise vector ( N ⌧ K ) generated from a uniform distribution, and  ( k )is the k -th row of a matrix  2 [0 , 1] K ⇥ N , which is referred to as the perturbation translation matrix(PTM).
Compared to vanilla FTPL, GFTPL only generates a small number ( N ) of random variables,and each expert is perturbed by a different linear combination of these random variables based on  .
The results of Dudík et al.
(2020) rely on the following assumption on  .
Deﬁnition 1.
(  -admissibility (Dudík et al., 2020) ) Let  2 [0 , 1] K ⇥ N be a matrix, and denote  ( k )as the k -th row of  , and  ( k,i ) the i -th element of  ( k ) .
Then,  is  -admissible if (a) 8 k, k 0 2 [ K ] ,9 i 2 [ N ] , such that  ( k,i ) {+
+}
=  ( k 0 ,i ) ; and (b) 8 i 2 [ N ] , k, k 0 2 [ K ] , such that  ( k,i ) 6 =
( k 0 ,i ) , then|  ( k,i )   ( k 0 ,i )
|   .
The  -admissibility guarantees that every two rows in  are signiﬁcantly distinct.
As pointed out by Dudík et al.
(2020), this is the essential property required by GFTPL, and is used to stabilize thealgorithm in the analysis, i.e., ensuring that P [ x t 6 = x t +1 ] is small.
However, motivated by adaptive
FTPL in the inefﬁcient setting (Hutter & Poland, 2005), we observe that the small-loss bound can notrest on this notion of stability, instead one needs to control the following 8 t and 8 i 2 [ K ] ,
[Equation]
the ratio of the probability of picking the i -th decision in two consecutive rounds.
We note that  -admissibility is not strong enough to ensure this quantity is bounded, as we establish in the followinglemma.
(See Appendix A.1 for proof).
Lemma 1.
There is an instance of a  -admissible  , and a sequence { y t : t = 1 , 2 , . . .
} , such that ifwe run GFTPL we can have P [ x t = x ( i ) ]
P [ x t +1 = x ( i ) ]
= 1 for some
i 2 [ K ] and some t > 0 .
To tackle this problem, we propose a new property for  .
Let B 1  := { s 2 R N : k s k 1   } .
Deﬁnition 2.
(  -approximability) Let  2 [0 , 1] K ⇥ N .
We say that  is  -approximable if
[Equation]
It may not be immediately obvious how we arrived at this condition, so let us provide some intuition.
The goal of perturbation methods in sequential decision problems, going back to the early work of
Hannan (1957), is to ensure that the algorithm is “hedging” across all available alternative decisions.
A new observed data point y may make expert j suddenly look more attractive than expert k , as wehave now introduced a new gap f
( x ( k ) , y )  f ( x ( j ) , y ) in their measured loss values.
With this inmind, we say that  is a “good” (i.e. approximable) choice for the PTM, if this gap can be overcome(hedged) by some small (i.e. likely) perturbation s , so that ⌦ [Equation] k
[Equation]
[Equation] j ) [Equation] s ↵ makes up the difference.
Later, we will show that  -approximability guarantees the required stability measure in (2), and thusis critical for the small-loss bound.
We want to emphasize two ﬁnal points.
First, the  -approximability condition is purely for analysispurposes and we need not compute the s in response to y and k .
Second, much of the computationaland decision-theoretic challenges rest heavily on the careful design of  .
The PTM allows thealgorithm to perform the appropriate hedging across an exponentially-sized set of experts with onlyan N ⌧ K dimensions of perturbation.
As we demonstrate in the following example, we can alwaysconstruct a  -approximable  , with N =
O (log K ) , but at the expense of computational efﬁciency.
The proposed  will not generally be compatible with the given oracle.
Simple Example
For any online learning problem we may construct  as follows.
Let N := d log 2 K e , and deﬁne the k th row  ( k ) to be the binary representation of the index k , with +1 /  1values instead of 0 / 1 .
We claim that this  is  -approximable, for  = d log 2 K e .
We can satisfythe condition of Deﬁnition 2, by setting s =
( k ) .
It is easy to see that for any j {+
+} =
k we have
[Equation]
x
[Equation]
j ) [Equation] y ) , where the last inequalityholds because | f ( x ( i ) , y ) |  1 for any i 2 [ K ] .
Comparison We note that, although  -approximability leads to a much tighter bound, it is notstronger than  -admissibility.
Instead, they are incomparable conditions.
Speciﬁcally:
• In Section 4.1 we demonstrate that when  is binary, admissibility leads to approximability.
As shown by Dudík et al.
(2020), a binary and admissible  exists in various online auctions problems, including VCG with bidder-speciﬁc reserves (Roughgarden & Wang, 2019),
envy-free item pricing (Guruswami et al., 2005), online welfare maximization in multi-unit
auction (Dobzinski & Nisan, 2010), and simultaneous second-price auctions (Daskalakis &
Syrgkanis, 2016).
We can directly obtain an approximable  in such cases.
• On the other hand, in problems such as level auction (Dudík et al., 2020), one can construct
both admissible and approximable  , although in different ways; we discuss in Section 4.1.
• In section 4.2, we show that, when the adversary’s action space is small, we can alwaysconstruct a  -approximable  , while  -admissible  in general does not exist.
• In Appendix A.2, we show that in some cases a  -admissible  can be obtained while -approximability cannot be achieved.
Equipped with the  -approximable PTM, we develop a generalized follow-the-perturbed-leaderalgorithm with the Laplace distribution 1 and a time-varying step size, which is summarized in Algorithm 1.
For the proposed algorithm, we successfully obtain the following stronger stabilityproperty.
1: Init:  2 [0 , 1] K ⇥ N 2: Draw IID vector ↵ =
[ ↵ (1) , . . .
, ↵ ( N ) ] ⇠ Lap (1) N ; that is, p ( ↵ ( i ) )
= 12 exp(  | ↵ ( i ) | ) 3: for t = 1 , . . .
, T do 4:
Set ↵ t ↵⌘ t , where ⌘ t > 0 a parameter computed online 5:
Choose x t argmin k {+
+} [ K ] t  X j = f ( x ( k ) , y j ) +
D  ( k ) , ↵ t
E 6: Observe y t 7: end for
Lemma 2.
Assume  is  -approximable.
Let x 0 t = argmin k
[ K ]
[Equation]
Then in each round t , we have 8 i 2 [ K ] ,
[Equation]
Note that we replace the term x t +1 in (2) with x 0 t , as a time-varying step-size is used.
Based on
Lemma 2, we obtain the regret bound of Algorithm 1 as follows.
Theorem 1.
Assume  is  -approximable, and let L ⇤ T = min
k 2 [ K ]
P
Tj =1 f
( x ( k ) , y j ) .
Algorithm 1,with ⌘ t = min ⇢ 1  , c p L ⇤ t  1 +1  for any c > 0 , achieves the following regret bound:
[Equation]
The proof of Lemma 2 and Theorem 1 can be found in Appendix A.3.
By setting c = ⇥ (1) , Theorem1 implies that our proposed algorithm achieves O (max {  , ln K, p N ln K } p
L ⇤ T ) regret bound.
Comparison to GFTPL (Dudík et al., 2020)
The original GFTPL algorithm has an O ( N  p T )regret bound.
For the dependence on T , our O ( p
L ⇤ T ) bound reduces to O ( p T ) in the worst-case,and automatically becomes tighter when L ⇤ T is small.
On the other hand, for the dependence on otherterms, we note that both N  and max {  , ln K, p N ln K } are lower bounded by ⌦ (ln K ) , and theirexact relationship depends on the speciﬁc problem.
In Section 4, we show that for many auctionapplications, the two terms are on the same order.
Moreover, in cases such as when |Y| is small,
Algorithm 1 with an appropriate c leads to O
[Equation]
regret bound, whilethe regret bound of GFTPL in Dudík et al.
(2020) can blow up since  can be inﬁnitely small.
Oracle-efﬁciency and Applications
In this section, we discuss how to run Algorithm 1 in an oracle-efﬁcient way.
Following Dudík et al.(2020), we introduce the following deﬁnition.
Deﬁnition 3 (Implementability) .
A matrix  is implementable with complexity M if for each j 2 [ N ] there exists a dataset S j , with | S j |  M , such that 8 k, k {+0 
+}
[ K ] ,
[Equation]
Based on Deﬁnition 3, it is easy to get the following theorem, which j =
N
X i = ↵ ( i ) ⌘ t4 X ( w,y ) 2 S i w · r ( x ( k ) , y )
1 p b
L ⇤ t +
Theorem 2.
If  is implementable, then Algorithm 1 is oracle-efﬁcient and has a per-round complexity
O ( T + NM ) .
In the following, we discuss how to construct approximable and implementable  matrices in differentapplications.
4.1 Applications in online auctions
In this part, we apply Algorithm 1 to online auction problems, which is the main focus of Dudík et al.(2020).
To deal with this sort of problems, we ﬁrst transform Algorithm 1 to online learning withrewards setting, i.e., in each round t , after choosing x t , instead of suffering a loss, the learner obtainsa reward r ( x t , y t ) 2 [0 , 1] .
For this case, it is straightforward to see that running Algorithm 1 on asurrogate loss f
( x, y ) = 1  r ( x, y ) directly leads to the small-loss bound.
To proceed, we slightlychange this procedure and obtain Algorithm 2.
The main difference is that, we implement  with thereward function r ( x, y ) , instead of the surrogate loss f ( x, y ) .
This makes the construction of  mucheasier .
We have the following regret bound for Algorithm 2.
Corollary 1.
Let f ( x, y )
= 1  r ( x, y ) .
Assume  is  -approximable w.r.t.
f ( x, y ) and implementable with function r ( x, y ) .
Then Algorithm 2 is oracle-efﬁcient and achieves the following regret bound:
[Equation]
where L ⇤ T = max i {+
+}
[ K ] P
Tt =1 r ( x ( i ) , y t ) is the cumulative reward of the best expert.
Next, we discuss how to construct the PTM in several auction problems.
Auctions with binary and admissible  .
As shown by Dudík et al.
(2020), in many online auctionproblems, such as VCG with bidder-speciﬁc reserves (Roughgarden & Wang, 2019), envy-free itempricing (Guruswami et al., 2005), online welfare maximization in multi-unit auction (Dobzinski & Nisan, 2010) and simultaneous second-price auctions (Daskalakis & Syrgkanis, 2016), there exists abinary PTM which is 1 -admissible and implementable with N rows where N ⌧ K .
For these cases,we have the following lemma.
The proof is deferred to Appendix B.1.
Lemma 3.
Let  2 [0 , 1] K ⇥ N be a binary matrix and 1 -admissible, then  is N -approximable.
Combining Lemma 3 with Corollary 1, and noticing the fact the N = ⌦ (ln
K ) , we can obtainan O ( N p
T  L ⇤ T ) bound for all of the above problems.
Compared to the original GFTPL, ourcondition leads to a similar dependence on N and a tighter dependence on T .
More details about theaforementioned auction problems and corresponding regret bounds can be found Appendix B is capable to get a (
✏ ) approximation compared with Myerson’s optimal auction the distribution of each valuation is independent from others
For this problem PTM Dudík al
(2020) does not lead to approximability.
To address this problem, we propose a novel way ofconstructing an approximable and implementable PTM.
The key idea is to utilize a coordinate-wisethrehold function to implement  .
Note that this kind of function can not be directly obtained.
Instead,we create an augmented problem with a surrogate loss to deal with this issue.
For level auction withsingle-item, n -bidders, s -level and m -discretization level, our method enjoys an O ( nsm p
T  L ⇤ T )regret bound, which is tighter than the O ( nm 2 p T ) (note that s  m ) of the original GFTPL.
Due tothe page limitation, we postpone the detailed problem description and proof to Appendix B.3.
4.2 Other applications
Oracle learning and ﬁnite parameter space
In many real-world applications, such as securitygame (Balcan et al., 2015) and online bidding with ﬁnite threshold vectors (Daskalakis & Syrgkanis,2016), the decision set X is extremely large, while the adversary’s action set |Y| is ﬁnite small.
For 281 these problems, we can construct an implementable PTM based the following Lemma, where theproof can be found in Appendix B.4.
Lemma 4.
Consider the setting with |Y| =
d ( d ⌧ K ), then there exists a 1 -approximable and 284 implementable  with d columns and complexity 1.
Combining Lemma 4 and Theorem 1, and conﬁguring c = q max { ln K, p d ln K } , we observe thatour algorithm achieves O ( q max { ln K, p d ln K } p
L ⇤ T ) small-loss bound.
On the other hand, inthis kind of problems, because of the continuity of the loss functions, a  -admissible PTM in generaldoes not exist (as  may approach 0 without any discretization).
Therefore, our proposed conditionnot only leads to a tighter bound, but can also solve problems that the original GFTPL (Dudík et al.
,2020) can not handle.
Transductive online classiﬁcation Finally, we consider the transductive online classiﬁcationproblem (Syrgkanis et al., 2016; Dudík et al., 2020).
In this setting, X contains K binary classiﬁers.
In each round t , ﬁrstly the adversary picks a feature vector w t 2 W , where |W| = m .
Then, the 294 learner chooses a classiﬁer x t ( · ) from X .
After that, the adversary reveals the label y t 2 { 0 , 1 } , and 295 the learner suffers a loss f ( x t , ( w t , y t ))
=
I [ x t ( w t ) 6 = y t ] .
We assume the problem is transductive,i.e., the learner has access to W at the beginning.
For this setting, we achieve the following results(the proof is in Appendix B.5).
Lemma 5.
Consider transductive online classiﬁcation with |W| = m .
Then there exists a 1- 299 approximable and implementable PTM with m columns and complexity 1.
Moreover, Algorithm 1with such a PTM and appropriately chosen parameters achieves O (
[Equation]
regret.
[Equation] Negative implementability In the this paper we assume that the ofﬂine oracle can solve theminimization problem in (1) given any real-weights.
In some cases, the oracle can only acceptpositive weights.
This problem can be solved by constructing negative implementable PTM (Dudíket al., 2020).
Note that in most cases discussed above negative implementable and approximable PTM exist, which is shown in Appendix B.6.
Best-of-Both-Worlds Bound
By setting all step sizes ⌘ t as 1 , Algorithm 1 reduces to the classical FTL algorithm, which sufferslinear regret in the adversarial setting and enjoys much tighter bounds when the data is IID or numberof leader changes is small.
To be more speciﬁc, we introduce the following regret bound for FTL.
Lemma 6 (Lemma 9, De Rooij et al.
Let x FTL t = argmin i {+
+}
[ K ]
P t  1 s =1 f ( x ( i ) , y s ) be theoutput of the FTL algorithm at round t , C T the set of rounds where the leader changes, and t
= f
( x FTL t , y t )  ( L ⇤ t  L ⇤ t  1 ) the “mixability gap” at round t .
Then for any T  1 , the regret of FTL is bounded by R FTL T  P
[Equation]
Note that since f 2 [0 , 1] and L ⇤ t  L ⇤ t  1 2 [0 , f ( x FTL t , y t )] , we know  t 2 [0 , 1] .
For the i.i.d case, ifthe mean loss of the best expert is smaller than that of other experts by a constant, then due to the law
Initialization: Alg 1 = FTL 1: for t = 1 , . .
, T do 2:
Get x t by Alg t , observe y t 3:
Compute b
U FTL t and b
U GFTPL t 4: if Alg t ==
FTL and b
U FTL t > ↵ b
U GFTPL t then 5:
Alg t +1 =
GFTPL 6: else if Alg t ==
GFTPL and b
U GFTPL t >  b
U FTL t then 7:
Alg t +1 =
FTL 8: end if 9:
Feed y t to Alg t +1 10: end for of large numbers, the number of leader changes would be small, which results in a constant regretbound (De Rooij et al., 2014).
Denote U GFTPL T as the attainable regret bound (as in Theorem 1) for running Algorithm 1 alone and
U FTL T =
P t 2 C T  t to be that of FTL.
In the following, we develop a new algorithm and prove that itis optimal in both worlds, that is, its regret is on the order of O (min { U FTL T , U GFTPL T } ) .
The proposed algorithm, named as oracle-efﬁcient ﬂipﬂop (OFF) algorithm, is summarized in Algorithm 3.
It is motivated by the Flipﬂop algorithm (De Rooij et al., 2014), and the core idea isto switch between FTL and GFTPL (Algorthm 1) based on the comparison of the estimated regret.
We optimistically start from FTL.
In each round t , we ﬁrstly pick x t based on the current algorithm
Alg t , and then obtain the adversary’s action y t (line 2).
Next, we compute the estimated bounds ofregret of both algorithms until round t (line 3).
Speciﬁcally, let I FTL t
= { i | i 2 [ t ] , Alg i = FTL } and
I GFTPL t
= { i | i 2 [ t ] , Alg i = GFTPL } be the set of rounds up to t in which we run FTL and GFTPL.
Then, the estimated regret of FTL in I FTL t is given by b
U FTL t
[Equation]
i 2 I FTL t  i , and the estimated regretof GFTPL in I GFTPL t can be bounded via Theorem 1:
[Equation]
where b
[Equation]
I GFTPL
[Equation]
f ( x, y i ) and we set c = 1 .
Note that, the two quantities deﬁnedabove are the exact regret upper bounds of the two algorithms on their sub-time intervals up to roundt , due to the fact that the regret bounds provided in Lemma 6 and Theorem 1 are timeless .
Moreover,note that the two values can be computed by the oracle.
We compare the estimated regret of bothalgorithms, and use the algorithm which performs better for the next round (lines 4-8).
For the proposed algorithm, we have the following theoretical guarantee (the proof can be found in
Appendix C).
Theorem 3.
Assume we have a  -approximable  , then Algorithm 3 is able to achieve the following bound:
[Equation]
where ⌧ = 4 p 2 max { 2 ln K, p N ln K } + 12  and ↵ =  = 1 .
Conclusion
In this paper, we propose a variant of the generalized follow-the-perturbed-leader algorithm andderive a sufﬁcient condition under which the small-loss bound can be attained in the oracle-efﬁcientsetting.
We also show the condition is satisﬁed in various applications.
Finally, we extend thealgorithm to adapt to IID losses and achieve a “best-of-both-worlds” bound.
In the future, we wouldlike to investigate how to achieve tighter results for oracle-efﬁcient setting, such as the second-orderbound (De Rooij et al., 2014).