Minimax Regret for Cascading Bandits
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Cascading bandits is a natural and popular model that frames the task of learning torank from Bernoulli click feedback in a bandit setting.
For the case of unstructuredrewards, we prove matching upper and lower bounds for the problem-independent(i.e.
, gap-free) regret, both of which strictly improve the best known.
A keyobservation is that the hard instances of this problem are those with small meanrewards, i.e., the small click-through rates that are most relevant in practice.
Basedon this, and the fact that small mean implies small variance for Bernoullis, ourkey technical result shows that variance-aware conﬁdence sets derived from the
Bernstein and Chernoff bounds lead to optimal algorithms (up to log terms),whereas Hoeffding-based algorithms suffer order-wise suboptimal regret.
Thissharply contrasts with the standard (non-cascading) bandit setting, where thevariance-aware algorithms only improve constants.
In light of this and as anadditional contribution, we propose a variance-aware algorithm for the structuredcase of linear rewards and show its regret strictly improves the state-of-the-art.
Introduction
The cascading click model describes users interacting with ranked lists, such as search results oronline advertisements [Craswell et al., 2008].
In this model, there is a set of items [ L ] = { 1 , . . .
, L } .
The user is given a list of K items
A = ( a 1 , . . .
, a K ) , sequentially examines the list, and clicks onthe ﬁrst attractive item (if any).
If a click occurs, the user leaves without examining the subsequentitems.
It is assumed that the e -th item has an attraction probability ¯ w ( e ) (which we also call the meanreward), and that the random (Bernoulli) clicks are conditionally independent given A .
Cascading bandits, introduced concurrently by Kveton et al.
[2015a] and Combes et al.
[2015], are asequential learning version of the model where the mean rewards { ¯ w ( e ) }
Le =1 are initially unknown.
At each round t 2 [ n ] , the learner chooses an action, which is a list of items A t =
( a t 1 , . . . , a tK ) .
Asin the basic click model, the user scans the list and clicks on the ﬁrst attractive item (if any).
Thus,if the C t -th item is clicked, the learner knows that the user was not attracted to { a tk } C t  1 k =1 but wasattracted to a t C t .
However, the learner receives no feedback on the items { a tk } Kk = C t +1 that the userdid not examine before leaving.
The objective for the learner is to choose the sequence { A t } nt =1 tomaximize the expected number of clicks, or equivalently, minimize the regret deﬁned in (1).
This work provides problem-independent (i.e., gap-free) regret bounds for cascading bandits thatstrictly improve the state-of-the-art.
In the case of unstructured rewards, our results provide the ﬁrstminimax-optimal regret bounds (up to log terms).
Our key insight is that, compared to the standardbandit problem, the reward variance plays an outsized role in the gap-free analysis.
In particular, weshow that for cascading bandits, the worst-case problem instances are those with low mean rewards –namely, ¯ w ( e )  1 /K , where the variance ¯ w ( e )(1  ¯ w ( e )) is also small.
We emphasize that theseworst-case instances are not pathological; rather, they model the low click-through rates that prevailin practice [ al., 2007].
Further, we argue that adapting to low variance is crucial to
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
cope with the worst-case instances .
Put differently, algorithms should be variance-aware , i
[Table]
e., moreexploitative when the variance is small.
We provide the intuition behind this key insight in Section1.2 and show how to formalize it with a proof sketch in Section 5.
More speciﬁcally, our ﬁrst goal is to establish upper bounds on the problem-independent regret (i [Table] e.,the maximum over ¯ w ) for cascading bandit algorithms, as well as minimax lower bounds (i.e., theinﬁmum over algorithms of their problem-independent regret).
As shown in Table 1, the tightestexisting such bounds are ˜ O
( p nLK ) and ⌦ ( p nL/K ) , respectively.
What is surprising is that, notonly is there a gap between the bounds, but they increase and decrease in K , respectively.
In otherwords, the following fundamental question is unresolved: as K grows, does the problem becomeharder (as suggested by the upper bound) or easier (as suggested by the lower bound) ?
As discussed by Zhong et al.
[2021], this question lacks an obvious answer.
On the one hand, larger
K means that the learner needs to identify more good items, which hints at a harder problem.
Onthe other hand, the learner receives more feedback as K grows, which intuitively makes the problemeasier.
As we show later, variance-aware algorithms are the key to resolving this tradeoff.
In addition to this basic version of the model – hereafter, the tabular case , where no structure isassumed for the mean rewards – we are also interested in the linear case , where ¯ w ( e ) =
h  ( e ) , ✓ i forsome known feature map  : [ L ] !
R d and unknown parameter vector ✓ 2 R d .
A second goal ofthis work is to apply our insights from the tabular case to the linear one, in hopes of improving thebest known upper bound ˜ O
( p ndK ) of Zong et al.
1.1 Main contributions
Tabular case.
First, we show no algorithm can achieve o ( p nL ) regret uniformly across ¯ w , assuming
L is large compared to K and n is large compared to L (see Theorem 1).
Next, we consider threealgorithms: CascadeKL-UCB , CascadeUCB-V , and CascadeUCB1 (the ﬁrst and third are due to
Kveton et al.
[2015a]; CascadeUCB-V is new).
All three rank the L items using upper conﬁdencebounds (UCBs) and choose A t as the K highest ranked items.
They differ in the choice of UCB.
As the names suggest, CascadeKL-UCB and CascadeUCB-V use KL-UCB [Garivier and Cappé, 2011,
Maillard et al., 2011, Cappé et al., 2013] and UCB-V
[Audibert et al., 2009], respectively.
Both arevariance-aware, in the sense that their respective UCBs are derived from the Chernoff and Bernsteininequalities.
We show that both algorithms have near-optimal regret ˜ O ( p nL ) for any ¯ w (see Theorem2).
In contrast, CascadeUCB1 relies on the Hoefﬁng-style UCB1 [Auer et al., 2002], and we show thatit suffers suboptimal regret ⌦ ( p nLK ) on some ¯ w (see Theorem 3).
In summary, we prove (i) the minimax regret is ˜ ⇥ ( p nL ) for cascading bandits, (ii) the variance-awarealgorithms CascadeKL-UCB and CascadeUCB-V are minimax-optimal up to log terms, and (iii) the variance-unaware algorithm CascadeUCB1 is decidely sub optimal.
Moreover, note from Table 1 thatwe strictly improve both the upper and lower bounds for this problem.
Discussion.
There are two surprising aspects to these results.
First, the minimax bound ˜ ⇥
( p nL )shows that (in a worst-case sense) the number of recommended items K plays no role.
In otherwords, the aforementioned tradeoff (identifying more good items but receiving more feedback) isperfectly balanced when the correct algorithm (e.g., CascadeKL-UCB ) is employed.
The second (and arguably more surprising) aspect is that CacadeKL-UCB and CascadeUCB-V areoptimal but CascadeUCB1 is not.
This stands in contrast to the standard L -armed bandit problem,where the analogous algorithms KL-UCB , UCB-V , and UCB1 all achieve the minimax regret ˜ ⇥ ( p nL ) ,and the main advantage of the former two is only to improve the constants in the gap-dependentbounds.
We discuss the intuition behind this contrast in Section 1.2.
Linear case.
Motivated by these ﬁndings, we also propose a variance-aware algorithm for the linearcase called CascadeWOFUL .
2 CascadeWOFUL proceeds in two steps.
First, we use the Hoeffdingstyle UCBs of Abbasi-Yadkori et al.
[2011] to upper bound the mean rewards ¯ w ( e ) , and thus the
Bernoulli variances ¯ w ( e )(1  ¯ w ( e )) , with high probability.
Second, we use these Hoeffding UCBsas proxies for the true variances in the WOFUL algorithm of Zhou et al.
[2021], which computes avariance-weighted estimate of ✓ that enjoys Bernstein-style concentration.
In Theorem 4, we showthe regret of CascadeWOFUL is ˜ O
( p nd ( d + K )) for large n , which improves existing bounds by afactor of at least p min { d, K } K (see Table 1).
The theorem also establishes ˜ o
( p ndK ) regret forsmall n , so our bound strictly improves the state-of-the-art in general.
1.2 Why do variance-aware algorithms succeed but variance-unaware algorithms fail?
To answer this question, we focus on the tabular case and contrast the standard L -armed bandit settingwith the cascading one.
We ﬁrst recall how the ˜ O ( p nL ) bound for L -armed bandits is derived.
Forsimplicity, we restrict to instances with ¯ w (1) =
p and ¯ w (2) = · · · = ¯ w ( L ) =
p   for somep 2 (0 , 1) and  2 (0 , p ) .
In this case, UCB1 plays each of the L  1 suboptimal items 1 /  2 times,up to constants and log factors [Auer et al., 2002].
Each such play costs regret  , for a total regretthat scales as L/  .
Alternatively, regret can simply be bounded by  n , since the n plays incur atmost  regret each.
Combining the bounds gives min { L/  ,  n } .
The worst case occurs when thetwo bounds are equal, i.e., when  = p
L/n , which implies p nL regret.
In contrast, UCB-V plays each suboptimal item  2 /  2 times (in an order sense), where  2  p isthe variance of the Bernoulli ( p   ) reward [Audibert et al., 2009].
Therefore, the argument of theprevious paragraph shows that regret grows as min { Lp/  ,  n } .
Here the worst case occurs when pis non-vanishing and  = p
L/n , which gives the same regret scaling as UCB1 .
A similar argumentholds for KL-UCB , because the number of plays grows as 1 /d ( p   , p ) [Cappé et al., 2013] and onecan show d ( p   , p ) =
⌦ (  2 /p )
(see Claim 1 in Appendix F).
The analysis is more complicated for cascading bandits, because regret is nonlinear in the meanrewards and the amount of feedback is random.
To oversimplify things, we draw an analogy withthe above and assume ¯ w (1) = · · · = ¯ w ( K ) =
p and ¯ w ( K + 1) = · · · = ¯ w ( L ) =
p   .
In thiscase, CascadeUCB1 similarly plays the L  K suboptimal items 1 /  2 times each, which costs L/ regret when L  K .
However, we can no longer bound regret by  n , because the total number of plays depends on the random number of items the user examines at each round, which is roughly min { 1 /p, K } (the mean of Geometric ( p ) random variable, truncated to the maximum K ).
Thus,the  n bound inﬂates to  n min { 1 /p, K } , which gives min { L/  ,  n min { 1 /p, K }} regret.
For p  1 /K and  = p
L/ ( nK ) , this yields the best known bound p nLK .
We emphasize that, unlike the previous paragraph, the worst case here occurs when p (i.e., the click-through rate) is small .
On the other hand, the analogous bound for CascadeUCB-V and CascadeKL-UCB scales as min { Lp/  ,  n min { 1 /p, K }} .
Crucially, the factor of p in the ﬁrst term – which arises due to the variance-aware nature of the algorithms – offsets the factor 1 /p in the second term.
Thus, in the hard case p  1 /K , the bound becomes min { L/ (  K ) ,  nK } .
Here the worst case is  = p
L/ ( nK 2 ) ,which yields the minimax regret p nL that we establish in Theorems 1 and 2.
Preliminaries
In this section, we precisely formulate our problem.
A cascading bandit instance is deﬁned by thetriple ( L, K, ¯ w ) , where L 2 N is the total number of items, K 2 [ L ] = { 1 , . . .
, L } is the numberof items the learner displays to the user at each round, and ¯ w 2 [0 , 1] L is the vector of attractionprobabilities.
We deﬁne a sequential game as follows.
At each round t 2 [ n ] , the learner chooses anaction A t =
( a t 1 , . . . , a tK ) , where a tk 2 [ L ] and a tk 6 = a tk 0 when k 6 = k 0 (i.e., A t is an ordered listof K distinct items).
The user sequentially examines this list, clicks on the ﬁrst item that attractsthem, and stops examining items after clicking.
Mathematically, we denote the ﬁrst attractive item by
C t = inf { k
[ K ] : w t ( a tk ) = 1 } , where w t ( e ) ⇠ Bernoulli ( ¯ w ( e )) for each e 2 [ L ] .
If no item isclicked, i.e., if w t ( a tk ) =
for all k , we set C t = 1 .
Note the learner only observes the realizationscorresponding to the items that the user examined, i.e., { w t ( a tk ) :
k 2 [min { C t , K } ] } .
We denote by H t =
[ t  1 s =
{ A s } [ { w s ( a sk ) : k 2 [min { C s , K } ] } the history of actions andobservations before time t .
We let P t ( · ) =
P ( ·|H t [ { A t } ) and E t [ · ] = E [ ·|H t [ { A t } ] denote 133 conditional probability and expectation given the history and current action.
In the cascading banditmodel, it is assumed that the feedback { w t ( a tk ) } Kk =1 (i.e., the presence or absence of clicks) isconditionally independent given H t and A t .
Therefore, the conditional click probability is
[Equation]
Mappings from H t to A t are called policies .
Let ⇧ be the set of all policies.
Given an instance¯ w 2 [0 , 1] L , a policy ⇡ 2 ⇧ , and a horizon n 2 N , the expected number of clicks is
[Equation]
where is the indicator function.
Let A ⇤ =
( a ⇤ 1 , . . . , a ⇤ K ) be any action A
= ( a 1 , . .
. , a K ) thatmaximizes the click probability 1
[Equation]
Our goal is to minimize regret , which is theexpected difference in the number of clicks between ⇡ and the policy that always plays A ⇤ , i.e.,
[Equation]
Results for the tabular case
We can now state our tabular results (the proofs are discussed in Section 5).
First, we have a minimaxlower bound showing no algorithm can achieve o ( p nL ) uniformly over the mean rewards ¯ w .
Theorem 1.
Suppose N , L/K 2 { 4 , 5 , . . . } and n  L .
Then for any policy ⇡ 2 ⇧ , there exists amean reward vector ¯ w 2 [0 , 1] L such that R ⇡ , ¯ w ( n )
= ⌦ ( p nL ) .
Remark 1.
The proof is essentially a reduction to Lattimore et al.
[2018]’s lower bound for theso-called document-based click model.
The assumptions are inherited from their proof approach.
Theprecise form of the assumptions differs but is not very comparable since the click models differ.
We next consider the algorithms CascadeKL-UCB and CascadeUCB1 from Kveton et al.
[2015a],along with a new one called CascadeUCB-V .
All follow a similar template, which is given in
Algorithm 1.
This is a natural generalization of upper conﬁdence bound (UCB) algorithms from thestandard L -armed bandit setting.
At each round t 2 [ n ] , it computes UCBs U t ( e ) in a manner to bespeciﬁed shortly, then chooses A t as the K items with the highest UCBs (in order of UCB).
Afterobserving the click feedback C t , the algorithm increments the number of observations T t ( e ) andupdates the empirical mean ˆ w T t ( e ) ( e ) for each item e that the user examined.
For CascadeKL-UCB , the UCBs U t ( e ) are computed as follows:
[Equation]
Initialize number of observations T t ( e ) = 0 and empirical mean ˆ w 0 ( e ) = 0 for each e 2 [ L ] for t = 1 , . .
. , n do
Compute U t ( e ) for each e 2 [ L ] (by (2), (3), and (4) for CascadeKL-UCB , CascadeUCB-V , and CascadeUCB1 , respectively) for k = 1 , . . .
K do
Let a tk = arg max e
[ L ] \{ a ti } k  1 i
=1 U t ( e ) be the item with the k -th highest UCB
Play A t = ( a t 1 , . .
. , a tK ) and observe C t = inf {
k 2 [ K ] : w t ( a tk ) = 1 } (where inf ; = 1 )
Let T t ( e ) =
T t  1 ( e ) for each e 2 [ L ] for k = 1 , . .
, min { C t , K } do e
[Equation]
e
[Equation]
where f ( t ) =
t (log t ) 3 and d ( p, q ) =
p log( p/q )+(1  p ) log((
p ) / (1  q )) is the relative entropybetween Bernoullis with means p, q 2 [0 , 1] .
The set in (2) is a conﬁdence interval for ¯ w ( e ) , which isderived from the Chernoff bound.
For CascadeUCB-V , the UCBs are instead given by
[Equation]
where ˆ v s ( e ) =
ˆ w s ( e )(1  ˆ w s ( e )) is the empirical variance from s observations of item e .
This UCBis derived from the coarser – but crucially, still variance-aware – Bernstein inequality.
Finally, for
CascadeUCB1 , the UCBs are derived from the Hoefﬁng bound and computed as follows:
[Equation]
where c t,s = p 1 .
[Equation]
log( t
[Equation]
We can now show the variance-aware UCBs are nearly optimal, while CascadeUCB1 is suboptimal.
Theorem 2.
Suppose ⇡ is CascadeKL-UCB or CascadeUCB-V , i.e., the policy from Algorithm 1 withthe UCBs given by (2) or (3) .
Then R ⇡ , ¯ w ( n ) =
˜ O ( p nL ) for any ¯ w 2 [0 , 1] L .
Remark 2.
The reader may wonder why we proposed CascadeUCB-V , since the CascadeKL-UCBbound is enough to establish the minimax regret.
The main reason is to demonstrate that varianceawareness alone (no additional information encoded by KL-UCB ) is enough to achieve the optimalregret, which helps motivate our linear algorithm.
Furthermore, we show empirically in Section 6that, while CascadeUCB-V is inferior to CascadeKL-UCB in terms of regret, its closed form natureleads to quicker computation, while still improving the regret of CascadeUCB1 .
Theorem 3.
Suppose n  max { LK, 49 K 4 } , L  800 K , and ⇡ is CascadeUCB1 , i.e., the policyfrom Algorithm 1 with the UCBs given by (4) .
Then R ⇡ , ¯ w ( n ) = ⌦ ( p nLK ) for some ¯ w 2 [0 , 1] L .
Remark 3.
The assumed bounds on n and L simplify the calculations and can be improved (see
Remark 8 in Appendix I).
The main point of Theorem 3 is to show that, unlike the variance-awarealgorithms, CascadeUCB1 cannot satisfy the conclusion of Theorem 2 for all choices of n and L .
Results for the linear case
In light of the previous section, we seek a variance-aware algorithm for the linear case.
Our methodis based on the WOFUL algorithm of Zhou et al.
[2021], which was designed for the standard linearbandit setting (the case K = 1 ).
In this section, we review WOFUL , discuss how to overcome itslimitations in the cascading setting, explain our CascadeWOFUL algorithm, and bound its regret.
Existing algorithm.
In Section 4.3 of their work, Zhou et al.
[2021] consider the following problem.
As above, there is a set of items [ L ] , a known feature map  : [ L ] !
R d , and an unknown parametervector ✓ 2 R d .
Successive plays of e 2 [ L ] give i.i.d.
rewards with mean ¯ w ( e ) =
h  ( e ) , ✓ i andvariance upper bounded by  2 e .
Thus, at each round t 2 [ n ] , the learner chooses a t 2 [ L ] and receivesa random reward r t = h  ( a t ) , ✓ i + ⌘ t , where E [ ⌘ t | a t ] = 0 and E [ ⌘ 2 t | a t ]   2 a t .
For this setting,the authors proposed the WOFUL algorithm, which is based on the (unweighted) OFUL algorithm[Abbasi-Yadkori et al., 2011].
At each round t 2 [ n ] , WOFUL chooses the item
[Equation]
where ↵ > 0 is an exploration parameter, k x k B = p x T Bx is the norm induced by a positive deﬁnitematrix B , and ˆ ✓ t is the regularized and variance-weighted least-squares estimate given by
[Equation]
[Equation]
where ⇤ t = I +
[Equation]
[Equation]
To gain some intuition, we assume momentarily that d = L and  ( e ) is the e -th standard basis vector,i.e., the vector with 1 in the e -th coordinate and 0 elsewhere.
In this case, one can easily calculate
[Equation]
Therefore, for large T t  1 ( e ) (large enough that 1 + T t  1 ( e ) /  2 e ⇡ T t  1 ( e ) /  2 e ), we have
[Equation]
Similar to UCB-V (3), the right side of this equation is the empirical mean plus an exploration bonusthat grows with the variance upper bound  2 e and decays in T t  1 ( e ) .
Hence, the term inside thearg max in (5) can be interpreted as a Bernstein-style UCB.
The analysis of this UCB relies on anovel concentration inequality for vector-valued martingales
[Zhou et al., 2021, Theorem 2], which isa Bernstein analogue of the Hoeffding-style bound due to Abbasi-Yadkori et al.
Limitations.
The fact that WOFUL (roughly) generalizes UCB-V is promising.
However, computing (6)requires knowledge of the variance upper bounds  2 e , and nontrivial bounds on the variance are rarelyavailable in practice.
We sidestep this issue with three simple observations: (i) cascading banditsonly involve Bernoulli rewards, (ii) for Bernoulli rewards, variances are upper bounded by means,and (iii) these means can be learned efﬁciently since they are linearly-parameterized.
This suggeststhe following algorithm: ﬁrst, compute Hoeffding-style UCBs; second, treat these UCBs as upperbounds for the true means, and thus the true variances, in WOFUL .
Proposed algorithm.
Algorithm 2 formalizes this approach.
It contains three steps.
First, step 1deﬁnes Hoeffding-style UCBs U t,H as in Abbasi-Yadkori et al.
Next, step 2 uses U t,H as anupper bound for the variance and computes the Bernstein-style UCBs U t,B as in WOFUL .
Finally, step3 chooses A t as the K items with the highest U t,B , analogous to Algorithm 1.
Two technical clariﬁcations are in order.
First, observe that in step 1, we clip the variance bound U t,Hbelow by 1 /K .
We do so to ensure that ⇤ t,B (which inverts U t,H ) remains bounded.
Additionally,we note the choice 1 /K is precisely motivated by Section 1.2, which shows this is a critical thresholdfor the small click-through rate.
Second, ⇤ t,B uses the regularizer KI .
This is to ensure that theregularizer is large enough compared to the summands  ( a sk )  ( a sk ) T / U s,H ( a sk ) , which scale as Kin the worst case where the variance upper bound U s,H ( a sk ) is clipped to 1 /K .
Remark 4.
Zong et al.
[2016] proposed a WOFUL -style cascading algorithm called CascadeLinUCB(see Appendix C), but they set the variance upper bound to ﬁxed  2 > 0 .
In fact, they remark “ideally, 2 should be the variance of the observation noises,” which CascadeWOFUL essentially learns .
Remark 5.
Appendix B contains an improved version of CascadeWOFUL .
It is more efﬁcient (forexample, the inverses are iteratively updated via Sherman-Morrison), satisﬁes the same theoreticalguarantee as Algorithm 2, and includes some tweaks that improve performance in practice.
Thetradeoff is that Algorithm 2 is simpler to explain, which is why we prefer it for the main text.
Regret bound.
We can now state our main result for the linear case.
Here B d ( M ) =
{ x 2 R d :k x k 2  M } denotes the Euclidean ball of radius M > 0 in R d .
Theorem 4.
Suppose ¯ w 2 [0 , 1] L ,  : [ L ] !
B d (1) , and ✓ 2 B d (1) satisfy ¯ w ( e ) =
h  ( e ) , ✓ i for alle 2 [ L ] .
Let ⇡ be the policy of Algorithm 2 with inputs ↵ t,H = p
d log(1 + tK/d ) + 2 log( n ) + 1and ↵ t,B =p
d log(1 + tK/d ) log( n 3 K )
+ 4 p K log( n 3 K )
+
p K .
Then
R ⇡ , ¯ w ( n ) =
˜
O ⇣ min np
[Equation]
o⌘ .
Algorithm 2:
Input: exploration parameters { ↵ t,H , ↵ t,B } nt =1 for t = 1 , . .
. , n do
Step 1: Deﬁne the (clipped) Hoeffding-style UCBs
[Equation]
o , where
[Equation]
Step 2: Deﬁne the Bernstein-style UCBs
[Equation]
where
[Equation]
Step 3: Let a t k = arg max e
[ L ] \{ a t i } k  1 i
=1 U t,B ( e ) be the item with the k -th highest UCB, play A t
= ( a t 1 , . . .
, a tK ) , observe C t = inf { k 2 [ K ] : w t ( a tk ) = 1 } (as in Algorithm 1)
Note this bound is completely independent of the number of items
L (the ˜ O ( · ) notation does notsuppress any L dependencies, as in the linear column of Table 1).
For large n (typically the case ofinterest), it becomes ˜ O
( p nd ( d + K )) , which improves the best known ˜ O
( p ndK ) by a factor ofp min { d, K } K .
The theorem also ensures regret remains ˜ o
( p ndK ) for general n .
Overview of the analysis
We next discuss the key ideas behind our proofs.
The details are deferred to Appendices D-I.
Theorem 2 proof.
For simplicity, we assume the arms are ordered by their means, i.e., ¯ w (1)  · · · ¯ w ( L ) .
Under this assumption, the optimal action is [ K ] .
We call those items optimal and [ L ] \ [ K ]suboptimal .
We let E t be the “bad event” that the empirical and true means differ substantially attime t , ¯ E t its complement, G e,e ⇤ ,t the event that suboptimal e > K was chosen in favor of optimal 238 e ⇤  K and subsequently examined by the user, and  e,e ⇤ =
¯ w ( e ⇤ )
¯ w ( e ) the reward gap .
Thenas in Appendix A.1 of Kveton et al.
[2015a], we “linearize” regret as follows:
[Equation]
The second term is small due to concentration.
For the ﬁrst term, deﬁne  = ¯ w ( K ) p
L/n andassume n  L (so  ⌧ ¯ w ( K ) ).
For each e > K , let K  ( e )
=
{ e ⇤  K :  e,e ⇤   } be theoptimal items with small gap relative to e and ¯ K  ( e ) =
[ K ] \ K  ( e ) the other optimal items.
Then
[Equation]
Therefore, we can upper bound the ﬁrst term in (7) by
[Equation]
For the ﬁrst term in (8), the inner double summation is the number of e > K that were chosen infavor of e ⇤ 2 K  ( e ) , and subsequently examined by the user, at round t .
Denote this number by O t .
Recall ¯ w ( e ⇤ )
¯ w ( K ) (by the assumed ordering) and  ⌧ ¯ w ( K ) , so for e ⇤ 2 K  ( e ) , we have
[Equation]
Hence, O t is bounded by the number of items e with ¯ w ( e ) ⇡ ¯ w ( K ) that the user examined.
Sincethe user stops examining at the ﬁrst attractive item, this number is dominated by a Geometric ( ¯ w ( K ))random variable.
Therefore, the ﬁrst term in (8) is ˜ O
(  n/ ¯ w ( K ))
=
˜ O ( p nL ) .
The second term in (8) accounts for choosing e instead of e ⇤ when  e,e ⇤   and the empiricalmeans are concentrated.
Building upon the intuition of Section 1.2, we exploit the variance-awarenessof KL-UCB and UCB-V (and use ¯ w ( e )  ¯ w ( K ) by the assumed ordering) to bound this term by˜ O
( L ¯ w ( e )(1  ¯ w ( e )) /  )  ˜ O
( L ¯ w ( K ) /  ) .
Thus, by choice of  , this term is ˜ O
( p nL )
as well.
Theorem 4 proof.
We decompose regret similar to (7) and (8), though with different choices of
E t and  .
6 To bound P ( E t ) , we use the aforementioned result of Abbasi-Yadkori et al.
[2011] toshow that the Hoeffding UCBs U t,H upper bound the variances with high probability, then prove aguarantee for the least-squares estimate ˆ ✓ t,B using the Bernstein-style bound from Zhou et al.
We bound the ﬁrst term in (8) using the exact same logic as the tabular case.
The second term hasa more complicated analysis, but (as in the tabular case) it amounts to bounding the number oftimes that items  -far from optimal are chosen when ˆ ✓ t,B is well-concentrated.
For this, we adapttechniques from the standard linear bandit setting (the case K = 1 ) to general K 2 [ L ] .
Theorem 1 proof.
We let H 0 t =
[ t  1 s =
{ A s , w s ( a s 1 ) , . . .
, w s ( a sK ) } be the entire history before time 263 t , which includes unobserved rewards w s ( a s k ) , k > C s .
We also let ⇧ 0 be the policies that map H
tto A t .
Note H t ⇢ H 0 t , so ⇧ ⇢ ⇧ 0 .
For any ⇡ 2 ⇧ 0 and ¯ w 2 [0 , 1] L , we deﬁne
[Equation]
where a ⇤ k = arg max e 2 [ L ] \{ a ⇤ i } k  1 i =1 ¯ w ( e ) .
Then a lower bound linearization analogous to (7) showsthat for any p 2 [0 , 1] and ¯ w 2 [0 , p ] L , R ⇡ , ¯ w ( n )  (1  p ) K  1 R 0 ⇡ , ¯ w ( n ) , which implies
[Equation]
If we choose p = 1 , the inf sup at right is the minimax regret for the document-based model thatwas analyzed by Lattimore et al.
[2018] (see Remark 1), but this makes (10) vacuous.
On the otherhand, by choosing p =
O (1 /K )
(again, the small click-through rate of Section 1.2), so that the term(1  p ) K  1 in (10) is ⌦ (1) , we can modify their analysis to prove Theorem 1.
Theorem 3 proof.
We linearize the regret similar to the proof of Theorem 1, then deﬁne a probleminstance reminiscent of Section 1.2 and (roughly) follow the intuition for UCB1 therein.
Experiments
Before closing, we conduct experiments on both synthetic and real data.
Some details regardingexperimental setup are deferred to Appendix C. Code is available in the supplementary material.
Synthetic data.
We let L = 100 and K 2 { 2 i } 8 i =2 .
For each K , we sample ¯ w ( e ) uniformlyin [ 23 K , 1 K ] for e  K and in [0 , 13 K ] for e > K .
Note this yields a positive gap with the smallclick-through rate of Section 1.2.
The top left plot in Figure 1 shows the regret at n = 10 5 for thetabular albums of Section 3 (the shaded regions are the standard deviations across ﬁve trials).
Aspredicted by Theorems 2 and 3, the CascadeUCB1 curve grows with K , while the variance-awarecurves do not.
The top middle plot shows the time spent computing UCBs for the same experiment,which conﬁrms the behavior mentioned in Remark 2.
For the linear case, we vary d 2 { 2 i } 8 i =2 , 283 generate the same ¯ w , then compute unit-norm vectors ✓ and  ( e ) satisfying ¯ w ( e )
=
h  ( e ) , ✓ i (see 284 Appendix C).
We compare our algorithm CascadeWOFUL to CascadeLinUCB [Zong et al., 2016],which has the strongest existing regret guarantee.
As suggested by Theorem 4, the left and middleplots on the bottom of Figure 1 show that our algorithm’s regret has superior dependence on K and d .
The rightmost plots show that regret is sublinear in n for the median values K = d = 10 .
Real data.
We replicate the ﬁrst experiment from Zong et al.
[2016] on the MovieLens-1M dataset( grouplens.org/datasets/movielens/1m/ ), which contains user ratings for L ⇡ 4000 movies.
In brief, the setup is as follows.
First, we use their default choices d = 20 and K = 4 .
Next, we dividethe ratings into train and test sets based on the user who provided the rating.
From the training dataand a rank- d SVD approximation, we learn a feature mapping  from movies to the probability that auniformly random training user rated the movie more than three stars.
Finally, we run the algorithmsas above, except at round t 2 [ n ] , we sample a uniformly random user J t from the test set and deﬁnew t ( a tk ) =
W ( J t , a tk ) , where W ( j, a ) =
( user j rated movie a more than 3 stars ) .
In other words,instead of the independent Bernoulli clicks of Section 2, we observe the actual feedback of user J t .
We point the reader to Section 4 of Zong et al.
[2016] and Appendix C for further details.
The left plotof Figure 2 shows that CascadeWOFUL outperforms CascadeLinUCB across n , eventually incurringless than 66% of the regret.
In addition to this setup from Zong et al.
[2016], we reran the experimentwhile restricting the set of items to movies of a particular genre, for each of 18 genres in the dataset.
This is intended to model platforms like Netﬂix that recommend movies in various categories.
Theright plot shows that CascadeWOFUL is superior for all genres; for some genres (e.g., fantasy) itsregret is about half of CascadeLinUCB ’s. Moreover, our experiments indicate that CascadeWOFULimproves CascadeLinUCB more dramatically for genres with smaller click-through rates (see Figure
and surrounding discussion in Appendix C), which reinforces a key message of this paper.
Conclusion
In this work, we proved matching upper and lower bounds for the problem-independent regret oftabular cascading bandits and an upper bound for the linear case, all of which improve the best known.
The algorithms we considered are all UCB-based, so it would be interesting to revisit Thompsonsampling for cascading bandits (see the recent paper by Zhong et al. [2021] and the references therein)in light of our variance-aware insight.
Finally, we note the paper is theoretical and has no immediatesocietal impact.
Nevertheless, we urge caution for the negative impacts that could arise in practice.
For example, our MovieLens experiments involved training on a subset of users, which could causepoor recommendations for demographics that are underrepresented in the training set.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] The claims are accurate.
Furthermore, we providereferences to the formal results alongside each claim stated in the introduction.(b) Did you describe the limitations of your work?
[Yes] The main limitation is that ourlower bounds (Theorems 1 and 3) require some additional assumptions.
These arestated clearly in the theorem statements and discussed in Remarks 1 and 3.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Section 7addresses societal impact.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem
[Yes
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] As mentionedabove, the assumptions are clearly stated in the theorem statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] Section 1.2 discusses
the high-level intuition behind our proofs, Section 5 contains a more detailed proofsketch, and the appendices include complete proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Complete codeto recreate all plots is available in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] The experimental setup is discussed in detail in Appendix C, andother training details can be found in the code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] All plots include error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] The experiments were run
over several hours on a laptop, so no signiﬁcant resources were used.
Nevertheless,
Appendix C mentions the approximate runtime needed to recreate the ﬁgures.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We only used theMovieLens dataset, which we cited and also provided a link to.
(b) Did you mention the license of the assets?
[Yes] The link to the MovieLens datasetcontains a README with license information.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]