Single-pass Streaming Lower Bounds for Multi-armed Bandits Exploration with Instance-sensitive Sample Complexity
Abstract
Motivated by applications to process massive datasets, we study streaming algorithms for pure exploration in Stochastic Multi Armed Bandits (MABs).
This problem was ﬁrst formulated by Assadi and Wang [STOC 2020] as follows:
A collection of n arms with unknown rewards are arriving one by one in a stream, and the algorithm is only allowed to store a limited number of arms at any point.
The goal is to ﬁnd the arm with the largest reward while minimizing the number of arm pulls (sample complexity) and the maximum number of stored arms (space complexity).
Assuming ∆ [2] is known, Assadi and Wang designed an algorithm that uses a memory of just one arm and still achieves the sample complexity of
O ( n/ ∆ 2[2] ) which is worst-case optimal even for non-streaming algorithms; here ∆ [ i ] is the gap between the rewards of the best and the i -th best arms.
In this paper, we extended this line of work to stochastic MABs in the streaming model with the instance-sensitive sample complexity, i.e. the sample complexity of O ( (cid:80) ni =
2 1∆ 2[ i ] log log ( 1∆ [ i ] )) , similar in spirit to Karnin et.al.
[ICML 2013] and
Jamieson et.al.
[COLT 2014] in the classical setting.
We devise strong negative results under this setting: our results show that any streaming algorithm under a single pass has to use either asymptotically higher sample complexity than the instance-sensitive bound, or a memory of Ω( n ) arms, even if the parameter ∆ [2] is known.
In fact, the lower bound holds under much stronger assumptions, including the random order streams or the knowledge of all gap parameters { ∆ [ i ] } ni =2 .
We complement our lower bounds by proposing a new algorithm that uses a memory of a single arm and achieves the instance-optimal sample complexity when all the strong assumptions hold simultaneously.
Our results are developed based on a novel arm trapping lemma .
This generic complexity result shows that any algorithm to trap the index of the best arm among o ( n ) indices (but not necessarily to ﬁnd it) has to use Θ( n/ ∆ 2[2] ) sample complexity.
This result is not restricted to the streaming setting, and to the best of our knowledge, this is the ﬁrst result that captures the sample-space trade-off for ‘trapping’ arms in multi-armed bandits, and it can be of independent interest.
Introduction
The stochastic multi-armed bandits (MABs) is one of the most well-known models in machine learning and theoretical computer science.
In this model, we are given n arms with unknown reward distributions, and we are required to ﬁnd some arms for certain goals.
The pure exploration version of the stochastic MABs studies how to ﬁnd the best arm(s), and it has been extensively investigated in the literature [21, 34, 6, 26, 27, 24, 14, 29, 11, 1, 16].
The model captures many fundamental problems in ﬁnding the best items among many, has various applications in experiment design [37, 9], crowdsourcing [17, 44, 10], search ranking [20, 3, 36], networking [43, 41], and others.
We study the problem of ﬁnding the best ( one ) arm in stochastic MABs.
Since the distribution of the rewards is unknown, the natural strategy is to pull each arm multiple times and use the empirical rewards to ﬁnd the best arm.
Accordingly, the goal of the algorithms in this problem is to identify the best arm with a high constant probability and a number of arm pulls (sample complexity) that is as small as possible.
Following the classical upper bound of [21] and lower bound of [34], it is well known that Θ( n/ ∆ 2[2] ) arm pulls are both necessary and sufﬁcient – throughout, ∆ [ i ] is the gap between the rewards of the best and the i -th-best arms.
Going beyond worst-case, the main focus of the literature has been on obtaining instance-sensitive algorithms for the problem, resulting in algorithms of [24] and [27] with O ( (cid:80) ni =2 1∆
[ i ] log log (cid:16) 1∆
[ i ] (cid:17) ) arm pulls (see also [16] and references therein for various extensions).
This upper bound has at most a doubly-logarithm gap from known instance-optimal lower bounds [34, 24, 16].
Almost the entirety of algorithms in the literature for best arm identiﬁcation require the entire set of the arms to be repeatedly visited, and therefore, a memory of Ω( n ) arms is inevitably needed.
More recently, motivated by applications to processing massive number of arms (e.g., in online advertisements) or the situations when the arms are arriving online (e.g. crowd-sourcing), the space requirements of algorithms for stochastic MABs has also received considerable attention [5, 32, 33, 13, 31, 2].
In particular, [5] recently formulated and studied the problem of pure exploration in streaming stochastic multi-armed bandits.
In their model, the arms arrive one by one in a stream, and the algorithm can only output or pull a past arm if it is stored.
The largest number of arms stored at any point is named the space complexity ; see also [32, 31, 25].
[5] designed an algorithm that given the knowledge of ∆ [2] , ﬁnds the best arm with high constant probability using O ( n/ ∆ 2[2] ) arm pulls and a memory of a single arm.
The algorithm of [5] achieves the worst-case optimal sample and space complexity simultaneously, thus asserting that no trade-off is needed between the sample and space for this problem.
On the ﬂip side, this upper bound is only worst-case optimal and not instance-sensitive.
In a recent result, [25] shows that if we are allowed to make O (log(1 / ∆ [2] )) passes over the stream, it is possible to ﬁnd the best arm with the instance-sensitive sample complexity in the same spirit of [24, 27], and only use a single-arm memory.
However, allowing O (log(1 / ∆ [2] )) passes seems to be a strong condition especially in streaming applications.
Therefore, a natural open question is whether we can achieve instance-sensitive sample complexity, while maintaining a small memory, preferably still a single arm, in a single pass of the streaming model .
The main result of this paper is that ﬁnding the best arm with the instance-sensitive sample complexity in a single pass is indeed hard : if we are only given the parameter ∆ [2] , any single-pass streaming algorithm must either use asymptotically higher sample complexity than the instance-sensitive bound, or use a memory of Ω( n ) arms.
Furthermore, the lower bound holds even if we are given additional strong assumption of either ( i ) the arms arrive in a random order or ( ii ) the parameters of all reward gaps { ∆ [ i ] } ni =2 .
En route to proving the above lower bounds, we also provide an even stronger lower bound for single-pass streaming algorithms without any parameter (not even ∆ [2] ): we show that in such a case, any single-pass streaming algorithm with a memory of o ( n ) has to use an unbounded sample complexity.
Finally, we complement our lower bounds with a single-pass streaming algorithm that uses a single-arm memory and achieves the instance-sensitive sample complexity, if given the parameter ∆ [2] and conditions of both random order stream and a known quantity of the instancesensitive sample complexity, where the latter is a weaker condition than ( ii ) .
1.1 Instance-sensitive Sample Complexity
As stated earlier, the worst-case sample complexity of stochastic MABs is Θ( n/ ∆ 2[2] ) .
However, in most instances of interest, one can hope for much better sample complexity that exploits the rewards of different arms in a more ﬁne-grained way.
This is the basic idea behind instancesensitive algorithms that have received signiﬁcant attention in the context of stochastic MABs.
We follow the same approach in this paper.
In particular, we aim to achieve the sample complexity of O (cid:0) (cid:80) ni
=2 1∆
[ i ] log log( 1∆ [ i ] ) (cid:1) arm pulls – the same bound achieved by classical algorithms for non-streaming stochastic MABs [24, 27].
Following the standard notation used in the literature [24, 27, 14], we denote the reward gap between the best and
the i -th best arm as ∆ [ i ] .
As such, we use { ∆ [ i ] } ni =2 to denote the set of all gap parameters.
The standard instance-sensitive complexity is as follows:
[Equation]
The algorithms of [24, 27] use O ( H 2 ) arm pulls.
It is worth noting that no known algorithm has achieved the O ( H 1 ) sample complexity, and whether ω ( H 1 ) sample complexity is necessary is an open problem even in the ofﬂine setting.
As such, we only focus on the O ( H 2 ) sample complexity in this paper.
The term ‘(almost) instance-sensitive’ is derived from ‘instance-optimal’ in the literature [14, 1, 5].
We use such a term as an analogy to the ‘truly instance-optimal’ O
( H 1 ) sample complexity, and this term is shared by recent work like [28].
Moreover, since there is a constant-factor difference between H 2 and the instance-sensitive sample complexity (i.e., H 2 vs.
O ( H 2 ) ), we denote the latter as INST-complexity in this work.
1.2 Our Contributions
Our main contribution is a set of lower bounds showing that it is almost impossible to achieve the instance-sensitive sample complexity with low memory for any single-pass streaming algorithm.
Moreover, if no parameter is provided (especially ∆ [2] is not known), it is impossible to achieve any bounded sample complexity with low memory:
Result 1 ( Informal ) .
There are families of stochastic multi-armed bandits instances such that:
( i )
If the knowledge of ∆ [2] is provided, but the knowledge of all other gap parameters { ∆ [ i ] } ni =3 and the random arrival of arms are not both provided, the sample complexity of any single-pass streaming algorithm with o ( n ) -arm memory is ω ( H 2 ) .
( ii )
If the knowledge of ∆ [2] is not provided and no other parameter is provided, the sample complexity of any single-pass streaming algorithm with o ( n ) -arm memory is unbounded as a function of n and ∆ [2] ;
Note that ( i ) of Result 1 captures three cases: only ∆ [2] is known; ∆ [2] is known and the stream is in a random order; and all the gap parameters { ∆ [ i ] } ni =2 are known (but the stream is in an adversarial order).
Compared to the positive results in [5], our lower bounds rule out any instance-sensitive algorithms under the same parameter settings.
Moreover, they provide a strong justiﬁcation for the knowledge of ∆ [2] in the algorithm of [5] (and any streaming MAB algorithms).
Another remark of Result 1 is the separation of the complexity for multi-armed bandits under the streaming and the classical models.
The algorithm in [5] implies that with the worst-case sample complexity and a known ∆ [2] , there is no separation of hardness for pure exploration MABs between the streaming and the classical settings, i.e. no sample-space trade-off.
On the other hand, our results suggest that if we switch to the regime of instance-sensitive sample complexity, or if we remove the assumption of ∆ [2] , a sample-space trade-off starts to emerge (note that there exist algorithms that achieve O ( H 2 ) sample complexity without the knowledge of ∆ [2] under the classical setting).
To complement the lower bounds, we design an algorithm that uses a single arm memory and achieves instance-sensitive sample complexity when ∆ [2] is not too small.
The algorithm is conditioned on the knowledge of ∆ [2] and both the random order stream and the knowledge of INST-complexity.
The latter condition, combined with the knowledge of ∆ [2] , is a weaker form of all gap parameters – one can certainly compute H 2 if parameters { ∆ [ i ] } ni =2 are given.
Result 2 ( Informal ) .
Assuming the knowledge of ∆ [2] and INST-complexity, and when the arms arrive in a random order, there is a streaming algorithm that ﬁnds the best arm in streaming stochastic multi-armed bandits with O (cid:16) H 2 + poly ( log n ∆ [2] ) (cid:17) arm pulls and a memory of a single arm with high (constant) probability.
We should note that the sample complexity of our algorithm in Result 2 has an additive term compared to the O ( H 2 ) obtained in the non-streaming setting [24, 27].
This term however is typically negligible except for pathological cases where ∆ [2] is extremely small compared to both average values of ∆ [ i ] ’s and the number of arms.
A key ingredient of our proof towards Result 1 is an arm-trapping lemma that characterizes the hardness to keep the arm in the memory, shown as below:
Result 3 ( Arm-trapping lemma, informal ) .
There are families of stochastic multi-armed bandits instances such that any algorithm that outputs o ( n ) arms that contains the index of the best arm (with high constant probability) has to use Ω (cid:18) n ∆ 2[2] (cid:19) arm pulls.
As a result, any single-pass streaming algorithm that keeps the best arm by the end of the stream (with high constant probability) has to use either Ω (cid:18) n ∆ 2[2] (cid:19) arm pulls or an Ω( n ) -arm memory.
To the best of our knowledge, this is the ﬁrst Multi-armed Bandits lower bound that explicitly establishes trade-off between sample and space complexity for trapping an arm 2 .
Note that this statement is stronger than the usual lower bounds on ﬁnding arms: our lower bound holds even if the algorithm is only required to output a collection of indices that contains the index of the best arm (possibly without knowing which arm is the best).
1.3 Related Work
Originated in the work of Robbins [37] almost 70 years ago, the study of stochastic MABs basically follows two popular branches: the pure exploration problem, which aims to ﬁnd the best arm [21, 34, 14, 29, 11, 1, 16]; and the exploration-exploitation problem, which focus on the regret minimization [42, 8, 30, 12, 19].
Covering the vast literature on regret minimization is beyond the scope of this paper but we should note that even regret minimization is studied in streaming setting although mostly for multi-pass algorithms [32, 13, 31, 2].
The worst-case optimal bounds on the sample complexity of exploration problem was established by [21] (upper bound) and [34] (lower bound).
Besides the best arm identiﬁcation, [21] also introduced the ε -best arm variation of the pure exploration problem.
The problem is motivated by learning under the PAC ( ε, δ ) -framework, where the algorithm aims to ﬁnd an arm with reward at most ε -far from the best arm.
The ε -best arm problem is also studied extensively.
It has recently been considered in the streaming model as well by [5, 32, 25] who designed streaming algorithms for ε -best arm problem with O (1) -arm memory and O ( n/ε 2 ) arm pulls 3 (we note that [32] also designed a heuristic for this problem on random streams with O (1) memory that works well empirically but does not have the same theoretical guarantee).
The instance-sensitive sample complexity of O ( H 2 ) was ﬁrst achieved by [27], followed up by [24] who achieved the same asymptotic complexity but a better performance in practice.
[24] also discovered that for the problem with 2 arms, Ω( 1∆ 2[2] log log( 1∆ [2] )) arm pulls are necessary – a lower bound that is due to a classical result of [22] from over half of a century ago.
[24] conjectured that the lower bound holds for n arms (and the complexity will be ‘instance- optimal ’); nonetheless, a later work by [14] shows that for some instances, an algorithm with O ( (cid:80) n i =2 1 ∆ 2 [ i ] ) arm pulls is possible and formulated a ‘gap-entropy’ conjecture for capturing the instance-optimal bounds (see [16] for further progress on this conjecture).
Preliminaries Problem Deﬁnition
The formal description of the stochastic multi-armed bandit model that we study is as follows.
Problem 1 (Pure exploration MABs) .
There is a set of n arms { arm i } ni =1 with unknown rewards, each following a sub-Gaussian distribution with means { p i } ni =1 .
Our goal is to ﬁnd the best arm with the highest expected reward, i.e., p i , by pulling them and observing the empirical rewards.
By deﬁnition, we need to solve Problem 1 with randomized algorithms.
As such, we aim to design algorithms that ﬁnd the best arm with high constant probability .
The reason to not consider the high probability of 1 − 1 poly ( n ) is that it is impossible to get such a high success probability without an
O (log( n )) multiplicative factor to the sample complexity, which is considered large in the literature.
We study Problem 1 under the streaming model, which means the arms arrive one by one from the set; for each arriving arm, the algorithm can pull it multiple times to observe the empirical reward or store it to the memory (so it can be pulled later).
However, if an arm is not stored, it is lost forever and the algorithm can neither pull nor restore it.
We refer to the number of arm pulls by the algorithm as the sample complexity , and the maximum number of arms stored as the space complexity .
The sample complexity of stochastic MABs is characterized by the gap parameters { ∆ [ i ] } ni =2 , which denotes the difference between the reward of the best arm and the arm with the i -th highest reward.
By deﬁnition, we have (cid:0) ∆ [2] ≤ ∆ [3] ≤ · · · ≤ ∆ [ n ] (cid:1) .
We assume that these parameters are greater than 0 – an assumption that is common in the literature [21, 26, 18, 39].
Note that the gap parameters are not necessarily given to the algorithm, although it is common to assume the knowledge of ∆ [2] .
In fact, we provide a comprehensive treatment to the necessity of the ∆ [2] in this work.
It is impossible to cover all of the wide range of assumptions used in the MABs literature.
As such, we restrict our attention to the following set of assumptions and conditions.
( i )
The knowledge of ∆ [2] .
( ii ) The (uniform) random order of arm arrival.
( iii )
The knowledge of additional gap parameters { ∆ [ i ] } ni =3 .
( iv ) The knowledge of the INST-complexity H 2 (or H δ 2 ): the instance-sensitive sample complexity.
Note that conditions ( i ) and ( iii ) forms a stronger condition than conditions ( i ) and ( iv ) – the latter can be computed given the former, but not vice versa.
The Arm Trapping Lemma
Our lower bounds are built upon an arm trapping lemma, which is a information-theoretic samplespace trade-off for any algorithm to ‘trap’ the best arm ( not restricted to the streaming setting).
Compared to the classical lower bounds for ﬁnding arms in stochastic MABs, our lemma stands out by showing that even trapping the best arm (possibly without knowing which one is the best among the trapped arms) requires a lot of arm pulls.
As a direct corollary, we can show that for any streaming algorithm with a memory of o ( n ) arms, storing the best arm in the stream takes Ω( n ∆ 2[2] ) arm pulls.
Our hard distribution for the arm-trapping lemma is quite natural: consider a collection of n arms with one of them having a reward of 12 , and all other arms having a reward of 12 − β for some β .
The high level idea behind the proof is a non-standard reduction from this problem to the problem of distinguishing whether the reward of a single arm is 1 / 2 or 1 / 2 − β ; a problem that provably requires Ω(1 /β 2 ) samples.
The reduction places this ‘special arm’ in a collection of n − 1 arms with reward 1 / 2 − β , uniformly permute them, and then gives them as input to any algorithm that can supposedly contradict Lemma 3.1.
The proof then shows that if the algorithm can only place this special arm with non-trivial probability among the output ‘trap’ set, this arm actually has reward of 1 / 2 and not 1 / 2 − β .
Combining this with a simple ‘direct sum style’ argument then proves the lower bound.
We now formally deﬁne the hard family of instances as follows.
DIST :
A hard
distribution
for trapping the best
arm 1.
An index i (cid:63) sampled uniform at random from [ n ] .
For i (cid:54) =
i (cid:63) , let the arms be with reward p i =
12 − β .
For i
= i (cid:63) , let the arm be with reward p
i (cid:63) = 12 .
Based on the deﬁnition of DIST , we can give the formal statement of our streaming arm trapping lemma.
Lemma 3.1 (Arm Trapping Lemma) .
Any algorithm that outputs (indices of) n 8 arms which contains the best arm on DIST with probability at least
has to use at least 1 1200 · nβ 2 arm pulls
The proof of Lemma 3.1 follows a (non-standard) reduction from the complexity to distinguish two arms with a β -reward gap, speciﬁed as follows:
Lemma 3.2.
Consider an arm with the reward from the following distribution
• p = 12 , with probability 12 ; • p = 12 − β , with probability 12 .
Any algorithm to determine the reward of the arm with β < 16 and a success probability of at leasthas to use 1144 · 1 β 2 arm pulls.
Lemma 3.2 is an information-theoretic lower bound similar to the ones in [34].
The bound shows that for any algorithm to distinguish an arm with reward either 12 or ( 12 − β ) , it has to pull the arm nβ 2 times (if β is sufﬁciently small).
This argument is standard and is provided for completeness as similar results, such as classical Farrell’s bound [22, 24, 14], have just subtle differences with this bound, which in fact prohibits us from using them.
Limited by space, we defer the proofs of Lemma 3.1 and Lemma 3.2 to Appendix C.
As a natural corollary of Lemma 3.1, any single-pass streaming algorithm to store the best arm with a memory of o ( n ) arms has to use Ω (cid:16) nβ 2 (cid:17) arm pulls.
Corollary 3.3.
Any streaming algorithm that stores the best arm on DIST with a memory of n 8 arms with probability at least 23 has to use Ω (cid:16) nβ 2 (cid:17) arm pulls.
Corollary 3.3 gives a powerful tool to prove space lower bounds in streaming MABs by exploiting the dilemma between necessity of the exploration of best arm in the early part of the stream and the cost of doing so.
In particular, consider instances where O ( n ) arms follow DIST in Corollary 3. appear in the ﬁrst part of the stream, and a few arms by the end of the stream are either with very high or with very low rewards.
If the algorithm explores the best arm in the earlier stream, it risks to ‘waste’ a large number of arm pulls since the late arm high-reward case renders the exploration unnecessary.
On the other hand, if the algorithm decides to forgo the best arm in the earlier part, it runs the risk that the late arms are with very low rewards, and the best arm will be missed.
Streaming Lower Bounds for Multi-armed Bandits with Instance-sensitive Sample Complexity
In this section, we provide the main lower bounds for single-pass streaming MABs.
On the high level, our lower bound states that for any streaming algorithm with a memory of o ( n ) arms that ﬁnds the best arm, if some parameters are not given, the sample complexity of the algorithm can be far more than the instance-sensitive Θ( H 2 ) or even unbounded.
Remark 4.1.
To be consistent with the deﬁnition of streaming MAB, in our lower bounds, we only measure the space of the algorithms in terms of the number of arms they store; the algorithms otherwise are allowed to maintain any arbitrary statistics on the input.
What we show in all these lower bounds is that if the sample complexity is “small”, then at some point the algorithm must have mistakenly discarded the best arm from its memory without the knowledge of even the index of this arm; thus at the end of the stream, the algorithm has no way of recovering the best arm (even given the arbitrary statistics it has collected).
4.1 A lower bound without the knowledge of ∆ [2] and other parameters
We begin with showing that if no additional parameter is provided other than the stream itself, the sample complexity can be arbitrarily high, and even O ( n/ ∆ 2[2] ) arm pulls are not sufﬁcient.
In fact, we show that in this case, the sample complexity cannot be bounded as a function of n and ∆ [2] .
In contrast, the algorithm in [5], which assumes the knowledge of ∆ [2] , achieves O ( n/ ∆ 2[2] ) arm pulls with the memory of a single arm.
Therefore, our lower bound asserts the importance of ∆ [2] for exploring the best arm under the streaming model.
Theorem 1.
For any function f : (0 , 1) → (0 , 1) , there always exists a family of streaming stochastic multi-armed bandit instances, such that any algorithm to ﬁnd the best arm with a memory of n − 1arms and a success probability of at least 56 has to use ω (cid:16) n f (∆ [2] ) (cid:17) arm pulls, if no further knowledge other than the arms themselves are provided.
On the high level, the adversarial distribution for Theorem 1 is as follows.
We let the ﬁrst ( n − 1) arms in the stream to follow DIST in Corollary 3.3, and the last arm to have either a very high or a very low reward, with probability 12 each.
As such, to guarantee a high enough success probability, the algorithm needs to trap the arm with the p = 12 reward among the ﬁrst ( n − 1) arms.
However, the sample complexity to keep such an arm becomes unbounded when the last arm is with a very high reward.
We defer the proof of Theorem 1 to Appendix B.1.
4.2 Instance-sensitive Lower bounds with the knowledge of ∆ [2]
Going beyond the case where none of the conditions are given, we now show the lower bound with ∆ [2] provided.
Recall that [5] already solved the problem with O ( n ∆ 2[2] ) arm pulls; therefore, it is pointless to expect any result as strong as Theorem 1.
Instead, we prove lower bound of ω ( H 2 ) by exploiting the gap between Θ (cid:16) n/ ∆ 2[2] (cid:17) and Θ ( H 2 ) .
We ﬁrst present the lower bound with only the knowledge of ∆ [2] .
Theorem 2.
There exists a family of streaming stochastic multi-armed bandit instances, such that given the knowledge of ∆ [2] , any algorithm to ﬁnd the best arm with a memory of n − 2 8 arms and a success probability of at least 56 has to use ω (cid:16) H
[Equation]
(cid:17)
[Equation]
pulls.
We have noted that it is pointless to expect any result as strong as Theorem 1.
Indeed, the hard distribution in Section 4.1 does not work anymore – by the change of the reward of the ﬁnal arm, the value of ∆ [2] is β in one case and 16 >> β in the other.
Therefore, by simply checking the parameter, an algorithm can determine whether to store the best arm among the ﬁrst ( n − 1) arms, which makes the distribution impossible to ‘fool’ the algorithm.
As an alternative idea, we modify the distribution of the instances to satisfy the following properties.
Firstly, it forces the algorithm to make Ω( n ∆ 2[2] ) arm pulls to ﬁnd the best arm.
Secondly, it leaves a large gap between ∆ [2] and other ∆ [ i ] , ∀ i ≥ 2 such that there is n ∆ 2[2] = ω
( H 2 ) .
And ﬁnally, ∆ [2] is static (the same version) for all the instances.
Limited by space, we defer the formal proof of Theorem 2 to Appendix B.2.
We also remark that the proof of Theorem 2 works under an approximate knowledge of ∆ [2] , which can lead to slightly stronger lower bounds (see Remark B.2 for details).
We now provide lower bounds with either the Random Arrival or the quantity of H 2 (but not both ) in
Theorem 3 and Theorem 4.
Theorem 3.
There exists a family of streaming stochastic multi-armed bandit instances, such that given the knowledge of ∆ [2] and the random arrival of arms, any algorithm to ﬁnd the best arm with a memory of ( n 24 − 1) arms and a success probability of at least 8081 has to use ω (cid:16) H 2 (cid:17)
=
ω
[Equation]
Theorem 4.
There exists a family of streaming stochastic multi-armed bandit instances, such that given the knowledge of all gap parameters { ∆ [ i ] } ni =2 , any algorithm to ﬁnd the best arm with a memory of t 8 arms for t = o ( n ) and a success probability of at least 89 has to use Ω (cid:16) nt · H 2 (cid:17)
= ω ( H 2 ) arm pulls.
The idea to prove the lower bound with random arrival is quite simple: we show that with at least a constant probability, the distribution of an instance with random arrival will resemble the instance we used for Theorem 2.
More speciﬁcally, suppose each arm arrives at any position with probabilityn .
Then, with probability 13 , the arm with reward 12 appears in the ﬁrst 13 -fraction of the stream;
Similarly, with probability 23 · 23 , both of the last two arms in the distribution of Theorem 2 will not pop up until the late 23 fraction of the stream.
The family of the instances with both of the above properties indeed resembles the distribution we used earlier, and we can show that the algorithm is forced to use a lot of arm pulls under this scenario.
The hard instance for Theorem 4 is developed with a different idea.
Due to space limits, we defer the proofs to Appendix B.
An Instance-sensitive Streaming Algorithm
We begin with introducing notations used in this section.
We denote the best arm with arm ∗ .
Furthermore, we denote the instance-sensitive complexity ( INST-complexity ) for solving the problem with probability of success at least 1 − δ as H δ 2 := (cid:80) ni =
2 1∆ 2[ i ] log (cid:16) 1 δ log (cid:16) 1∆ [ i ] (cid:17)(cid:17) .
Note H 2 and H δonly differ from the correct probability; if one sets δ to be a constant, H 2 and H δ 2 become the same asymptotically.
As mentioned before, our algorithm makes two assumptions on the stream of arms, namely:
( i ) Random arrival stream : the stream is a random permutation of the arms { arm i } ni =1 .
( ii ) Known INST-complexity : The parameter H δ 2 (or an α -approximation of it with α = O (1) being a ﬁxed and known constant) is given;
Assumption ( ii ) is a weaker version of all gap parameters { ∆ [ i ] } ni
=2 used in the lower bound of
Theorem 4.
If the gap parameters are given, we can simply compute H 2 and H δ 2 .
The reverse direction is not generally possible.
Although the assumptions are strong, they are common in existing work of streaming models and stochastic MABs.
The random order stream is a standard assumption in the streaming setting (see [7, 4, 35, 40], to name a few).
The knowledge of H δ 2 is similar-in-spirit (albeit not identical) to the ﬁxed-budget stochastic MABs, which has been studied intensively in the literature; see, e.g.
Furthermore, from a practical point of view, the knowledge of ∆ [2] can be satisﬁed by an estimation that is at most ∆ [2] .
The stronger assumptions of ( i ) and ( ii ) can be satisﬁed under the following scenarios:
• The arms are stored in a ‘bank’ and a local machine reads the arms with a small working memory.
This allows the arms to be ‘served’ in a random order from the external bank.
• There exists historical data such that one can roughly estimate the quantity of INST-complexity.
While the scenarios are restricted, it does capture a non-negligible range of applications.
The formal guarantees of our algorithm is as follows.
Theorem 5.
There is a streaming algorithm that for any conﬁdence parameter δ > 0 , given n arms in a random order, gap parameter ∆ [2] , and INST-complexity H δ 2 = (cid:80) ni =
2 1∆ 2[ i ] log (cid:16) 1 δ log (cid:16) 1∆ [ i ] (cid:17)(cid:17) , ﬁnds the best arm w.p.
at least 1 − δ using O (cid:18) H δ 2 + 1∆ 7[2] · log( nδ ) 2 · log 2 (cid:16) 1 δ log (cid:16) 1∆ [2] (cid:17)(cid:17)(cid:19) arm pulls and a memory of a single arm.
The algorithm in Theorem 5 contains an additive term in sample complexity beside the standard
H δ 2 term.
For most reasonable choices of parameters, this additive term is in fact negligible — in other words, our streaming algorithm in Theorem 5 achieves the same sample complexity as that of non-streaming algorithms of [27, 24].
From a practical perspective, assuming H 2 ≥ 100 n and a constant success probability, we only need n to be in 10 6 ∼ 10 7 and ∆ [2] be around 0 .
1 ∼ 0 .
2 to make H 2 the dominating term, which seems reasonable in practice.
5.1 The Algorithm
Since we only maintain a single arm memory through the stream, we deﬁne the stored arm as the king in a manner similar to the G AME - OF -C OINS algorithm of [5] (an arm with Bernoulli reward is equivalent to a coin ).
We also adopt the challenge procedure ﬁrst developed by [5]: for each arriving arm, if there already exists a king , the new arm will begin a challenge procedure.
If the challenge is successful, the challenger becomes the new king to be challenged by the later arms; otherwise, the challenger is discarded, and the existing king retains its position and will be challenged by later arms.
By the end of the stream, the algorithm outputs the king that is ﬁnally stored.
The crucial part of our algorithm (similar to G AME - OF -C OINS of [5]) is how to design the challenge process.
Following [5], the generic strategy for this is to compare the empirical reward on many levels .
That is, on each level, we pull both the king and the challenger for a certain times, and observe which arm has a lower empirical reward, to identify the defeated arm.
If the challenger is defeated, we can simply discard it; but, if the king is defeated, we increase the number of arm pulls and compare the empirical rewards again.
The king will not be discarded until some certain conditions are met.
The challenge procedure in [5] inherently requires Ω(1 / ∆ 2[2] ) arm pulls for each challenger even on level 1.
Therefore, the algorithm inevitably requires Θ( n/ ∆ 2[2] ) arm pulls.
Our main task here is to design a new challenge with fewer arm pulls, while maintaining the correctness of the algorithm.
To this end, we now formally introduce the algorithm.
Let us ﬁrst set up the parameters as follows.
[Equation]
We also need some new deﬁnitions.
We refer to the period of time between the introduction of a new king and the time it is discarded (or the end of the stream) as an epoch .
For each epoch, we deﬁne the early arms as the ﬁrst m early arms, and the the late arms as the arms afterwards.
During each epoch, we maintain a ﬁxed estimate for the value of p ∗ as a parameter p est that will only be updated after this epoch ends (and p est is increasing throughout the updates of the algorithm).
We can now present the algorithm:
Main Algorithm:
I NSTANCE -S ENSITIVE -G
AME - OF -A RMS :
a) Initialize the ﬁrst arm as king and the ﬁrst guess of p ∗ as p est = ∆ [2] 20 .
b) For each epoch: ( i ) Run G AME - OF -C OINS of [5] on early arms with parameters: n (cid:48) :
= m early ∆ (cid:48) [2] = ∆ [2]and δ (cid:48)
[Equation]
while setting the reward of the king to be p est (start their algorithm with an “artiﬁcial” ﬁrst arm/coin with reward p est ).
Terminate if G AME - OF -C OINS discards king and jump to Line ( iii ) ; otherwise, initialize the budget of the king as
B ← b · m early .
( ii )
For each late arm, give the king a budget of b , i.e., B ← B + b ; run InstanceSensitive Challenge Subroutine (below) for king , challenger, budget B , and p est .
( iii )
If the king is discarded, set the challenger as king and update p est as follows: (1) Pull the challenger 64∆ 2[2] · log( nδ ) times and record the new (cid:98) p challenger .
(2) Set p est ← max( (cid:98) p challenger , p est + ∆ [2] 20 ) ; c) Run till the end of the stream, and output the king as the best arm.
Instance-Sensitive Challenge Subroutine :
Input: the king and its budget B ; a challenger arm ; the estimation parameter p est .
For levels (cid:96) = 1 , 2 , · · · , as long as s (cid:96) < 64∆ 2[2] · log( nδ ) : (a) If B > s (cid:96) , let B ← B − s (cid:96) and pull challenger s (cid:96) times to get empirical reward (cid:98) p .
(b) If (cid:98) p < p est − ∆ (cid:96) / 4 , drop the challenger and terminate.
Pull the challenger ( 64 ∆ 2 [2] · log( n δ ) times and record its empirical reward (cid:98) p : (a) If (cid:98) p > p est − ∆ [2] / 4 , declare the challenge successful, and discard the king ; otherwise, declare the challenge unsuccessful, and discard the challenger.
In a nutshell, the intuitions for the algorithm are as follows.
Classical ofﬂine algorithms for instancesensitive explorations (e.g. the algorithm in [27]) often use estimations ˜∆ (cid:96) for gaps ∆ [ i ] , and argue that by iteratively comparing the arm rewards with the best arm, the distinction between the best and the i -th best arms becomes very clear after ˜∆ (cid:96) ≤ ∆ [ i ] .
The value of ˜∆ (cid:96) takes O (log 1∆
[ i ] ) iterations to converge; as such, it takes O (cid:0) 1∆
[ i ] log log( 1∆ [ i ] ) (cid:1) arm pulls to distinguish the best and the i -th best arms.
The strategy inherently requires the best arm to be stored in the beginning of the algorithm.
Alas, migrating to the streaming setting, the above strategy no longer works: we cannot assume the best arm is stored in the beginning – it may appear in a very late part of the stream.
As such, we face the challenges of ( i ) .
it is unclear how to use the reward of the best arm as in the ofﬂine setting; and ( ii ) .
the comparisons between sub-optimal arms may take too many arm pulls.
It turns out our assumptions are helpful to tackle the above challenges.
On the very high level, we use the knowledge of ∆ [2] to ‘guess’ the reward of the best arm – this allows us to iteratively update the value p est in the algorithm, and eventually arrive at an estimation that is (cid:0) c · ∆ [2] (cid:1) -close to the best reward ( c < 1 ).
Furthermore, we use the the random order of the stream and the knowledge of H δ 2 to ‘budget’ the number of arm pulls for each arriving arm.
It turns out this allows us to argue the budget is sufﬁcient for the real best arm in an amortized manner.
Limited by space, we defer the detailed overview and the formal analysis of the algorithm to
Appendix D.
Remark 5.1.
Our lower bounds in Section 4 shows that if we are given the knowledge of ∆ [2] , but not both the random order of arrival and the knowledge of H δ 2 (in the stronger form of { ∆ [ i ] } ni =3 ), it is impossible to arrive at the INST-complexity sample complexity.
On the other hand, it leaves the possibility for an algorithm to ﬁnd the best arm with o ( n ) memory, assuming no knowledge of ∆ [2] but with given parameters { ∆ [ i ] } ni =3 and the random arm arrival.
Indeed, it appears the following modiﬁcation of our algorithm works for such a scenario: sample each arm in the stream with probability 12 , and run I NSTANCE -S ENSITIVE -G
AME - OF -A RMS by replacing ∆ [2] with ∆ [3] and the H δ 2 with the quantity (cid:80) ni =
1∆ 2[ i ] log (cid:16) 1 δ log (cid:16) 1∆
[ i ] (cid:17)(cid:17) .
By the sampling of arms, with probability 18 , the best arm and third-best arm join the stream, while the second-best arm does not.
Therefore, we can run O (log(1 /δ ))
copies of the algorithm, and output the top arm among the O (log(1 /δ ))
copies by running the ofﬂine algorithm in [27] at the end of the stream.
Verifying whether the above strategy works will be an interesting immediate next-step for the work.
Discussion and Conclusion
In this paper, we studied the limitations for streaming MABs algorithm with the instance-sensitive sample complexity in a single pass.
We proved a collection of lower bounds showing that achieving the instance-sensitive sample complexity with a small memory in a single-pass stream is hard : with the absence of various (subsets of) assumptions, including the knowledge of ∆ [2] , the additional gap parameters { ∆ [ i ] } ni =3 , and the random arrival of arms, no algorithm can achieve the instance-sensitive bound with o ( n ) -arm memory.
Furthermore, as a byproduct of the lower bounds, we establish the crucial role of ∆ [2] in streaming MABs: without such a parameter, any algorithm with a memory even as large as a constant fraction of n cannot achieve any bounded sample complexity.
We complemented our lower bounds by an algorithm that achieves the instance-sensitive bound under most realistic parameters and with a memory of a single arm when (almost) all assumptions hold.
Our lower and upper bounds demonstrate a sharp dichotomy.
Our results open the door for various future work in this line of research.
One open problem posed by our results is whether one can reﬁne the streaming algorithm (or design a new one) that drops the additive term on the sample complexity.
Currently, we do not rule out the possibility that the additive term is necessary .
Moreover, from the application side, an interesting problem is to explore the empirical performances of our algorithm, and observe the numerical advantages compared to the algorithm in [5].
And ﬁnally, a more challenging open problem is to understand the tight number of passes that are necessary and sufﬁcient to achieve the instance-sensitive sample complexity (with no or minimum assumptions) under the streaming MABs model.
Acknowledgments and Disclosure of Funding
Research supported in part by a NSF CAREER Grant CCF-2047061, a gift from Google Research, and a Rutgers Research Council Fulcrum Award.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We have discussed the problems left open by our work both in the corresponding sections and in Section 6.
All the assumptions are explicitly stated in the paper.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is a theoretical work and we cannot identify any immediate positive or negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Please see the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]