On Elimination Strategies for Bandit Fixed-Confidence Identification
Anonymous Author(s)
AffiliationAddress email
Abstract
Elimination algorithms for bandit identification, which prune the plausible correctanswers sequentially until only one remains, are computationally convenient sincethey reduce the problem size over time.
However, existing elimination strategies areoften not fully adaptive (they update their sampling rule infrequently) and are noteasy to extend to combinatorial settings, where the set of answers is exponentiallylarge in the problem dimension.
On the other hand, most existing fully-adaptivestrategies to tackle general identification problems are computationally demandingsince they repeatedly test the correctness of every answer, without ever reducing theproblem size.
We show that adaptive methods can be modified to use eliminationin both their stopping and sampling rules, hence obtaining the best of these twoworlds: the algorithms (1) remain fully adaptive, (2) suffer a sample complexitythat is never worse of their non-elimination counterpart, and (3) provably eliminatecertain wrong answers early.
We confirm these benefits experimentally, where elimination improves significantly the computational complexity of adaptive methodson common tasks like best-arm identification in linear bandits.
Introduction
The multi-armed bandit is a sequential decision-making task which is now extensively studied(see, e.g., [1] for a recent review).
In this problem, an algorithm interacts with its environmentby sequentially “pulling” one among K ∈ N arms and observing a sample from a correspondingdistribution.
Among the possible objectives, we focus on fixed-confidence identification [2, 3, 4, 5].
In this setting the algorithm successively collects samples until it decides to stop and return an answerto a given query about the distributions.
Its task is to return the correct answer with at most a givenprobability of error δ and its secondary goal is to do so while stopping as early as possible.
Thisproblem is called “fixed-confidence” as opposed to “fixed-budget”, where the goal is to minimize theerror probability with at most a given number of samples [6, 7, 8, 9, 10].
The most studied query is best arm identification (BAI), where the aim is to return the arm whosedistribution has highest mean.
A variant is Top-m identification [11], where the goal is to find them arms with highest means.
While these are the most common, other queries have been studied,including thresholding bandits [9], minimum threshold [12], and multiple correct answers [13].
Algorithms for fixed-confidence identification can be generally divided into two classes: those basedon adaptive sampling and those based on elimination .
Adaptive algorithms
[e.g., 8, 11, 4, 14] updatetheir sampling strategy at each round and typically stop when they can simultaneously assess thecorrectness of every answer.
They often enjoy strong theoretical guarantees.
For instance, some ofthem [4, 15, 16, 17] have been shown to be optimal as δ → 0 .
However, since they repeatedly test thecorrectness of every answer, they are often computationally demanding.
Elimination-based strategies[e.g., 2, 18, 19, 20, 21] maintain a set of “active” answers (those that are still likely to be the correct one) and stop when only one remains.
They typically update their sampling rules and/or the activeanswers infrequently.
This, together with the fact that eliminations reduce the problem size overtime, makes them more computationally efficient but also yields large sample complexity in practice.
Moreover, while adaptive algorithms for general identification problems (i.e., with arbitrary queries)exist [4, 15, 17], elimination-based strategies are not easy to design at such a level of generality.
Inparticular, they are not easy to extend to combinatorial problems (such as Top-m), where the numberof answers is exponential in the problem dimension.
In this paper, we design a novel elimination rule for general identification problems which we callselective elimination.
It can be easily combined with existing adaptive strategies, both in theirstopping and sampling rules, making them achieve the best properties of the two classes mentionedabove.
In particular, we prove that (1) selective elimination never suffers worse sample complexitythan the original algorithm, and hence remain asymptotically optimal whenever the base algorithmis; (2) It provably discards some answers much earlier than the stopping time; (3) It improvesthe computational complexity of the original algorithm when some answers are eliminated early.
Experimentally, we compare several existing algorithms for three identification problems (BAI,
Top-m, and thresholding bandits) on two bandit structures (linear and unstructured).
We find that,coherently across all experiments, existing adaptive strategies achieve significant gains in computationtime and, to a smaller extent, in sample complexity when combined with selective elimination.
1.1 Bandit fixed-confidence identification
An algorithm interacts with an environment composed of K > 1 arms .
At each time t ∈ N , thealgorithm picks an arm k t and observes X k t t ∼ ν k t , where ν k t is the distribution of arm k t .
At a timeτ , the algorithm stops and returns an answer ˆ ı from a finite set I .
Formally, let F t be the σ -algebragenerated by the observations up to time t .
An identification algorithm is composed of
Sampling rule : the sequence ( k t ) t ∈ N , where k
t
is F t − 1 -measurable.
Stopping rule : a stopping time τ with respect to ( F t ) t ∈ N and a random variable ˆ ı ∈ I , i.e.,the answer returned when stopping at time τ .
Note that, while it is common to decouple τ and ˆ ı , we group them to emphasize that the time at whichan algorithm stops depends strongly on the answer it plans on returning.
We assume that the arm distributions depend on some unknown parameter θ ∈ M , where M ⊆ R dis the set of possible parameters, and write ν k ( θ ) for k ∈ [ K ] to make this dependence explicit.
Forsimplicity, we shall use θ to refer to the bandit problem ( ν k ( θ ))
k ∈ [ K ] .
This assumption allows us toinclude linear bandits in our analysis.
We let i ⋆ : M → I be the function, known to the algorithm,which returns the unique correct answer for each problem.
The algorithm is correct on θ if ˆ ı = i ⋆ ( θ ) .
Definition 1.1 ( δ -correct algorithm) .
An algorithm is said to be δ -correct on M ⊆ R d if for allθ ∈ M , τ < + ∞ almost surely and P θ (ˆ ı ̸
= i ⋆ ( θ ))
≤ δ .
We want to design algorithms that, given a value δ , are δ -correct on M and have minimal expectedsample complexity E θ
[ τ ] for all θ ∈ M .
A lower bound on E θ [ τ ] was proved in [4].
In order to presentit, we introduce the concept of alternative set to an answer i ∈
I : Λ( i ) :
= { λ ∈ M | i ⋆ ( λ )
̸ = i } , theset of parameters for which the correct answer is not i .
Let us denote by KL k ( θ, λ ) the KullbackLeibler (KL) divergence between the distribution of arm k under θ and λ .
Then the lower boundstates that for any algorithm that is δ -correct on M and any problem θ ∈ M ,
E θ
[ τ ] ≥ log(1 / (2 . 4 δ )) /H
⋆ ( θ ) , with H ⋆ ( θ ) :
= max ω ∈ ∆ K inf λ ∈ Λ( i ⋆
[Equation]
Example: BAI in Gaussian linear bandits While our results apply to general queries, we illustrateall statements of this paper on the widely-studied task of BAI in Gaussian linear bandits [19, 14, 23,16, 24].
In this setting, each arm k ∈
[ K ] has a Gaussian distribution N ( µ k ( θ ) , 1) with mean µ k ( θ ) =
ϕ
⊤ k θ , a linear function of the unknown parameter θ ∈ R d (and M = R d ) and of known arm featuresϕ k ∈ R d .
The set of answers is I =
[ K ] and the correct answer is i ⋆ ( θ ) :
= arg max k ∈
[ K ] ϕ ⊤ k θ .
Finally, for x ∈ R d and A ∈ R d × d , we define ∥ x
∥ A := √ x ⊤ Ax .
For ω ∈ R K , let V ω :=(
cid:
Kk =1 ω k ϕ k ϕ ⊤ k .
With this notation, we have (cid:80) k ∈
[ K ] ω k KL k ( θ, λ ) =
12 ∥ θ − λ ∥ 2 V ω .
1.2 Log-likelihood ratio stopping rules
Most existing adaptive algorithms use a log-likelihood ratio (LLR) test in order to decide whento stop.
Informally, they check whether sufficient information has been collected to confidentlydiscard at once all answers except one.
Since such LLR tests are crucial for the design of our generalelimination rules, we now describe their principle.
Given two parameters θ, λ ∈ M , the LLR of observations X [ t ] =
( X k 1 1 , . . .
, X k t t ) between modelsθ and λ is L t ( θ, λ ) :
=
log d P θ d P λ ( X [ t ] )
= (cid:80) ts =1 log d P θ d P λ ( X k s s ) .
Let ˆ θ t := arg max λ ∈M log P λ
( X [ t ] )be the maximum likelihood estimator of θ from t observations.
In Gaussian linear bandits, wehave L t ( θ, λ ) =
12 ∥ θ − λ ∥ 2 V Nt + ( θ − λ )
⊤ V N t (ˆ θ t − θ ) , where N kt := (cid:80) ts =1 1 ( k s = k ) .
See
Appendix C for more details.
L t ( θ, λ ) is closely related to (cid:80)
Kk =
N kt KL k
( θ, λ ) , a quantity thatappears frequently in our results.
Indeed, the difference between these quantities is a martingale,which is a lower order term compared to them.
The LLR stopping rule was introduced to the banditliterature in [4].
At each step t ∈ N , the algorithm computes the infimum LLR to the alternative setof i ⋆
(ˆ θ t ) and stops if it exceeds a threshold, i.e., if
[Equation]
(2) where the function β t,δ can vary, notably based on the shape of the alternative sets.
The recommendation rule is then ˆ ı = i ⋆ (ˆ θ t ) .
Informally, the algorithm stops if it has enough information to exclude allpoints λ for which the answer is not i ⋆
(ˆ θ t ) .
This stopping rule enforces δ -correctness, provided thatthe sampling rule ensures τ <
+ ∞ a.s. and that β t,δ is properly chosen.
The most popular choice isto ensure a concentration property of L t (ˆ θ t , θ ) .
For example, if for all δ , β t,δ guarantees that
[Equation]
LLR stopping with that threshold returns a wrong answer with probability at most δ .
Such concentration bounds can be found in [25, 26] for linear and unstructured bandits, respectively.
This LLRstopping rule is used in many algorithms [4, 14, 15, 16, 24, 17] 2 .
Some of them have been proven tobe asymptotically optimal : their sample complexity upper bound matches the lower bound (1) whenδ → 0 .
However, improvements are still possible: their sample complexity for moderate δ may notbe optimal and their computational complexity may be reduced, as we will see.
Elimination stopping rules for adaptive algorithms
We show how to modify the stopping rule of adaptive algorithms using LLR stopping to performelimination.
We assume that the alternatives sets Λ( i ) can be decomposed into a union of sets whichwe refer to as alternative pieces (or simply pieces), with the property that computing the infimum
LLR over these sets is computationally easy.
Assumption 2.1.
For all i ∈ I , there exist pieces (Λ p ( i ))
p ∈P ( i ) , where P ( i ) is a finite set of pieceindexes, such that Λ( i ) =
(cid:83) p ∈P ( i ) Λ p ( i ) and inf λ ∈
Λ p ( i )
L t (ˆ θ t , λ ) can be efficiently computed for allp ∈ P ( i ) and t > 0 .
This assumption is satisfied in many problems of interest, including BAI, Top- m identification, andthresholding bandits (see Appendix B).
Indeed, in all applications we consider in this paper, the setsof Assumption 2.1 are half-spaces.
In our linear BAI example, the piece indexes are simply arms.
For i, j ∈ [ K ] we can define Λ j ( i ) =
{ λ ∈ M | ϕ ⊤ j λ > ϕ ⊤ i λ } .
Then, Λ( i ) =
(cid:83) j ∈ [ K ] \{ i } Λ j ( i ) .
Moreover, the infimum LLR (and the corresponding minimizer) can be computed in closed form as[e.g.
, 20] inf λ ∈ Λ j ( i )
L t ( ˆ [Equation] t , λ ) = max {
[Equation]
t ( ϕ i − ϕ j ) , 0 } 2 /
[Equation]
ϕ i − ϕ j ∥
[Equation] V − 1 Nt .
Elimination stopping The main idea is that it is not necessary to exclude all Λ p ( i ) for p ∈ P ( i )at the same time , as LLR stopping (2) does, in order to know that the algorithm can stop and returnanswer i .
Instead, each piece can be pruned as soon as we have enough information to do so.
Definition 2.2.
A set S ⊆
R d is said to be eliminated at time t if, for all λ ∈ S , L t (ˆ θ t , λ ) ≥ β t,δ .
From the concentration property (3), we obtain that the probability that θ ∈ S and S is eliminatedis less than δ .
LLR stopping interrupts the algorithm when the alternative set Λ( i ⋆ (ˆ θ t )) can beeliminated.
In elimination stopping, we eliminate smaller sets gradually, instead of the wholealternative at once.
Formally, let us define, for all
i ∈
I ,
[Equation]
as the subset of pieces for answer i ∈ I whose infimum LLR at time t is below a threshold β t,δ .
Thatis, the indexes of pieces that are not eliminated at time t .
Moreover, we define, for all i ∈ I , a set ofactive pieces P stp t ( i ) which is initialized as P stp0 ( i )
=
P ( i ) (all piece indexes).
Our selective elimination rule updates, at each time t , only the active pieces of the empirical answeri ⋆ (ˆ θ t ) .
That is, for i
=
i ⋆
(ˆ θ t ) , it sets
[Equation] [Equation]
[Equation]
while it sets P stp t ( i ) :
= P stp t − 1 ( i ) for all i ̸
= i ⋆
(ˆ θ t ) .
One might be wondering why not updating allanswers at each round.
The main reason is computational: as we better discuss at the end of thissection, checking LLR stopping requires one minimization for each piece p ∈ P ( i ⋆ (ˆ θ t )) , whileselective elimination requires only one for each active piece p ∈ P stp t − 1 ( i ⋆ (ˆ θ t )) .
Thus, the latterbecomes increasingly more computationally efficient as pieces are eliminated.
For completeness,we also analyze the variant, that we call full elimination , which updates the active pieces accordingto (5) for all answers i ∈
I at each round.
While we shall establish slightly better theoreticalguarantees for this rule, it is computationally demanding and, as we shall see in our experiments, itdoes not significantly improve sample complexity w.r.t.
selective elimination, which remains ourrecommended choice.
Let τ s .
elim = inf t ≥ 1 { t | P stp t ( i ⋆ (ˆ θ t )) = ∅} and τ f . elim :
= inf t ≥ 1 { t | ∃ i ∈ I : P stp t ( i ) = ∅} be thestopping times of selective and full elimination, respectively.
Intuitively, these two rules stop whenone of the updated answers has all its pieces eliminated (and return that answer).
We show that, as faras β t,δ is chosen to ensure concentration of ˆ θ t to θ , those two stopping rules are δ -correct.
Lemma 2.3 ( δ -correctness) .
Suppose that β t,δ guarantees (3) and that the algorithm verifies that,whenever it stops, there exists i ∅ ∈
I such that P stp τ ( i ∅ ) = ∅ and ˆ ı = i ∅ .
Then, P θ (ˆ ı ̸
= i ⋆ ( θ )) ≤ δ .
All proofs for this section are in Appendix D.
If an algorithm verifies the conditions of Lemma 2.3and has a sampling rule that makes it stop almost surely, then it is δ -correct.
Interestingly, we canprove a stronger result than δ -correctness: under the same sampling rule, the elimination stoppingrules never trigger later than the LLR one almost surely .
In other words, any algorithm equippedwith elimination stopping suffers a sample complexity that is never worse than the one of the samealgorithm equipped with LLR stopping.
Let τ llr :=
inf t ≥ 1 { t | inf λ ∈ Λ( i ⋆ (ˆ θ t ))
L t (ˆ θ t , λ ) ≥ β t,δ } .
Theorem 2.4.
For any sampling rule, almost surely τ f . elim ≤ τ s .
elim ≤ τ llr .
The proof of this theorem is very simple: if τ llr = t , then at t all pieces Λ p ( i ⋆ (ˆ θ t )) for p ∈ P ( i ⋆ (ˆ θ t ))can be eliminated, hence τ s .
elim ≤ t .
The proof that τ f . elim ≤ τ s .
elim follows from the observationthat full elimination always has less active pieces than selective elimination.
Note that all threestopping rules must use the same threshold β t,δ to be comparable.
Although simple, Theorem 2.4has an important implication: we can take any existing algorithm that uses LLR stopping, equip itwith elimination stopping instead, and obtain a new strategy that is never worse in terms of samplecomplexity and for which the original theoretical results on the stopping time still hold.
Finally, it is important to note that, while defining the elimination rule in the general form (5) allowsus to unify many settings, storing/iterating over all sets P stp t ( i ) would be intractable in problemswith large number of answers (e.g., top-m identification or thresholding bandits, where the latteris exponential in K ).
Fortunately, we show in Appendix B that this is not needed and efficientimplementations exist for these problems that take only polynomial time and memory.
2.1 Elimination time of alternative pieces
We now show that elimination stopping can indeed discard certain alternative pieces much earlierthat the stopping time.
While all results so far hold for any distribution and bandit structure, in theremaining we focus on Gaussian linear bandits.
Other distribution classes beyond Gaussians could beused with minor modifications (see Appendix C.2) but the Gaussian case simplifies the exposition.
Since most existing adaptive sampling rules target the optimal proportions from the lower bound of[4], we unify them under the following assumption.
Assumption 2.5.
Consider the concentration events
[Equation] [Equation]
A sampling rule is said to have low information regret if there exists a problem-dependent function
R ( θ, t ) which is sub-linear in t such that for each time t where E t holds,
[Equation]
The left-hand side of (7) can be understood as the information collected by the sampling rule at timet to discriminate θ with all its alternatives.
Therefore, Assumption 2.5 requires that information tobe comparable (up to a low-order term R ( θ, t ) ) with the maximal one from the lower bound.
In
Appendix F, we show that this is satisfied by both Track-and-Stop [4] and the approach in [15].
Let H p ( ω, θ ) :
= inf λ ∈ Λ p
( i ⋆ ( θ ))
(cid:80)
k ∈
[ K ] ω k KL k ( θ, λ ) , the information that sampling withproportions ω brings to discriminate θ from the alternative piece Λ p ( i ⋆ ( θ )) .
Note that
H ⋆
[Equation] [Equation]
For ϵ ≥ 0 , let Ω ϵ ( θ ) :
= { ω ∈ ∆ K |inf λ ∈ Λ( i ⋆ ( θ ))
(cid:80) k ω k KL k
( θ, λ ) ≥ H ⋆ ( θ ) − ϵ } be the set of ϵ -optimal proportions.
Theorem 2.6 (Piece elimination) .
The stopping time of any sampling rule having low informationregret, combined with LLR stopping, satisfies E [ τ ] ≤ ¯ t + 2 , where ¯ t is the first integer such that
[Equation]
When the same sampling rule is combined with elimination stopping, let τ p be the time at whichp ∈ P
( i ⋆ ( θ )) is eliminated.
Then, E [ τ p ] ≤ min { ¯ t p , ¯ t } + 2 , where ¯ t p is the first integer such that
[Equation]
with G ( θ, t ) = 0 for full elimination and G ( θ, t ) = 4 β t, 1 /t 2 + R ( θ
[Equation]
t
[Equation]
for selective elimination.
First, the bound we obtain on the elimination time of pieces in P ( i ⋆ ( θ )) is not worse than the boundwe obtain on the stopping time of LLR stopping.
Second, with elimination stopping, such eliminationscan actually happen much sooner.
Intuitively, sampling rules with low information regret play armswith proportions that are close to the optimal ones.
If all of such “good” proportions provide largeinformation for eliminating some piece p ∈ P ( i ⋆ ( θ )) , then p is eliminated much sooner than theactual stopping time (which requires eliminating the worst-case piece in the same set).
While both elimination rules are provably efficient, with full elimination enjoying slighly betterguarantees 3 , selective elimination provably never worsens (and possibly improves) the computational complexity over LLR stopping.
In all applications we consider, implementing LLR stoppingrequires one minimization for each of the same alternative pieces we use for elimination stopping.
Therefore, the total number of minimizations required by LLR stopping is (cid:80) τ llr t =1 |P
( i ⋆ (ˆ θ t )) | versus(cid:80) τ s .
elim t =
|P stp t ( i ⋆ ( ˆ θ t ))
| for selective elimination.
The second is never larger since τ s .
elim ≤ τ llrby Theorem 2.4 and P stp t ( i ⋆ (ˆ θ t )) ⊆
P ( i ⋆ (ˆ θ t )) for all t , and much smaller if eliminations happenearly, as we shall verify in experiments.
In our linear BAI example we need to perform ( K − 1)minimizations at each step, one for each sub-optimal arm, in order to implement LLR stopping.
On the other hand, we need only |P
stp t ( i ⋆ (ˆ θ t )) | minimizations with selective elimination, one for eachactive sub-optimal arm, while full elimination takes (cid:80)
i ∈
[ K ] |P stp t ( i ) | to update all the sets.
Note that Theorem 2.6 does not provide a better bound on E [ τ ] for elimination stopping than for LLRstopping.
In fact, when evaluating the bound on E [ τ p ] for the worst-case piece in p ∈ P
( i ⋆ ( θ )) , werecover the one on E [ τ ] .
This is intuitive since the sampling rule is playing proportions that try toeliminate all alternative pieces at once.
The following result formalizes this intuition.
Theorem 2.7.
Suppose that we can write β t,δ = log 1 δ + ξ ( t, δ ) with lim δ
→ 0 ξ ( t, δ ) / log(1 /δ )
= 0 .
Then for any sampling rule that satisfies Assumption 2.5,
[Equation]
with lim δ → 0 f ( θ, δ ) / log(1 /δ )
= 0 .
Here τ elim can stand for either full or selective elimination.
See Appendix D.4 for f .
This result shows that when the sampling rule is tailored to the LLR stoppingrule, the expected LLR and elimination stopping times differ by at most low-order (in log(1 /δ ) )
terms.
As δ → 0 the two expected stopping times converge to the same value H ⋆ ( θ ) − 1 log(1 /δ ) ,which is the asymptotically-optimal sample complexity prescribed by the lower bound (1).
We showed that, for both elimination rules, some pieces of the alternative are discarded sooner thanthe stopping time, and that the overall sample complexity of the method can only improve over LLRstopping.
However, since the sampling rule of the algorithm was not changed, elimination does notchange the computational cost of each sampling step, only the cost of checking the stopping rule.
Elimination at sampling
We show how to adapt sampling rules in order to accommodate piece elimination.
There are tworeasons for doing this: first, adapting the sampling to ignore pieces that have been discarded couldreduce the sample complexity; second, the amount of computations needed to update the samplingstrategy is often proportional to the number of pieces and decreasing it can reduce the overall time.
We start from an algorithm using LLR stopping, for which we change the stopping rule as above.
Thesampling strategies that we can adapt are those that aggregate information from each alternative piece.
For example, in linear BAI, methods that mimic the lower bound allocation (1), like Track-and-Stop[4], LinGame [16], or FWS [17], and even LinGapE [14], all compute distances or closest pointsto each piece in the decomposition { λ | ϕ ⊤ j λ ≥ ϕ ⊤ i ⋆ (ˆ θ t ) λ } .
Eliminating pieces at sampling simplymeans omitting from such computations the arms that were deemed sub-optimal.
Algorithm 1 showshow Track-and-Stop [4] can be modified to incorporate elimination at sampling and stopping. Algorithm -and-Stop [4]: vanilla (left) and with selective elimination (right) while not stopped do for p ∈ P ( i ⋆ (ˆ θ t )) do ▷ stopping
L p,t = inf λ ∈ Λ p ( i ⋆ (ˆ θ t )) L t (ˆ θ t , λ ) end for if ∀ p ∈ P ( i ⋆ (ˆ θ t )) :
L p,t > β t,δ then STOP w t = arg max ω min p ∈P ( i ⋆ (ˆ θ t )) H p ( ω, ˆ θ t ) if ∃ k : N kt < √ t pull k t +1 = arg min
k N kt else pull k t +1 = arg min k ( N kt − tw kt ) end while while not stopped do Set P stp t ( i ⋆ ( ˆ θ t )) = P stp t − 1 ( i ⋆ ( ˆ θ t )) for p ∈ P stp t − 1 ( i ⋆ (ˆ θ t )) do ▷ stopping
L p,t = inf λ ∈ Λ p ( i ⋆ (ˆ θ t )) L t (ˆ θ t , λ ) if L p,t > β t,δ delete p from P stp t ( i ⋆ (ˆ θ t )) end for if P stp t ( i ⋆ (ˆ θ t )) = ∅ then STOP w t = arg max ω min p ∈P smp t ( i ⋆ (ˆ θ t ))
H p ( ω, ˆ θ t ) if ∃ k : N kt < √ t pull k t +1 = arg min k N kt else pull k t +1 = arg min k ( N kt − tw kt )
{+Update P smp t +
+}
( i ⋆ (ˆ θ t )) (Algorithm 3) end while
Similarly to elimination stopping, the idea is to maintain sets of active pieces at sampling P smp t ( i )for each i ∈ I .
Note that these are different from the ones introduced in Section 2 for the stoppingrule.
The set is updated at each step like we did for the stopping sets, but with a different thresholdα t,δ (see Appendix E for details). Additionally, we reset it very infrequently at steps t ∈ { ¯ t 2 j 0 } j ≥ 0 , where ¯ t 0 ≥ 2 .
Formally, let us define the helper sets ˜ P smp t ( i ) as ˜ P smp0 ( i ) :
= P ( i ) and˜ P smp t ( i ) :
= (cid:40) ˜
P smp t − 1 ( i ) ∩ P t ( i ; α t,δ ) if t / ∈ { ¯ t 2 j 0 } j ≥
P t ( i ; α t,δ ) otherwise , where P t was defined in (4).
Let t j :=
¯ t 2 j 0 be the time step at which the j -th reset is performedand j ( t ) :=
⌊ log 2 log ¯ t 0 t ⌋ be the index of the last reset before t .
We define P smp t ( i ) :
=
˜ P smp t ( i ) ∩˜
P smp t j ( t ) − 1 ( i ) , such that P smp t ( i ) is the intersection of all active pieces from the second-last reset up to t ,i.e., P smp t ( i )
= (cid:84) ts =
t j ( t ) − 1 P s ( i ; α s,δ ) .
Since the resets are very infrequent, this definition only dropsa small number of rounds from the intersection (less than √ t ).
The detailed procedure to update thesesets is summarized in Algorithm 3.
As before, we can instantiate both selective and full elimination. The reason for the resets is two-fold. First, they ensure that the algorithm stops almost surely asrequired by Definition 1.1.
In fact, without resets, it might happen with some small (less than δ )probability that pieces containing the true parameter are eliminated, in which case the sampling rulecould diverge. Second, they guarantee that the thresholds ( α s,δ ) ts
= t j ( t ) − 1 used in P smp t ( i ) are withina constant factor of each other. This is crucial to relate the LLR of active pieces at different times.
3.1 Properties
We consider a counterpart of Assumption 2.5 for sampling rules combined with piece elimination.
Assumption 3.1.
There exists a sub-linear (in t ) problem-dependent function R ( θ, t ) such that, foreach time t where E t (defined in Equation 6) holds,
[Equation]
Intuitively, the sampling rule maximizes the information for discriminating θ with all its alternativesfrom the sequence of active pieces ( P smp s − 1 ( i ⋆ ( θ )))
ts =1 .
We prove in Appendix F that the algorithmsfor which we proved Assumption 2.5 also satisfy Assumption 3.1 when their sampling rules arecombined with either full or selective elimination.
Theorem 3.2.
Consider a sampling rule that verifies Assumption 3.1 and uses either full or selectiveelimination with the sets P smp t .
Then, Assumption 2.5 holds as well.
Moreover, when using thesame elimination rule at stopping, such a sampling rule verifies Theorem 2.6, i.e., it enjoys the sameguarantees as without elimination at sampling.
The proof is in Appendix E. Theorem 3.2 shows that for an algorithm using elimination at samplingand stopping, we get bounds on the times at which pieces of Λ( i ⋆ ( θ )) are discarded from thestopping rule which are not worse than those we obtained for the same algorithm without eliminationat sampling.
This result is non-trivial.
We know that the sampling rule collects information todiscriminate θ with its closest alternatives, and eliminating a piece cannot make the resulting “optimal”proportions worse at this task.
However, it could make them worse at discriminating θ with alternativesthat are not the closest.
This would imply that the elimination times for certain pieces could actuallyincrease w.r.t.
not eliminating at sampling.
Theorem 3.2 guarantees that this does not happen:eliminating pieces at sampling cannot worsen our guarantees.
We shall see in our experiments thateliminating pieces in both the sampling and stopping rules often yields improved sample complexity.
Experiments
Our experiments aim at addressing the following questions: (1) how do existing adaptive strategiesbehave when combined with elimination at stopping and (when possible) at sampling?
How do theycompare with native elimination-based methods?
(2) What is the difference between selective andfull elimination?
(3) How do LLR and elimination stopping compare as a function of δ ?
We ran experiments on two bandit structures: linear (where d < K ) and unstructured (where K = dand the arms are the canonical basis of R d ).
For each of them, we considered 3 pure exploration problems: BAI, Top-m, and online sign identification (OSI) [9, 27], called thresholding bandits.
All experiments use δ = 0 .
and are averaged over 100 runs.
We combined adaptive algorithms which are natively based on LLR stopping with our eliminationstopping rules and, whenever possible, we extended their sampling rule to use elimination.
Theselected baselines are the following.
For linear BAI, LinGapE [14], LinGame [16],
Frank-Wolfe
Sampling (FWS) [17], Lazy Track-and-Stop (TaS) [24], XY-Adaptive [19], and RAGE [20] (the lattertwo are natively elimination based).
For linear Top-m, m-LinGapE [28], MisLid [29], FWS, Lazy
TaS 4 , and LinGIFA [28].
For linear OSI, LinGapE 5 , LinGame, FWS, and Lazy TaS. For unstructuredinstances linear algorithms are still applicable, and we further implemented LUCB [11], UGapE [8],and the Racing algorithm [18] for BAI and Top-m. We also tested an “oracle” sampling rule whichuses the optimal proportions from the lower bound.
Due to space constraints, we present only theresults on linear structures.
Those on unstructured problems can be found in Appendix G.
The firstexperiments use randomly generated instances with K = 50 arms and dimension d = 10 .
Comparison of elimination times.
We analyze how different adaptive algorithms eliminate pieceswhen combined with selective elimination at stopping.
To this purpose we focus on BAI, where thesets of pieces can be conveniently reduced to a set of active arms, those that are still likely to be theoptimal one.
Figure 1 (left) shows how the set of active arms evolves over time for the 5 adaptivebaselines.
Notably, many arms are eliminated very quickly, with most baselines able to halve the setof active arms in the first 3000 steps.
The problem size is quickly reduced over time.
As we shallsee in the last experiment, this will yield significant computational gains.
We further note that the“oracle” strategy, which plays fixed proportions, seems the slowest at eliminating arms.
The reason isthat the optimal proportions from the lower bound focus on discriminating the “hardest” arms, whilethe extra randomization in adaptive rules might indeed eliminate certain “easier” arms sooner.
Full versus selective elimination.
We combine the different algorithms with full and selectiveelimination, both at sampling and stopping.
Due to space constrains, Figure 1 (middle) shows theresults only for LinGame (see Appendix G for the others).
We note that full elimination seems fasterat discarding arms in earlier steps, as we would expect theoretically.
However, it never stops earlierthan its selective counterpart.
Moreover, its computational overhead is not advantageous.
Overall, weconcluded that our selective elimination rule is the best choice and we shall thus focus on it in theremaining.
Finally, we remark that combining the sampling rule with elimination (no matter of whattype) seems to discard arms faster in later steps, and could eventually make the algorithm stop sooner.
LLR versus elimination stopping.
We now compare LLR and elimination stopping as a functionof δ .
We know from theory that both stopping rules allow to achieve asymptotic optimality.
Hencefor asymptotically optimal sampling rules the resulting stopping times with LLR and eliminationshould tend to the same quantity as δ → 0 .
Figure 1 (right) , where we report the ratio between the
LLR stopping time and the elimination one for different algorithms, confirms that this is the case.
Some algorithms (LinGapE and FWS) seem to benefit less from elimination stopping than the others,i.e., they achieve smaller ratios of stopping times.
We believe this to be a consequence of their mostly“greedy” nature, while the extra randomization of the other algorithms might help in this aspect.
[Table]
Sample complexities and computation times.
We finally compare our baselines in all threeexploration tasks, in terms of sample complexity and computation time.
For this experiment, weselected a larger linear instance with K = 50 and d = 20 , randomly generated (see the protocolin Appendix G).
From the results in Table 1, we highlight three points.
(1) The computation timesof all adaptive algorithms decrease when using selective elimination stopping instead of LLR andfurther decrease when also using elimination at sampling.
In the case of Top-m (i.e., the hardestcombinatorial problem), most adaptive algorithms become at least twice faster with elimination atstopping and sampling instead of LLR.
(2) Elimination at sampling improves the sample complexityof all algorithms.
(3) For BAI, the natively elimination-based algorithm RAGE, which updates itsstrategy infrequently, is the fastest in terms of computation time but the slowest in terms of samples.
Adaptive algorithms using elimination achieve run times that are within an order of magnitude ofthose of RAGE, while outperforming it in terms of sample complexity by a factor 2 to 3.
Conclusion
We proposed a selective elimination rule, which successively prunes the pieces of the empiricalanswer, that can be easily combined with existing adaptive algorithms for general identificationproblems.
We proved that it reduces their computational complexity, it never worsens their samplecomplexity guarantees, and it provably discards certain answers early.
Our experiments on differentpure exploration problems and bandit structures show that existing adaptive algorithms often benefitfrom a reduced sample complexity when combined with selective elimination, while achievingsignificant gains in computation time.
Moreover, they show that selective elimination is overall better(in terms of samples vs time) than its full variant which repeatedly updates the pieces of all answers.
Interesting directions for future work include investigating whether better guarantees on the stoppingtime can be derived for algorithms combined with elimination as compared to their LLR counterparts,and designing adaptive algorithms which are specifically tailored for elimination.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] In appendix D and E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In appendix G.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]