Minimax Optimal Algorithms for Fixed-Budget Best Arm Identification
Abstract
We consider the fixed-budget best arm identification problem where the goal is to find the arm of the largest mean with a fixed number of samples.
It is known that the probability of misidentifying the best arm is exponentially small to the number of rounds.
However, limited characterizations have been discussed on the rate (exponent) of this value.
In this paper, we characterize the minimax optimal rate as a result of an optimization over all possible parameters.
We introduce two rates, R go and R go ∞ , corresponding to lower bounds on the probability of misidentification, each of which is associated with a proposed algorithm.
The rate R go is associated with R go -tracking, which can be efficiently implemented by a neural network and is shown to outperform existing algorithms.
However, this rate requires a nontrivial condition to be achievable.
To address this issue, we introduce the second rate R go ∞ .
We show that this rate is indeed achievable by introducing a conceptual algorithm called delayed optimal tracking (DOT).
Introduction
We consider K -armed best arm identification problem with T samples.
In this problem, each arm i ∈
[ K ] = { 1 , 2 , . .
, K } is associated with (unknown) distribution P i ∈ P for some class of distributions
P .
Upon choosing arm i , the forecaster observes reward X ( t ) , which is drawn independently from P i .
The forecaster then tries to identify (one of) the best arm
I ∗ = arg max i µ i with the largest mean µ ∗ =
max i µ i for µ i =
E X ∼ P
i
[ X ] .
The problem 2 is called the best arm identification (BAI, Audibert et al. (2010)) or the ranking and selection (R&S, Hong et al.
To this aim, the forecaster uses some algorithm that would adaptively choose an arm based on its history of rewards.
At each round t , the algorithm chooses one of the arms
I ( t ) ∈
[ K ] and receives the corresponding reward X ( t ) .
After the T -th round, the algorithm outputs a recommendation arm J ( T ) ∈
[ K ] , which corresponds to an estimator of the best arm.
The probability of misidentification is expressed by P [ J ( T ) / ∈
I ∗ ] , which will be referred to as the probability of the error (PoE) throughout the paper.
Best arm identification has two settings.
In the fixed confidence (FC) setting, the forecaster minimizes the number of draws
T until the confidence level on the PoE reaches a given value δ ∈ (0 , 1) .
In this case, T is a stopping time that can be chosen adaptively.
In the fixed-budget (FB) setting, the forecaster tries to minimize the PoE given a constant T .
In this paper, we shall focus on the FB setting.
In general, a good algorithm for the FC setting is very different from that for the FB setting.
To be more specific, an algorithm for the FC setting can be uniformly optimal.
Namely, several FC algorithms exist (Garivier and Kaufmann, 2016) that is able to adapt to each instance of distributions P
=
( P 1 , P 2 , . . .
, P K ) .
On the contrary, an algorithm for the FB setting requires consideration of the tradeoff in that improving the PoE for an instance P can compromise the PoE for another instance P ′ .
Thus, we must consider an optimization of the performance over all possible P ∈ P K .
See Appendix C for a demonstration of the asymptotic inconsistency between the two settings.
1. Minimax optimality in the fixed-budget setting
In the FB setting, the PoE decays exponentially to T as exp( − RT ) for some rate R > 0 .
The uniform optimality given above is no longer available here.
To demonstrate this, assume that we make an estimate of P based on the initial o ( T ) rounds, say, √ T rounds.
In this case, we can obtain the estimate of P that is ϵ -correct with probability exp( − ϵ 2 O ( √ T )) =
exp( − o ( T )) .
However, this estimation does not help to improve the rate of exponential convergence.
In other words, estimating P requires non-negligible (i.e. O ( T ) ) cost for exploration.
As a result, we cannot fully adapt the PoE to each instance P unlike the FC setting.
Instead, to discuss optimality in the FB setting, we must choose a complexity function H ( P ) , and the performance of an algorithm must be evaluated on the rate normalized by the complexity exp( − RT/H ( P )) .
In the literature, little is known about the optimal rate of the exponent.
Several papers considered what rate R is achievable for all P given a complexity function H ( P ) .
Kaufmann et al.
(2016) showed that this rate is larger than the corresponding rate of the FC setting.
Audibert et al.
(2010) proposed the successive rejects (SR) algorithm, which has the rate of 1 / (log K ) with the complexity function H 2 ( P ) :
= max i ∈
[ K ] i ∆ 2 i for ∆ i = max j µ j − µ i satisfying ∆ 1 ≤ ∆ 2 ≤ · · · ≤ ∆ K .
Carpentier and Locatelli (2016) showed a particular set of instances such that this rate matches the lower bound up to a constant factor.
However, the constant used there is by far loose 5 , and there is limited discussion on the actual rate of such algorithms.
1. Contributions
This paper tightly characterizes the optimal minimax rate of the PoE as a result of an optimization given P .
Let
H =
H ( P ) :
P K → R + ∪ {∞} be any complexity measure.
We then discuss the best possible rate
R > 0 such that the PoE is bounded by exp(
− RT/H ( P )+ o ( T )) for all P ∈ P K and make the following contributions
• We derive an upper bound on R (corresponding to a lower bound of the PoE), denoted by R go , which we obtain by considering a class of oracle algorithms that can determine the allocation of trials to each arm knowing the final empirical distribution after T rounds (Theorem 1).
• We propose an algorithm ( R go -tracking) that tracks this oracle allocation based on the current empirical distribution (Section 2.1).
Although this oracle allocation is expressed by a complicated minimax optimization, we propose a technique to learn this by a neural network and empirically confirm that the PoE of the learned algorithm is close to the lower bound (Sections 3 and 4).
We also discuss that the algorithm is
includes a
unlikely to provably achieve the bound even when the minimax problem is perfectly solved because of the impossibility of tracking.
• We tighten the PoE lower bound by weakening the oracle algorithms to obtain a new rate R go ∞ .
We propose the delayed optimal tracking (DOT) algorithm that asymptotically achieves this rate for Bernoulli and Gaussian arms.
While DOT is minimax, the algorithm is computationally almost infeasible (Sections 2.2 and 2.3).
In summary, we propose a nearly tight minimax lower bound rate on the PoE with a computationally feasible algorithm that is empirically close to this bound.
We also propose exact minimax rate and a matching algorithm in a computationally infeasible form.
Notation is listed in the Section A in the appendix.
1. Related work
Compared with the works of the fixed-confidence (FC) BAI, less is known about the fixedbudget (FB) BAI.
For example, a book on this subject (Lattimore and Szepesvári, 2020) spends only two pages on the FB-BAI.
6 Many algorithms designed for the FC-BAI, such as D-tracking (Kaufmann et al., 2016), do not have a finite-time PoE guarantee when we apply them to the FB setting.
Nevertheless, there are two well-known FB BAI algorithms:
Successive rejection (SR, Audibert et al.
and successive halving (SH, Shahrampour et al.
Both SR and SH decompose T time steps to a finite number of time segments, and then progressively narrow the candidate of the best arm at the end of each segment.
While SR discards one arm after each segment, SH discards half of the remaining arms after each segment.
SR and SH have the guarantee on the PoE of the rate exp ( − RT/H 2 ( P )) for some constant R > 0 .
Other FB BAI algorithms, such as UCB-E (Audibert et al., 2010) and
UGapE (Gabillon et al., 2012), require the knowledge of minimum gap min i ∆ i , and thus are not universal to all best arm identification instances.
Another literature on this topic is the ranking and selection (R&S) problems (Bechhofer and Sobel, 1954; McDonald, 1998; Powell and Ryzhov, 2018;
Hong et al., 2021).
Although the goal of R&S problems is to identify the best arm, many R&S papers do not consider the estimation error of P in a finite time.
As a result, algorithms therein do not have the guarantee on the PoE in the best arm identification setting.
The optimal computing budget allocation (OCBA, Chen et al. (2000); Glynn and Juneja (2004))
algorithm tries to minimize the PoE assuming the plug-in estimator matches the true parameter.
Bayesian
R&S algorithms try to solve the dynamic programming of minimizing the PoE marginalized by a prior (Bayesian PoE), which is computationally prohibitive, and thus approximated solutions have been sought (Frazier et al., 2008; Powell and Ryzhov, 2018).
Minimax optimal algorithm
In this section, we derive several lower bounds on the PoE and propose algorithms to empirically or theoretically achieve these bounds.
First, we formalize the problem.
Let P be a known class of reward distributions.
We consider the case where P is the set of Bernoulli distributions with mean Θ ⊂ [0 , 1] (including the case Θ = [0 , 1] ), or Gaussian distributions with mean in Θ ⊂ R (including the case Θ = R ) and known variance σ 2 > 0 .
It should be noted that many parts of the results in this paper can be generalized to much wider classes of distributions, but it makes the notation much longer and is discussed in Appendix D.
When we derive lower bounds and construct algorithms, we introduce Q as a class of distributions corresponding to the estimated distributions of the arms.
Namely, we set Q as the set of all Bernoulli (resp.
Gaussian) distributions with mean in [0 , 1] (resp.
R ) when P is the set of Bernoulli (resp.
Gaussian) distributions with mean in Θ .
As such, we take Q ⊃ P so that the estimator of P i is always in Q .
In these models, we identify the distribution
P i ∈
P with its mean parameter in Θ ⊂ R .
Our interest lies in the rate
lim T →∞ 1 T log(1 / P
[ J ( T ) / ∈
I ∗ ( P )]) of convergence of the PoE. Since we are interested in lower and upper bounds of the rate of algorithms including those requiring the knowledge of T , we define the rate for a sequence 7 of algorithms { π T } by
[Equation]
Here, a larger R ( { π T } ) indicates a faster convergence of the PoE.
2. PoE for oracle algorithms
First, we derive a lower bound on the PoE that is unlikely to be achievable but strongly related to an optimal algorithm.
Let D ( P ∥ Q ) =
E X ∼ P [ d P d Q ( X )] be the Kullback–Leibler (KL) divergence between P and Q .
In the case of Bernoulli distributions, D ( P ∥ Q ) = µ P log( µ P /µ Q )
+(1 − µ P ) log((
− µ P ) / (1 − µ Q )) where µ P , µ Q are the means of P, Q .
In the case of Gaussian distributions,
[Equation]
We then have the following bound.
Theorem 1.
Under any sequence of algorithm { π T } it holds that
[Equation]
where the outer supremum is taken over all functions r ( · ) :
Q K → ∆ K , J ( · ) : Q K → [ K ] .
All proofs are provided in the appendix.
This theorem states that under any algorithm there exists an instance P such that the PoE is at least exp( − TR go /H
( P ) + o ( T )) .
Intuitively speaking, the bound in Theorem 1 corresponds to the best possible rate of oracle algorithms that can determine the allocation as r = r ∗ ( Q ) ∈
∆ K knowing the final empirical mean
Q = Q ( T ) , where r ∗ ( · ) is the ( ϵ -)optimal 9 solution of Eq.
However, whether the rate R go or not by some algorithm is highly nontrivial.
In the actual trial, the algorithm can only know the empirical mean Q ( t − 1) at the beginning of the current round t ; we cannot ensure the achievability of the bound for oracle algorithms.
Despite this, one reasonable choice of the algorithm would be to keep tracking this optimal allocation r ∗ ( Q ( t − 1)) , expecting that the current empirical mean Q ( t − 1) is close to Q ( T ) .
R go -tracking in Algorithm 1 is the algorithm based on this idea.
Here, N i ( t − 1) is the number of times that the arm i is drawn at the beginning of the t -th round, and it draws the arm such that the current fraction of the allocation
N i ( t − 1) /
( t − 1) is the most insufficient compared with the estimated optimal allocation r ∗ ( Q ( t − 1)) .
As we will see in Section 4, the empirical performance of Algorithm 1 is very close to the
PoE lower bound stated above.
However, it is difficult to expect that this algorithm provably achieves this bound in general because of the following: We could prove that R go -tracking is optimal if the fraction of allocation always satisfies N i ( t ) /t =
r ∗ ( Q ( t ))
+ o (1) , that is, the algorithm is able to track the ideal allocation r ∗ ( Q ( t )) .
However, this does not hold in general.
For example, the empirical mean Q ( t ) sometimes changes rapidly in the Gaussian case.
Whilst such an event occurs with an exponentially small probability, the PoE itself is also an exponentially small probability and it is highly nontrivial to specify in which case the tracking failure probability becomes negligible.
Remark 1.
(Derivation of Theorem 1)
From a technical point of view, the main difference from the lower bound in the FC setting is that we also have to consider candidates for empirical distributions Q and true distributions P .
This makes the analysis much more difficult because a slight difference of the empirical distribution might (possibly discontinuously) affect the allocation unlike the difference of the true distribution P unknown to the algorithm.
A naive analysis only depending on the empirical distribution fails because of this discontinuity of the allocation.
To overcome this difficulty, we adopt a technique inspired by the typical set analysis that is often used in the field of information theory (Cover and Thomas, 2006).
We define the typical allocation for each candidate of empirical distribution Q and prove the theorem by evaluating the error probability based on the typical allocation.
Remark 2.
(Trivial solutions)
We can take arbitrary H ( P ) > 0 as a complexity measure, but R go can become zero if H ( P ) is not taken reasonably.
For example, if we take H ( P ) = (cid:80) i 1 / ∆ i (rather than (cid:80) i 1 / ∆ 2 i that is widely used), any positive rate is not achievable around a small gap min i ∆ i , and thus R go = 0 .
When R go =
any algorithm trivially satisfies
PoE ≤ exp( − TR go /H
( P ) + o ( T )) .
This means that any algorithm is minimax optimal in terms of
H ( P ) , that is, such a choice of H ( P ) gives meaningless results.
Remark 3.
(Empirical best arm) Eq.
(2) also involves the optimization of the recommendation arm J ( Q ) and r ( Q ) .
We can easily see that it is optimal to set J ( Q )
= i ∗ ( Q ) , that is, to take the empirical best arm as the recommendation arm when P = Q since R ( π ) otherwise becomes zero.
However, J ( Q ) = i ∗ ( Q ) might not hold for Q / ∈ P when P ⊊ Q .
Sections 2.2–2.4 that follow this section are for achieving the minimax optimal rate and are of theoretical interest.
A reader who is mainly interested in an actual implementation may skip these sections.
2. PoE considering trackability
To construct a provably optimal algorithm, we begin by refining the lower bound of the PoE by weakening the “strength” of the oracle algorithm.
We consider splitting T rounds into B batches of size ⌊ T/B ⌋ or ⌊ T/B ⌋ + 1 .
Let r B
=
( r 1 ( Q 1 ) , r 2 ( Q 1 , Q 2 ) , r 3 ( Q 1 , Q 2 , Q 3 ) , . . .
, r B ( Q 1 , . . .
, Q B )) be a sequence of B functions, where r b : Q Kb → ∆ K corresponds the allocation in the b -th batch when the empirical means of the first b batches are Q b =
( Q 1 , Q 2 , . . .
, Q b ) .
Based on this class of allocation rule, we have the following PoE lower bound.
Theorem 2.
(PoE Bound for batch-oracle algorithms)
Under any sequence of algorithms π T and B ∈ N ,
[Equation]
Here, the outer supremum is taken over all functions r B ( · ) =
( r 1 ( · ) , r 2 ( · ) , . . .
, r B ( · )) for r b ( · ) : Q Kb → ∆ K and J ( · ) : Q KB → [ K ] .
Theorem 1 is the special case of this theorem with B = 1 .
This bound corresponds to the best bound of oracle algorithms that can determine the allocation of the b -th batch knowing the empirical distribution of this batch.
It is tighter than that given in Theorem 1, as the oracle considered here cannot know the empirical distribution of the later batches b + 1 , b + 2 , . . .
, B .
It follows that we can obtain the following result.
Corollary 3.
We have R go B ≤ R go for any B ∈ N .
We will show that R go ∞ :=
lim B →∞ R go B exists and is the best possible rate.
input : ϵ -optimal solution r B, ∗ ( · ) =
( r ∗ 1 ( · ) , r ∗ 2 ( · ) , . .
, r ∗ B ( · ) , J ∗ ( · )) of Eq.
n b i ≥ r b,i ( T B − K ) is taken so
Update the stored empirical average as
[Equation]
where r b Q denotes the element-wise product.
2. Matching algorithm
In this section, we derive an algorithm that has a rate that almost matches R go B .
For any ϵ > 0 , let an ( ϵ -)optimal solution 11 of Eq.
(3) be ( r B, ∗ ( · ) , J ∗ ( · )) =
( r ∗ 1 ( · ) , r ∗ 2 ( · ) , r ∗ 3 ( · ) , . .
, r ∗ B ( · ) , J ∗ ( · )) with its objective at least
[Equation]
We cannot naively follow the allocation r ∗ b,i ( Q b ) because it requires the empirical mean of the current batch Q b , which is not fully available until the end of the current batch.
The delayed optimal tracking algorithm (DOT, Algorithm 2) addresses this issue.
This algorithm divides T rounds into B + K − 1 batches, where the b -th batch corresponds to ( bT B + 1 , bT B + 2 , . . .
, ( b + 1) T B ) -th rounds for T B =
T/
( B + K − 1) .
Here, for simplicity, we assume that
T is a multiple of B + K − 1 .
In the other case, we can reach almost the same result by just ignoring the last T − ( B + K − 1)
⌊ T/ ( B + K − 1) ⌋ rounds.
The crux of Algorithm 2 is to determine allocation r b by using the stored empirical mean
Q ′ 1 , Q ′ 2 , . . .
Q ′ B rather than the true empirical mean Q 1 , Q 2 , . . .
Q B + K − 1 : The first K batches are devoted to uniform exploration and the samples are stored in a queue (though this explanation is not strict, in that the actual procedure is done after taking the mean of the stored samples).
At the b -th batch for b ≥ K + 1 , we draw each arm i for n b,i ≈ T B r b,i times 12 , where r b = ( r b,i ) i ∈
[ K ] is determined based on the stored samples in the queue.
When drawing arm i for n b,i times, we dequeue and open n b,i stored samples instead of opening actual n b,i samples, the latter of which are enqueued and kept unopened.
By the nature of this algorithm, we can ensure the following property.
Lemma 4.
Assume that we run Algorithm 2.
Then, the following inequality always holds:
[Equation]
Lemma 4 states that the empirical divergence of DOT given in the LHS of Eq.
(4) almost matches the upper bound R go B /H ( P ) for sufficiently large B despite delayed allocation.
Using this property, we obtain the following bound
Theorem 5.
(Performance bound of Algorithm 2)
The PoE of the DOT algorithm satisfies
[Equation]
where T ′ =
T − ( B + K − 1) K and f ( K, B, T ) =
BK log(2 T ) .
The following corollary is immediate since f ( K, B, T ) =
o ( T ) holds for fixed K, B .
Corollary 6.
The worst-case rate of the DOT algorithm π DOT ,T satisfies
[Equation]
2. Optimality
In this section, we show that the rate R ( π DOT ) of DOT becomes arbitrarily close to optimal when we take a sufficiently large number of batches B .
Theorem 7.
(Optimality of DOT)
Assume H ( P ) be such that R go < ∞ .
Then, the limit
[Equation]
exists.
Moreover, for any η > 0 , there exist parameters B, ϵ such that the following holds on the performance of the DOT algorithm:
[Equation]
Remark 4.
( η -optimality)
Since R ( { π T } ) ≤ R go B holds for any sequence of algorithms { π T } and
B ∈ N , we have
[Equation]
from Eq.
Remark 4 states that the rate of the DOT algorithm given in Eq.
(6) is optimal up to η for arbitrary small η > 0 .
This essentially states that, no algorithm can be η -better than DOT in terms of the rate against the worst-case instance P .
In that sense, DOT is asymptotically minimax optimal.
Note that the discussion here first takes T → ∞ , and then takes a large
B , which implies that B should be taken as o ( T ) (i.e., each batch has a sufficiently large number of samples).
Remark 5.
(Utility of the DOT algorithm)
Although DOT (Algorithm 2) has an asymptotically optimal rate R go ∞ , it is difficult to calculate, or to even approximate, the optimal solution of Eq.
(3) since it is not an optimization of a finite-dimensional vector but an optimization of function r B , which has a high input dimension proportional to B .
In this sense, the
DOT algorithm as well as Theorem 7 is purely theoretical thus far, and the existence of a computationally tractable and provably optimal algorithm is an important open question.
Instead of pursuing the direction of batching, the subsequent sections rather focus on implementing Algorithm 1.
Learning
In this section, we propose a method to learn r ( Q ) of Eq.
(2) by utilizing a neural network to practically realize R go -tracking in Algorithm 1.
Throughout this section, we assume a class of algorithms that recommend the empirical best arm, which is guaranteed to be optimal when P = Q (see Remark 3).
This section focuses on Bernoulli arms, and with a slight abuse of notation, we use P i and Q i to denote the true mean and the empirical mean of arm i , respectively.
: learning rate η do /* Compute P min and Q min which minimizes the negative exponent */
Set E min ← ∞ .
for
. , N true do true random.
N emp do
Sample
∗ if then
P min ← P
Q min ← Q min ← E
Update parameters θ ← θ − η ∇ θ min once.
do
3. Learning allocation
Let r θ ( Q ) : Q K → ∆ K be a neural network with a set of parameters θ .
We consider alternately optimizing r θ ( · ) and the worst pair of instances ( P , Q ) , and we update θ via mini-batch gradient descent.
Given a complexity function H ( P ) , Eq.
(2) is defined as the minimum over all ( P , Q ) such that the best arm is different.
Our learning method (Algorithm 3) uses L mini-batches.
Let
[Equation] [Equation]
Given allocation r θ , Eq.
(7) is the negative log-likelihood (rate) of the bandit instance P against the empirical means Q .
At each batch, it obtains the pair P min , Q min such that
Eq.
(7) is minimized.
Specifically, for each iteration, we sample N true candidates of true means P uniformly from P K , then for each P , we sample N emp values of empirical means
Q ∈ Q K such that I ∗ ( Q )
∩ I ∗ ( P ) = ∅ uniformly at random.
3. Tracking by neural network
Having trained r θ , we propose the R go -Tracking by the neural network (TNN) algorithm (Algorithm 4), which is an implementation of R go -Tracking by the trained neural network.
This algorithm draws the arm such that the current fraction of samples
N i ( t − 1) /
( t − 1) is the most insufficient compared to the learned allocation r
θ ( Q ( t − 1)) .
Simulation
This section numerically tests the performance of the TNN algorithm.
14 We compared the performance of TNN (Algorithm 4) with two algorithms: Uniform algorithm, which samples each arm in a round-robin fashion, and Successive Rejects (SR, Audibert et al., 2010), where the entire trial is divided into segments before the game starts, and one arm with the smallest estimated mean reward is removed for each segment.
[Figure]
We consider Bernoulli bandits with K = 3 arms, where each mean parameter is in [0 , 1] .
In particular, we consider the three sets of true parameters: (instance 1) P = (0 . 5 , 0 .
45 , 0 . 3) , (instance 2)
P = (0 . 5 , 0 . 45 , 0 . 05)
[Figure]
and (instance 3) P = (0 . 5 , 0 . 45 , 0 . 45) .
The number of the rounds
T is fixed to
[Figure]
and we repeated the experiments for 10 5 times
[Figure]
4. Training neural networks
Here, we show experimental details for training neural networks for the TNN algorithm discussed in Section 3.2.
We used the complexity measure H 1 ( P ) =
(cid:80) i ̸
= i ∗ ( P ) ( P ∗ − P i ) − 2 as a standard choice of H ( P ) .
We used the neural network with four layers (including the input layer and output layer), where we used the ReLU for the activation functions and introduced the skip-connection (He et al., 2016) between each hidden layer to make training the network easier.
To obtain the map to ∆ K , we adopted the softmax function.
The number of nodes in the hidden layers was fixed to K × 3 .
We used AdamW (Loshchilov and Hutter, 2019) with a learning rate 10 − 3 and weight decay 10 − 7 to update the parameters.
For training the neural network, we ran Algorithm 3 with N true =
32 and N emp = 90 .
Furthermore, to allow the neural network to easily learn r , the elements of P =
( P 1 , P 2 , . . .
, P K ) were sorted beforehand.
4. Experimental results
Figure 1 illustrates the results of our simulations.
Each column corresponds to the result for each instance.
The first row ((a)–(c)) shows the PoE of the compared methods when the arm with the largest empirical mean is regarded as the estimated best arm J ( t ) at each round t .
Here, the black line represents exp( − t inf Q (cid:80) i r θ ,i ( Q ) D ( Q i ∥ P i )) , which corresponds to the exponent of the oracle algorithm that can perfectly track the allocation r θ ,i ( Q ) .
Therefore, the asymptotic slope of TNN cannot be better than that of the black line.
We can see from the figures that the slope of the TNN is close to the oracle algorithm and performs better than or comparable to the other algorithms.
Note that this is the result for fixed time horizon T .
Although the final slope of SR seems to be steeper than that of TNN, it is just due to the fact that SR is not anytime and is an algorithm that divides T rounds into several segments.
The second row ((d)–(f)) shows the tracking error of the TNN algorithm, which is defined as disc( t ) =
max i ∈
[ K ] | r i ( Q ( t ))
− N i ( t ) /t | , which measures the discrepancy between the ideal allocation r
i ( Q ( t )) and the actual allocation N i ( t ) /t .
If this quantity is o ( T ) in almost all tests (including those in which the algorithm failed to recommend the best arm) and in all instances, then we can guarantee R go =
R go ∞ .
The labels TNN (average), TNN (worst), and
TNN (average in fail) correspond to the average tracking error of all trials, the worst-case tracking error and the average tracking error of all failed trials, respectively.
The fact that ‘TNN (worst)’ is small at T = 2,000 implies that the gap between R go and R go ∞ is small, which supports the reasonableness of algorithms based on R go .
Conclusion
This paper considered the fixed-budget best arm identification problem.
We identified the minimax rate R go ∞ on the exponent of the probability of error by introducing a matching algorithm (DOT algorithm).
Optimization of rate R go ∞ is very challenging to implement, and we considered learning a simpler optimization problem of rate R go by using a neural network (TNN algorithm).
The TNN algorithm outperformed existing algorithms.
Several possible lines of future work include the following points.
• More scalable learning of r ( Q ) :
TNN adopted a neural network to obtain the oracle allocation r ( Q ) .
While its empirical results are promising and support our theoretical findings, the current experiment is limited to the case of K = 3 arms because the learning is very costly even for small K .
A more sophisticated learning algorithm is desired to realize R go -tracking for larger K .
• Learning a complexity measure H ( P ) :
We have assumed the complexity measure H ( P ) is given exogenously.
A principled way to choose H ( P ) is an interesting future work.
• Identifying the existence (or nonexistence) of the gap: Although the empirical results suggest that R go is very close (or maybe equal) to the optimal rate R go ∞ for the Bernoulli case, a formal analysis of this gap for general cases is demanded since the DOT algorithm to achieve R go ∞ is computationally almost infeasible.
A bound for another rate measure: we defined the worst-case rate by (1), which first takes the limit of T and then takes the worst-case instance P .
Another natural choice of the rate would be to exchange them, that is, to consider
[Equation]
Whereas Theorems 1 and 2 on the upper bounds of R ( π ) are still valid for R ′
( { π T } ) ≤
R ( { π T } ) , the current achievability analysis does not apply and analyzing the tightness of R go ∞ for R ′
( { π T } ) is an open problem.
Acknowledgement
The authors thank the reviewers and the associated editor for constructive discussion and suggested corrections.
To include the discussion during the review process, we have rewritten
Sections B and G.1.
We have also added Section C.
All errors in the paper are our own.
TT was supported by JST, ACT-X Grant Number JPMJAX210E, Japan and JSPS, KAKENHI
[-To be more specific, for example, even if we consider a ( δ -correct) algorithm that has a suboptimal sample complexity of 2 C conf ( P ) log δ − 
-]
[-+ o (log δ − 1 ) for some instance P , it is still impossible to achieve sample complexity better than C conf ( Q ) log δ − 
-]
Grant Number JP21J21272, Japan.
[-which implies r conf
-]
[-The black lines in Figure 1 (a)–(c) representing exp( − t inf Q (cid:80) i r θ ,i ( Q ) D ( Q i ∥ P i )) are computed by the grid search of Q with each Q i separated by intervals of 5 . 
-]
JH was supported by JSPS KAKENHI Grant Number
[-Let ϵ > 
-]
[-Let ϵ > 
-]
[-In batch B C + 1 , the algorithm draws arms in accordance with allocation r B C +
-]
JP21K11747.