Beyond the Best: Estimating Distribution Functionals in Infinite-Armed Bandits
Abstract
In the infinite-armed bandit problem, each arm’s average reward is sampled from an unknown distribution, and each arm can be sampled further to obtain noisy estimates of the average reward of that arm.
Prior work focuses on identifying the best arm, i.e., estimating the maximum of the average reward distribution.
We consider a general class of distribution functionals beyond the maximum, and propose unified meta algorithms for both the offline and online settings, achieving optimal sample complexities.
We show that online estimation, where the learner can sequentially choose whether to sample a new or existing arm, offers no advantage over the offline setting for estimating the mean functional, but significantly reduces the sample complexity for other functionals such as the median, maximum, and trimmed mean.
The matching lower bounds utilize several different
Wasserstein distances.
For the special case of median estimation, we identify a curious thresholding phenomenon on the indistinguishability between Gaussian convolutions with respect to the noise level, which may be of independent interest.
Introduction
In the infinite-armed bandit problem (Berry et al., 1997), at each time instance the learner can either sample an arm that has been previously observed, or sample from a new arm, whose average reward is drawn from an unknown distribution F .
The learner’s goal is to identify arms with large average reward, with the objective being achieving either small cumulative regret (Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013), or small simple regret (Carpentier and Valko, 2015).
This setting differs from the classical multi-armed bandit formulation as the number of observed arms is not fixed a priori and needs to be carefully chosen by the algorithm.
We consider the problem of estimating some functional g ( F ) of an underlying distribution F .
From this point of view, the classical infinite-armed bandit problem can be viewed as an online sampling algorithm to estimate the maximum of the distribution F .
Once we cast the infinite-armed bandit problem in this manner, it immediately suggests several additional questions.
For example, what about offline sampling algorithms?
Indeed, online sampling requires continual interactions with the environment which may be infeasible in certain applications, and recent work in online and offline reinforcement learning have demonstrated the significant value of both formulations (Rashidinejad et al., 2021;
Zhang et al., 2021; Schrittwieser et al., 2021).
Additionally, it is worth estimating functionals beyond the maximum: in many practical scenarios, including mean estimation in singlecell RNA-sequencing (Zhang et al., 2020) and Benjamini Hochberg (BH) threshold estimation in multiple hypothesis testing (Zhang et al., 2019), we are interested in the mean, median (quantile), or trimmed mean of the underlying distribution F .
The estimation of quantiles is similar to estimation of the BH threshold, as both depend on the order statistics of the underlying distribution.
Estimating the median or trimmed mean has further applications in robust statistic for instance, maintaining the fidelity of an estimator in the presence of adversarial corruption or outliers.
Another natural setting where such problems arise is in large-scale distributed learning (Son and Simon, 2012).
Here, a server / platform wants to estimate how much test-users like their newly released product.
Users return a noisy realization of their affinity for the product, and the platform can decide to pay the user further to spend more time with the product, to test it further.
For many natural objectives which are robust to a small fraction of adversarial users, e.g. trimmed mean, median, or quantile estimation, we see that our algorithm will enable estimation of the desired quantity to high accuracy while minimizing the total cost (number of samples taken).
Since sampling is expensive, it is critical to identify the optimal method to collect samples, and identify the improvements afforded by adaptivity.
For example, do online methods offer significant gains over offline methods?
Are the fundamental limits of estimating the median and trimmed mean different from that of the maximum?
In this paper we initiate the study of distribution functional estimation in both online and offline settings and obtain both information theoretic limits and efficient algorithms for estimating the mean, median, trimmed mean, and maximum.
We propose unified meta algorithms for both offline and online settings, and provide matching upper and lower boundsfor the sample complexity of estimating the aforementioned functionals in Table 1.
We also reveal new insights on the fundamental differences between the offline and online algorithms, as well as the fundamental differences between different functionals.
To determine these sharp statistical limits, we use the Wasserstein- 2 distance to upper bound the KL divergence in the offline setting, while the Wasserstein- ∞ distance is applied in the online setting instead.
This approach leads to valid sample complexity lower bounds for general functionals g , which turn out to be tight for estimating the mean and maximum.
However, a curious thresholding phenomenon, which is not captured by the previous approach and does not occur for the mean and maximum , appears in the median and trimmed mean analyses: the KL divergence does not change smoothly with the noise level and enjoys a phase transition after the noise level exceeds some threshold.
This phenomenon calls for different treatments under different estimation targets and could be of independent interest.
The rest of this paper is structured as follows.
In Section 1.1 we discuss the relevant literature.
We then formulate our distribution functional estimation problem in Section 2.
Our unified meta algorithms for the offline and online settings are presented in Section 3, where we show the sample complexity upper bounds.
We present information theoretic lower bounds proofs via Wasserstein distance for the online and offline settings in Section 4, and discuss a special thresholding phenomenon arising in median estimation in Section 5.
Section 6 concludes this work.
[Table]
Θ
[Table]
ε
[Table]
) Θ( ε
[Table]
No gain from online sampling
[Table]
Θ( ε − 3 ) ˜Θ( ε − 2 . 5 )
Holds for any quantile not on the boundary
Maximum Θ( ε −
[Table]
2+ β ) ) ˜Θ( ε − max( β, 2) )
Depends on the tail regularity β
Trimmed mean ˜Θ( ε − 3 ) ˜Θ( ε − 2 . 5 ) g ( F ) =
E { X | X ∈
[ F − 1 ( α ) , F − 1 (1 − α )] }
1.1 Related works
The field of multi-armed bandits has seen broad interest and utility since its formalization in 1985 (Lai et al., 1985).
Across clinical trials, multi-agent learning, online recommendation systems, and beyond (Lattimore and Szepesvári, 2020), multi-armed bandits have proven to be an excellent framework for modeling and solving complex tasks regarding exploration in an unknown environment.
In the classical multi-armed bandit setting we have a set of n distributions, where the player sequentially pulls one arm per round and observes a sample drawn from the associated reward distribution.
In the infinite-armed bandit setting (Berry et al., 1997), the average arm reward for each arm is sampled i.i.d.
from an unknown distribution, i.e., we have infinitely many available arms.
There are many possible objectives that can be formulated in this online learning problem, from cumulative/simple regret minimization (Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015;
Li and Xia, 2017) to identification tasks (for example identifying an arm whose average reward is ε close to the largest average reward) (Aziz et al., 2018; Chaudhuri and Kalyanakrishnan, 2017, 2019).
Many works have studied best-arm identification, and we now have essentially matching instance-dependent upper and lower bounds (Jamieson and Nowak, 2014; Kaufmann et al., 2016).
One could also use the average reward estimate of the identified best arm to estimate the maximum of the average reward distribution in the infinite-armed bandit setting (Carpentier and Valko, 2015;
Aziz et al., 2018; Chaudhuri and Kalyanakrishnan, 2017, 2019).
From a statistical perspective, the sample complexity in the offline setting is closely related to deconvolution distribution estimation (Cordy and Thomas, 1997; Wasserman, 2004; Hall and Lahiri, 2008; Delaigle et al., 2008; Dattner et al., 2011).
Nevertheless, these previous works mainly focus on the expected L2 difference between the underlying distribution function and its estimation.
This simplified setting does not allow for consideration of the trade-off inherent in our setting between the number of points and the (variable) number of observations per point.
Additionally, these past works did not calculate the specific sample complexity for more general functionals like quantile and trimmed mean.
Since the noise is treated as fixed and uniform, there has been no study of the online setting where adaptive resampling can enable dramatic sample complexity improvements.
In particular, the challenge is that we have noisy observations, which makes deriving lower bounds even in offline cases a significant challenge that has not been dealt with in the past, let alone analyzing the online case.
The dramatic performance gains afforded by adaptive resampling for functional estimation, combined with its lack of formal study, motivates the focus of this work.
Problem formulation
We are interested in estimating the distribution functional g ( F ) ∈ R of an underlying distribution with cumulative distribution function (CDF) F .
We study a class of indicator-based functionals g defined as follows.
Definition 1 (Indicator-based functionals) .
The functional g can be represented as
[Equation]
for some set S ( F ) , where X ∼ F .
The set S ( F ) is defined as follows:
[Equation]
We denote S ( F ) by S throughout this work when F is clear from context.
This class encompasses many natural functionals of interest, which we formulate in Table 2.
In Appendix A, we discuss extending our results to more general functionals, and show that our approach can extend to smooth reweighting functions h ( X ) and more complex sets S .
[Table]
As in the infinite-armed bandit setting, we only have access to noisy observations of samples drawn from the distribution with CDF F .
We can either choose to sample from a point X which we already have some noisy observations of, or draw a new point X from F .
We then observe Y =
X + Z , where Z ∼ N (0 , 1) is independent of everything observed so far.
In this paper we characterize the online and offline sample complexities of these problems, and in
Section 3 propose online and offline algorithms achieving them.
For ε > 0 and δ ∈ (0 , 1) , we call an estimator ˆ G an ( ε, δ ) -PAC approximation of g ( F ) if P ( | ˆ G − g ( F ) | > ε ) ≤ δ.
Offline and online algorithms 3.1 Offline estimation algorithms
We study a special class of offline algorithms, which uniformly obtain observations of the points following the underlying distribution.
To be precise, based on prior information regarding the distribution in question, F , it will choose an appropriate number of points n and number of samples per point m to obtain an ( ε, δ ) -PAC approximation of g ( F ) .
Specifically, the latent variables
X 1 , . . .
, X n are drawn from F , and our observations { Y i,j } mj =1 are drawn i.i.d.
from N ( X i , 1) , independently for each i .
For i ∈ [ n ] , denote ˆ X i = m − 1 (cid:80) mj
=1 Y i,j as the empirical mean of the observations for arm i .
Then, we can write ˆ X
i
=
X i
+
ˆ Z i where ˆ
Z i ∼ N (0 , 1 /m ) , independent across i .
Define S n ≜ { i : X ( ⌊ α 1 n ⌋ ) ≤ X i ≤ X ( ⌊ α 2 n ⌋ ) } as the set of arms relevant for estimating the functional g , and define our n sample estimate of g as g n ( ˆ X 1 , . . .
, ˆ X n ) ≜ | S n | − 1 (cid:80) i ∈
S n ˆ X i .
Here X ( i ) denotes the i -th order statistic, that is the i -th smallest entry in X 1 , . . .
, X n .
Then,
G n,m = g n ( ˆ X 1 , . .
, ˆ X n ) , where each X i has been sampled m times, serves as a natural estimator for g ( F ) from the noisy observations.
With this, we can state the following theorem:
Theorem 1 (Offline PAC sample complexity) .
An ( ε, δ ) -PAC offline uniform-sampling-based algorithm for estimating g ( F ) requires Θ( nm ) samples where n, m depend on ε, δ , the functional g , and information about F , with orderwise dependence on ε detailed in Table 3.
For the rest of this section, we discuss in greater detail our assumptions on the underlying distribution.
We defer the proofs and calculations for n and m to Appendix C, as well as discussion regarding the trimmed mean to Appendix B.
[Table]
ε .
3.1.1 Mean
To guarantee that the empirical mean is a good estimator for the true mean, we impose assumptions on the tail of the distribution F :
Assumption 1.
The distribution F satisfies Var X ∼ F [ X ] ≤ c .
Assumption 1 ensures that estimation of the mean of the distribution can be accomplished with finite samples.
The following proposition gives the sample complexity of the offline algorithm.
Proposition 1.
Suppose that Assumption 1 is satisfied.
By choosing m = 1 and n ≥ δ − 1 (1 + c ) ε − 2 , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Thus, the offline algorithm requires
O ( ε − 2 ) samples.
3.1.2 Median
For median estimation we require different assumptions than the mean, as listed below.
Assumption 2.
There exist constants c 1 , c 2 > 0 such that
• F ′ ( x ) ≥ c 1 for | x − median( F ) | ≲ ε .
• | F ′′ ( x ) | ≤ c 2 for | x − median( F ) | ≲ √ ε .
The first assumption ensures that the median of F is unique.
The second assumption precludes the distribution from being dumbbell-shaped (very little mass near the median), in which case estimating the true median is meaningless and can be arbitrarily difficult.
The following proposition gives a suitable choice of ( n, m ) for providing an ( ε, δ ) -PAC approximation of g ( F ) .
Proposition 2.
Suppose that Assumption 2 holds.
Then, by choosing m ≥ 4( c 2 + 1) / ( c 1 ε ) and n ≥ 28 log(1 /δ ) /
( c 1 ε ) 2 , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Thus, the offline algorithm requires O ( ε − 3 ) samples.
3.1.3 Maximum
For maximum estimation, we require an assumption on the tail of F as is common in the infinite-armed bandit literature.
Assumption 3.
There exist constants 0 < c 1 < c 2 and β > 0 such that
• 1 − F ( F − 1 (1) − t ) ∈
[ c 1 t β , c 2 t β ] , for all
≤ t ≲ ε .
This assumption is also known as the β -regularity of F around F − 1 (1) , see (Wang et al., 2008).
We present a suitable choice of ( n, m ) in the following proposition.
Proposition 3.
Suppose that Assumption 3 holds.
By choosing n ≥ c − 1 1 2 β ε − β log(2 /δ ) and m ≥ 4 ε − 2 log(2 n/δ ) , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Therefore, the offline algorithm requires O ( ε − β − 2 ) samples.
3.2 Online estimation algorithm
We now present our general algorithm (Algorithm 1), an elimination-based ( ε, δ ) -PAC algorithm that efficiently estimates g ( F ) , where g is a known input functional and F is an unknown distribution from which we are able to sample X i independently, and observe noisy observations Y i,j of X i .
In order to exploit the Bayesian nature of the problem, we analyze the algorithm in two parts.
First, we use the fact that our arms are drawn from a common distribution to find some n, m as in Theoremsuch that the plug-in estimator G n,m will be an ( ε/ 2 , δ/ 2) -PAC approximation of g ( F ) .
Second, we show that our adaptive algorithm is an ( ε/ 2 , δ/ 2) -PAC approximation of G n,m , but is able to accomplish this using significantly fewer than n
× m samples.
Notationally, we denote by ˆ µ ( r ) the estimated mean vector of all arms at round r , and denote the i -th entry of this vector by ˆ µ i ( r ) .
We have that with high probability each arm
’s mean estimate stays within its width b r = 2 − r confidence interval for each round r .
To analyze our algorithm, we denote µ uni1 , . . .
, µ uni n as the estimates of X 1 , . . .
, X n generated by the offline algorithm after sampling each arm m times.
Then, we see that for the offline algorithm the arms relevant for the estimation task and the corresponding n sample estimator are
[Equation]
Here S n indicates the arms that the offline algorithm believes are in S .
We show that our online algorithm is able to efficiently estimate the set S n as ˆ S n , determining whether or not arms are in
S n , sampling these arms in S n sufficiently, and returning a plug-in estimator.
By construction each arm is only pulled by the adaptive algorithm at most m times, as we know from the analysis of the offline algorithm that for the utilized n, m , if each arm is pulled m times then the output is an ( ε/ 2 , δ/ 2) -PAC estimate of g ( F ) .
Thus, the online algorithm’s objective is essentially emulating the output of the offline algorithm, for which it only needs to sample any arm at most m times.
Note that when α 1 ̸
= α 2 , we have many samples
X i that are within S , with | S n | ≥ ⌊ n ( α 2 − α 1 ) ⌋ .
In order to avoid issues of dependence, we discard all previous samples (as arms in ˆ S n will have been sampled different numbers of times), and see that since we have Θ( n ) arms in ˆ S n we can construct a sufficiently accurate estimate by sampling each arm in ˆ S n once.
Algorithmically, we denote this as obtaining one fresh observation and constructing ˜ µ i .
To upper bound the sample complexity of our algorithm, we see that each arm only needs to be sampled to determine whether it is in S n or not.
As we show in Appendix D, the number of samples
N ( i ) needed for point
X i satisfies
[Equation]
with probability at least 1 − δ/ 4 for all arms simultaneously, where ∂A denotes the boundary of a set A , Conv ( A ) denotes the convex hull of a set A , and dist ( X, A ) =
min a ∈ A
| X − a | .
In the limit as ε → 0 we show that µ uni ( ⌊ α 1 n ⌋ ) → F − 1 ( α 1 ) (similarly with µ uni ( ⌊ α 2 n ⌋ ) ).
This allows us to state the following theorem regarding the expected sample complexity of Algorithm 1 with respect to the distribution’s relevant set of values S rather than the estimated indices S n .
Theorem 2 (Meta algorithm) .
For a functional g satisfying Definition 1, Algorithm 1 provides an ( ε, δ ) -PAC estimate of g ( F ) with M samples when given the requisite inputs.
Here m and n are calculated as in Theorem 1, and the number of samples M required satisfies
[Equation]
The proof of this Theorem is deferred to Appendix D.
Evaluating this expression for different functionals under their corresponding assumptions yields the stated sample complexity upper bounds, as we show in Appendix D.3.
Lower bounds via Wasserstein distance
In this section we derive general lower bounds on the sample complexity of functional estimation for both offline and online algorithms, where two different Wasserstein distances play important roles.
These Wasserstein-based lower bounds yield tight results for mean and maximum estimation.
4.1 General lower bounds based on Wasserstein distance
A classical technique for proving minimax lower bounds is Le Cam’s two-point method (Le Cam et al., 2000): let F 1 and F 2 be two distributions with | g ( F 1 )
− g ( F 2 ) | ≥ 2 ε , and let p π,F 1 and p π,Fbe the probability distributions of all observations queried by policy π under the true population distributions F 1 and F 2 , respectively.
One version of Le Cam’s two-point lower bound (Tsybakov, 2009, Theorem 2.2) gives
[Equation]
Consequently, to construct a lower bound on the PAC sample complexity of estimating g ( F ) , it suffices to find the largest ε such that there exist F 1 , F 2 with | g ( F 1 ) − g ( F 2 )
| ≥ 2 ε while
D KL ( p π,F 1 ∥ p π,F 2 ) =
O (1) .
A key step in the above analysis is to upper bound the KL divergence D KL ( p π,F 1 ∥ p π,F 2 ) , which differs significantly between offline and online algorithms.
For offline algorithms, the learner samples n arms i.i.d.
from F with average rewards X 1 , · · · , X n ∼ F , and each arm is pulled m times with
Gaussian observations.
Consequently, p π,F ==
( F ∗ N (0 , 1 /m )) ⊗ n , where p ⊗ n denotes the n -fold product distribution and ∗ denotes the convolution operation.
The following lemma presents an upper bound on the KL divergence for offline algorithms.
Lemma 1.
For any offline algorithm π defined in Section 3.1, it holds that
[Equation]
where W 2 ( P, Q ) is the Wasserstein-2 distance defined as W 22 ( P, Q )
=
inf γ ∈ Γ E
( X,Y ) ∼ Γ [( X − Y ) 2 ] , with Γ being the class of all couplings between P and Q .
For online algorithms the distribution p π,F is no longer a product distribution as actions can depend on past observations.
As a result, the KL divergence becomes larger, but still enjoys an upper bound based on another Wasserstein distance.
Lemma 2.
For any online algorithm π which queries T samples, it holds that
[Equation]
where W ∞ ( P, Q ) is the Wasserstein- ∞ distance: W ∞ ( P, Q )
=
inf γ ∈ Γ esssup ( X,Y ) ∼ Γ | X − Y | , with Γ being the class of all couplings between P and Q .
As W 2 ( P, Q ) ≤ W ∞ ( P, Q ) , the upper bound of Lemma 2 is no smaller than that of Lemma 1, showing the stronger power of online algorithms.
The following corollary is then immediate from
Lemmas 1 and 2.
Corollary 2.1.
The sample complexity of ( ε, . 1) -PAC estimation of g ( F ) is
[Equation]
for offline algorithms, and is
[Equation]
for online algorithms.
In the remainder of this section, we show that Corollary 2.1 leads to tight lower bounds for mean and maximum estimations for both offline and online settings.
4.2 Lower bounds for mean estimation
Consider two distributions F 1 and F 2 which are Dirac masses supported on 1 / 2 − ε and 1 / 2 + ε , respectively.
Clearly W 2 ( F 1 , F 2 ) =
W ∞ ( F 1 , F 2 ) =
2 ε , which is the best possible as W 2 ( F 1 , F 2 ) ≥ | mean( F 1 ) − mean( F 2 )
| ≥ 2 ε .
Corollary 2.1 gives the following lower bounds.
Corollary 2.2.
The ( ε, .
1) -PAC sample complexity for mean estimation is Ω( ε − 2 ) for both offline and online algorithms.
4.3 Lower bounds for maximum estimation
For maximum estimation, the Wasserstein distances W 2 and W ∞ behave differently, as summarized in the following lemma.
Let F β be the class of densities satisfying Assumption 3.
Lemma 3.
For ε ∈ (0 , 1 / 2) , it holds that
[Equation]
Note that we have included another term D KL ( F 1 ∥ F 2 ) in Lemma 3 as it can provide a better lower bound than using W ∞ if β ≥ 2 , as D KL ( p π,F 1 ∥ p π,F 2 ) ≤ T · D KL ( F 1 ∥ F 2 ) always holds due to the data-processing inequality (i.e. assuming that all arm rewards are independent).
Consequently, we have the following corollary on the sample complexity of maximum estimation.
Corollary 2.3.
The ( ε, .
1) -PAC sample complexity for maximum estimation over F β is Ω( ε − ( β +2) ) for offline algorithms, and Ω( ε − max { β, 2 } ) for online algorithms.
Lower bounds via thresholding phenomenon
Although the Wasserstein distance-based approach in Section 4 provides general lower bounds for both offline and online algorithms, and these lower bounds are tight for mean and maximum estimation, sometimes this approach can be loose.
For example, Lemma 3 shows that using the W ∞ distance might be looser than using the original KL divergence for maximum estimation.
This section provides tight lower bounds for median estimation, revealing a curious thresholding phenomenon.
5.1 Thresholding phenomenon for offline algorithms
Let F denote the set of distributions satisfying Assumption 2.
To use Le Cam’s two-point method to prove lower bounds for offline algorithms for median estimation, the key quantity is:
[Equation]
Its inverse KL − 1 σ ( ε ) is referred to as the modulus of smoothness of the median with respect to the KL divergence under Gaussian convolution.
The Wasserstein-based approach to upper bound KL σ ( ε ) in Lemma 1 is the following: let W 2 ,σ ( ε ) be the counterpart of the above quantity with the KL divergence replaced by the Wasserstein-2 distance, Lemma 1 shows that
[Equation]
an upper bound decreasing continuously with σ , where the proof of the last identity is presented in the Appendix.
However, this upper bound is not tight, as shown in the following lemma.
Lemma 4.
For ε ∈ (0 , 1 / 4) , KL σ ( ε ) can be characterized as follows:
[Equation]
where θ ∈ (0 , 1 / 4) , κ ∈ N are arbitrary fixed parameters, and c, C 1 , C 2 , C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
Lemma 4 shows a thresholding phenomenon as follows: when σ increases from 0 to 1 , the quantity
KL σ ( ε ) stabilizes at Θ( ε 2 ) whenever σ ≲ ε 1 / 2 ; however, when σ exceeds this threshold slightly (i.e. σ ≳ ε 1 / 2 − θ for any constant θ > 0 ), this quantity immediately drops to o ( ε κ ) for every possible κ .
The main intuition behind this thresholding phenomenon is that, if σ =
O ( ε 1 / 2 )
, the “bandwidth” of F 1 − F 2 exceeds that of N (0 , σ 2 ) , and the convolution is effectively using N (0 , σ 2 ) as a Gaussian kernel (which preserves polynomials up to order 2 ) for smoothing F 1 − F 2 (which is second-order differentiable).
In contrast, when σ ≫ ε 1 / 2 , the “bandwidth” of F 1 − F 2 could be smaller than N (0 , σ 2 ) , and the convolution is effectively using F 1 − F 2 as a kernel (which could preserve polynomials up to any desired order) for smoothing N (0 , 1) (which is infinitely differentiable).
Approximation theory tells us that the latter approximation error could be much smaller than the former, leading to the thresholding phenomenon.
We remark that this phenomenon is not captured by using the W 2 distance.
This thresholding behavior has an important consequence for median estimation.
By Lemma 4 with σ = 1 / √ m , PAC learning requires that m =
Ω( ε 2 θ − 1 ) for any offline algorithm, as otherwise the
KL divergence could be made arbitrarily small.
When m is large enough, the first line of Lemmathen requires n = Ω( ε − 2 ) to result in a large KL divergence for PAC learning, which comes from the idendity that
[Equation]
Consequently, we have the following corollary for median estimation using offline algorithms.
Corollary 2.4.
Fix any θ > 0 .
The ( ε, .
1) -PAC sample complexity for median estimation is Ω( ε − 3+ θ ) for any offline algorithm.
5.2 Thresholding phenomenon for online algorithms
To prove the PAC lower bound for online algorithms, one first wonders if the same observation in
Lemma 4 could still work.
However, a close inspection of the proof reveals an issue: the optimizers ( F 1 , F 2 ) in the definition of KL σ ( ε ) are different under the regimes σ
= O ( ε 1 / 2 ) and σ = Ω( ε 1 / 2 − θ ) .
An online learning algorithm could first identify the right scenario and then choose a proper sample size to tackle the problem, and thus the above lower bound arguments break down.
To resolve this issue, we aim to choose a proper pair of distributions ( F 1 , F 2 ) with | median( F 1 ) − median( F 2 )
| ≥ 2 ε , and investigate the behavior of D KL ( F 1 ∗N (0 , σ 2 )
∥ F 2 ∗N (0 , σ 2 ))
as a function of σ with ( F 1 , F 2 ) fixed along the line .
The following lemma shows that, even for some fixed pair ( F 1 , F 2 ) , a similar thresholding phenomenon still holds for the KL divergence.
Lemma 5.
Fix any ε, θ ∈ (0 , 1 / 4) , and κ ∈ N .
There exist two distributions F 1 , F 2 ∈ F with | median( F 1 ) − median( F 2 )
| ≥ 2 ε , and
[Equation]
where c, C 1 , C 2 , C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
Compared with Lemma 4, Lemma 5 still shows a similar thresholding phenomenon for the KL divergence when σ ≫ ε 1 / 2 , but the KL divergence becomes larger for small σ due to the additional constraint that ( F 1 , F 2 ) is held fixed.
Under the choice of ( F 1 , F 2 ) in Lemma 5, each arm should be pulled at least Ω( ε 2 θ − 1 ) times, while Ω( ε θ − 3 / 2 ) arms need to be pulled in view of the first line.
The following theorem makes the above intuition formal.
Theorem 3.
The ( ε, .
1) -PAC sample complexity for median estimation is Ω( ε − 5 / 2+ θ ) for any fixed θ > 0 and any online algorithm.
The formal proof of Theorem 3 is more complicated and requires an explicit computation of the
KL divergence D KL ( p π,F 1 ∥ p π,F 2 ) .
We relegate the full proof to Appendix F.
This thresholding phenomenon of the noise level also applies to the case of trimmed mean, which is discussed further and an analogous result is proved in Appendix B.
Conclusion
In this work we formulated and studied offline and online algorithms for estimating functionals of distributions.
We developed unified algorithms for estimating the mean, median, maximum, and trimmed mean, providing sample complexity upper bounds.
We additionally proved information theoretic lower bounds in these settings, which show that our algorithms are optimal up to ε c where c is a fixed constant arbitrarily close to zero.
We used different Wasserstein distances to construct information theoretic lower bounds for mean and maximum estimation, and showed how fundamentally different techniques are required for median and trimmed mean estimation.
The lower bounds for median and trimmed mean estimation elucidate an interesting thresholding phenomenon of the noise level to distinguish two distributions after Gaussian convolution, which may be of independent interest.
Interesting directions of future work include extending our analysis to nonindicator-based functionals, such as the BH threshold and analyzing the limiting behavior as δ → ∞ .
A Extensions of the formulation
The formulation of the functional g can be extended in several different ways.
First, we note that we can extend the set S ( F ) to a finite union of disjoint closed intervals, i.e., S ( F ) = ∪ ki
=1 S i
( F ) , where S i ( F ) is a closed interval for i ∈ [ k ] .
This is because E [ X | S ( F )] can be estimated based on estimations of E
[ X | S i ( F )] via
[Equation]
Observe that this definition naturally extends to cases where the distribution is continuous, where the density of F at X can be substituted for P ( X ∈ S ( F )) for singleton sets S ( F ) .
We can also consider a more general class of functionals
[Equation]
where h is a differentiable function.
However, when we take the limit ε → 0 , we see that for any fixed distribution F and fixed function h the reweighting induced by h does not matter.
Assuming that we knew whether X i ∈ S ( F ) for each i , we would simply want to sample X i ∝ h ′
( X i ) ε − r for some r .
Since h is differentiable, this is simply reweighting by a constant factor, which does not show up in our ε dependence.
Thus, we can safely only consider the weighting functional h ( x ) = x , which retains the central elimination aspect of this setting (determining whether a point is relevant or not).
Loosely speaking, for any differentiable function h and smooth and compactly supported F , we have that in the limit as ε → 0 it degenerates to one of these settings.
B Results for trimmed mean
In this section, we present our upper and lower bound analysis for trimmed mean via both online and offline sampling algorithms.
B.1 Upper bound for offline algorithms
For trimmed mean, the following statements are assumed to hold:
Assumption 4.
There exist constants c 0 , c 1 , . . .
, c 5 such that
• (cid:82) x 2 dF ( x ) ≤ c 0 .
• F ′ ( x ) ≥ c 1 for | x − F − 1 ( α ) | ≲ ε and | x − F − 1 (1 − α ) | ≲ ε .
• | F (2) ( x ) | ≤ c 2 for | x − F − 1 ( α ) | ≲ √ ε and | x − F − 1 (1 − α ) | ≲ √ ε .
• F ′ ( x ) ≤ c 3 for | x − F − 1 ( α ) | ≲ ε and | x − F − 1 (1 − α ) | ≲ ε .
• max {| F − 1 ( α ) | , | F − 1 (1 − α ) |} ≤ c 4 , min {| F − 1 ( α ) | , | F − 1 (1 − α ) |} ≥ c 5 .
The first assumption is to ensure that the mean and variance of F is upper bounded, which is slightly stronger than the assumption for mean.
The second assumption is to ensure that the α and 1 − α quantiles of F is well-defined.
The third assumption ensures that the distribution has Lipschitzcontinuous density around the quantiles.
The forth assumption precludes the distributions which have lots of mass around the α and 1 − α quantiles.
The fifth assumption ensures that the α and 1 − α quantiles are upper-bounded and bounded away from 0 .
The following proposition gives the choice of ( n, m ) to obtain the ( ε, δ ) -PAC approximation of the trimmed mean.
Proposition 4.
Suppose that Assumption 4 holds.
Then, by choosing m ≥ C 1 ε − 1 log ε − 1 and n ≥ C 2 ε − 2 δ − 1 , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Here C 1 , C 2 are constants which can be expressed by c 0 , . .
c 5 .
Thus, the offline sampling algorithm takes overall
O ( ε − 3 log(1 /ε ))
samples.
B.2 Lower bounds for offline algorithms
Similar to the analysis for estimating median, we consider the following quantity
[Equation]
Analogously, we have the following bounds on the above quantity with respect to the magnitude of noise σ .
Lemma 6.
For ε ∈ (0 , 1 / 4) , the following characterization of KL σ ( ε ) holds as a function of σ :
[Equation]
where θ ∈ (0 , 1 / 4) , κ ∈ N are arbitrary parameters, and c, C 1 , C 2 ,
C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
In the same manner, we have the following corollary.
Corollary 3.1.
Fix any θ > 0 .
The ( ε, .
1) -PAC sample complexity for trimmed mean estimation is Ω( ε − 3+ θ ) for any offline algorithm.
B.3 Lower bounds for online algorithms
Analogous to the results for median, we start with the following lemma to give bounds of KL divergence between two distributions after the convolution.
Lemma 7.
Fix any ε, θ ∈ (0 , 1 / 4) , and κ ∈ N .
There exists two distributions F 1 , F 2 ∈ F with | g ( F 1 ) − g ( F 2 )
| ≥ 2 ε , and
[Equation]
where c, C 1 , C 2 , C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
We then show the lower bound for trimmed mean via online sampling algorithms.
Theorem 4.
Suppose that ε > 0 .
Denote F as the set of distributions satisfying Assumption 4.
Consider an online algorithm π with a fixed budget t which outputs ˆ G .
Then, for any θ ∈ (0 , 1 / 4) , there exists at least one distribution F ∈ F , such that
[Equation]
where c > 0 is a constant.
C Proofs of upper bounds for offline algorithms C.1 Mean
Here we present the proof of Proposition 1.
Proof.
Let X ∼ F and Z ∼ N (0 , 1 /m ) are independent random variables.
Then, we have
[Equation]
This implies that g ( F m ) =
g ( F ) for any m ≥ 1 .
Therefore, we can simply take m = 1 .
Then, we note that
[Equation]
This implies that Var ˆ X ∼ F 1 [ ˆ X ] ≤ c + 1 .
According to the Chebyshev inequality, we have
[Equation]
Therefore, by taking n ≥ δ − 1 ( c + 1) ε − 2 , we have
[Equation]
Hence, it takes mn
= O ( ε − 2 )
samples to provide an ( ε, δ ) -PAC approximation of g ( F ) .
C.2 Median
Consider the following conditions
(A1)
For x ∈ R , there exists c 1 , t 1 > 0 such that for all t satisfying 0 ≤ | t − x | ≤ t 1 , F ′ ( t ) ≥ c 1 .
(A2) For x ∈ R , there exists c 2 , t 2 > 0 such that
[Equation]
for x 1 , x 2 ∈ [ x − t 2 , x + t 2 ] .
We can view Assumption 2 as follows.
Let η = g ( F )
=
F − 1 (0 . 5) .
F satisfies (A1) with ( η, c 1 , t 1 ) and satisfies (A2) with ( η, c 2 , t 2 ) while t 1 ≳ ε and t 2 ≳ √ ε .
Denote ρ ( x ) =
F ′ ( x ) .
Let ρ m =
ρ ∗ φ 1 /m as the pdf of the distribution of ˆ X i .
We start with Lemma 8 to show that under suitable choice of m , F m also satisfies (A1).
Lemma 8.
Let η = g ( F ) .
Assume that F satisfies (A1) with ( η, c 1 , t 1 ) .
Suppose that m − 1 / 2 ≤ t 1 / 2 .
Then, F m satisfies (A1) with ( η, c 1 / 4 , t 1 ) .
Proof.
It is sufficient to show that for x ∈ [ η − t 1 , η + t 1 ] , ρ m ( x ) > c 1 / 4 .
As m − 1 / 2 < t 1 / 2 ,
[Equation]
Therefore, for x ∈ [ η, η + t 1 ] , we have
[Equation]
Similarly, for x ∈ [ η − t 1 , η ] , we have
[Equation]
This completes the proof.
To prove Proposition 2, we introduce the following proposition to give a point-wise bound on the difference between F ( x ) and F m ( x ) .
Proposition 5.
Suppose that F satisfies (A2) with ( x, c 2 , t 2 ) and (cid:112) 4 log(2 m − 1 / 2 ) m − 1 / 2 ≤ t 2 .
Then, we have
[Equation]
Proof.
With k = (cid:112) 4 log(2 m − 1 / 2 ) , we have
[Equation]
For | y | ≤ t 2 , as | F (2) ( x − y )
| ≤ c 2 , it follows that
[Equation]
Note that km − 1 / 2 = (cid:112) 4 log(2 m − 1 / 2 ) m − 1 / 2 ≤ t 2 , we have
[Equation]
This completes the proof.
We restate Proposition 2 as follows and present the proof.
Proposition 6.
Suppose that δ ∈ (0 , 1) .
Assume that (A1) holds at η with ( c 1 , t 1 ) and (A2) holds at η with ( c 2 , t 2 ) .
Suppose that t 1 ≥ ε/ 2 and t 2 ≳ √ ε .
Then, with m ≥ 4( c 2 +1) ε − 1 cand n ≥ 28 ε − 2 log δ − 1 c,
G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Proof.
Suppose that we use n points and m samples per point.
From our choice of m , we have
[Equation]
Let η m =
g ( F m ) and η = g ( F ) .
From Proposition 5, we have
[Equation]
From Lemma 8, we note that F m satisfies (A1) with ( η, c 1 / 4 , t 1 ) .
If | η σ − η | ≥ t 1 , then, we have
[Equation]
which leads to a contradiction.
Therefore, we have
[Equation]
This implies that | η − η m | ≤ ε/ 2 .
As ε ≤ t 1 / 2 , we note that F m satisfies (A1) with ( η m , c 1 / 4 , t 1 / 2) .
From the choice of n , according to Lemma 13, we have
[Equation]
Under the event | G n,m − η m | ≤ ε/ 2 , we have
[Equation]
This completes the proof.
C.3 Maximum
In this case, the estimator for the noiseless samples writes G n = max i ∈ [ n ] X n .
We first show that for sufficiently large n , F ( G n ) can be close to 1 .
Proposition 7.
Suppose that ε > 0 and δ ∈ (0 , 1) .
Then, for n ≥ ε − 1 log(2 /δ ) , we have P ( F ( G n ) ≥ 1 − ε ) ≥ 1 − δ/ 2 .
Proof.
Consider a fixed number of points n .
Note that G n = max i ∈ [ n ] X i .
Therefore, we have
[Equation]
Here we utilize that log(1 − ε ) ≤ − ε .
This completes the proof.
Then, based on the β -regularity of F , we show that G n can be close to g ( F ) when n is large.
Proposition 8.
Let ε > 0 and δ ∈
Denote η = g ( F ) .
Suppose that Assumption 3 holds.
Then, with n = c − 1 1 ε − β log(2 /δ ) points, we have P ( | G n − η | ≤ ε ) ≥ 1 − δ/ 2 .
Proof.
From Proposition 7, we note that
[Equation]
According to Assumption 3, F ( G n ) ≥ 1 − c 1 ε β implies that G n ≥ η − ε .
As G n = max i ∈ [ n ] X i ≤ η , this completes the proof.
We first choose n ≥ c − 1 1 ( ε/ 2) − β log(2 /δ ) .
From Proposition 8, this guarantees that P ( | G n − η | ≤ ε/ 2) ≥ 1 − δ/ 2 .
Then, by choosing m ≥ 4 ε − 2 log(2 nδ − 1 ) , we have
[Equation]
Here we utilize the tail bound of Gaussian distributions and the fact that X i − ˆ X i ∼ N (0 , 1 /m ) .
As G n = max i ∈ [ n ] X i and G n,m = max i ∈ [ n ] ˆ X i , conditioned on {| X i −
ˆ X i | ≤ ε/ 2 ,
∀ i ∈
[ n ] } , we have | G n − G n,m | ≤ ε/ 2 .
Therefore, it follows that
[Equation]
In summary, we have P ( | η − G n,m | ≤ ε ) ≥ P ( | G n − G n,m | ≤
[Equation] [Equation]
+ P ( |
[Equation]
n − η |
[Equation]
ε/ 2) − 1 ≥ 1 − δ .
C.4 Trimmed mean
Consider the following conditions
(B1)
There exists constant c 0 > 0 such that (cid:82) ∞
−∞ x 2 dF ( x ) ≤ c 0 .
(B2)
For x ∈ R , there exists c 1 , t 1 > 0 such that for all t satisfying 0 ≤ | t − x | ≤ t 1 , F ′ ( t ) ≥ c 1 .
(B3)
For x ∈ R , there exists c 2 , t 2 > 0 such that for x 1 , x 2 ∈ [ x − t 2 , x + t 2 ] ,
(B4) For x ∈ R , there exists c 3 , t 3 > 0 such that for all t satisfying 0 ≤ | t − x | ≤ t 3 , F ′ ( t ) ≤ c 3 .
(B5)
There exists constants c 4 , c 5 > 0 such that max {| F − 1 ( α ) | , | F − 1 (1 − α ) |} ≤ c 4 , min {| F − 1 ( α ) | , | F − 1 (1 − α ) |} ≥ c 5 .
We can view Assumption 4 as follows.
F satisfies (B1) with c 0 and (B5) with c 4 , c 5 .
At F − 1 ( α ) and
F − 1 (1 − α ) , F satisfies (B2) with ( c 1 , t 1 ) , satisfies (B3) with ( c 2 , t 2 ) and satisfies (B4) with ( c 3 , t 3 ) .
Here t 1 , t 3 ≳ ε and t 2 ≳ √ ε .
We first show that for n ≥ O ( ε − 2 ) , the empirical estimator of the trimmed mean from noiseless samples will be close to the trimmed mean.
Proposition 9.
Suppose that Assumption 4 holds.
Let δ ∈ (0 , 1) .
Suppose that ε > 0 is sufficiently small.
For
[Equation]
with probability at least 1 − δ , we have
[Equation]
We defer the proof to Appendix C.4.1.
Then, we prove for the noisy case.
Lemma 9.
Suppose that Assumption 4 holds.
Then, we have
[Equation]
where k = (cid:112) 2 log ( m/ 2) .
We leave the proof in Appendix C.4.2.
From the median proof, analogously, we also have
[Equation]
Then, we have the bound
[Equation]
Here we utilize that | x | F ′ m ( x ) is upper bounded.
Therefore, by choosing m =
O ( ε − 1 log(1 /ε )) , we have
[Equation]
By choosing ε sufficiently small,
F m also satisfies Assumption 4 with constants (2 c 0 , c 1 / 2 , 2 c 2 , 2 c 3 , 2 c 4 , c 5 / 2) .
Therefore, with n ≥ O ( ε − 2 δ − 1 ) , we have
[Equation]
This completes the proof.
C.4.1 Proof of Proposition
Proof.
For ξ > 0 , denote the event
[Equation]
Choose ξ < c 5 .
From Lemma 14, with n ≥ 4 log(4 /δ )
c − 2 1 ξ − 2 , we have P ( E ( ξ )) ≥ 1 − δ/ 2 .
Conditioned on E ( ξ ) , we note that n
[Equation]
i
[Equation] [Equation]
α )
[Equation]
sign
[Equation]
F
[Equation]
1 − α ))
ξ ]) , and n
[Equation]
[Equation]
i
[Equation]
We introduce the following lemma to show the convergence of trimmed mean.
Lemma 10.
Let a < b .
Then, with n ≥ c 0 ε − 2 δ − 1 , we have
[Equation]
Proof.
Denote Y i
=
X i I
( X i ∈ [ a, b ]) .
Then, E [ Y i ] =
(cid:82) b a xdF ( x ) .
Note that Var[ Y i ] ≤ (cid:82) b
a x 2 dF ( x ) ≤ (cid:82) ∞
−∞ x 2 dF ( x ) ≤ c 0 .
Therefore, from the Chebyshev inequality, we have
[Equation]
This completes the proof.
From Lemma 10, for n ≥ 16 c 0 ξ − 2 δ − 1 , with probability at least 1 − δ/ 4 , we have
[Equation]
and
[Equation]
Here we utilize that ξ ≤ c 5 ≤ c 4 and | xF ′ ( x ) | ≤ 2 c 3 c 4 around F − 1 ( α ) and F − 1 (1 − α ) .
Combining the above bound with (C.4.1) and (C.4.1), with probability at least 1 − 3 δ/ 4 , we have
[Equation]
Therefore, by letting ξ = 1 (2 α − 1)(4 c 3 c 5 +1) ε , we complete the proof.
C.4.2 Proof of Lemma
Proof.
Denote σ = 1 / √ m .
We also denote F σ =: F m .
According to the Cauchy-Schwartz inequality, we have
[Equation]
which implies that (cid:82) | x | dF ( x ) ≤ √ c 0 .
Let k > 0 be a constant.
Note that
[Equation]
It follows that (cid:90)
[Equation]
We note that (cid:82) ∞ k φ 1 ( x ) dx ≤ e − k 2 / 2 .
By taking k = (cid:112) − 2 log ( σ 2 / 2) , then, we have
[Equation]
∞
[Equation]
We can compute that
[Equation]
b a
[Equation]
In the following two lemmas
[Equation]
we show that both terms in the last line are upper bounded by O ( σ 2 ) .
Lemma 11.
We have the bound
[Equation]
Lemma 12.
We have the bound
[Equation]
The proofs are left in Appendix C.4.3 and C.4.4.
In summary, we have the bound
[Equation]
b
[Equation]
This completes the proof.
C.4.3 Proof of Lemma
We first upper-bound the LHS in (11) by the following parts:
[Equation]
For the term (10), as y ∈ [ a + kσ, b − kσ ] , we have [ − kσ, kσ ] ⊆
[ a − y, b − y ] , which implies that
[Equation]
Hence, we have
[Equation]
For the term (11), we note that
[Equation]
Here we utilize that for y ≥ b + kσ or y ≤ a − kσ , we have
[Equation]
For the term (12), we note that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Similarly, for the term (13)
[Equation]
we have the bound
[Equation]
C.4.4 Proof of Lemma
For the LHS in (12), we can decompose it into
[Equation]
(cid:12) (cid:12)(cid:12) (14)
[Equation]
[Equation]
[Equation]
For the term (14), we note that (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:90)
[Equation]
a
[Equation]
′ ( y ) (cid:32)(cid:90)
[Equation]
Here we utilize that for y ∈ [ a + kσ, b − kσ ] ,
[Equation]
We can bound the term (15) by
[Equation]
Here we utilize that for y ≤ a − kσ or y ≥ b + kσ ,
[Equation]
For the term (16), we note that
[Equation]
Here we utilize that for sufficiently small σ such that 2 kσ ≤ b − a ,
[Equation]
Similarly
[Equation]
we can bound the term (17) by
[Equation]
This completes the proof.
C.5 Auxiliary results
We introduce the following auxilary lemmas to extend the median results to quantile
[Equation]
Lemma 13.
Let α ∈ (0 , 1) .
Suppose that 0 < ε < min { α, 1 − α } / 5 and δ ∈ (0 , 1) .
For n ≥ 4 ε − 2 log(2 /δ ) points, with probability at least 1 − δ , ˆ η = X
( ⌊ αn ⌋ ) satisfies | F (ˆ η
[Equation]
− α | ≤ ε
[Equation]
Proof.
Assume that ε < min { α, 1 − α } / 5 .
Consider the random variable Z i = 1 if F ( X i ) ≤ α − ε and 0 otherwise.
Let Z = (cid:80) ni
=1 Z i .
By the Chernoff bound,
[Equation]
On the other hand, consider the random variable Z ′
i =
1 if F ( X i ) ≥ α + ε and 0 otherwise.
Let
Z ′ = (cid:80) ni =1 Z ′ i .
According to the Chernoff bound,
[Equation]
In summary, we have
[Equation]
Therefore, by taking n = 4 ε − 2 log(2 /δ ) , we have P ( | F (ˆ η ) − α | ≤ ε ) ≤ δ .
This completes the proof.
Lemma 14.
Assume that F satisfies (B2)
At F − 1 ( α ) with ( c 1 , t 1 ) .
Suppose that ε ≤ min { t 1 , min { α, 1 − α } / (5 c 1 ) } and δ ∈ (0 , 1) .
With n ≥ 4 log(2 /δ )
c − 2 1 ε − 2 points, we have P ( | X ( ⌊ αn ⌋ ) − F − 1 ( α ) | ≤ ε ) ≥ 1 − δ .
Proof.
Note that c 1 ε ≤ c 1 t 1 ≤ 15 min { α, 1 − α } .
From Lemma 13, with n ≥ 4 log(2 /δ )
c − 2 1 ε − 2 , we have
[Equation]
Let η = F − 1 ( α ) and ˆ η = X
( ⌊ αn ⌋ ) .
If | ˆ η − η | > c 1 t 1 , as F satisfies (B2) at η with ( c 1 , c 3 , t 1 ) , we have
[Equation]
, which leads to a contradiction.
If | ˆ η − η | ≤ c 1 t 1 , then, in the same manner,
[Equation]
This implies that | X ( ⌊ αn ⌋ ) − F − 1 ( α ) | ≤ ε .
D Proofs of upper bounds for online algorithms
In this Appendix we provide the proof of Theorem 2.
As discussed, in order to exploit the Bayesian nature of the problem, we analyze the algorithm in two parts.
First, we use the fact that our arms are drawn from a common distribution to find some n, m as in Theorem 1 such that the plug-in estimator G n,m will be an ( ε/ 2 , δ/ 2) -PAC approximation of g ( F ) .
Second, we show that our adaptive algorithm is an ( ε/ 2 , δ/ 2) -PAC approximation of G n,m , but is able to accomplish this using significantly fewer samples.
We begin by proving the correctness of our algorithm, afterwards analyzing its sample complexity.
D.1 Correctness
To show correctness, we need to condition on the n × m matrix of observed samples Y , where
A i,j = X i + Z i,j , where Z i,j are i.i.d.
N (0 , 1) .
We couple the randomness in the analysis of the offline and online algorithms, considering our random arm pulls for both to be jointly generated, and the same matrix Y fed into each algorithm.
Analyzing the online algorithm, we show that it recovers the result of the offline sampling algorithm within error ε/ 2 with probability at least 1 − δ/ 4 .
Notationally, let µ uni1 , . . .
, µ uni n be the estimates of samples of offline sampling algorithm and online sampling algorithms with given ( m, n ) .
Let N ( i ) be the number of samples for point X i from the online algorithm.
Defining g n as the n -sample version of the functional g , we proceed by showing that the output of our algorithm is close to the output of the n, m offline sampling algorithm, which is close to g ( F ) .
Concretely, for our algorithm output ˆ G , we have that
[Equation]
We see from the previous arguments regarding offline sampling that for n, m as selected, we have that
[Equation]
Now all that remains is to show that the second term is small.
We show that when α 1 = α 2 , the online algorithm exactly recovers the output of the offline sampling algorithm on the event that the confidence intervals hold.
When α 1 ̸ =
α 2 (the case of the trimmed mean), we show that our estimate ˆ
G is within ε/ 2 of G n,m with probability at least 1 − δ/ 4 on the event that the confidence intervals hold.
We begin by defining ξ 1 as the good event where our arms stay within their confidence intervals:
[Equation]
We give a lower bound on the probability of the good event ξ in the following lemma.
Lemma 15 (Confidence intervals) .
The event ξ 1 defined in (D.1) , where the confidence intervals of ˆ µ ri about µ uni i hold, satisfies P ( ξ 1 ) ≥ 1 − δ/ 4 .
Proof.
Recall that
[Equation]
With this choice of t r , we have
[Equation]
This completes the proof.
On this good event ξ 1 , we show that our online algorithm exactly recovers the partitioning of arms performed by the offline sampling algorithm.
That is, for a given matrix of observations Y overloading notation we can see that g n ( Y ) , the output of the offline sampling algorithm, satisfies
[Equation]
where S n is the set of relevant arms (i.e. those close to the boundary of S ).
Note that S n is a function of g .
In the following lemma, we show that the online algorithm correctly identifies the arms in this set.
Lemma 16.
On the event ξ 1 we have that ˆ S n is identical to S n .
The proof of this Lemma conditions on the good event where all confidence intervals hold, and shows that in this case the boundaries of ˆ S n stay accurate throughout the course of the algorithm, and no arms are spuriously eliminated.
Proof.
In this proof we focus on showing that ˆ S n correctly partitions those elements smaller than µ uni( ⌊ α 1 n ⌋ ) from those greater than this threshold.
Identical arguments hold for the analysis of µ uni( ⌊ α 2 n ⌋ ) , which together imply the correctness of ˆ S n .
In round r , we use ( r ) to represent the corresponding values before the sampling, for example, ˆ µ ( r ) .
Let t i be the round that the i -th arm is eliminated from the active set.
Suppose that the algorithm ends in T rounds.
We denote
[Equation]
From the definition of G ( r ) and L ( r ) , it is easy to observe that | L ( T +1)
| ≤ ⌊ α 1 n ⌋ and | G ( T +1)
| ≤ n − ⌊ α 1 n ⌋ .
We note that | U ( T + 1) | = 0 and this implies that | G ( T + 1) | = n − ⌊ α 1 n ⌋ and | L ( T + 1) | =
⌊ α 1 n ⌋ .
From the definition of L ( T + 1) , it consists of ⌊ α 1 n ⌋ points with minimal ˆ µ
i
( T + 1) , i.e.,
[Equation]
Conditioned on the good event ξ 1 , we have
[Equation]
Note that b T +1 = 0 .
Therefore, for arbitrary i ∈ L ( T + 1) and j ∈ G ( T + 1) , we have µ
[Equation]
Hence, the maximal element in { µ uni i }
i ∈ L ( T +1) is µ uni( ⌊ α 1 n ⌋ ) , which is the α 1 -th quantile of { µ uni i } ni =1 .
This also implies that
[Equation]
On the other hand, analogously, we note that
[Equation]
By combining the above two equations together, we completes the proof.
We now split our analysis into cases.
When α 1 = α 2 , we see that all arms in ˆ S n will be pulled exactly m times, and so for i ∈
ˆ S n we have that µ i ( r ) =
µ uni i for the final round r .
This implies that ˆ G = G n,m for this given Y .
When α 1 ̸ = α 2 , we have that some arms in the set ˆ S n have not been pulled m times; they were determined to be in S n using fewer samples, and removed from the active set as they did not require further sampling.
Thus, we will not have that ˆ G = G n,m .
Instead, we show that because there are so many points in ˆ S n , by sampling each of them only once and averaging the results, we obtain ˆ G which is within ε/ 2 of G n,m with probability at least 1 − δ/ 4 .
Lemma 17.
On the event ξ 1 , Algorithm 1 satisfies P ( | G n,m − ˆ G | ≥ ε/ 2) ≤ δ/ 4 .
Proof.
If α 1 = α 2 then G n,m = ˆ G , and so the result holds trivially.
If α 1 ̸ = α 2 , then on the good event ξ 1 where our confidence intervals hold, we have that our online algorithm correctly identifies S n ( Y ) .
Then,
[Equation]
as this sum is normally distributed with variance decaying with S n , and so for sufficiently large n we have the desired result (as when α 1 ̸
= α 2 , | S n | ≥ ⌊ n ( α 2 − α 1 ) ⌋ ).
Thus, we see that our algorithm’s output ˆ G will be close to g ( F ) with high probability.
D.2 Sample complexity analysis
We now turn to bounding the sample complexity of our online algorithm.
For simplicity, we overload
S n in our analysis as S n = Conv ( { µ uni i : i ∈ S n } ) .
Useful in this analysis will be the distance from
X to the boundary of S n (essentially the gap of X ), which we define as
[Equation]
To this end, we provide the following Lemma:
Lemma 18.
On the good event ξ 1 , we have that a given arm X i will be pulled N ( i ) times where
[Equation]
Proof.
To begin, no arm can be pulled more than m times by our adaptive algorithm, due to the structure of t r .
We now additionally see that by the construction of our b r confidence intervals, we have that On the good event where our confidence intervals hold, we see that an arm X i must be eliminated when 2 b r ≤ dist ( X i , ∂S n ) .
Due to the iterative halving of b r , this means that arm i must be eliminated by round r where 4 b r ≥ dist ( X i , ∂S n ) , and so b − 2 r ≤ 16 [ dist ( X i , ∂S n )] − 2 .
This is to say, it cannot be pulled more than m times, and if it is far from the boundary of S n then it can be determined whether it is in the set or not using many fewer samples, only scaling with [ dist ( X, ∂S n )] − 2 .
Thus, the total sample complexity of our online algorithm (for a given matrix of observed samples Y with corresponding arm mean vector µ uni ) is upper bounded by [Equation]
[Equation]
  . We know by the Glivenko-Cantelli theorem that in the limit µ uni ( ⌊ α 1 n ⌋ ) → F − 1 ( α 1 ) , but we require finite sample rates to give a useful bound.
Lemma 19.
For n =
Θ( ε − 2 ) , with probability at least 1 − δ/ 8 in the randomness in Y , we have for i ∈ { 1 , 2 } that simultaneously | [Equation] uni [Equation]
⌊ α i n ⌋ )
[Equation]
F
[Equation]
Denote this event as ξ 2 .
Proof.
This lemma is simply a statement about the correctness of the offline sampling algorithm for estimating the α 1 / α 2 -th quantiles, which we have already proven.
Note that we are not conditioning on ξ 2 occurring in order for our algorithm to provide the correct output; we are simply utilizing this event to bound our algorithm’s sample complexity.
On the good event in Lemma 19 and the good event where our algorithm correctly outputs an ε accurate estimate and has sample complexity as in (18), we have that
[Equation]
(cid:17) [Equation] n
[Equation]
n ⌋ )
[Equation]
(cid:17)(cid:105)  ≤   m when dist ( X, ∂S n ) ≤ C (cid:113) log(1 /δ ) m , 1024log ( 16 n log m δ )
[ dist ( X i ,∂S )]when dist ( X, ∂S n ) > C (cid:113) log(1 /δ ) m , ≤ min   m, log (cid:16) n log m δ (cid:17) [ dist ( X i , ∂S )]
 , where C is an absolute constant and dist ( X i , ∂S ) is defined analogously as dist ( X i , ∂S ) =
min (cid:0) |
X i − F − 1 ( α 1 ) | , | X i − F − 1 ( α 2 ) | (cid:1) .
We then have that our sample complexity M is bounded as, conditioned on ξ 1 we have that
[Equation]
where we defined the good event ξ 2 as in Lemma 19 where our µ uni ( ⌊ α i n ⌋ ) are within (cid:112) log(1 /δ )
/m of their distributional values, i.e. F − 1 ( α i ) .
We utilize the fact that n ≥ m to simplify the sample complexity.
(a) comes from that P ( ξ c 2 ) ≤ δ/ 8 from Lemma 19 and that for events E with probability greater than 1 / 2 and
positive random variables
X , we have that E [ X | E ] ≤ 2 E [ X ] .
This gives us the desired result.
Theorem 5 (Restating Theorem 2) .
Algorithm 1 succeeds in estimating g ( F ) to within accuracy ε with probability at least 1 − δ , and requires at most
[Equation]
observations in expectation.
D.3 Functional-specific upper bounds
From Theorem 2, we are able to derive the upper bound sampling complexity of online algorithms in Table 1 for mean, median, maximum and trimmed mean estimation by analyzing (18) under the functional specific assumptions.
D.3.1 Mean estimation
Proof.
For mean estimation, from Theorem 1 we have that n =
Θ( ε − 2 ) and m = Θ(1) is sufficient.
Therefore, we have an expected sample complexity of
[Equation]
This completes the proof.
D.3.2 Median estimation
Proof.
For median estimation, from Theorem 1 we have that n =
Θ
( ε − 2 ) and m =
Θ( ε − 1 ) is sufficient.
We can compute that
[Equation]
−∞
[Equation]
+ O (1)
[Equation]
The first term can be bounded using integration by parts, where we note that
[Equation]
Here we utilize that F ′ ( x ) is upper bounded at F − 1 (0 . 5) − √ ε .
Similarly, we have
[Equation]
In summary
[Equation]
we have
[Equation]
and this implies that
[Equation]
This completes the proof.
D.3.3 Maximum estimation
Proof.
For maximum estimation, from Theorem 1 we have that n =
Θ( ε − β ) and m =
Θ( ε − 2 ) .
We note that
[Equation]
For β <
[Equation]
we can compute that
[Equation]
This implies that
[Equation]
As a result
[Equation]
we have
[Equation]
log /ε )) .
For β = 2 , we can compute that
[Equation]
Hence, we have
[Equation]
For β > 2 , we note that
[Equation]
Hence [Equation] we have
[Equation]
In summary, we have
[Equation]
, ∂S ) − 2 (cid:1)(cid:3)(cid:1) ≤
O ( ε − max { β, 2 } log(1 /ε )) .
This completes the proof.
D.3.4 Trimmed mean estimation
Proof.
For trimmed mean, we note that the analysis is similar to the case of median.
This gives that
O (cid:0) n log( n/δ ) E (cid:2) min (cid:0) m, dist ( X, ∂S ) − 2 (cid:1)(cid:3)(cid:1) ≤ O ( n log( n )
ε − 0 .
5 log(1 /ε ))
=
O ( ε − 2 .
5 log 2 (1 /ε )) .
E Proofs in SectionE.1 Proof of Lemma
Proof.
Denote p F 1 and p F 2 as the pdf of F 1 and F 2 respectively.
Let σ = 1 / √ m and let φ σ 2 be the pdf of N (0 , σ 2 ) .
As p π,F 1 = ( p F 1 ∗ φ σ 2 ) ⊗ n and p π,F 2 = ( p F 2 ∗ φ σ 2 ) ⊗ n , we have
[Equation]
On the other hand, we note that
[Equation]
Similarly, we have p F 1 ∗ φ σ 2 ( y ) =
E X ′ ∼ F 2 [ φ σ 2 ( y − X ′ )] .
Let γ ∈ Γ be a coupling of F 1 and F 2 .
Namely, it is a joint distribution of ( X, X ′ ) and its marginal distribution on X ( X ′ ) are F 1 ( F 2 ).
Then, utilizing the convexity of KL divergence, we have
[Equation]
By taking the infimum w.r.t.
all possible coupling γ , we note that
[Equation]
This completes the proof.
E.2 Proof of Lemma
Proof.
For a given underlying distribution F 1 and a given algorithm π , the joint distribution of { ( X i , A i , Y i ) } ti =1 has the following probability density function
[Equation]
Thus, we can also write
[Equation]
where
[Equation]
Thus, the marginal distribution on { ( a i , y i ) } ti =1 follows
[Equation]
Let F 2 be a distribution different from F 1 .
We want to bound the KL divergence from p π,F 1 ( { ( a i , y i ) } ti =1 ) to p π,F 2 ( { ( a i , y i ) } ti =1 ) .
Let γ ∈ Γ be a joint distribution with marginals F 1 and F 2 .
For simplicity, we write E γ =
E
( X i ,X ′ i ) ti =1 ∼ γ .
Utilizing the convexity of KL divergence, we note that
[Equation]
Given the pair of underlying states ( { X i } ti =1 , { X ′ i } ti =1 ) , we can compute that
[Equation]
Here we let C i ( { X j } tj =1 ) =
E { A i } ti =1 ∼ p π ( ·|{ X i } ti =1 ) (cid:104)(cid:80) t
j =1 I ( A j = i ) (cid:105) .
This implies that
[Equation] [Equation]
We note that (cid:80) ti =1 C i ( { X j } tj =1 ) =
t and
[Equation]
t
This implies that
[Equation]
By taking the infinimum w.r.t.
γ , we have
[Equation]
This completes the proof.
E.3 Proof of Lemma
Proof.
We first give the example for the Wasserstein-2 distance.
Let G ( s ) be defined as
[Equation]
where g ( s ) is a monotonic cubic interpolation satisfying that
[Equation]
We note that the image of G − 1 is [ ε, 1] .
Therefore, the domain of G is [ ε, 1] .
We also note that for s ∈ [0 , ε ] , we have |
[Equation]
Consider the following two distributions.
We consider a distribution with
[Equation]
in its support [
[Equation]
1] and another distribution with CDF
[Equation]
in its support [
[Equation]
1 − ε ]
[Equation]
We can verify that F 1 and F 2 satisfy Assumption 3 and | max( F 1 ) − max( F 2 ) | = ε .
We note that
[Equation]
We can verify that F 1 and F 2 satisfy Assumption 3 and | max( F 1 ) − max( F 2 ) | = ε .
The Wassersteindistance between F 1 and F 2 can be computed as
[Equation]
We then give the example for the Wasserstein- ∞ distance.
Consider a distribution with CDF
[Equation]
in its support [0 , 1] and another distribution with CDF
[Equation]
in its support [ ε, 1 + ε ] .
Let γ be the joint distribution of ( X, X + ε ) , where X follows F 1 .
Then, γ ∈ Γ is the coupling of F 1 and F 2 .
We can compute that esssup
[Equation]
X
[Equation]
X
[Equation]
) ∼ γ |
[Equation]
− X ′ | = ε.
This implies that W ∞ ( F 1 , F 2 ) ≤ ε .
Finally, we give the example for the KL divergence.
Consider two distributions with following CDFs:
[Equation]
We can verify that F 1 and F 2 satisfy Assumption 3 and | max( F 1 ) − max( F 2 ) | = ε .
We note that
[Equation]
Thus, according to the reverse Pinsker inequality, we have
[Equation]
We note that lim ε → 0 log ζ 1 − ζ
[Equation]
1 = lim
[Equation]
→ 0 − log
[Equation]
1 −
[Equation]
Therefore, we have
[Equation]
This implies that
[Equation]
This completes the proof.
F Proof of lower bounds for median estimation
We start with an auxiliary lemma to pointwisely bound the log-likelihood difference of two distributions.
Lemma 20.
Consider two densities supported on [ − 1 , 1] with pdf p ( x ) and q ( x ) such that p ( x ) , q ( x ) ≥ 1 / 4 and | p ( x ) − q ( x ) | ≤ ε · 1 ( | x | ≤ ζ ) for all x ∈ [ − 1 , 1] , and (cid:82) ζ − ζ x ℓ ( p ( x ) − q ( x ))
dx = 0 for all ℓ = 0 , · · · , k .
Then for σ ≤ 1 / 2 ,
[Equation]
where φ σ 2 is the density function of N (0 , σ 2 )
[Equation]
and C > 0 is an absolute constant
[Equation]
Proof.
Write h = p − q , then
[Equation]
where H ℓ ( x ) = ℓ ! · (cid:80) ⌊
[Equation] [Equation]
m
=
( − 1) m x
[Equation]
2 m m !
[Equation]
− 2 m )!
2 m is the Hermite polynomial, (c) uses its analytical form, and (a) uses its exponential generating function:
[Equation]
As for the step (b)
[Equation]
we use the assumed property of h to conclude that
[Equation]
On the other hand
[Equation]
to lower bound the denominator q ∗ ϕ σ 2 ( x )
[Equation]
we have the following observations: as σ ≤ 1 / 2 ,
[Equation]
Consequently, by Markov’s inequality,
[Equation]
A combination of the above inequalities leads to
[Equation]
and therefore the claimed result
[Equation]
Then, we introduce a lemma for constructing two distributions with matched moments.
Lemma 21.
Let ε > 0 .
For any k = 1 , 2 , . . .
, there exists a constant b > 0 and a function h ( x ) supported in [ −√ ε, √ ε ] such that
[Equation]
For i > k ,
[Equation]
We further have | h ( x ) | ≤ b √ ε and h ( x ) is b -Lipschitz continuous.
Here the constant b only depends on k and not on ε .
Proof.
Note that we only need to prove the lemma for ε = 1 , as h ( x ) = √ εh 0 ( x/ √ ε ) only properly scales the moments and preserves Lipschitzness.
For h 0 , consider the following form
[Equation]
] 0 , otherwise , where h 1 ( x ) is a polynomial taking the form
[Equation]
Let ( a 1 , . . .
, a k +2 ) be the unique solution to the following linear system:
[Equation]
Let b ≜ (cid:80)
k +2 i
=
i | a i | .
Then clearly h 1 (0)
=
h 1 (1) = 0 , | h 1 ( x ) | ≤ (cid:80) k
+2 i
=
| a i | ≤ b , and | h ′ 1 ( x ) | ≤ (cid:80) k
+2 i
=1 i | a i | ≤ b .
It remains to check the odd moments of h 0 (all even moments are zero by symmetry).
Specifically,
[Equation]
This completes the proof.
F.1 Proof of Lemma
For σ ≤ cε 1 / 2 , we consider two Gaussian distribution F 1 as the CDF of N (0 , 1) and F 2 as the CDF of N (3 ε, 1) .
Then, g ( F 2 ) − g ( F 1 ) = 3 ε and D KL ( F 1 ∥ F 2 )
=
O ( ε 2 ) .
From the data-processing inequality, we have
[Equation]
To show the lower bound Ω( ε 2 ) , without loss of generality we assume that F − 1 1 (0 . 5)
=
and
F − 1 2 (0 . 5) ≥ ε .
Proposition 5 and the density lower bound in Assumption 2 show that
[Equation]
Consequently, for σ ≤ cε 1 / 2 with a small constant c > 0 , it holds that
[Equation]
Therefore, Pinsker’s inequality gives
[Equation]
For σ ≥ ε 1 / 2 − θ , let F 1 be uniform on [ − 1 , 1] .
Then, g ( F 1 ) = 0 .
Clearly F 1 satisfies Assumption 2.
To construct F 2 , we take the construction of h ( x ) in Lemma 21 with k ≥ 2 κ/θ and support on [ − ε 1 , ε 1 ] , with ε 1 = bε/c 2 .
Here b is the Lipschitz constant in Lemma 21, and c 2 is the smoothness constant in Assumption 2.
The density of F 2 is then taken to be
[Equation]
As long as ε is sufficiently small, we have | p F 2 ( x ) − 1 / 2 | ≤ c 2 /b · b √ ε 1 ≤ 1 / 4 everywhere on x ∈ [ − 1 , 1] .
In other words, p F 2 ( x ) ∈ [1 / 4 , 3 / 4] on its support.
Moreover, p ′ F 2 ( x ) ≤ c 2 /b · b = c 2 .
This shows that F 2 satisfies Assumption 2 as well.
We first show that the median difference between F 1 and F 2 is at least ε .
In fact, by the density upper bound p F 2 ( x ) ≤ 3 / 4 , we have
[Equation]
Next we upper bound the KL divergence between Gaussian convolutions.
By choosing ε sufficiently small, we have ε 1 ≤ ε 1 − θ .
From Lemma 20 and the property of h ( x ) , we immediately have
[Equation]
F.2 Proof of Lemma
We construct the same pair of distributions ( F 1 , F 2 ) as in Lemma
[Equation]
and it suffices to prove that when σ ≤ cε 1 / 2 , we have
[Equation]
For the upper bound
[Equation]
we simply use the data-processing inequality:
[Equation]
For the lower bound, the same proof of Lemma 4 shows that D TV ( F 1 ∗ N (0 , σ 2 ) , F 2 ∗ N (0 , σ 2 ))
=
Ω( ε ) .
A naïve application of Pinsker’s inequality only leads to an Ω( ε 2 ) lower bound on the KL divergence.
A better lower bound is obtained by noticing that the signed measure ( F 1 − F 2 ) ∗N (0 , σ 2 ) is effectively supported on [ − Θ( √ ε 1 ) , Θ( √ ε 1 )] .
To this end, recall from the proof of Lemma 4 that
[Equation]
On the other hand, Proposition 5 tells that
[Equation]
Therefore, for σ ≤ cε 1 / 2 with a small enough c > 0 , we have
[Equation]
=Ω( ε )
[Equation]
Let p ( x ) and q ( x ) be the shorthands of p F 2 ∗ N (0 , σ 2 )( x ) and p F 1 ∗ N (0 , σ 2 )( x ) , respectively.
The
KL-divergence can be lower bounded as follows:
[Equation]
where
• (a) is due to the non-negativity of a log( a/b ) − a + b ≥ 0 ; • (b) follows from a log( a/b ) − a + b ≍ ( a − b ) 2 whenever a, b = Θ(1) .
The latter follows from | p ( x ) − p F 1 ( x ) |
= O ( σ ) =
O (1) from Proposition 5, and similarly for q ( x ) ; • (c) makes use of the Cauchy-Schwarz inequality.
This completes the proof.
F.3 Proof of Theorem
From Le Cam’s two-point lower bound, it is sufficient to show that the following proposition holds.
Proposition 10.
Suppose that ε > 0 .
Let F denote the set of distributions satisfying Assumption 2.
Consider an online algorithm π with a fixed budget t which outputs ˆ G .
Given the distribution F ∈ F of the underlying arms and the algorithm π , let p π,F
( { ( a i , y i ) } ti =1 ) denote the distribution of the action-observation pairs up to the t -th iteration.
Then, for any θ ∈ (0 , 1 / 4) , there exist F 1 , F 2 ∈ F with median g ( F 1 ) and g ( F 2 ) such that | g ( F 1 ) − g ( F 2 ) | ≥ ε and
D KL ( p π,F 1 ∥ p π,F 2 ) ≤ O ( ε 2 .
5 − θ t ) .
We start with a general log-sum inequality.
Lemma 22.
Suppose that p, q, K are probability density functions.
Then, we have the following inequality:
[Equation]
Then, we observe that the pair of distributions ( F 1 , F 2 ) constructed in the proof of Lemma 5 satisfies the following property.
Proposition 11.
Suppose that θ > 0 is a given constant.
Let F denote the set of distributions satisfying Assumption 2.
Then, there exists two distribution F 1 , F 2 ∈ F with median g ( F 1 ) and g ( F 2 ) such that g ( F 2 ) =
g ( F 1 ) + ε
and they satisfy that
[Equation]
Denote φ σ 2 as the pdf of N (0 , σ 2 ) .
For sufficiently small ε and for σ satisfying σ 2 ≥ ε 1 − θ , we further have
[Equation]
We then continue with the proof of Proposition 10.
Proof.
Consider two densities defined in Proposition 11 with the parameter θ .
Suppose that x 1 , . . .
, x t are i.i.d.
samples from either F 1 or F 2 .
Then, we note that a i ∈ N for i ∈ [ t ] and y i ∼ N ( x a i , 1) for i ∈ [ t ] .
For simplicity, we write a t =
( a 1 , . .
. , a t ) and y t
= ( y 1 , . . .
, y t ) .
Hence, we can write the probability distribution of ( a t , y t ) as follows
[Equation]
Let we write n j = (cid:80) i ≤ t 1 ( a i = j ) and ¯ y j = 1 n j (cid:80) i ≤ t,a i = j y i .
Then we can write
[Equation]
Here we denote
[Equation]
and
[Equation]
Therefore
[Equation]
we can write the log-likelihood ratio as
[Equation]
Then
[Equation]
we can compute that
[Equation]
For n j ≤ ε θ − 1 , from Proposition 11, we have
[Equation]
On the other hand
[Equation]
for n j ≥ ε θ − 1 , by utilizing Lemma
[Equation]
we note that
[Equation]
Therefore
[Equation]
we have
[Equation]
Note that
[Equation]
=
[Equation]
and t ≥
[Equation]
i
[Equation]
N
I
[Equation]
n
[Equation] [Equation]
The above inequalities imply that
[Equation] [Equation]
O (
[Equation] [Equation]
− θ t ) .
This completes the proof.
F.4 Proof of Lemma
Proof.
We note that the function f ( x ) = x log x is strictly convex.
Suppose that x ∈ R .
We note that
[Equation]
By the Jensen’s inequality
[Equation]
we have
[Equation]
This implies that
[Equation]
This completes the proof.
G Proof of Lower Bounds for trimmed mean estimation G.1 Proof of Lemma
Firstly, for σ ≤ Cε 1 / 2 , we consider two Gaussian distribution F 1 as the CDF of N (0 , 1) and F 2 as the CDF of N (3 ε, 1) .
Then, g ( F 2 ) − g ( F 1 ) = 3 ε and D KL ( F 1 ∥ F 2 )
= 9 ε 2 2 .
From the data-processing inequality, we have
[Equation]
To show the lower bound Ω( ε 2 ) , without loss of generality we assume that (cid:82)
[Equation]
xdF 1 ( x
[Equation]
=
[Equation]
and (cid:82) F − 1 2 (1 − α )
F
[Equation]
1 2 ( α ) xdF 2 ( x ) ≤ − ε .
Lemma 9 and the density lower bound in Assumption 4 show that
[Equation]
Here ˜ C > 0 is a constant.
Consequently, for σ ≤ cε 1 / 2 with a small constant c > 0 , it holds that
[Equation]
From the algorithm for trimmed mean estimation, we can distinguish F 1 ∗ φ σ 2 and F 2 ∗ φ σ 2 using
O ( ε − 2 ) samples.
This implies that D KL ( F 1 ∗ N (0 , σ 2 ) ∥ F 2 ∗ N (0 , σ 2 ))
≥ Ω( ε 2 ) .
For σ ≥ ε 1 − θ , let F 1 be uniform on [1 , 2] .
Without the loss of generality, we may assume that σ 2 ≥ 4 , c 1 ≤ 0 . 5 , c 3 ≥ 2 , c 4 ≥ 2 , c 5 ≤ 1 .
Then, F 1 satisfies Assumption 4.
By taking k ≥ 2 κθ , we can construct h ( x ) satisfying the conditions in Lemma 21 with ε 1 = 4 b ′ ε , where b ′ =
max { bc 2 , 4 } .
Consider the distribution F 2 with pdf p F 2 ( x ) =
p F 1 ( x ) + ( b ′ )
− 1 h ( x − 1 − α ) .
By choosing ε sufficiently small such that √ ε 1 ≤ min { α, 1 − 2 α } , the density function p F 2 is supported in [1 , 2] and F − 1 2 (1 − α )
=
2 − α .
As | h ( x ) | ≤ b √ ε 1 , for ε 1 ≤ 1 4 c 22 , we have ( b ′ ) − 1 h ( x ) ≤ c 2 b h ( x ) ≤ c 2 √ ε ≤ 1 2 .
This implies that p F 2 ( x ) ∈ [1 / 2 , 3 / 2] for x ∈ [1 , 2] .
Therefore, p F 2 is a density function.
Because p ′ F 1 ( x ) = 0 for x ∈ [1 , 2] , p F 2 ( x ) is c 2 -Lipschitz continuous in [1 , 2] .
Note that F 2 (1 + α )
=
F 1 (1 + α ) + ( b ′ )
− 1 (cid:82)
−∞ h ( x ) dx =
α − 4 ε .
Hence, it follows that
[Equation]
This implies that F − 1 2 ( α ) ≥ 1 + α + 23 · 4 ε ≥ 1 + α + 2 ε .
Therefore,
[Equation]
Here we utilize that b ′ ≥ 4 .
Note that (cid:82) F − 1 (1 − α ) F − 1 1 ( α ) xdF
[Equation]
This implies that
[Equation]
By choosing ε sufficiently small, we have ε 1 ≤ ε 1 − θ/ 2 .
From Lemma 20 and the property of h ( x ) , we immediately have
[Equation]
This completes the proof.
G.2 Proof of Lemma
We construct the same pair of F 1 and F 2 as in Lemma 6.
It is sufficient to prove that when σ ≤ cε 1 / 2 , [Equation]
For the upper bound, we simply use the data-processing inequality:
[Equation]
For the lower bound
[Equation] we note that F 2 (1 + α ) =
α − 4 ε = F 1 (1 + α ) − 4 ε .
Similar to the proof of Lemma [Equation] we have
[Equation]
Analogously, we can derive the same lower bound
[Equation]
σ 2 )) ≥ Ω( ε 3 / 2 ) .
G.3 Proof of Theorem
By applying the Le Cam’s two point lower bound, it is sufficient to show that the following proposition holds.
Proposition 12.
Suppose that ε > 0 .
Denote F as the set the set of distributions satisfying Assumption 4. Consider an online algorithm π with a fixed budget t which outputs ˆ G .
Given the distribution with CDF F ∈ F of the underlying arms and the algorithm π , let p π,F
( { ( a i , y i ) } ti =1 ) denote the distribution of the action-observation pairs up to the t -th iteration.
Then, for any θ ∈ (0 , 1 / 4) , there exist F 1 , F 2 ∈ F with trimmed means g ( F 1 ) and g ( F 2 ) such that | g ( F 1 ) − g ( F 2 ) | ≥ ε and
[Equation]
Similar to the proof of Proposition 10, we start with the following proposition.
Proposition 13.
Suppose that θ > 0 is a given constant.
Let F denote the set of distributions satisfying Assumption 4.
Then, there exists two distribution F 1 , F 2 ∈ F with trimmed mean g ( F 1 ) and g ( F 2 ) such that g ( F 2 ) ≤ g ( F 1 ) − ε
and they satisfy that
[Equation]
Denote φ σ 2 as the pdf of N (0 , σ 2 ) .
For sufficiently small ε and for σ satisfying σ 2 ≥ ε 1 − θ , we further have
[Equation]
Then, we present the proof of Proposition 12.
Proof.
Consider F 1 and F 2 as distributions constructed in Proposition 13.
Based on Propositionand Lemma 22, analogously, we have
[Equation]
Note that
[Equation]
This implies that
[Equation]
This completes the proof.
G.4 Proof of Proposition
Consider two distributions constructed in Lemma 6.
It is sufficient to show the bound on KL divergence and the pointwise bound.
Firstly
[Equation]
we note that F 1 is uniform on [1 , 2] and the density of F 2 only differs from the density of F 1 in [1 + α − √ ε 1 , 1 + α + √ ε 1 ] .
The difference is upper bounded by b √ ε 1 .
We note that for sufficiently small ε
[Equation]
we have ε 1 ≤ ε 1 − θ/ 2 .
Therefore, we have
[Equation]
We note that F 1 and F 2 have 2 k matched moments, where k > 6 /θ .
Analogous to the result in
Lemma 20, for all x ∈ R ,
[Equation]
Here C > 0 is an absolute constant.
This completes the proof.