Beyond the Best: Estimating Distribution Functionals in Infinite-Armed Bandits
Abstract
In the infinite-armed bandit problem, each arm’s average reward is sampled from an unknown distribution, and each arm can be sampled further to obtain noisy estimates of the average reward of that arm.
Prior work focuses on identifying the best arm, i.e., estimating the maximum of the average reward distribution.
We consider a general class of distribution functionals beyond the maximum, and propose unified meta algorithms for both the offline and online settings, achieving optimal sample complexities.
We show that online estimation, where the learner can sequentially choose whether to sample a new or existing arm, offers no advantage over the offline setting for estimating the mean functional, but significantly reduces the sample complexity for other functionals such as the median, maximum, and trimmed mean.
The matching lower bounds utilize several different
Wasserstein distances.
For the special case of median estimation, we identify a curious thresholding phenomenon on the indistinguishability between Gaussian convolutions with respect to the noise level, which may be of independent interest.
Introduction
In the infinite-armed bandit problem (Berry et al., 1997), at each time instance the learner can either sample an arm that has been previously observed, or sample from a new arm, whose average reward is drawn from an unknown distribution F .
The learner’s goal is to identify arms with large average reward, with the objective being achieving either small cumulative regret (Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013), or small simple regret (Carpentier and Valko, 2015).
This setting differs from the classical multi-armed bandit formulation as the number of observed arms is not fixed a priori and needs to be carefully chosen by the algorithm.
We consider the problem of estimating some functional g ( F ) of an underlying distribution F , as is illusrated in Figure 1.
From this point of view, the classical infinite-armed bandit problem can be viewed as an online sampling algorithm to estimate the maximum of the distribution F .
Once we cast the infinite-armed bandit problem in this manner, it immediately suggests several additional questions.
For example, what about offline sampling algorithms?
Indeed, online sampling requires continual interactions with the environment which may be infeasible in certain applications, and recent work in online and offline reinforcement learning have demonstrated the significant value of both formulations (Rashidinejad et al., 2021;
Zhang et al., 2021; Schrittwieser et al., 2021).
Additionally, it is worth estimating functionals beyond the maximum: in many practical scenarios, including mean estimation in single-cell RNA-sequencing (Zhang et al., 2020) and Benjamini Hochberg (BH) threshold estimation in multiple hypothesis testing (Zhang et al., 2019), we are interested in the mean, median (quantile), or trimmed mean of the underlying distribution F .
The estimation of quantiles is similar to estimation of the BH threshold, as both depend on the order statistics of the underlying distribution.
Estimating the median or trimmed mean has further applications in robust statistic for instance, maintaining the fidelity of an estimator in the presence of adversarial corruption or outliers.
Another natural setting where such problems arise is in large-scale distributed learning (Son and
Simon, 2012).
Here, a server / platform wants to estimate how much test-users like their newly released product.
Users return a noisy realization of their affinity for the product, and the platform can decide to pay the user further to spend more time with the product, to test it further.
For many natural objectives which are robust to a small fraction of adversarial users, e.g. trimmed mean, median, or quantile estimation, we see that our algorithm will enable estimation of the desired quantity to high accuracy while minimizing the total cost (number of samples taken).
Since sampling is expensive, it is critical to identify the optimal method to collect samples, and identify the improvements afforded by adaptivity.
For example, do online methods offer significant gains over offline methods?
Are the fundamental limits of estimating the median and trimmed mean different from that of the maximum?
[Equation]
[Figure]
In this paper we initiate the study of distribution functional estimation in both online and offline settings and obtain both information theoretic limits and efficient algorithms for estimating the mean, median, trimmed mean, and maximum.
We propose unified meta algorithms for both offline and online settings, and provide matching upper and lower boundsfor the sample complexity of estimating the aforementioned functionals in Table 1.
We also reveal new insights on the fundamental differences between the offline and online algorithms, as well as the fundamental differences between different functionals.
To determine these sharp statistical limits, we use the Wasserstein- 2 distance to upper bound the KL divergence in the offline setting, while the Wasserstein- ∞ distance is applied in the online setting instead.
This approach leads to valid sample complexity lower bounds for general functionals g , which turn out to be tight for estimating the mean and maximum.
However, a curious thresholding phenomenon, which is not captured by the previous approach and does not occur for the mean and maximum , appears in the median and trimmed mean analyses: the KL divergence does not change smoothly with the noise level and enjoys a phase transition after the noise level exceeds some threshold.
This phenomenon calls for different treatments under different estimation targets and could be of independent interest.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] No gain from online sampling
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Holds for any quantile not on the boundary
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Depends on the tail regularity
[Table]
[Equation]
The rest of this paper is structured as follows.
In Section 1.1 we discuss the relevant literature.
We then formulate our distribution functional estimation problem in Section 2.
Our unified meta algorithms for the offline and online settings are presented in Section 3, where we show the sample complexity upper bounds.
We present information theoretic lower bounds proofs via Wasserstein distance for the online and offline settings in Section 4, and discuss a special thresholding phenomenon arising in median estimation in Section 5.
Section 6 concludes this work.
1.1 Related works
The field of multi-armed bandits has seen broad interest and utility since its formalization in 1985 (Lai et al., 1985).
Across clinical trials, multi-agent learning, online recommendation systems, and beyond (Lattimore and Szepesvári, 2020), multi-armed bandits have proven to be an excellent framework for modeling and solving complex tasks regarding exploration in an unknown environment.
In the classical multi-armed bandit setting we have a set of n distributions, where the player sequentially pulls one arm per round and observes a sample drawn from the associated reward distribution.
In the infinite-armed bandit setting (Berry et al., 1997), the average arm reward for each arm is sampled i.i.d.
from an unknown distribution, i.e., we have infinitely many available arms.
There are many possible objectives that can be formulated in this online learning problem, from cumulative/simple regret minimization (Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015;
Li and Xia, 2017) to identification tasks (for example identifying an arm whose average reward is ε close to the largest average reward) (Aziz et al., 2018; Chaudhuri and Kalyanakrishnan, 2017, 2019).
Many works have studied best-arm identification, and we now have essentially matching instance-dependent upper and lower bounds (Jamieson and Nowak, 2014; Kaufmann et al., 2016).
One could also use the average reward estimate of the identified best arm to estimate the maximum of the average reward distribution in the infinite-armed bandit setting (Carpentier and Valko, 2015;
Aziz et al., 2018; Chaudhuri and Kalyanakrishnan, 2017, 2019).
From a statistical perspective, the sample complexity in the offline setting is closely related to deconvolution distribution estimation (Cordy and Thomas, 1997; Wasserman, 2004; Hall and Lahiri, 2008; Delaigle et al., 2008; Dattner et al., 2011).
Nevertheless, these previous works mainly focus on the expected L2 difference between the underlying distribution function and its estimation.
This simplified setting does not allow for consideration of the trade-off inherent in our setting between the number of points and the (variable) number of observations per point.
Additionally, these past works did not calculate the specific sample complexity for more general functionals like quantile and trimmed mean.
Since the noise is treated as fixed and uniform, there has been no study of the online setting where adaptive resampling can enable dramatic sample complexity improvements.
In particular, the challenge is that we have noisy observations, which makes deriving lower bounds even in offline cases a significant challenge that has not been dealt with in the past, let alone analyzing the online case.
The dramatic performance gains afforded by adaptive resampling for functional estimation, combined with its lack of formal study, motivates the focus of this work.
Problem formulation
We are interested in estimating the distribution functional g ( F ) ∈ R of an underlying distribution with cumulative distribution function (CDF) F .
We study a class of indicator-based functionals g defined as follows.
Definition 1 (Indicator-based functionals) .
The functional g can be represented as
[Equation]
for some set S ( F ) , where X ∼ F .
The set S ( F ) is defined as follows:
[Equation]
We denote S ( F ) by S throughout this work when F is clear from context.
This class encompasses many natural functionals of interest, which we formulate in Table 2.
In Appendix ?? , we discuss extending our results to more general functionals, and show that our approach can extend to smooth reweighting functions h ( X ) and more complex sets S . Functional [Table]
Mean
[Table]
As in the infinite-armed bandit setting, we only have access to noisy observations of samples drawn from the distribution with CDF F .
We can either choose to sample from a point X which we already have some noisy observations of, or draw a new point X from F .
We then observe Y =
X + Z , where Z ∼ N (0 , 1) is independent of everything observed so far.
In this paper we characterize the online and offline sample complexities of these problems, and in
Section 3 propose online and offline algorithms achieving them.
For ε > 0 and δ ∈ (0 , 1) , we call an estimator ˆ G an ( ε, δ ) -PAC approximation of g ( F ) if P ( | ˆ G − g ( F ) | > ε ) ≤ δ.
Offline and online algorithms 3.1 Offline estimation algorithms
We study a special class of offline algorithms, which uniformly obtain observations of the points following the underlying distribution.
To be precise, based on prior information regarding the distribution in question, F , it will choose an appropriate number of points n and number of samples per point m to obtain an ( ε, δ ) -PAC approximation of g ( F ) .
Specifically, the latent variables
X 1 , . . .
, X n are drawn from F , and our observations { Y i,j } mj =1 are drawn i.i.d.
from N ( X i , 1) , independently for each i .
For i ∈ [ n ] , denote ˆ X i = m − 1 (cid:80) mj[-=
-]
=1 Y i,j as the empirical mean of the observations for arm i .
Then, we can write ˆ X
i
=
X i
+
ˆ Z i where ˆ
Z i ∼ N (0 , 1 /m ) , independent across i .
Define S n ≜ { i : X ( ⌊ α 1 n ⌋ ) ≤ X i ≤ X ( ⌊ α 2 n ⌋ ) } as the set of arms relevant for estimating the functional g , and define our n sample estimate of g as g n ( ˆ X 1 , . . .
, ˆ X n ) ≜ | S n | − 1 (cid:80) i ∈
S n ˆ X i .
Here X ( i ) denotes the i -th order statistic, that is the i -th smallest entry in X 1 , . . .
, X n .
Then,
G n,m = g n ( ˆ X 1 , . .
, ˆ X n ) , where each X i has been sampled m times, serves as a natural estimator for g ( F ) from the noisy observations.
With this, we can state the following theorem:
Theorem 1 (Offline PAC sample complexity) .
An ( ε, δ ) -PAC offline uniform-sampling-based algorithm for estimating g ( F ) requires Θ( nm ) samples where n, m depend on ε, δ , the functional g , and information about F , with orderwise dependence on ε detailed in Table 3.
For the rest of this section, we discuss in greater detail our assumptions on the underlying distribution.
We defer the proofs and calculations for n and m to ?? , as well as discussion regarding the trimmed mean to ?? .
[Table]
3.1.1 Mean
To guarantee that the empirical mean is a good estimator for the true mean, we impose assumptions on the tail of the distribution F :
Assumption 1.
The distribution F satisfies Var X ∼ F [ X ] ≤ c .
Assumption 1 ensures that estimation of the mean of the distribution can be accomplished with finite samples.
The following proposition gives the sample complexity of the offline algorithm.
Proposition 1.
Suppose that Assumption 1 is satisfied.
By choosing m = 1 and n ≥ δ − 1 (1 + c ) ε − 2 , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Thus, the offline algorithm requires
O ( ε − 2 ) samples.
3.1.2 Median
For median estimation we require different assumptions than the mean, as listed below.
Assumption 2.
There exist constants c 1 , c 2 > 0 such that
• F ′ ( x ) ≥ c 1 for | x − median( F ) | ≲ ε .
• | F ′′ ( x ) | ≤ c 2 for | x − median( F ) | ≲ √ ε .
The first assumption ensures that the median of F is unique.
The second assumption precludes the distribution from being dumbbell-shaped (very little mass near the median), in which case estimating the true median is meaningless and can be arbitrarily difficult.
The following proposition gives a suitable choice of ( n, m ) for providing an ( ε, δ ) -PAC approximation of g ( F ) .
Proposition 2.
Suppose that Assumption 2 holds.
Then, by choosing m ≥ 4( c 2 + 1) / ( c 1 ε ) and n ≥ 28 log(1 /δ ) /
( c 1 ε ) 2 , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Thus, the offline algorithm requires O ( ε − 3 ) samples.
3.1.3 Maximum
For maximum estimation, we require an assumption on the tail of F as is common in the infinite-armed bandit literature.
Assumption 3.
There exist constants 0 < c 1 < c 2 and β > 0 such that
• 1 − F ( F − 1 (1) − t ) ∈
[ c 1 t β , c 2 t β ] , for all 0 ≤ t ≲ ε .
This assumption is also known as the β -regularity of F around F − 1 (1) , see (Wang et al., 2008).
We present a suitable choice of ( n, m ) in the following proposition.
Proposition 3.
Suppose that Assumption 3 holds.
By choosing n ≥ c − 1 1 2 β ε − β log(2 /δ ) and m ≥ 4 ε − 2 log(2 n/δ ) , the estimator G n,m is an ( ε, δ ) -PAC approximation of g ( F ) .
Therefore, the offline algorithm requires O ( ε − β − 2 ) samples.
3.2 Online estimation algorithm
We now present our general algorithm (Algorithm 1), an elimination-based ( ε, δ ) -PAC algorithm that efficiently estimates g ( F ) , where g is a known input functional and F is an unknown distribution from which we are able to sample X i independently, and observe noisy observations Y i,j of X i .
In order to exploit the Bayesian nature of the problem, we analyze the algorithm in two parts.
First, we use the fact that our arms are drawn from a common distribution to find some n, m as in Theoremsuch that the plug-in estimator G n,m will be an ( ε/ 2 , δ/ 2) -PAC approximation of g ( F ) .
Second, we show that our adaptive algorithm is an ( ε/ 2 , δ/ 2) -PAC approximation of G n,m , but is able to accomplish this using significantly fewer than n
× m samples.
Notationally, we denote by ˆ µ ( r ) the estimated mean vector of all arms at round r , and denote the i -th entry of this vector by ˆ µ i ( r ) .
We have that with high probability each arm
’s mean estimate stays within its width b r = 2 − r confidence interval for each round r .
To analyze our algorithm, we denote µ uni1 , . . .
, µ uni n as the estimates of X 1 , . . .
, X n generated by the offline algorithm after sampling each arm m times.
Then, we see that for the offline algorithm the arms relevant for the estimation task and the corresponding n sample estimator are
[Equation]
Here S n indicates the arms that the offline algorithm believes are in S .
We show that our online algorithm is able to efficiently estimate the set S n as ˆ S n , determining whether or not arms are in
S n , sampling these arms in S n sufficiently, and returning a plug-in estimator.
By construction each arm is only pulled by the adaptive algorithm at most m times, as we know from the analysis of the offline algorithm that for the utilized n, m , if each arm is pulled m times then the output is an ( ε/ 2 , δ/ 2) -PAC estimate of g ( F ) .
Thus, the online algorithm’s objective is essentially emulating the output of the offline algorithm, for which it only needs to sample any arm at most m times.
Note that when α 1 ̸
= α 2 , we have many samples
X i that are within S , with | S n | ≥ ⌊ n ( α 2 − α 1 ) ⌋ .
In order to avoid issues of dependence, we discard all previous samples (as arms in ˆ S n will have been sampled different numbers of times), and see that since we have Θ( n ) arms in ˆ S n we can construct a sufficiently accurate estimate by sampling each arm in ˆ S n once.
Algorithmically, we denote this as obtaining one fresh observation and constructing ˜ µ i .
To upper bound the sample complexity of our algorithm, we see that each arm only needs to be sampled to determine whether it is in S n or not.
As we show in ?? , the number of samples N ( i ) needed for point
X i satisfies
[Equation]
with probability at least 1 − δ/ 4 for all arms simultaneously, where ∂A denotes the boundary of a set A , Conv ( A ) denotes the convex hull of a set A , and dist ( X, A ) =
min a ∈ A
| X − a | .
In the limit as ε → 0 we show that µ uni ( ⌊ α 1 n ⌋ ) → F − 1 ( α 1 ) (similarly with µ uni ( ⌊ α 2 n ⌋ ) ).
This allows us to state the following theorem regarding the expected sample complexity of Algorithm 1 with respect to the distribution’s relevant set of values S rather than the estimated indices S n .
Theorem 2 (Meta algorithm) .
For a functional g satisfying Definition 1, Algorithm 1 provides an ( ε, δ ) -PAC estimate of g ( F ) with M samples when given the requisite inputs.
Here m and n are calculated as in Theorem 1, and the number of samples M required satisfies
[Equation]
The proof of this Theorem is deferred to ?? .
Evaluating this expression for different functionals under their corresponding assumptions yields the stated sample complexity upper bounds, as we show in ?? .
Lower bounds via Wasserstein distance
In this section we derive general lower bounds on the sample complexity of functional estimation for both offline and online algorithms, where two different Wasserstein distances play important roles.
These Wasserstein-based lower bounds yield tight results for mean and maximum estimation.
4.1 General lower bounds based on Wasserstein distance
A classical technique for proving minimax lower bounds is Le Cam’s two-point method (Le Cam et al., 2000): let F 1 and F 2 be two distributions with | g ( F 1 )
− g ( F 2 ) | ≥ 2 ε , and let p π,F 1 and p π,Fbe the probability distributions of all observations queried by policy π under the true population distributions F 1 and F 2 , respectively.
One version of Le Cam’s two-point lower bound (Tsybakov, 2009, Theorem 2.2) gives
[Equation]
D KL ( p π,F 1 ∥ p π,F 2 )) . Consequently, to construct a lower bound on the PAC sample complexity of estimating g ( F ) , it suffices to find the largest ε such that there exist F 1 , F 2 with | g ( F 1 ) − g ( F 2 )
| ≥ 2 ε while
D KL ( p π,F 1 ∥ p π,F 2 ) =
O (1) .
A key step in the above analysis is to upper bound the KL divergence D KL ( p π,F 1 ∥ p π,F 2 ) , which differs significantly between offline and online algorithms.
For offline algorithms, the learner samples n arms i.i.d.
from F with average rewards X 1 , · · · , X n ∼ F , and each arm is pulled m times with Gaussian observations.
Consequently, p π,F ==
( F ∗ N (0 , 1 /m )) ⊗ n , where p ⊗ n denotes the n -fold product distribution and ∗ denotes the convolution operation.
The following lemma presents an upper bound on the KL divergence for offline algorithms.
Lemma 1.
For any offline algorithm π defined in Section 3.1, it holds that
[Equation]
where W 2 ( P, Q ) is the Wasserstein-2 distance defined as W 22 ( P, Q )
=
inf γ ∈ Γ E
( X,Y ) ∼ Γ [( X − Y ) 2 ] , with Γ being the class of all couplings between P and Q .
For online algorithms the distribution p π,F is no longer a product distribution as actions can depend on past observations.
As a result, the KL divergence becomes larger, but still enjoys an upper bound based on another Wasserstein distance.
Lemma 2.
For any online algorithm π which queries T samples, it holds that
[Equation]
where W ∞ ( P, Q ) is the Wasserstein- ∞ distance: W ∞ ( P, Q )
=
inf γ ∈ Γ esssup ( X,Y ) ∼ Γ | X − Y | , with Γ being the class of all couplings between P and Q .
As W 2 ( P, Q ) ≤ W ∞ ( P, Q ) , the upper bound of Lemma 2 is no smaller than that of Lemma 1, showing the stronger power of online algorithms.
The following corollary is then immediate from
Lemmas 1 and 2.
Corollary 2.1.
The sample complexity of ( ε, . 1) -PAC estimation of g ( F ) is
[Equation]
for offline algorithms, and is
[Equation]
for online algorithms.
In the remainder of this section, we show that Corollary 2.1 leads to tight lower bounds for mean and maximum estimations for both offline and online settings.
4.2 Lower bounds for mean estimation
Consider two distributions F 1 and F 2 which are Dirac masses supported on 1 / 2 − ε and 1 / 2 + ε , respectively.
Clearly W 2 ( F 1 , F 2 ) =
W ∞ ( F 1 , F 2 ) =
2 ε , which is the best possible as W 2 ( F 1 , F 2 ) ≥ | mean( F 1 ) − mean( F 2 )
| ≥ 2 ε .
Corollary 2.1 gives the following lower bounds.
Corollary 2.2.
The ( ε, .
1) -PAC sample complexity for mean estimation is Ω( ε − 2 ) for both offline and online algorithms.
4.3 Lower bounds for maximum estimation
For maximum estimation, the Wasserstein distances W 2 and W ∞ behave differently, as summarized in the following lemma.
Let F β be the class of densities satisfying Assumption 3.
Lemma 3.
For ε ∈ (0 , 1 / 2) , it holds that
[Equation]
β , | max( F 1 ) − max( F 2 ) | ≥ 2 ε } = O ( ε ); min { D KL ( F 1 ∥ F 2 ) : F 1 , F 2 ∈ F β , | max( F 1 ) − max( F 2 ) | ≥ 2 ε } = O ( ε β ) . Note that we have included another term D KL ( F 1 ∥ F 2 ) in Lemma 3 as it can provide a better lower bound than using W ∞ if β ≥ 2 , as D KL ( p π,F 1 ∥ p π,F 2 ) ≤ T · D KL ( F 1 ∥ F 2 ) always holds due to the data-processing inequality (i.e. assuming that all arm rewards are independent).
Consequently, we have the following corollary on the sample complexity of maximum estimation.
Corollary 2.3.
The ( ε, .
1) -PAC sample complexity for maximum estimation over F β is Ω( ε − ( β +2) ) for offline algorithms, and Ω( ε − max { β, 2 } ) for online algorithms.
Lower bounds via thresholding phenomenon
Although the Wasserstein distance-based approach in Section 4 provides general lower bounds for both offline and online algorithms, and these lower bounds are tight for mean and maximum estimation, sometimes this approach can be loose.
For example, Lemma 3 shows that using the W ∞ distance might be looser than using the original KL divergence for maximum estimation.
This section provides tight lower bounds for median estimation, revealing a curious thresholding phenomenon.
5.1 Thresholding phenomenon for offline algorithms
Let F denote the set of distributions satisfying Assumption 2.
To use Le Cam’s two-point method to prove lower bounds for offline algorithms for median estimation, the key quantity is: [Equation] [Equation] [Equation]
−
[Equation]
2 (1 / 2) | ≥ 2 ε } .
Its inverse KL − 1 σ ( ε ) is referred to as the modulus of smoothness of the median with respect to the KL divergence under Gaussian convolution. The
[Table] [Table] in Lemma 1 is the following: let W 2 ,σ ( ε ) be the counterpart of the above quantity with the KL divergence replaced by the Wasserstein-2 distance, Lemma 1 shows that
[Equation]
an upper bound decreasing continuously with σ , where the proof of the last identity is presented in the Appendix.
However, this upper bound is not tight, as shown in the following lemma.
Lemma 4.
For ε ∈ (0 , 1 / 4) , KL σ ( ε ) can be characterized as follows:
[Equation]
where θ ∈ (0 , 1 / 4) , κ ∈ N are arbitrary fixed parameters, and c, C 1 , C 2 , C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
Lemma 4 shows a thresholding phenomenon as follows: when σ increases from 0 to 1 , the quantity
KL σ ( ε ) stabilizes at Θ( ε 2 ) whenever σ ≲ ε 1 / 2 ; however, when σ exceeds this threshold slightly (i.e. σ ≳ ε 1 / 2 − θ for any constant θ > 0 ), this quantity immediately drops to o ( ε κ ) for every possible κ .
The main intuition behind this thresholding phenomenon is that, if σ =
O ( ε 1 / 2 )
, the “bandwidth” of F 1 − F 2 exceeds that of N (0 , σ 2 ) , and the convolution is effectively using N (0 , σ 2 ) as a Gaussian kernel (which preserves polynomials up to order 2 ) for smoothing F 1 − F 2 (which is second-order differentiable).
In contrast, when σ ≫ ε 1 / 2 , the “bandwidth” of F 1 − F 2 could be smaller than N (0 , σ 2 ) , and the convolution is effectively using F 1 − F 2 as a kernel (which could preserve polynomials up to any desired order) for smoothing N (0 , 1) (which is infinitely differentiable).
Approximation theory tells us that the latter approximation error could be much smaller than the former, leading to the thresholding phenomenon.
We remark that this phenomenon is not captured by using the W 2 distance.
This thresholding behavior has an important consequence for median estimation.
By Lemma 4 with σ = 1 / √ m , PAC learning requires that m =
Ω( ε 2 θ − 1 ) for any offline algorithm, as otherwise the
KL divergence could be made arbitrarily small.
When m is large enough, the first line of Lemmathen requires n = Ω( ε − 2 ) to result in a large KL divergence for PAC learning, which comes from the idendity that
[Equation]
Consequently, we have the following corollary for median estimation using offline algorithms.
Corollary 2.4.
Fix any θ > 0 .
The ( ε, .
1) -PAC sample complexity for median estimation is Ω( ε − 3+ θ ) for any offline algorithm.
5.2 Thresholding phenomenon for online algorithms
To prove the PAC lower bound for online algorithms, one first wonders if the same observation in
Lemma 4 could still work.
However, a close inspection of the proof reveals an issue: the optimizers ( F 1 , F 2 ) in the definition of KL σ ( ε ) are different under the regimes σ
= O ( ε 1 / 2 ) and σ = Ω( ε 1 / 2 − θ ) .
An online learning algorithm could first identify the right scenario and then choose a proper sample size to tackle the problem, and thus the above lower bound arguments break down.
To resolve this issue, we aim to choose a proper pair of distributions ( F 1 , F 2 ) with | median( F 1 ) − median( F 2 )
| ≥ 2 ε , and investigate the behavior of D KL ( F 1 ∗N (0 , σ 2 )
∥ F 2 ∗N (0 , σ 2 ))
as a function of σ with ( F 1 , F 2 ) fixed along the line .
The following lemma shows that, even for some fixed pair ( F 1 , F 2 ) , a similar thresholding phenomenon still holds for the KL divergence.
Lemma 5.
Fix any ε, θ ∈ (0 , 1 / 4) , and κ ∈ N .
There exist two distributions F 1 , F 2 ∈ F with | median( F 1 ) − median( F 2 )
| ≥ 2 ε , and
[Equation]
where c, C 1 , C 2 , C ( θ, κ ) are absolute constants with the last one depending only on ( θ, κ ) .
Compared with Lemma 4, Lemma 5 still shows a similar thresholding phenomenon for the KL divergence when σ ≫ ε 1 / 2 , but the KL divergence becomes larger for small σ due to the additional constraint that ( F 1 , F 2 ) is held fixed.
Under the choice of ( F 1 , F 2 ) in Lemma 5, each arm should be pulled at least Ω( ε 2 θ − 1 ) times, while Ω( ε θ − 3 / 2 ) arms need to be pulled in view of the first line.
The following theorem makes the above intuition formal.
Theorem 3.
The ( ε, .
1) -PAC sample complexity for median estimation is Ω( ε − 5 / 2+ θ ) for any fixed θ > 0 and any online algorithm.
The formal proof of Theorem 3 is more complicated and requires an explicit computation of the KL divergence D KL ( p π,F 1 ∥ p π,F 2 ) .
We relegate the full proof to ?? .
This thresholding phenomenon of the noise level also applies to the case of trimmed mean, which is discussed further and an analogous result is proved in Appendix ?? .
Conclusion
In this work we formulated and studied offline and online algorithms for estimating functionals of distributions.
We developed unified algorithms for estimating the mean, median, maximum, and trimmed mean, providing sample complexity upper bounds.
We additionally proved information theoretic lower bounds in these settings, which show that our algorithms are optimal up to ε c where c is a fixed constant arbitrarily close to zero.
We used different Wasserstein distances to construct information theoretic lower bounds for mean and maximum estimation, and showed how fundamentally different techniques are required for median and trimmed mean estimation.
The lower bounds for median and trimmed mean estimation elucidate an interesting thresholding phenomenon of the noise level to distinguish two distributions after Gaussian convolution, which may be of independent interest.
Interesting directions of future work include extending our analysis to nonindicator-based functionals, such as the BH threshold and analyzing the limiting behavior as δ → ∞ .
Acknowledgements
Yifei Wang and David Tse were partially supported by NSF Grants CCF-1909499.
Tavor Z. Baharav was supported in part by the NSF GRFP and the Alcatel-Lucent Stanford Graduate Fellowship.
Yanjun Han is supported by a Simons-Berkeley research fellowship and Norbert Wiener postdoctoral fellowship.
Jiantao Jiao was partially supported by NSF Grants IIS-1901252, and CCF-