Estimation of Entropy in Constant Space with Improved Sample Complexity
Abstract
Recent work of Acharya et al.
(NeurIPS 2019) showed how to estimate the entropy of a distribution D over an alphabet of size k up to ± (cid:15) additive error by streaming over ( k/(cid:15) 3 )
· polylog(1 /(cid:15) )
i.i.d.
samples and using only O (1) words of memory.
In this work, we give a new constant memory scheme that reduces the sample complexity to ( k/(cid:15) 2 ) · polylog(1 /(cid:15) ) .
We conjecture that this is optimal up to polylog(1 /(cid:15) ) factors.
Introduction
In the ﬁeld of streaming algorithms , an algorithm makes one pass (or few passes) over a database while using memory sublinear in the data it sees to then answer queries along the way or at the data stream’s end.
Researchers have developed various algorithms, as well as memory lower bounds, for such problems for over four decades [MP80, MG82, AMS99].
For the vast majority of research in the ﬁeld, the database is assumed to be ﬁxed, and algorithms are then analyzed through the lens of worst case analysis.
In this work, we look to further develop the relationship between streaming algorithms and statistics, speciﬁcally studying statistical inference through low-memory streaming algorithms.
In this setup, rather than processing a worst-case instance of some ﬁxed database, our input is instead a distribution
D , and our algorithm processes i.i.d.
samples from D with the goal of inferring its properties.
Natural questions then arise, such as understanding the tradeoffs between sample complexity, memory, accuracy, and conﬁdence, or even understanding whether a low-memory algorithm exists at all for a particular inference problem even if we allow the streaming algorithm to draw an unlimited number of samples.
Work on streaming algorithms for statistical inference problems began in [GM07], which studied nonparameteric distribution learning, followed by the work of [CLM10], studying low-memory streaming algorithms for use in robust statistics and distribution property testing.
Interest in the area later exploded off after work of [SVW16], which explicitly raised the question of whether low memory might place fundamental limits on learning rates, with a ﬂurry of works proving such limitations in response [CMVW16, MM17, KRT17, Raz17, GRT18, SSV19, GRT19, GKR20,
GKLR21], starting with a work of [Raz19] on memory/sample tradeoff lower bounds for learning parities ( D generates ( x, (cid:104) w, x (cid:105) ) for x uniform in the hypercube with w an unknown parameter, and the goal is to learn w ).
In this work, following [ABIS19], we focus speciﬁcally on the problem of estimating the entropy of an unknown distribution D over { 1 , . . .
, k } using a low-memory streaming algorithm over i.i.d.
samples.
It is known that to estimate the entropy up to (cid:15) additive error with large constant success probability, without memory constraints the optimal sample complexity is
[Equation]
[VV17, VV11, JVHW15, WY16].
Prior work by [BDKR05] also shows that a sublinear number of samples is possible for multiplicative approximation of entropy for distributions whose entropy is sufﬁciently large.
The known optimal algorithms from prior work, however, must remember all samples and hence use Ω( n ) words of memory 5 .
The algorithm of [ABIS19] uses only O (1) words of memory, though at the cost of requiring an increased sample complexity of k · ˜ O (1 /(cid:15) 3 ) 6 .
In this work, our goal is to address the question: to what extent was the worsening of sample complexity in previous work necessary to achieve constant memory?
Our Contribution.
We show that using O (1) words of memory 7 , it is possible to obtain a sample complexity of k · ˜ O (1 /(cid:15) 2 ) , which is an improvement over the previous memory-efﬁcient sample complexity bound which had cubic dependence on 1 /(cid:15) .
The starting point of our algorithm revisits a simple estimator proposed by [ABIS19].
Their simple estimator uses O ( k log 2 ( k/(cid:15) ) /(cid:15) 3 ) samples to estimate the entropy in constant space.
Our novel contribution is a modiﬁcation which estimates a bias incurred by the estimator; this change allows us to use only O ( k log 2 k log 2 (1 /(cid:15) ) /(cid:15) 2 ) samples.
With the simple estimator with improved sampled complexity in hand, we show how an “interval-based” algorithm, similar to the one in [ABIS19], improves the dependence on k to k · ˜ O
(1 /(cid:15) 2 ) .
We remark that there has been other work on estimating entropy in the data streaming model [BG06, CCM10, HNO08, KNW10, JW19], but those works are qualitatively different from our own current work and that of [ABIS19].
Speciﬁcally, they take the worst case point of view, where the stream items are not drawn i.i.d.
from a distribution, but rather the stream itself is viewed as a worst-case input and the goal is to estimate its empirical entropy.
In that model, O (1) memory algorithms for ± (cid:15) additive estimation to entropy provably do not exist, as there is a known memory lower bound of Ω(1 /(cid:15) 2 ) bits [JW19].
Overview of Approach.
We start by describing the basic algorithm of [ABIS19].
Their basic estimator takes a single random sample i ∼ D , followed by N more i.i.d.
samples.
Then, they deﬁne
N x to be the number of these N samples equal to i .
The estimate ˆ p i :=
N x /N is an unbiased estimator of of the probability p
i of i according to D , and for large N , log(1 /
ˆ p i ) is a reasonable estimator for the entropy
H = H ( D ) =
E [[[log(1 /p i )]]] of D .
One can then average many such independent estimates.
There is an additional technical detail, that ˆ p i may be zero (if N x is zero), which is ﬁxed via a “one-smoothing” trick of actually setting ˆ p i :=
( N x + 1) /N
(which introduces an acceptably small amount of additional bias when N is sufﬁciently large).
Our improvement begins with the observation
that log(1 / ˆ p i ) is not an unbiased estimator for H .
We ﬁrst propose a similar but different estimator to the previous simple estimator.
We also begin by taking a random sample i ∼ D ; however, rather than letting N x be sampled from the binomial distribution Bin ( N, p i ) , we sample a negative binomial random variable X , which is the number of additional draws to see i exactly t more times ( t is a parameter of the algorithm).
Henceforth we let NB ( t, p ) denote such a negative binomial random variable, where the underlying Bernoulli experiment has success probability p .
Then E [[[ X ]]]
= t/p i , and we will use log( X /t ) as a reasonable estimate of log(1 /p i ) .
This estimator is also biased, but we can correct for this bias using a few more samples.
Speciﬁcally, let Y =
X p
i /t and consider the degree- r Taylor expansion of our estimate log( X /t ) and the ideal quantity log(1 /p
i ) .
As it will turn out, the expectation of the degree- r Taylor expansion of log( X /t )
− log(1 /p
i ) =
log Y is a degree- r polynomial in p i .
By drawing r additional samples, we may design an estimator for this polynomial, and subtract it from log( X /t ) .
Correcting some of the bias in this way gives us our improved estimate for log(1 /p
i ) .
Our analysis of this scheme shows that a sample complexity of ( k/(cid:15) 2 ) · polylog( k/(cid:15) ) sufﬁces.
We then describe and analyze an improved algorithm in Section 3, which achieves ( k/(cid:15) 2 ) · polylog(1 /(cid:15) )
sample complexity by additionally incorporating a “bucketing” scheme, similar to one proposed in [ABIS19].
The idea is to partition the possibilities for values of X into disjoint intervals I (cid:96) =
[ b (cid:96) − 1 , b (cid:96) ) for (cid:96) = 1 , 2 , . . .
, L and optimized choices of the breakpoints b (cid:96) , then estimate both Pr [[[ X ∈
I
(cid:96) ]]] and the conditional contributions to entropy conditioned on X ∈ I (cid:96) for each (cid:96) .
By estimating separately for each I (cid:96) , one can show that the conditional variance is reduced to obtain an overall smaller sample complexity of ( k/(cid:15) 2 ) · polylog(1 /(cid:15) ) , a strict improvement over that of [ABIS19]; details are in Section 3.
Lower bounds: Diakonikolas et al.
[DGKR19] show lower bounds for sample-memory tradeoffs for testing uniformity of distributions.
They construct a distribution p over [2 k ] such that for every i ∈ [ k ] , the probabilities of the elements 2 i and 2 i − 1 are (1 + √ (cid:15) ) /
(2 k ) and (1 − √ (cid:15) ) / (2 k ) while the order is picked randomly.
They show that any streaming algorithm that uses m bits of memory, n samples, and can distinguish p from the uniform distribution over [2 k ] with high constant probability requires: (i) m · n = Ω( k(cid:15) ) , and (ii) if n ≤ k 0 .
9 and m ≥ n 2 /k 0 .
9 , m · n = Ω( k log k (cid:15) 2 ) .
It is not hard to see that the entropy of the difference between uniform distribution and p is Θ( (cid:15) ) .
While this trade-off indicates a more general relation between the memory usage and samples, only (i) applies to constant memory algorithms (since (ii) requires n ≤ k 0 . 9 ).
With a word being log( k/(cid:15) ) bits, (i) leads to a lower bound of Ω( k/ ( (cid:15) log( k/(cid:15) )))
samples (the same as with unbounded memory).
This leaves open whether the sample complexity for O (1) -word algorithms is Ω( k/(cid:15) ) or Ω( k/(cid:15) 2 ) .
We speculate the latter, and we discuss our conjecture in Section 4.
A Simple Algorithm and Analysis
Let k ∈ N , and D be an unknown distribution supported on [ k ] .
For any i ∈ [ k ] , we denote the probability that i ∈
[ k ] is sampled by D as p i .
The goal is to design a low-space streaming algorithm which receives independent samples from D and outputs an estimate to the entropy:
[Equation]
where logarithms above and throughout this paper are base- 2 , unless otherwise stated.
2.1 An Estimator for log(1 /p i ) log(1 /p i ) log(1 /p i )
As mentioned in Section 1, similarly to [ABIS19] the algorithm aims to estimate H ( D ) by taking a sample i ∼ D and estimating log(1 /p
i ) .
Then, averaging these estimates will give an estimator for H ( D ) (albeit with a super-linear dependence on k , which we ﬁx in Section 3).
We describe the estimator in Figure 1.
There are three main steps in the analysis.
In the ﬁrst, we show that the estimator has small bias.
The second is showing that the above estimator has low variance.
Finally, we show that the estimator may be computed with few bits.
In Figure 1, we set r = Θ(log(1 /(cid:15) )) and t = Θ(log 2 (1 /(cid:15) )) to obtain an estimator whose bias is at most (cid:15) and variance is at most O (log 2 k ) .
It then follows that repeating the estimate of log(1 /p
i ) for O (log 2 k/(cid:15) 2 ) i.i.d.
chosen i ∼ D gives the desired estimate with probability at least 2 / 3 .
These parameter settings establish the following theorem:
Theorem 1.
There exists a single-pass data stream algorithm using O (1) words of working memory that processes a stream of O ( k(cid:15) − 2 log 2 k log 2 (1 /(cid:15) ))
i.i.d.
samples from an unknown distribution D on [ k ] and returns an additive (cid:15) approximation of H ( D ) with probability 2 / 3 .
The space complexity in the theorem above follows since computing the estimator just requires maintaining integers in the sets [ k ] , [ t ] , and [ r ] , as well as computing a low-degree polynomial.
To compute the average of multiple estimators in small space it sufﬁces to compute the sum of the estimates where each estimator is computed in series.
The sample complexity bound (given the speciﬁed parameters) in the above theorem follows directly from the sample complexity of
LogEstimator .
By virtue of the fact our estimators are based on negative binomial distributions ( X
Input: Sample access to a distribution D supported on [ k ] , an index i ∈ [ k ] where p i (cid:54) = 0 .
Output: A number η ∈ R ≥ 0 , which is our bias estimate.
We draw enough samples from D so that i is sampled exactly t times, and let X ∈ N denote the number of samples taken.
For r ∈ N , let f : R → R denote the degree- r Taylor expansion of log z centered at 1 , and h t : [0 , 1] → R be the degree- r polynomial satisfying
[Equation]
Finally, g : [0 , 1] r → R is the linear function with g ( ρ, ρ 2 , . . .
, ρ r ) =
h t ( ρ ) .
We take r additional independent samples from D , and for j ∈ [ r ] , we let B j be the
indicator random variable that the ﬁrst j samples were all i .
Note that { B j } j ∈ [ r ] can be encoded using a single counter requiring log r bits.
We return
[Equation]
in Figure 1 is the number of Bernoulli trials until t successes), this in turn follows directly from the expectation of negative binomial distributions:
Fact 2.1 (Expected Sample Complexity of LogEstimator ) .
Suppose we draw i ∼ D and execute
LogEstimator ( D , i ) .
Then, the expected sample complexity is
[Equation]
Although the number of samples we draw is a random variable that is only bounded in expectation, note that it implies the existence of a good algorithm that always has a bounded sample complexity: namely, we can simply terminate the algorithm early and output Fail if it draws a large constant factor times more samples than we expect, which happens with low probability by Markov’s inequality.
Before moving on to the showing the properties of the estimator, we verify that h t ( ρ ) is a degree- r polynomial.
Lemma 2.2.
For any r ∈ N , let f : R → R denote the degree- r Taylor expansion of log( z ) centered at 1 .
Then, for any ρ > 0 and t ∈ N ,
[Equation]
is a polynomial of degree at most r .
Proof.
Recall that the random variable Z ∼ NB ( t, ρ ) is the number of independent trials from a
Ber( ρ ) distribution before one sees t successes.
Furthermore, f is the degree- r Taylor expansion of log z centered at 1 , and
[Equation]
By linearity of expectation, it sufﬁces to show that for every j ∈ { 1 , . . .
, r } , E Z (cid:2)(cid:2)(cid:2) ( Z ρ
[Equation]
t − 1) j (cid:3)(cid:3)(cid:3) is a degree- j polynomial in ρ .
Note that Z is a sum of t independent Geo( ρ ) random variables, so by expanding ( 1 t (cid:80) ti =1 G i ρ − 1) j and applying linearity of expectation once more, it sufﬁces to show that
[Equation]
is a degree- j polynomial in ρ .
We note that this latter term, E G (cid:2)(cid:2)(cid:2) G j (cid:3)(cid:3)(cid:3) may be expressed as ρ ·
Li − j (1 − ρ ) , where Li − j ( · ) is the polylogarithm function (see [Wei]).
Li − j (1 − ρ ) happens to be a rational function, where the denominator is exactly ρ j +1 , which cancels the ρ j +1 term .
In addition, the numerator of Li − j (1 − ρ ) is a degree- j polynomial in ρ , which gives the desired polynomial representation.
Finally, it will be useful for the variance calculation to show that the correction term is always bounded, which we show here.
Lemma 2.3.
There exists a universal constant c > 0 such that, for any r, t ∈ N , if we let g : [0 , 1] r → R be the linear function where g ( ρ, ρ 2 , . . .
, ρ r ) =
h t ( ρ ) , then g ( b ) ∈ [ − c, c ] for all b ∈ { 0 , 1 } r .
Proof.
Recall g : [0 , 1] r
→ R is the linear function where g ( ρ, ρ 2 , . .
, ρ r ) =
h t ( ρ ) .
Hence, in order to show that g : { 0 , 1 } r → R is bounded, it sufﬁces to show that the sum-of-magnitudes of the r +coefﬁcients of h t is bounded.
Since we have
[Equation]
(cid:19) i (cid:35)(cid:35)(cid:35) .
Notice that in Lemma 2.2, we showed that each E Z (cid:2)(cid:2)(cid:2) ( Z ρ/t − 1) i (cid:3)(cid:3)(cid:3) is a degree- i polynomial in ρ , and the bound (3) implies that, for each i ∈ { 1 , . .
, r } these polynomials are at most (cid:0) O
( i/ √ t ) (cid:1) i in magnitude.
Furthermore, since these are degree- i polynomials bounded in [0 , 1] , we conclude (by Lemma 4.1 in [She13]), that the coefﬁcients in E Z (cid:104)(cid:104)(cid:104) ( Z ρ/t − 1) i (cid:105)(cid:105)(cid:105) are at most (cid:0) O
( i/ √ t ) (cid:1) i .
In particular, we have that the r coefﬁcients of h t ( ρ ) are at most
[Equation]
because r/ √ t can be made an arbitrarily small constant.
To show that g : { 0 , 1 } r → R is bounded, we add the magnitudes of the r coefﬁcients, which is O ( r/ √ t )
=
O (1) when r
= O (log(1 /(cid:15) )) and t
=
O (log 2 (1 /(cid:15) )) .
2.2 Bounding Bias of Estimator
Lemma 2.4.
Let D be any distribution and consider any i ∈ [ k ] .
If, for (cid:15) ∈ (0 , 1) , we instantiate
LogEstimator ( D , i ) with r = Θ(log(1 /(cid:15) )) and t = Θ(log 2 (1 /(cid:15) )) , which produces the random variable η , then (cid:12)(cid:12)(cid:12) E [[[ η ]]] − log (cid:16) 1 p
i (cid:17)(cid:12)(cid:12)(cid:12) ≤ (cid:15).
The remainder of the section constitutes the proof of Lemma 2.4, which will follow from a sequence of claims.
Claim 2.5.
In an execution of LogEstimator ( D , i ) , let X and η be deﬁned as in Line 1 and Lineof Figure 1, and let Y = X · p
i /t .
Then,
[Equation]
where h ( z ) is the error in the degree- r Taylor expansion of log z at 1 .
Lemma 2.6.
For any (cid:15) ∈ (0 , 1) , letting r = Θ(log(1 /(cid:15) ) and t = Θ(log 2 (1 /(cid:15) )) , we have that for p i > 0 , | E X [[[ h ( Y )]]] | ≤ (cid:15) .
Proof.
Whenever z ∈ (0 , 2) , we may write log( z ) as its Taylor expansion centered at 1 .
In particular, we have
[Equation]
First, consider the case that z ∈ (1 , 2) .
Then, we may re-write (1) as
[Equation]
Whenever z ∈ (1 , 2) , then every term in the right-most summation of (2) is positive; indeed, ( z − 1) 2 (cid:96) − 1 / ( r + 2 (cid:96) ) > 0 because z > 1 , and ( r + 2 (cid:96) ) /
( r + 2 (cid:96) − 1) > 1 while ( z − 1) < 1 .
In particular, for z ∈ (1 , 2) , we may upper bound the right-most summation in (2) by upper bounding each term.
For every (cid:96) ∈ N , we may upper bound each term
[Equation] [Equation]
Plugging this upper bound into each term of (2), we have that z ∈ (1 , 2) satisﬁes
[Equation]
We now consider z ∈ (0 , 1) .
Here, every term in the right-most summation in (1), ( − 1) (cid:96) ( z − 1) (cid:96) /
( r + (cid:96) ) , is positive.
So we upper bound
[Equation]
= ( − 1) (cid:96) · ( z − 1) (cid:96) r + (cid:96) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(2) ∞ (cid:88) (cid:96) =
( − 1) (cid:96) · ( z − 1) (cid:96) (cid:96) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
| z − 1 | r | log z | .
The ﬁnal case occurs when z ≥ 2 , and we may no longer use the series representation.
However, in this case, we have
[Equation]
In all cases, we have | E X
[[[ h ( Y )]]] | ≤ E X [[[ | h ( Y ) | ]]] ≤ O ( r ) · E X (cid:104)(cid:104)(cid:104) | Y − 1 | r +1 (cid:105)(cid:105)(cid:105) + (cid:15)/ 2 + E X
[[[ 1 { Y ≤ 1 / 10 } log(1 / Y )]]] , where we used the fact that Y > 0 and r = Θ(log(1 /(cid:15) )) to say (9 / 10) r < (cid:15)/ 2 .
In order to bound the above two quantities, we use the fact that the random variable Y is a subgamma random variable and thus has good concentration around its mean (which is 1 for the case of Y ), giving the desired inequality.
Deﬁnition 2.7 (Subgamma Random Variable) .
For σ, B ∈ R , a random variable Z with expectation µ is ( σ, B ) -subgamma
if for all λ ∈ R with | λ | < 1 / | B | ,
[Equation]
It is not hard to verify (see Section B) that the random variable Y is centered at 1 , and that there are constants α, β ∈ R ≥
so Y is ( α/ √ t, β/t ) -subgamma.
Then, by taking the Taylor expansion of
E (cid:2)(cid:2)(cid:2) e λ
( Y − 1) (cid:3)(cid:3)(cid:3) , we have that for any | λ | < t/β , and any j ∈ N ,
[Equation]
by picking λ = √ t/α , which is less than t/β for large enough t .
Letting j = r + 1 and setting t
= O ( r 2 ) , we get the desired bound of o ( (cid:15)/r ) .
In order to bound E X [[[ 1 { Y ≤ 1 / 10 } log(1 / Y )]]] , we compute it explicitly, and recall that X ≥ t , so that the above event is satisﬁed only if p i ≤ 1 / 10 .
[Equation]
Improving Sample Complexity via Bucketing
In this section, we focus on estimating the expected value of log( X /t ) with error at most (cid:15) .
Our goal here is to remove the poly(log k ) dependencies in the sample complexity of estimation.
In particular, we prove the following theorem, which improves on the dependence of k in Theorem 1.
Theorem 2.
There exists a single-pass data stream algorithm using O (1) words of working memory that processes a stream of O ( k log 4 (1 /(cid:15) ) /(cid:15) 2 ) i.i.d.
samples from an unknown distribution D on [ k ] and returns an additive (cid:15) approximation of H ( D ) with probability at least 2 / 3 .
Given the work done in Section 2, it will sufﬁce to estimate the quantity H (we give the explicit reduction in Lemma 3.1 shortly):
[Equation]
where t is set to Θ(log 2 (1 /(cid:15) )) , such that we can then apply the correction term of Section 2.
Recall that the randomness in the above expectation is taken over the random choice of i ∼ D , and X is a negative binomial random variable drawn from NB ( t, p i ) .
First, we show that it sufﬁces to estimate (4) in order to estimate the entropy, given our tools from Section 2.
Lemma 3.1. Consider a ﬁxed distribution D , and for (cid:15) > 0 suppose ˆ H ∈ R is such that | H − ˆ H | ≤ (cid:15) .
Then, there exists a O (1) word streaming algorithm which given ˆ H and using an additional
O (log(1 /(cid:15) ) /(cid:15) 2 ) independent samples from D , outputs an estimate to the entropy of D up to error ± 2 (cid:15) with probability at least 0 .
It thus sufﬁces to design an algorithm to estimate (4).
Our approach is to use a bucketing scheme.
At a high level, we partition the range of X into L intervals: I 1 , I 2 , . . .
, I L .
We compute the conditional expectation of log( X /t ) in each interval separately.
Then, we take the weighted average of these conditional expectations, where the weights are determined by the probability of the intervals.
Unbounded XXX : As speciﬁed above, the random variable X is a mixture of negative binomial random variables, so X may be unbounded.
In addition, if we had sampled i ∼ D where p i was very small, X ’s value will tend to be very large.
It will be convenient to introduce a parameter X max ∈ N and consider the random variable X (cid:48) :
= min( X , X max ) .
Let ˜ H denotes the expected value of X (cid:48) :
[Equation]
For the rest of the section, we will seek to approximate ˜ H , and the fact that this is a good estimate for
H follows from the following lemma.
Lemma 3.2.
Let i ∼ D , and let X and be a negative binomial random variable from NB ( t, p i ) .
Let X (cid:48) be the bounded version of X : X (cid:48) :
= min( X , X max ) .
Let t ∈ N and (cid:15) ∈ (0 , 1) .
If we set
X max = tk/ (ln(2) (cid:15) ) , then
[Equation]
Comparison to related work: It is worth noting that the proofs in this section are inspired by the work of [ABIS19].
The authors used a similar bucketing technique to estimate entropy.
While the structure of our proof is similar, there are subtle differences between our work and what they did.
First, we are focusing on estimating different quantities.
In particular, we work with an unbounded random variable while their estimator is bounded.
Moreover, they have a two-step bucketing system where they draw a sample i and two estimates for p i ; they use one estimate for detecting which bucket falls into and the second one to estimate entropy in that bucket.
One of the complications of this approach is that the second estimator may fall into a different bucket; Thus, they have to “clip" the second estimator to make sure it is close to the bucket of the ﬁrst estimator.
We have circumvented these hurdles by using the same estimate for detecting which bucket we are in and estimating log( X /t ) in that bucket.
The algorithm: We write ˜ H in terms of conditional expectation in the intervals.
[Equation]
Let q (cid:96) denote the probability of X (cid:48) being in I (cid:96) , and H (cid:96) denote the conditional expectation in I (cid:96) .
Our algorithm estimate q (cid:96) and H (cid:96) for each interval to ﬁnd an estimate for ˜ H .
Below we give a brief description of our algorithm, and the pseudocode can be found in Algorithm 1.
Below, we deﬁne b
=
t < b 1 < · · · < b L = X max to be L + 1 parameters (which we will set shortly) that denote the boundary points of the intervals:
[Equation]
− 1 , b (cid:96) )
∀ i ∈
[ L − 1] ,
I L =
[ b L − 1 , b L ] .
For each interval I (cid:96) , we draw r (cid:96) samples from D , namely i 1 , . .
, i r (cid:96) ∼ D .
For each i j , we start drawing samples from D in the process of drawing a negative binomial random variable
X j ∼ NB
( t, p i j ) ; then, we will set X (cid:48)
j = min( X j , X max ) .
Furthermore, we will only consider
X (cid:48)
j ’s that fall in I (cid:96) , which means that we can stop early if we already know X (cid:48)
j will be too large.
In particular, if we draw b (cid:96) samples and have not observed t instances of i j , we can already conclude
X (cid:48)
j is not in I (cid:96) and stop sampling.
Among these r (cid:96) samples { i 1 , . . .
, i r (cid:96) } , let c (cid:96) denote the number of X (cid:48)
j ’s that fall into I (cid:96) .
We estimate the weight of each bucket by ˆ q ˆ q ˆ q :=
c (cid:96) /r
(cid:96) .
For the last bucket, we set ˆ q ˆ q ˆ q (cid:96) in a way that the sum of the weight is one:
[Equation]
Also, we compute an average of log( X (cid:48)
j /t ) of such X (cid:48)
j ’s and denote it by ˆ H
ˆ H ˆ H (cid:96) :
[Equation]
j
[Equation] [Equation]
c
[Equation]
∀ (cid:96) = 1 , . . .
, L .
In these deﬁnitions, we take ˆ H
ˆ H
ˆ H L = log( b L /t )
if c L = 0 and for the sake of analysis ˆ H
ˆ H ˆ H (cid:96) =
H (cid:96) if c (cid:96) = 0 for (cid:96) <
L ; note that the value of ˆ H
ˆ H ˆ H (cid:96) will be multiplied by 0 in this case and hence we may deﬁne ˆ H
ˆ H ˆ H (cid:96) arbitrarily.
Our estimate for ˜ H is the weighted sum of ˆ H
ˆ H ˆ H (cid:96) , i.e., ˆ H
ˆ H
ˆ H = (cid:80) L(cid:96)
=
1 ˆ q ˆ q ˆ q
cid:96) ·
ˆ H
ˆ H ˆ H (cid:96) .
1: procedure L OG E STIMATOR ( k , (cid:15) , sample access to D ) 2: ˆ H ˆ H ˆ H ←3: for (cid:96) = 1 , 2 , . . .
, L do 4: c (cid:96) ← 0 , ˆ H
ˆ H ˆ H (cid:96) ←5: for r (cid:96) times do 6:
Draw i ∼ D 7:
Draw b (cid:96) samples from D but terminate early if t occurrences of i are observed.
X ← number of samples drawn 9: ˆ H ˆ H ˆ H (cid:96) ← ˆ H ˆ H ˆ H (cid:96) + log( X /t ) · 1 { X ∈
I (cid:96) } and c (cid:96) ← c (cid:96) + 1 { X ∈
I (cid:96) } 10:
Deﬁne ˆ q ˆ q ˆ q (cid:96) ← (cid:40) c (cid:96) /r
(cid:96) if (cid:96) <
L 1 − (cid:80) L − 1 (cid:96) =
1 ˆ q ˆ q ˆ q
cid:96) if (cid:96) =
L and ˆ H ˆ H ˆ H (cid:96) ←  
ˆ H
ˆ H ˆ H (cid:96) /c
(cid:96) if c (cid:96) >
H (cid:96) if c (cid:96) = 0 , (cid:96) <
L log( b L /t )
if c (cid:96) = 0 , (cid:96) =
L (cid:46) Note that if c (cid:96) = 0 and (cid:96) <
L , the deﬁnition of ˆ H
ˆ H ˆ H (cid:96) is just for analysis since in that case ˆ q ˆ q ˆ q (cid:96) =
and the output does not depend on ˆ H
ˆ H ˆ H (cid:96) .
11: ˆ H
ˆ H
ˆ H ← ˆ H
ˆ H ˆ H + ˆ q ˆ q ˆ q (cid:96)
ˆ H
ˆ H ˆ H (cid:96)
Lemma 3.3.
Deﬁne Error :
= (cid:80) L(cid:96)
=
1 ˆ q ˆ q ˆ q
cid:96) ·
ˆ H
ˆ H ˆ H (cid:96) − (cid:80) L(cid:96)
=1 q (cid:96) · H (cid:96) .
For any setting of t = b 0 < · · · < b L =
X max and { r (cid:96) ∈ N : (cid:96) ∈
[ L ] } , we have
E (cid:2)(cid:2)(cid:2) Error 2 (cid:3)(cid:3)(cid:3) ≤ 2 ·
L − 1 (cid:88) (cid:96) =
q (cid:96) log 2 ( b L /b (cid:96) − 1 ) r (cid:96) + 6 · L (cid:88) (cid:96) = q (cid:96) log 2 ( b (cid:96) /b (cid:96) − 1 ) r (cid:96) +
Proof of Lemma 3.3.
We start by rewriting Error as follows:
Error =
L (cid:88) (cid:96) =
[Equation]
(cid:96)
[Equation]
(cid:96) )
ˆ H
[Equation]
H
[Equation]
H (cid:96) +
L (cid:88) (cid:96) =
q (cid:96) (cid:16) ˆ H
ˆ H ˆ H (cid:96) − H (cid:96) (cid:17) =
L − 1 (cid:88) (cid:96) =
(ˆ
[Equation]
ˆ
[Equation]
ˆ
[Equation]
(cid:96) − q (cid:96) )
ˆ H
ˆ H ˆ H (cid:96) + (cid:32) 1 −
L − 1 (cid:88) (cid:96) =
ˆ q ˆ q ˆ q (cid:96) − 1 +
L − 1 (cid:88) (cid:96)
[Equation]
Using the fact that ˆ q ˆ q ˆ q (cid:96) is an unbiased estimator for q (cid:96) and that for (cid:96) ≤ L − 1 , ˆ H
ˆ H ˆ H (cid:96) is an unbiased estimator of H (cid:96) even when conditioned on a speciﬁc value of ˆ q ˆ q ˆ q (cid:96) , we have
[Equation]
and ∀
[Equation]
[Table]
Hence,
E (cid:2)(cid:2)(cid:2) Error 2 (cid:3)(cid:3)(cid:3) ≤ 2 · E       (cid:32)
L − 1 (cid:88) (cid:96) =
(ˆ q ˆ q ˆ q (cid:96) − q (cid:96) ) (cid:16)
ˆ H
ˆ H ˆ H (cid:96) − ˆ H
ˆ H
ˆ H L (cid:17)(cid:33) 2   
 
 + 2 · E       (cid:32) L (cid:88) (cid:96) =
q (cid:96) (cid:16) ˆ H
ˆ H ˆ H (cid:96) − H (cid:96) (cid:17)(cid:33) 2   
 
 = 2 · E (cid:34)(cid:34)(cid:34) L − 1 (cid:88) (cid:96) =
(ˆ q ˆ q ˆ q (cid:96) − q (cid:96) ) 2 (cid:16)
ˆ H
ˆ H ˆ H (cid:96) − ˆ H
ˆ H
ˆ H L (cid:17)(cid:35)(cid:35)(cid:35) + 2 · E (cid:34)(cid:34)(cid:34) L (cid:88) (cid:96) =
q 2 (cid:96) (cid:16) ˆ H
ˆ H ˆ H (cid:96) − H (cid:96)
[Equation]
Recall ˆ q ˆ q ˆ q (cid:96) is the average of r (cid:96) Bernoulli random variables, each set to 1 with probability q (cid:96) .
Hence,
[Equation]
To bound the expectation of ( ˆ H
ˆ H ˆ H (cid:96) − H (cid:96) )
2 recall the deﬁnition of ˆ H
ˆ H ˆ H (cid:96) : we take a sample i ∼ D , then
X ∼ NB ( t, p i ) to check whether X (cid:48) ∈
I (cid:96) ; if so, we increment c (cid:96) and use log( X (cid:48) /t ) as an estimate for H (cid:96) .
Crucially, if we condition on any non-zero value of c (cid:96) , ˆ H
ˆ H ˆ H (cid:96) is an unbiased estimator given by averaging c (cid:96) values, all of which are bounded in the interval [log( b (cid:96) − 1 /t ) , log( b (cid:96) /t )] .
If c (cid:96) = 0 , the estimator sets ˆ H
ˆ H ˆ H (cid:96) to log( b (cid:96) /t ) .
In particular, since for all non-zero positive integers, 1 / c ≤ 2 / ( c +1) , we may write
[Equation]
where the second last inequality used that q (1 − q ) r ≤ 1 / ( r + 1) for any q ∈ [0 , 1] and that since c (cid:96) is distributed as Bin ( q (cid:96) , r (cid:96) ) , we have E c (cid:96) (cid:2)(cid:2)(cid:2)
( c (cid:96) + 1) − 1 (cid:3)(cid:3)(cid:3) ≤ 1 / ( q (cid:96) ( r (cid:96) + 1) .
The proof of this last inequality appears as Lemma 6 in [ABIS19], which we reproduce below for convenience:
[Equation]
Substituting Eq. 7 and Eq. 8 into Eq. 6 gives the lemma. We consider the following setting of parameters, where we let L = log ∗ k , 8 such that we have b 0 = t , and
[Equation]
and let
[Equation]
It is fairly straightforward to show that Algorithm 1 uses a constant number of words.
Note that r (cid:96) (similarly b (cid:96) ’s) can be computed from r (cid:96) − 1 , so we do not need to calculate and store all the r (cid:96) ’s beforehand.
Also, to compute ˆ q ˆ q ˆ q L , we do not need all the ˆ q ˆ q ˆ q (cid:96) ’s. We only need to keep a running sum of ˆ q ˆ q ˆ q (cid:96) .
Thus, we only need a constant number of words of memory.
We analyze correctness simply by bounding the variance.
Namely, the remainder of the section will be devoted to proving the following lemma, which will imply that our estimator will be within ± (cid:15) of ˜ H with constant probability.
Lemma 3.4.
The expected sample complexity of the algorithm is O
( k log 4 (1 /(cid:15) ) /(cid:15) 2 ) and returns an ± O ( (cid:15) ) approximation with probability 0 .
Conclusions
We presented an algorithm for returning an additive (cid:15) approximation of the Shannon entropy of a distribution over [ k ] .
The algorithm required O ( k (cid:15) − 2 log 4 (1 /(cid:15) ))
i.i.d.
samples from the unknown distribution and a constant number of words of memory.
In terms of the (cid:15) dependence, this improves over the state-of-the-art [ABIS19] by a factor 1 /(cid:15) in the sample complexity.
More generally, we expect that the technique used, that of correcting the bias via low-degree polynomials will be useful in the context of the other inference problems in the data stream setting.
The main open problem is determining whether the sample complexity of our result is optimal.
We conjecture that this is the case, up to the poly(log(1 /(cid:15) )) factors.
See Section D for a discussion of this conjecture.
A Variance of LogEstimator
We now bound the variance of our estimator by O (log 2 k ) .
Recall that the output of LogEstimator is given by log( X /t )
− g ( B 1 , . . . , B r ) , where the function g is bounded.
Since the variance we seek is O (log 2 k ) , it sufﬁces to show that the variance of log( X /t ) is O (log 2 k ) with i ∼ D , since subtracting g changes the estimate by at most a constant (see Lemma 2.3).
Lemma A.1.
Let i ∼ D and X denote the number of independent trials from Ber( p i ) before we see t successes.
Then, Var [[[log( X /t )]]]
= O (log 2 k ) .
Proof.
Let X max
= 2 kt
[Equation]
and consider the random variable X (cid:48)
=
min { X
[Equation]
X max }
[Equation]
Then
[Equation]
where we used that log( X (cid:48) /t ) ≤ log(2 k ) always, and that log( z ) ≤ √ z − 1 / ln(2) for all z ≥ 1 .
Then,
[Equation]
B Omitted Details from Section
Proof of Claim 2.5.
Notice that X is the number of trials from Ber( p i ) until we see t successes.
We now have the following string of equalities:
[Equation]
where we used the fact that g is a linear function, and that E [[[ B (cid:96) ]]]
= p (cid:96)i in order to substitute
[Equation]
B 1 ,..., B r
[[[ g ( B 1 , . . .
, B r )]]]
= g ( p i , p 2 i , . .
, p r i ) .
Furthermore, we divide log Y =
f ( Y )
+ h ( Y ) , where f ( z ) is the degree- r Taylor expansion of log
z at 1 , and h ( z )
= log z − f ( z ) is the error in the degree- r Taylor expansion of log( z ) , i.e., h ( z )
= log( z )
− f ( z ) .
Finally, by construction of g , E
[[[ f ( Y )]]] =
g ( p i , p 2 i , . .
, p ri ) , which gives the desired equality.
Verifying Y is subgamma.
Recall that X is the number of independent draws from a Ber( p ) distribution until we see t successes.
In other words, we may express X =
X 1 + · · · + X t , where
X i is the number of draws of Ber( p ) before we get a single success.
Then, we always satisfy
[Equation]
This, in turn, implies that for any r ≥
[Equation]
where the ﬁrst line is by Jensen’s inequality, and the second is by the triangle inequality and Hölder inequality.
Finally, we use the tail bound on X
i to upper bound the expectation of | X i | r .
Then, we have
[Equation]
when | λ | sufﬁciently smaller than p
[Equation]
Then
[Equation]
since X 1 , . . .
, X t are all independent
[Equation]
we have
[Equation]
and this bound is valid whenever | λ | is sufﬁciently smaller than t .
C Omitted Proofs from Section
Proof of Lemma 3.1.
The approach is to estimate
[Equation]
There exists an algorithm using O (log(1 /(cid:15) ) /(cid:15) 2 ) samples to estimate the above quantity: for j ∈ { 0 , . . .
, O (1 /(cid:15) 2 ) } , one takes a sample i j ∼ D and uses r
= O (log(1 /(cid:15) ))
additional samples s 1 , . . .
, s r ∼ D to deﬁne
[Equation]
and
[Equation]
Then, let Z be the average of all Z j ’s, which is an unbiased estimate to E i ∼D (cid:2)(cid:2)(cid:2) g ( p i , p 2 i , . .
, p r i ) (cid:3)(cid:3)(cid:3) .
Since g is bounded (from Lemma 2.3), the variance of O (1 /(cid:15) 2 ) such values is a large constant factor smaller than (cid:15) 2 .
By Chebyshev’s inequality, we estimate (10) to error ± (cid:15) with probability at least 0 .
With that estimate, we will now use Lemma 2.4.
Speciﬁcally, the entropy of D is exactly
E i ∼D [[[log(1 /p i )]]] , and we have
[Equation]
where η i is the result of running LogEstimator ( D , i ) .
Proof of Lemma 3.2.
We note that since log( · ) is monotone increasing, we must have H ≥ ˜ H .
To see that it is not much larger, note that we always have log z
= ln( z ) / ln(2) ≤ ( z − 1) / ln(2) , which means
[Equation]
Proof of Lemma 3.4.
Substituting the r (cid:96) values into Lemma 3.3 ensures E (cid:2)(cid:2)(cid:2)
Error 2 (cid:3)(cid:3)(cid:3) ≤ (cid:15) 2 / 10 .
Hence the estimator is within ± (cid:15) of ˜ H with probability 0 .
9 by Chebyshev’s inequality.
For the intervals (cid:96) = { 1 , . . .
, L − 1 } , we always spend r (cid:96) tries to determine whether a sample falls within a particular interval.
Note that we take one sample to determine i ∼ D , and then we take at most b (cid:96) samples.
Therefore, the sample complexity for these is
[Equation]
where we used the fact that
[Equation]
Finally, it remains to bound the expected sample complexity of the bucket L .
Here, we note
[Equation]
= O (
[Equation] [Equation]
Therefore, the expected sample complexity for interval L is r L · (cid:80) ki
=1 p i · t p i
=
O
( k log 4 (1 /(cid:15) ) /(cid:15) 2 ) .
D Conjectured Lower Bound
Recall that without a memory constraint the sample complexity is known to be n = Θ(max { (cid:15) − 1 · k/ log( k/(cid:15) ) , (cid:15) − 2 log 2 k } )
[VV17, VV11, JVHW15, WY16].
To prove a Ω( k/(cid:15) 2 ) lower bound for the memory constrained version, we conjecture the following randomized process can be used to generate distributions over [2 k ] that look alike to any constant space algorithm that uses o ( k/(cid:15) 2 ) samples but they have different entropies.
Suppose we have k Bernoulli random variables with parameter α : Y 1 , . . .
, Y k .
And, we have k
Rademacher random variables Z 1 , . . .
, Z k (that are +1 or − 1 with probability 1 / 2 ).
We construct distribution p in such a way that it is uniform over k pairs of elements (1 , 2) , (3 , 4) , . . .
, (2 k − 1 , 2 k ) .
However, conditioning on pair (2 i − 1 , 2 i ) , we may have a constant bias based on the random variable Y i .
And, we decide about the direction of the bias based on Z i .
More precisely, we set the probabilities in p as follows:
[Equation]
Now, it is not hard to show that if we generate two distributions as above with α = (1 + (cid:15) ) /and
α =
(1 − (cid:15) ) / 2 , then their entropies are Θ( (cid:15) ) separated with a constant probability.
Thus, any algorithm that can estimate the entropy has to distinguish α
= (1 + (cid:15) ) / 2 from α
= (1 − (cid:15) ) / 2 .
Intuitively, to learn α , we would require to determine Ω(1 /(cid:15) 2 ) many of Y i ’s. Since we have only a constant words of memory, we cannot perform the estimation of the Y i ’s in parallels.
Thus, any natural algorithm would require to draw Ω( k/(cid:15) 2 ) samples.