Estimating entropy on m bins given fewer than m samples
Consider a sequence p(N) of discrete probability measures, supported on m(N) points, and assume that we observe N independent and identically distributed (i.i.d.) samples from each p(N). We demonstrate the existence of an estimator of the entropy, H (p(N)),which is consistent even if the ratio N/m(N) is bounded (and, as a corollary, even if this ratio tends to zero, albeit at a sufficiently slow rate).
estimation entropy consistency approximation theory bias distribution-free bounds