Estimation of Nonlinear Functionals of Densities With Confidence
This paper introduces a class of k-nearest neighbor (k-NN) estimators called bipartite plug-in (BPI) estimators for estimating integrals of nonlinear functions of a probability density, such as Shannon entropy and Renyi entropy. The density is assumed to be smooth, have bounded support, and be uniformly bounded from below on this set. Unlike previous k-NN estimators of nonlinear density functionals, the proposed estimator uses data-splitting and boundary correction to achieve lower mean square error. Specifically, we assume that i.i.d. samples from the density are split into two pieces of cardinality and, respectively, with samples used for computing a k-NN density estimate and the remaining samples used for empirical estimation of the integral of the density functional. By studying the statistical properties of k-NN balls, explicit rates for the bias and variance of the BPI estimator are derived in terms of the sample size, the dimension of the samples, and the underlying probability distribution. Based on these results, it is possible to specify optimal choice of tuning parameters, for maximizing the rate of decrease of the mean square error. The resultant optimized BPI estimator converges faster and achieves lower mean squared error than previous k-NN entropy estimators. In addition, a central limit theorem is established for the BPI estimator that allows us to specify tight asymptotic confidence intervals.
entropy estimation adaptive estimators bias and variance tradeoff bipartite k-nearest neighbor (k-nn) graphs concentration bounds convergence rates data-splitting estimators