We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences of continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We analyze the rates of convergence for our estimators and show that the parametric rate of n −1/2 is achievable when the densities' smoothness s are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n −1/2 rate of convergence. We confirm our theoretical guarantees with a number of simulations.
In contrast, our estimator undersmooths the density, and converges at a slower rate of O n −β/(β+d) when β < d (and the parametric rate O n −1/2 when β ≥ d), but obeys an exponential concentration inequality, which is not known for the estimators of REF .
Nonparametric Estimation of Renyi Divergence and Friends
ICML
Mathematics Computer Science