  Energy-based models (EBMs) are a simple yet powerful framework for generative
modeling. They are based on a trainable energy function which defines an
associated Gibbs measure, and they can be trained and sampled from via
well-established statistical tools, such as MCMC. Neural networks may be used
as energy function approximators, providing both a rich class of expressive
models as well as a flexible device to incorporate data structure. In this work
we focus on shallow neural networks. Building from the incipient theory of
overparametrized neural networks, we show that models trained in the so-called
"active" regime provide a statistical advantage over their associated "lazy" or
kernel regime, leading to improved adaptivity to hidden low-dimensional
structure in the data distribution, as already observed in supervised learning.
Our study covers both maximum likelihood and Stein Discrepancy estimators, and
we validate our theoretical results with numerical experiments on synthetic
data.
