In this note, we show that exponentially deep belief networks can approximate any distribution over binary vectors to arbitrary accuracy, even when the width of each layer is limited to the dimensionality of the data. We further show that such networks can be greedily learned in an easy yet impractical way.
Sutskever and Hinton REF showed that deep belief networks whose hidden layers have the same dimension as the input space can approximate any function over binary vectors.
Deep, Narrow Sigmoid Belief Networks Are Universal Approximators
Neural Computation
Neural Computation
Mathematics Medicine Computer Science