We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining. Figure 1 . Top: A denoising auto-encoder defines an estimated Markov chain where the transition operator first samples a corruptedX from C(X|X) and then samples a reconstruction from P θ (X|X), which is trained to estimate the ground truth P (X|X). Note how for any givenX, P (X|X) is a much simpler (roughly unimodal) distribution than the ground truth P (X) and its partition function is thus easier to approximate. Bottom: More generally, a GSN allows the use of arbitrary latent variables H in addition to X, with the Markov chain state (and mixing) involving both X and H. Here H is the angle about the origin. The GSN inherits the benefit of a simpler conditional and adds latent variables, which allow far more powerful deep representations in which mixing is easier (Bengio et al., 2013b) . P(X)
Generative Stochastic Networks Like GibbsNet, generative stochastic networks (GSNs, REF ) also directly parameterizes a transition operator of a Markov chain using deep neural networks.
Deep Generative Stochastic Networks Trainable by Backprop
ArXiv
ArXiv
Mathematics Computer Science