Q: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
Recent work (Bengio et al., 2013) has shown how Denoising Auto-Encoders(DAE) become generative models as a density estimator. However, in practice, the framework suffers from a mixing problem in the MCMC sampling process and no direct method to estimate the test loglikelihood. We consider a directed model with an stochastic identity mapping (simple corruption process) as an inference model and a DAE as a generative model. By cascading these models, we propose Cascading Denoising AutoEncoders(CDAE) which can generate samples of data distribution from tractable prior distribution under the assumption that probabilistic distribution of corrupted data approaches tractable prior distribution as the level of corruption increases. This work tries to answer two questions. On the one hand, can deep directed models be successfully trained without intractable posterior inference and difficult optimization of very deep neural networks in inference and generative models? These are unavoidable when recent successful directed model like VAE (Kingma & Welling, 2014) is trained on complex dataset like real images. On the other hand, can DAEs get clean samples of data distribution from heavily corrupted samples which can be considered of tractable prior distribution far from data manifold? so-called global denoising scheme. Our results show positive responses of these questions and this work can provide fairly simple framework for generative models of very complex dataset. Proceedings of the 32 International Conference on Machine Learning, Lille, France, 2015. JMLR: W&CP volume 37. Copyright 2015 by the author(s).
A:
Cascading Denoising Auto-Encoder as a Deep Directed Generative Model