Paper:
Learning Structured Output Representation using Deep Conditional Generative Models
Supervised deep learning has been successfully applied to many recognition problems. Although it can approximate a complex many-to-one function well when a large amount of training data is provided, it is still challenging to model complex structured output representations that effectively perform probabilistic inference and make diverse predictions. In this work, we develop a deep conditional generative model for structured output prediction using Gaussian latent variables. The model is trained efficiently in the framework of stochastic gradient variational Bayes, and allows for fast prediction using stochastic feed-forward inference. In addition, we provide novel strategies to build robust structured prediction algorithms, such as input noise-injection and multi-scale prediction objective at training. In experiments, we demonstrate the effectiveness of our proposed algorithm in comparison to the deterministic deep neural network counterparts in generating diverse but realistic structured output predictions using stochastic inference. Furthermore, the proposed training methods are complimentary, which leads to strong pixel-level object segmentation and semantic labeling performance on Caltech-UCSD Birds 200 and the subset of Labeled Faces in the Wild dataset.

Review:
Summary: A framework for learning complex structured output representations is presented. To this end variational auto-encoders (VAE) are extended to `conditional VAEs,' i.e., conditioned on the input data x.Quality - The paper is mostly well written, could however be improved occasionally. Clarity - The idea is clearly presented but some details are missing. Originality - Conditional VAEs seem to be a straightforward extension of standard VAEs, but certainly worth a discussion Significance - The significance could be improved by a more extensive evaluation showing results for various modificationsComments: - I think the term generative is typically used when learning distributions that also involve the input data. This seems different from the proposed conditional variational auto-encoder technique that is not capable of generating data via p(x). Hence readers could get confused.- Three different methods are possible for inference, i.e., estimation of y given x. Two of them are evaluated using toy data. But which of the ones discussed in Sec. 3.1. do the authors recommend?- The authors mention a gap (l. 148) measurable by the regularization term during/after learning, i.e., the proposal q(z|x,y) and the prior p(z|x) don't exactly agree. How big is this gap?- The proposed solution to close the gap is to introduce a Gaussian stochastic neural network which models the reconstruction term directly, i.e., no regularization using a KL divergence. A weighted combination of the conditional VAE (computes regularization+reconstruction) and the newly introduced network (computes reconstruction) is then proposed as the desirable cost function. This however seems counter-intuitive to me. I would have expected the regularization gap between both distributions to be closer to zero if more weight is placed on the regularization rather than on the reconstruction term. Could the authors provide intuition? How did the authors set/cross-validate \alpha and what was the resulting value?- There is a significant amount of semantic image segmentation techniques using CNNs. For fairness the authors might consider citing a few more recent ones.- More details regarding the noise injection into data x is useful for a reader. What exactly did the authors do?- Moreover as a reader I'm very interested in quantitative results regarding the modifications described in Sec. 3.3.2, i.e., how important is the `latent-to-output' pathway really? In addition I'm curious as to how much performance improved by using the direct output prediction as the input for the conditional prior network? Further I'd appreciate if the authors could clarify whether the conditional prior network as trained using only the direct output prediction as its input or whether both data x and prediction \tilde{y} were used? Although some of the modifications are investigated in Tab. 3, a more careful ablation analysis would seems very useful to a reader.- Since the conditional VAE involves multiple networks I'm wondering whether its improved performance is a result of the larger amount of trainable parameters. Can the authors comment?- Since efficiency is claimed in the abstract I'm wondering about the time for training and inference. Can the authors comment?------------- As pointed out by fellow reviewers some citations could be added for completeness.