Paper:
Conditional Image Generation with PixelCNN Decoders
This work explores conditional image generation with a new image density model based on the PixelCNN architecture. The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks. When conditioned on class labels from the ImageNet database, the model is able to generate diverse, realistic scenes representing distinct animals, objects, landscapes and structures. When conditioned on an embedding produced by a convolutional network given a single image of an unseen face, it generates a variety of new portraits of the same person with different facial expressions, poses and lighting conditions. We also show that conditional PixelCNN can serve as a powerful decoder in an image autoencoder. Additionally, the gated convolutional layers in the proposed model improve the log-likelihood of PixelCNN to match the state-ofthe-art performance of PixelRNN on ImageNet, with greatly reduced computational cost.

Review:
This paper investigates the problem of conditional image generation based on the pixel (R/C)NN framework. Building upon the previous pixel (R/C)NN framework, this paper proposes a gated extension of pixel CNN using multiplicative interactions which can be trained efficiently. The main focus of this paper lies in the conditioning modeling of image distribution p(x|h), where h is assumed to be a vector (e.g., one-hot vector or feature vector from pre-trained encoder networks). In addition, the proposed gated pixel CNN can be jointly trained with an encoder network, which is referred as pixel CNN auto-encoders. The model is validated in experiments on CIFAR-10, ImageNet, and Portraits dataset. For unconditional modeling, the performance of gated extension is on par with (slightly better on ImageNet) original pixel CNN. For conditional modeling, the model is able to generate reasonably-looking and diverse samples that agrees with the conditioning vector (either class label or face embedding).
The paper solves a significant problem in generative modeling and the paper is quite interesting. However, reviewer feels the current version is not polished well due to several issues in the experimental section. For rebuttal, please focus on the (*), (**), (***) and (***) mentioned in the following paragraphs. Reviewer is willing to change the score if all the concerns are addressed in the rebuttal.  Novelty: The proposed model is technically novel in the sense that it explores the conditional modeling with the recent pixel (R/C)NN framework. The extension of pixel-CNN to auto-encoders is interesting.  In a high-level, however, generating images with conditioning variables has been explored to some extent: -- Learning to Generate Chairs with Convolutional Neural Networks by Dosovitskiy et al. -- Deep Generative Image Modeling using a Laplacian Pyramid of Adversarial Networks by Denton et al. -- Deep Convolutional Inverse Graphics Network by Kulkarni et al. -- Attribute2Image: Conditional Image Generation from Visual Attributes by Yan et al. -- Generating Images from Captions with Attention by Mansimov et al.  Clarity: The paper is easy to follow and clearly written in general. However, it is hard to reproduce the results solely based on the description. Reviewer suggests the authors to submit supplementary document or consider code release in the future. References are incomplete.  Technical quality: The current experimental result fails to convince reviewer for the following reasons: -- In unconditional modeling, whether the proposed gated pixel CNN is indeed significant enough compared to original pixel CNN [21]? In the paper, the only comparisons between original pixel CNN and gated pixel CNN are detailed in Table 1 and Table 2. However, it is possible that performance gap comes from more parameters and deeper architecture of gated pixel CNN. *For rebuttal, please describe the quality of samples (comment the difference compared to original pixel CNN) and report the numbers under fair comparison. -- In conditional modeling, whether the improvements in visual quality is a sign of overfitting? It would be more convincing if the nearest training examples are added along with generated samples (as has been done in LAP-GAN paper). **For rebuttal, please comment on the overfitting risk and visual similarity between generated samples and nearest training examples. -- In conditional modeling, whether the proposed model is superior to original pixel CNN and other existing previous work is unknown. It seems that Eq.(4) and Eq.(5) can also be applied to original pixel CNN [21] as well. For other existing work on conditional image generation, please refer to class-conditional generation on CIFAR-10 dataset (LAP-GAN) and text-conditional generation on MS-COCO dataset (alignDRAW [10]). It would be more convincing if the proposed model is evaluated against the original pixel CNN [21] or other existing previous work. ***For rebuttal, please comment on the performance of conditional generation using original pixel CNN.   Usefulness: One issue with the original pixel (R/C)NN framework is that the generation cannot be made in parallel (one needs to generate one pixel at a time). This would be an issue in practice since the competitor models (condVAE and condGAN) can generate images in a much more efficient way. ****For rebuttal, please make it clear if this problem has been resolved in the proposed method.  Detailed Comments: -- Eq. (2): “W_{g,f}” → “W_{k,g}” -- Eq. (4) vs. Eq. (5): are you applying conditioning variable to every pixel CNN layer or not? Which one has been used in the experiments (imagenet and portraits)? Any suggestions when choosing Eq. (4) or Eq. (5) in practice? -- The description of the “portraits” dataset that being used is missing. How many images are there in the dataset? Whether the identities are overlapping between training set and testing set?