Paper:
Visual Object Networks: Image Generation with Disentangled 3D Representation
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors-shape, viewpoint, and texture-and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.

Review:
This paper describes a generative model for image formation, with disentangled latent parameters for shape, viewpoint and texture. This is in keeping with the vision as an inverse graphics problem, where image generation is formulated as a parameter search in model space, that when rendered, produces the given image. The difference between the rendered image and the original image is used to train the model. Using inverse graphics as inspiration, this paper learns the following models: 1. An voxel generator that can map the latent 3D shape code to a voxellized 3D shape 2.A differentiable projection module that converts the output of 1 to a 2.5D sketch (depth map) and a silhouette mask, conditional on a latent representation of the required viewpoint 3.A texture generator, that can map the output of 2 to a realistic textured image, conditional on a latent representation of the required texture 4.A 2.5D sketch encoder that can map a 2D image to a 2.5D sketch 5.A texture encoder that maps the texture of the object in a 2D image into a texture latent code The models are learnt adversarially using GANs, and without the need for paired image and shape data using the now common cycle consistency constraints. The training of the GANs are improved using the recently described WGAN that enforces the Lipshitz constraint with a gradient normalizing term, which leads to a better optimization landscape and prevents mode collapse. Results on car and chair datasets indicate qualitative and quantitative (Frechet Inception distance) improvements in image generation by varying pose, texture and shape, compared to competing techniques.  The trained models and disentangled latent parameters of shape, texture and viewpoint allow a mix and match between the same.   Quality This work demonstrates a generative model of image formation with disentangled factors that can be trained without paired training data between factors. The work is well-reasoned and represents an advance in tying together the factors responsible for image formation, and how to train generative models to train each of those, end-to-end.  Clarity The paper is well written, with the motivation GAN losses well-explained.  The following points require the writers’ attention: Line 26 - you mean virtual reality and not visual reality Line 103 - do you classify individual voxels, and not the whole voxel shape as real/fake? More explanation required. Section 3.2 - More details required on the projection module, perhaps a diagram. And how is it differentiable? Figure 5 (b) Texture transfer does not seem to work as well in the first row, where the white car’s texture is transferred. The two cars to whom the texture has been transferred look more grey/silver than white. This could be a consequence of lighting change. Perhaps lighting is another factor that requires consideration as a factor in a disentangled generative model of image formation. Figure 5 (c) It would clarify things if you show the 2 objects you are interpolating between. Results on the 3rd row are not very impressive - the front of the car seems mashed up, and not a very smooth transition between shapes. An explanation in the text is required. Why do you not use the W-GAN loss in the loss functions for 2.5D GAN and texture GAN, when you use it for the shape GAN? In addition to the applications listed in 4.3, it would be useful to see if you can apply this to objects for which the 3D model does not exist during training. Can you go from 2D images (from 1 or 2 viewpoints) of a new object (unseen during training), and go from the 2D images to the 3D shape, and then generate 2D images of the object from a new viewpoint? This would be a test of how well your model is able to generalize over shapes.   Originality  This paper describes an incremental advance in the development of generative models of image formation. It has enough originality to warrant publication at NIPS.  Significance The inverse graphics style generative model with disentangled factors for image formation, that can be trained without paired training data between the factors is a fundamental development in the field of Computer Vision, and has the potential for significant advances in applications like virtual reality, augmented reality and robotics. This paper presents a well written and comprehensive treatment of an incremental improvement in this direction.