Paper:
We aim to model 3D properties of the scenes from single 2D images. Learning 3D scenes from 2D images is a long-standing problem in computer vision with applications in related fields such as simulation and robotics. We propose pix2scene, a deep generative-based approach that represents the 3D scene in a learnt latent variable decoded into a viewpoint-dependent representation that can be rendered. Our method learns the depth of the scene and leverages a local smoothness assumption to extract the orientation of visible scene points. We achieve this using an encoderdecoder adversarial learning mechanism and a novel differentiable renderer to train the 3D model in an end-to-end fashion, using only images. We showcase the generative ability of our model qualitatively on the ShapeNet dataset (Chang et al., 2015). We also demonstrate that our model can predict the structure of scenes from various, previously unseen view points. Finally, we evaluate the effectiveness of the learned 3D scene representation in supporting 3D spatial reasoning.

Review:
This paper introduces a method to create a 3D scene model given a 2D image and a camera pose. The method is: (1) an "encoder" network maps the image to some latent code vector, (2) a "decoder" network uses the code and the camera pose to create a depthmap, (3) surface normals are computed from the depthmap, and (4) these outputs are fed to a differentiable renderer which reconstructs the input image. At training time, a discriminator provides feedback to (and simultaneously trains on) the latent code and the reconstructions. The model is self-supervised by the reconstruction error and the GAN setup. Experiments show compelling results in 3D scene generation for simple monochromatic synthetic scenes composed of an empty room corner and floating ShapeNet shapes. 
This is a nice problem, and if the approach ever works in the real world, it will be useful. On synthetic environments, the results are impressive.
The paper seems to claim more ground than it actually covers. The abstract says "Our method learns the depth and orientation of scene points visible in images", but really only the depth is learned, and the "orientation" is an automatically-computed surface normal, which is a free byproduct of any depth estimate. The "surfel" description includes a reflectance vector, but this is never estimated or further described in the paper, so my guess is that it is simply treated as a scalar (which equals 1). Taking this reflectance issue together with the orientation issue, the model is not really estimating surfels at all, but rather just a depthmap, which makes the method seem considerably less novel. Furthermore, the differentiable rendering (eq. 1) appears to assume that all light sources are known exactly -- this is not a trivial assumption, and yet it is never mentioned in the paper. The text suggests that only an image is required to run the model, but Figure 3 shows that the networks are conditioned on the camera pose -- exact knowledge of the camera pose is difficult to obtain precisely in real settings, so this again is not an assumption to ignore. 
To rewrite the paper more plainly, one might say that it receives a monochrome image as input, estimates a depthmap, and then shades this depthmap using perfect knowledge of lighting and camera pose, which reconstructs the input. This may sound less appealing, but it also seems more accurate.
The paper is also missing some details of the method and evaluation, which I hope can be cleared up easily.
- What is happening with the light source? This is critical in the shading equation (eq. 1), and yet no information is given on it -- we need the color and the position of every light in the scene. 
- How is the camera pose represented? Section 3.3.3 says conditional normalization is used, but what exactly is fed to the network that estimates these conditional normalization parameters? 
- What is the exact form of the reconstruction error? An equation would be great.
- How is the class-conditioning done in 4.2?
- In Eq. 4, the first usage of D_\theta should use only the object part of the vectors, and the second usage should use only the geometric part, right? Maybe this can be cleared up with a second D_subscript.
- I do not understand the "interleaved" training setup in 4.4.1. Please explain that more. 
- It is not clear to me why the task in 4.4.2 needs any supervised training at all, if the classification is just done by computing L2 distances in the latent space. What happens with "0 sampled labels"?
Overall, I like the paper, and I can imagine others in my group liking it. I hope it gets in, assuming the technical details get cleaned up and the language gets softer.