Paper:
Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
We address the problem of learning accurate 3D shape and camera pose from a collection of unlabeled category-specific images. We train a convolutional network to predict both the shape and the pose from a single image by minimizing the reprojection error: given several views of an object, the projections of the predicted shapes to the predicted camera poses should match the provided views. To deal with pose ambiguity, we introduce an ensemble of pose predictors which we then distill to a single "student" model. To allow for efficient learning of high-fidelity shapes, we represent the shapes by point clouds and devise a formulation allowing for differentiable projection of these. Our experiments show that the distilled ensemble of pose predictors learns to estimate the pose accurately, while the point cloud representation allows to predict detailed shape models.

Review:
This paper introduces an approach to learning to predict 3D shape with 2D images only as input. The core idea is to make use of a 3D points-based representation of shape and to additionally predict the pose of the observed object, so as not to require this information during training.  In general, the paper is clearly written and does not suffer from any major flaw. My main concern is that I find the contribution of this paper a bit weak for NIPS. See detailed comments below.  Novelty: - As acknowledged by the authors, there has been an increasing interest in learning to predict 3D shape from 2D information only. In this context, [6] and [11] also represent shape as a 3D point cloud. Furthermore, [13] and [20] also work in the setting where pose is not available during training. I find this to limit the contributions of this submission. - I acknowledge that there can be different approaches to solving the same problem, which could be the focus here. However, the proposed method does not seem to be of great originality, essentially defining a network that directly outputs 3D points and pose. I might be missing something, but currently the paper fails to convince me of the novelty of the method.  Method: - Section 4 discusses the notion of differentiable point clouds, which is achieved by converting the point cloud to a volume. Considering that [6] and [11] have also worked with point cloud representations within deep networks, I do not see why this different strategy is necessary or beneficial. I would appreciate if the authors could clarify this. - When predicting a point cloud, the order of the points in the set is irrelevant. This suggests that there are ambiguities in what the network can predict. I was wondering if the authors have observed problems related to these ambiguities, or if at least they could discuss this issue.  Experiments: - In Section 5.2, it would be interesting to compare with [11], which also uses a point-cloud representation and works with known pose. - In Section 5.3, it would be interesting to compare with [13], which also handles the unknown pose scenario. - For the classification metric, 30 degrees seems like a very large threshold. It would be worth evaluating what happens with smaller thresholds. - In practice, at test time, is there any reason to also predict the pose? It seems it is mostly the shape that is of interest. - In Section 5.3, the authors report results of Ours-basic and Ours, referred to as the full model. Is Ours what was described as "fast" in Section 4.1? If so, considering that it performs better than Ours-basic, is there any value in the basic algorithm? From Section 4.1, it is supposed to be more flexible, but the quantitative experiments don't really reflect this (although Fig. 5 briefly illustrates this).  Related work: - It might be worth citing Cohen & Welling, 2014, "Transformation Properties of Learned Visual Representations", and Worrall et al., ICCV 2017, "Interpretable transformations with encoder-decoder networks". These methods also exploit a point-based representation but in the context of novel view synthesis. - Rhodin et al., CVPR 2018, "Learning Monocular 3D Human Pose Estimation From Multi-View Images" also make use of multiview data with unknown pose 3D human pose estimation.