Paper:
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D. The existing methods either are ineffective or only tackle the problem partially. In this paper, we propose an end-to-end model that simultaneously solves all three tasks in realtime given only a single RGB image. The essence of the proposed method is to improve the prediction by i) parametrizing the targets (e.g., 3D boxes) instead of directly estimating the targets, and ii) cooperative training across different modules in contrast to training these modules individually. Specifically, we parametrize the 3D object bounding boxes by the predictions from several modules, i.e., 3D camera pose and object attributes. The proposed method provides two major advantages: i) The parametrization helps maintain the consistency between the 2D image and the 3D world, thus largely reducing the prediction variances in 3D coordinates. ii) Constraints can be imposed on the parametrization to train different modules simultaneously. We call these constraints "cooperative losses" as they enable the joint training and inference. We employ three cooperative losses for 3D bounding boxes, 2D projections, and physical constraints to estimate a geometrically consistent and physically plausible 3D scene. Experiments on the SUN RGB-D dataset shows that the proposed method significantly outperforms prior approaches on 3D object detection, 3D layout estimation, 3D camera pose estimation, and holistic scene understanding.

Review:
I have read the rebuttal and I am satisfied with the authors responses. I am happy accepting the paper.  ----------- This paper proposes to take an image as input and output: (a) camera pose (b) room layout as a 3D box, and (c) for every object, the 3D box. The method consists of a global network that predicts (a,b) and a local network to predict (c). In addition to the standard losses, the paper proposes physical losses that entangle the global and local predictions by using the predicted camera pose in the 3D bounding box loss and a 2D projected bounding box, as well as a loss that ensures that each object is inside the room.  I like many aspects of the submission but I am concerned that the experiments do not clearly support the claims of the paper.  + The parametrization of the 3D box is clever and, while it is simple geometry, I do not think I have seen it before. It quite cleanly refactors the estimation of the coordinates so that it's easy to actually get the box to align with the 2D box.  + Forcing the two networks to work together makes a lot of sense and the particular formulation of how the two should work together is clean: factoring it so that there's only one camera pose so that the objects are not independently implicitly predicting a camera pose is quite nice. Similarly, the non-intersection loss is quite nice.   Experimentally, the paper is tricky to evaluate -- there isn't a lot of work on this task, so it's difficult to do meaningful comparisons. 3DGP and Hedau et al.'s VP estimator are very much out of date, leaving IM2CAD and ablations. My overall concern with the experiments is that I'm not convinced the paper clearly demonstrates the benefits of the system.  -It's not clear that the "individual" vs "cooperative" result is explained by the cooperative estimation: estimating 3D pose from a cropped 2D bounding box is a difficult problem, and so it's not surprising that the setting that can see the full image outperforms the network that can only see cropped bounding boxes. Indeed this was observed in Tulsiani et al. CVPR 2018, in which not providing the context of the full image for estimating object pose led to performance losses.  -The comparison to IM2CAD is difficult to assess. It's clear that the paper improves on it by a little bit in terms of holistic understanding and by a lot in terms of the layout estimation, but it's unclear where that performance gain comes from. The base architectures change from VGG16 -> Resnet34 for object detection (and maybe also the object detection pipeline) and VGG16 + hand-crafted optimization -> Direct estimation via resnet34 for layout estimation.   -The ablative analysis does not clearly support the importance of most of the proposed constraints. For example, S3 ablation (without L_PHY), the drop without L_PHY is not particularly big, mainly <0.5% and sometimes not including L_PHY improves things (precision, for instance). This could easily be explained by plenty of other effects like the natural variance of training runs. Similarly, in the S2 ablation, getting rid of the 2D projection loss improves things dramatically.   Smaller stuff that needs to be fixed but which does not impact my review a lot:  -The very first sentence is misciting Oliva 2005. Oliva 2005 has nothing to do with getting full, metric 3D out of an image and indeed, comments "Common to these representations is their holistic nature: the structure of the scene is inferred, with no need to represent the shape or meaning of the objects". -SoftNMS is not a detector. SoftNMS is a post-detection heuristic used to improve the output of a detector. The paper should identify which detector was used. - Missing citations. The paper is missing an important line of work from the 2010-2014 era of joint optimization of objects and layout, e.g., Lee, et al. NIPS 2010; Schwing et al. Box in the Box ICCV 2013. (which is notably not slow) -Paragraph discussing the ablative analysis S1/S2/S3 has the order of S2/S3 flipped.