Paper:
Learning to Exploit Stability for 3D Scene Parsing
Human scene understanding uses a variety of visual and non-visual cues to perform inference on object types, poses, and relations. Physics is a rich and universal cue that we exploit to enhance scene understanding. In this paper, we integrate the physical cue of stability into the learning process by looping in a physics engine into bottom-up recognition models, and apply it to the problem of 3D scene parsing. We first show that applying physics supervision to an existing scene understanding model increases performance, produces more stable predictions, and allows training to an equivalent performance level with fewer annotated training examples. We then present a novel architecture for 3D scene parsing named Prim R-CNN, learning to predict bounding boxes as well as their 3D size, translation, and rotation. With physics supervision, Prim R-CNN outperforms existing scene understanding approaches on this problem. Finally, we show that finetuning with physics supervision on unlabeled real images improves real domain transfer of models training on synthetic data.

Review:
The goal of this paper is to output a set of 3D bounding boxes and set of dominant planes for a scene depicted in a single image. The key insight is to incorporate stability constraints in the 3D layout, i.e., the reconstructed 3D boxes should not move too far under simulation (in Bullet) with physical forces (gravity, friction). Parameters for 3D boxes are regressed using a modified R-CNN training loss and dominant planes for the walls and floors are regressed via a RNN. A stability criterion is used to update the output 3D scene (via REINFORCE) where the predicted 3D layout is run through Bullet simulator and 3D displacements are checked. Results are shown on synthetic (SUNCG, SceneNet RGB-D) and real (SUN RGB-D) datasets, out-performing the factored 3D approach of [Tulsiani18].  Positives: I like the direction this paper takes towards incorporating physics for this problem. The paper demonstrates an interesting approach that leverages modern recognition components and shows positive results on a hard problem. I also like the analysis performed on synthetic and transfer to real scenes. The paper is well written (particularly given the amount of conveyed information in tight space) and reasonably cites prior work (although I have a couple more reference suggestions below).   Negatives: My comments are mostly minor.   A. Probably the biggest issue is the submission missed a highly relevant paper that also used stability constraints for inferring 3D scene layout:  Abhinav Gupta, Alexei A. Efros and M. Hebert, Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics. In ECCV 2010.  There are big differences between the current submission and the above paper ([Gupta10] infers layout for outdoor scenes, have a different mechanism for reasoning about stability, and use 2010 pre-deep learning tech). I think it would be good to tone down a bit the first contribution in the introduction in light of this paper.  B. The second contribution mentions that “a end to end [sic x2] scene reconstruction network” is proposed. Since the simulator is not differentiable (L172), is this still considered an end-to-end network?  In general, I think this would be a good poster. I like the overall story and the execution looks good. There are many system components, so I think it would be hard to reproduce without source code.   Minor comments:  + L161 “parameterizated”  + Why is there a big bend around ~5x10^-2 in Fig 3(a) and at 10^-1 in Fig 5(a)?  + The paper mentions that the model is trained with unlabeled synthetic data. However, it looks like ground truth bounding box annotations are used (L211). If this is the case, it would be great to rephrase this to avoid confusion since it’s not really unlabeled.  + It may be good to also include some relevant work on higher-order 3D scene layout modeling, e.g.,   Characterizing Structural Relationships in Scenes Using Graph Kernels. Matthew Fisher, Manolis Savva, Pat Hanrahan. SIGGRAPH 2011.