Paper:
Learning to See Physics via Visual De-animation
We introduce a paradigm for understanding physical scenes without human annotations. At the core of our system is a physical world representation that is first recovered by a perception module and then utilized by physics and graphics engines. During training, the perception module and the generative models learn by visual de-animation -interpreting and reconstructing the visual information stream. During testing, the system first recovers the physical world state, and then uses the generative models for reasoning and future prediction. Even more so than forward simulation, inverting a physics or graphics engine is a computationally hard problem; we overcome this challenge by using a convolutional inversion network. Our system quickly recognizes the physical world state from appearance and motion cues, and has the flexibility to incorporate both differentiable and non-differentiable physics and graphics engines. We evaluate our system on both synthetic and real datasets involving multiple physical scenes, and demonstrate that our system performs well on both physical state estimation and reasoning problems. We further show that the knowledge learned on the synthetic dataset generalizes to constrained real images.

Review:
This paper presents an approach to physical scene understanding which utilizes the combination of physics engines and renderers.  The approach trains a network to predict a scene representation based on a set of input frames and subject to the constraints of the physics engine.  This is trained using either SGD or reinforcement learning depending on whether a differentiable physics engine is used.  The approach is applied to several simple scenarios with both real and synthetic data.
Overall the idea in the paper is interesting and promising, however, the paper as it's written has several issues.  Some of these may be able to be addressed in a rebuttal.
First, there are some significant missing references, specifically related to using physical models in human tracking.  Kyriazis and Argyros (CVPR2013, CVPR 2014); Pham et al (PAMI 2016); Brubaker et al (ICCV 2009, IJCV 2010); Vondrak et al (SIGGRAPH 2012, PAMI 2013).  In terms of simple billiards-like scenarios, see also Salzmann and Urtasun (ICCV 2011).  This is a significant blind spot in the related works and should be addressed.
Second, the paper is missing some important technical details.  In particular, it is never specified how the system actually works at test time.  Is it simply run on successive sets of three input frames independently using a forward pass of the perception model?  Does this mean that the physics and graphics rendering engines are not used at all at test time?  If this is the case, why not use the physics engine at test time?  It could, for instance, be used as a smoothing prior on the per-frame predictions of the perception model?  This issue is particularly relevant for prediction of stability in the block world scenario.  Is the network only run on a single set of three frames?  This makes it particularly difficult to assess the technical aspects of this work.
Other comments:
 - Absolute mass is never predictable without knowing something about what an object is made of.  Only relative mass plays a role in the physics and visual perception can only tell you volume (assuming calibration), not density.  Any network which appears to "learn" to predict absolute mass is really just learning biases about mass/density from the training set.  This is fine, but important to recognize so that overboard claims aren't made.
 - The table entries and captions are inconsistent in Fig 7(b-c).  The table used "full" and "full+", whereas the caption references "joint" and "full".