Paper:
REASONING ABOUT PHYSICAL INTERACTIONS WITH OBJECT-ORIENTED PREDICTION AND PLANNING
Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

Review:
A method is proposed, which learns to reason on physical interactions of different objects (solids like cuboids, tetrahedrons etc.). Traditionally in related work the goal is to predict/forecast future observations, correctly predicting (and thus learning) physics. This is also the case in this paper, but the authors explicitly state that the target is to evaluate the learned model on downstream tasks requiring a physical understanding of the modelled environment.
The main contribution here lies in the fact that no supervision is used for object properties. Instead, a mask predictor is trained without supervision, directly connected to the rest of the model, ie. to the physics predictor and the output renderer. The method involves a planning phase, were different objects are dropped on the scene in the right order, targeting bottom objects first and top objects later. The premise here is that predicting the right order of the planning actions requires understanding the physics of the underlying scene.
I particularly appreciated the fact, that object instance renderers are combined with a global renderer, which puts individual images together using predicted heatmaps for each object. With a particular parametrization, these heatmaps could be related to depth maps allowing correct depth ordering, but depth information has not been explicitly provided during training.
Important issues:
One of the biggest concerns is the presentation of the planning algorithm, and more importantly, a proper formalization of what is calculated, and thus a proper justification of this part. The whole algorithm is very vaguely described in a series of 4 items on page 4. It is intuitively almost clear how these steps are performed, but the exact details are vague. At several steps, calculated entities are “compared” to other entities, but it is never said what this comparison really results in. The procedure is reminiscent of particle filtering, in that states (here: actions) are sampled from a distribution and then evaluated through a likelihood function, resulting in resampling. However, whereas in particle filtering there is clear probabilistic formalization of all key quantities, in this paper we only have a couple of phrases which describe sampling and “comparisons” in a vague manner.
Since the procedure performs planning by predicting a sequence of actions whose output at the end can be evaluated, thus translated into a reward, I would have also liked a discussion (or at least a remark) why reinforcement learning has not been considered here.
I am also concerned by an overclaim of the paper. As opposed to what the paper states in various places, the authors really only evaluate the model on video prediction and not on other downstream tasks. A single downstream task is very briefly mentioned in the experimental section, but it is only very vaguely described, it is unclear what experiments have been performed and there is no evaluation whatsoever.
Open questions:
Why is the proposed method better than one of the oracles?
Minor remarks:
It is unclear what we see in image 4, as there is only a single image for each case (=row) and method (=column). 
The paper is not fully self-contained. Several important aspects are only referred to by citing work, e.g. CEM sampling and perceptual loss. These are concepts which are easy to explain and which do not take much space. They should be added to the paper.
A threshold is mentioned in the evaluation section. A plot should be given showing the criterion as a function of this threshold, as is standard in, for instance, pose estimation literature.
I encourage the authors to use the technical terms “unary terms” and “binary terms” in the equation in section 2.2. This is the way how the community referred to interactions in graphical models for relational reasoning long before deep learning showed up on the horizon, let’s be consistent with the past.
I do not think that the physics module can be reasonable be called a “physics simulator” as has been done throughout the paper. It does not simulate physics, it predicts physics after learning, which is not a simulation.
A cube has not been confused with a rectangle, as mentioned in the paper, but with a rectangular cuboid. A rectangle is a 2D shape, a rectangular cuboid is a 3D polyhedron.