Paper:
Visual Interaction Networks: Learning a Physics Simulator from Video
From just a glance, humans can make rich predictions about the future of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains or require information about the underlying state. We introduce the Visual Interaction Network, a generalpurpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.

Review:
Summary
---
Interaction Networks (INs) [2] and the Neural Physics Engine [7] are two recent and successful instantiations of a physics engine. In each case a hand programed 2d physics engine was used to simulate dynamics of 2d balls that might bounce off each other like billiards, orbit around a made up gravitational well, or a number of other settings. Given ball states (positions and velocities), these neural approaches predict states at subsequent time steps and thus simulate simple entity interaction. This paper continues in the same vein of research but rather than taking states as input, the proposed architecture takes images directly. As such, the proposed Visual Interaction Networks (VINs) must learn to embed images into a space which captures state information and learn how to simulate 2d physics in that embedding space.
The VIN comes in three parts.
1) The visual encoder is a CNN which takes a sequence of frames which show the three most recent states of a simulation and embeds them into a different embedding for each object (the number of objects is fixed). Some tricks help the CNN focus on pairs of adjacent frames and do position computations more easily.
2) The dynamics predictor does the same job as an IN but takes state embeddings as input (from the visual encoder) and predicts state embeddings. It's implemented as a triplet of INs which operate at different time scales and are combined via an MLP.
3) The state decoder is a simple MLP which decodes state embeddings into hand coded state vectors (positions and velocities).
Hand coded states are be extracted from the ground truth 2d simulation and a loss is computed which forces decoded state embeddings to align to ground truth states during training, thus allowing the dynamics predictor to learn an appropriate 2d physics model.
This model is tested across a range of dynamics similar to that used in INs. Performance is reported both in terms of MSE (how close are state predictions to ground truth states) and relative to a baseline which only needs to extract states from images but not predict dynamics. Across different number of objects and differing dynamics VINs outperform all baselines except in one setting. In the drift setting, where objects do not interact, VINs are equivalent to an ablated version that removes the relational part from each IN used in the dynamics predictor. When compared to the simulation rolled out by the ground truth simulator using MSE all models show increased errors longer from the initial state (as expected), but VIN is smallest at every time step. A particularly interesting finding is that VINs even outperform INs which are given ground truth states and need not decode them from images. This is hypothesized to be due to noise from the visual component of VINs. Finally, videos of VIN rollouts are provided and appear qualitatively to convincingly simulate each intended 2d dynamics.
Strengths
---
This is a well written paper which presents the next step in the line of work on neural physics simulators, evaluates the proposed method somewhat thoroughly, and produces clean and convincing results. The presentation also appropriately orients the work relative to old and new neural physics simulators.
Weaknesses
---
This paper is very clean, so I mainly have nits to pick and suggestions for material that would be interesting to see. In roughly decreasing order of importance:
1. A seemingly important novel feature of the model is the use of multiple INs at different speeds in the dynamics predictor. This design choice is not
ablated. How important is the added complexity? Will one IN do?
2. Section 4.2: To what extent should long term rollouts be predictable? After a certain amount of time it seems MSE becomes meaningless because too many small errors have accumulated.  This is a subtle point that could mislead readers who see relatively large MSEs in figure 4, so perhaps a discussion should be added in section 4.2.
3. The images used in this paper sample have randomly sampled CIFAR images as backgrounds to make the task harder.
While more difficult tasks are more interesting modulo all other factors of interest, this choice is not well motivated.
Why is this particular dimension of difficulty interesting?
4. line 232: This hypothesis could be specified a bit more clearly. How do noisy rollouts contribute to lower rollout error?
5. Are the learned object state embeddings interpretable in any way before decoding?
6. It may be beneficial to spend more time discussing model limitations and other dimensions of generalization. Some suggestions:
    * The number of entities is fixed and it's not clear how to generalize a model to different numbers of entities (e.g., as shown in figure 3 of INs).
    * How many different kinds of physical interaction can be in one simulation?
    * How sensitive is the visual encoder to shorter/longer sequence lengths? Does the model deal well with different frame rates?
Preliminary Evaluation
---
Clear accept. The only thing which I feel is really missing is the first point in the weaknesses section, but its lack would not merit rejection.