Paper:
BOUNCE AND LEARN: MODELING SCENE DYNAMICS WITH REAL-WORLD BOUNCES
We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajectories and infer two underlying physical properties that govern bouncing -restitution and effective collision normals. Our model, Bounce and Learn, comprises two modules -a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model physical interactions for the prediction task given physical parameters and observed pre-collision 3D trajectories. To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices. Our proposed model learns from our collected dataset of real-world bounces and is bootstrapped with additional information from simple physics simulations. We show on our newly collected dataset that our model out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.

Review:
The authors present both a dataset of videos of a real-world foam ball bouncing and a model to learn the trajectory of the ball at collision (bounce) points in these videos.  The model is comprised of a Physics Inference Module (PIM) and a Visual Inference Module (VIM).  The PIM takes in both a vector of physical parameters (coefficient of restitution and collision normal) and a point cloud representation of the pre-bounce trajectory, and produces a point cloud representation of the post-bounce trajectory (or, rather, an encoded version of such).  The VIM takes in an image and ground-truth bounce location and produces the physical parameters of the surface at that location.
I find the paper well-written and clear.  The motivation in the introduction is persuasive and the related work section is complete.  However, the authors are introducing both a new training paradigm (to my knowledge unused in the literature) and a new model, and without any existing baselines to compare against I find it a bit difficult to understand how well the model works.  
Overall, the authors’ model is somewhat complicated and not as general as it initially seems.  To justify this complication I would like to see more convincing results and benchmarking or application to more than one single dataset (e.g. non-spheres bouncing).
Here are some specific concerns:
1)  I could not find a link to an open-sourced version of the dataset(s).  Given that the authors emphasize the dataset as a main contribution of the paper, they should open-source it and make the link prominent in the main text (apologies if I somehow missed it).
2)  The authors claim in multiple places that the model is trained end-to-end, but this does not seem to be the case.  Specifically, the PIM is pre-trained on an auxiliary dataset from simulation.  The trajectory encoder also seems to be pre-trained (though I could be wrong about that, see my question below).  Furthermore, there is a bit of hand-holding:  The PIM uses ground-truth state for pre-training, and the VIM gets the ground-truth bounce location.  In light of this, the model seems a lot less general and end-to-end than implied in the abstract and introduction.
3)  No comparison to existing baselines.  I would like to see how the authors’ model compares to standard video prediction algorithms.  The authors could evaluate their model with respect to pixel loss (after ground-truth rendering) and compare to a video prediction algorithm (such as PredNet by Lotter, Kreiman, & Cox, 2016).  Given that the authors’ method uses some extra “privileged” information (as described in point 2), it should far out-perform algorithms that train only on video data, and such a result would strengthen the paper a lot.
4)  Table 1 is not a very convincing demonstration of performance.  Regardless of baselines, the table does not show confidence intervals.  I would love to see training curves with errorbars of the models on the most important metrics (e.g. Dist and COR Median Absolute Error).
I also was confused about a couple of things:
1)  How was the PointNet trajectory encoder trained?  I did not see this mentioned anywhere.  Were gradients passed through from the PIM?  Was the same network used for both the simulation and real-world data?
2)  The performance of the center-based model in Table 1 seems surprisingly low.  The center-based model should be as good at the Train core, Fix traj. enc. model, since it has access to the ball’s position.  Why is it worse?  Is the VIM at fault?  Or is the sphere-fitting sub-optimal?  How does it compare on the simulated data with ground truth physical parameters?
3)  Lastly, the color-scheme is a bit confusing.  It looks like the foam ball in the videos was rainbow-colored.  However, in the model outputs in trajectory figures time is also rainbow-colored.  This was initially a bit confusing.  Perhaps grayscale for the model outputs would be clearer.