L EARNING L ONG - TERM V ISUAL D YNAMICS WITH R EGION P ROPOSAL I NTERACTION N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning long-term dynamics models is the key to understanding physical common sense.
Most existing approaches on learning dynamics from visual input sidestep long-term predictions by resorting to rapid re-planning with short-term models.
This not only requires such models to be super accurate but also limits them only to tasks where an agent can continuously obtain feedback and take action at each step until completion.
In this paper, we aim to leverage the ideas from success stories in visual recognition tasks to build object representations that can capture inter-object and object-environment interactions over a long range.
To this end, we propose Region Proposal Interaction Networks (RPIN) , which reason about each object’s trajectory in a latent region-proposal feature space.
Thanks to the simple yet effective object representation, our approach outperforms prior methods by a signiﬁcant margin both in terms of prediction quality and their ability to plan for downstream tasks, and also generalize well to novel environments.
Results are available at https://sites.google.com/view/iclr21-rpin.
I NTRODUCTION
As argued by Kenneth Craik, if an organism carries a model of external reality and its own possible actions within its head, it is able to react in much fuller, safer and more competent manner to emergencies which face it (Craik, 1952).
Indeed, building prediction models has been long studied in computer vision and intuitive physics.
In vision, most approaches make predictions in pixelspace (Denton & Fergus, 2018; Lee et al., 2018b; Ebert et al., 2018b; Jayaraman et al., 2018; Walker et al., 2016), which ends up capturing the optical ﬂow (Walker et al., 2016) and is difﬁcult to generalize to long-horizon.
In intuitive physics, a common approach is to learn the dynamics directly in an abstracted state space of objects to capture Newtonian physics (Battaglia et al., 2016;
Chang et al., 2016; Sanchez-Gonzalez et al., 2020).
However, the states end up being detached from raw sensory perception.
Unfortunately, these two extremes have barely been connected.
In this paper, we argue for a middle-ground to treat images as a window into the world, i.e., objects exist but can be accessed only via images.
Images are neither to be used for predicting pixels nor to be isolated from dynamics.
We operationalize it by learning to extract a rich state representation directly from images and build dynamics models using the extracted state representations.
It is difﬁcult to make predictions, especially about the future — Niels Bohr
Contrary to Niels Bohr, predictions are, in fact, easy if made only for the short-term.
Predictions that are indeed difﬁcult to make and actually matter are the ones made over the long-term.
Consider the example of “Three-cushion Billiards” in Figure 1.
The goal is to hit the cue in such a way that it touches the other two balls and contacts the wall thrice before hitting the last ball.
This task is extremely challenging even for human experts because the number of successful trajectories is very sparse.
Do players perform classical Newtonian physics calculations to obtain the best action before each shot, or do they just memorize the solution by practicing through exponentially many conﬁgurations?
Both extremes are not impossible, but often impractical.
Players rather build a physical understanding by experience (McCloskey, 1983;
McCloskey et al., 1983; Kubricht et al., 2017) and plan by making intuitive, yet accurate predictions in the long-term.
Learning such long-term prediction models is arguably the “Achilles’ heel” of modern machine learning methods.
Current approaches on learning physical dynamics of the world cleverly side-step the long-term dependency by re-planning at each step via model-predictive control (MPC) (Allgöwer & Zheng, 2012; Camacho & Alba, 2013).
The common practice is to train short-term dynamical
[Figure]
models (usually 1-step) in a simulator.
However, small errors in short-term predictions can accumulate over time in MPC.
Hence, in this work, we focus primarily on the long-term aspect of prediction by just considering environments, such as the three-cushion billiards example or the PHYRE (Bakhtin et al., 2019) in Figure 1, where an agent is allowed to take only one action in the beginning so as to preclude any scope of re-planning.
How to learn an accurate dynamics model has been a popular research topic for years.
Recently, there are a series of work trying to represent video frames using object-centric representations (Battaglia et al., 2016; Watters et al., 2017;
Chang et al., 2016; Janner et al., 2019;
Ye et al., 2019; Kipf et al., 2020).
However, those methods are either operates in the state space, or ignore the environment information, both of which are not practical in real-world scenarios.
In contrast, our objective is to build a data-driven prediction model that can both: (a) model long-term interactions over time to plan successfully for new instances, and (b) work from raw visual input in complex real-world environments.
Therefore, the question we ask is: how to extract such an effective object representation and perform long-term predictions?
We propose Region Proposal Interaction Network (RPIN) which contains two key components.
Firstly, we leverage the region of interests pooling (RoIPooling) operator (Girshick, 2015) to extract object features maps from the frame-level feature.
Object feature extraction based on region proposals have achieved huge success in computer vision (Girshick, 2015;
He et al., 2017; Dai et al., 2017;
Gkioxari et al., 2019), and yet, surprisingly under-explored in the ﬁeld of intuitive physics.
By using
RoIPooling, each object feature not only contains its own information but also the context of the environment.
Secondly, we extend the Interaction Network and propose Convolutional Interaction
Networks that perform interaction reasoning on the extracted RoI feature maps.
Interaction Networks is originally proposed in (Battaglia et al., 2016), where the interaction reasoning is conducted via
MLPs.
By changing MLPs to convolutions, we can effectively utilize the spatial information of an object and make accurate future prediction of object location and shapes changes.
Notably, our approach is simple, yet outperforms the state-of-the-art object feature extraction methods in both simulation and real datasets.
In Section 5, we thoroughly evaluate our approach across four datasets to study scientiﬁc questions related to a) prediction quality, b) generalization to time horizons longer than training, c) generalization to unseen conﬁgurations, d) planning ability for downstream tasks.
Our method reduces the prediction error by 75% in the complex PHYRE environment and achieves state-of-the-art on the PHYRE cross-task generalization setting.
R ELATED W ORK
Physical Reasoning and Intuitive Physics.
Learning models that can predict the changing dynamics of the scene is the key to building physical common-sense.
Such models date back to “NeuroAnimator” (Grzeszczuk et al., 1998) for simulating articulated objects.
Several methods in recent years have leveraged deep networks to build data-driven models of intuitive physics (Bhattacharyya et al., 2016; Ehrhardt et al., 2017;
Fragkiadaki et al., 2015; Chang et al., 2016; Stewart & Ermon, 2017).
However, these methods either require access to the underlying ground-truth state-space or do not scale to long-range due to absence of interaction reasoning.
A more generic yet explicit approach has been to leverage graph neural networks (Scarselli et al., 2009) to capture interactions between entities in a scene (Battaglia et al., 2018;
Chang et al., 2016).
Closest to our approach are interaction models that scale to pixels and reason about object interaction (Watters et al., 2017; Ye et al., 2019).
However, these approaches either reason about object crops with no context around or can only deal with a predetermined number and order of objects.
A concurrent work (Girdhar et al., 2020) studies using prediction for physical reasoning, but their prediction model is either in the state space or in the pixel space.
Other common ways to measure physical understanding are to predict future judgments given a scene image, e.g., predicting the stability of a conﬁguration (Groth et al., 2018;
Jia et al., 2015; Lerer et al., 2016;
Li et al., 2016a;b).
Several hybrid methods take a data-driven approach to estimate
Newtonian parameters from raw images (Brubaker et al., 2009; Wu et al., 2016;
Bhat et al., 2002;
Wu et al., 2015), or model Newtonian physics via latent variable to predict motion trajectory in images (Mottaghi et al., 2016a;b; Ye et al., 2018).
An extreme example is to use an actual simulator to do inference over objects (Hamrick et al., 2011).
The reliance on explicit Newtonian physics makes them infeasible on real-world data and un-instrumented settings.
In contrast, we take into account the context around each object via RoIPooling and explicitly model their interaction with each other or with the environment without relying on Newtonian physics, and hence, easily scalable to real videos for long-range predictions.
Video Prediction.
Instead of modeling physics from raw images, an alternative is to treat visual reasoning as an image translation problem.
This approach has been adopted in the line of work that falls under video prediction.
The most common theme is to leverage latent-variable models for predicting future (Lee et al., 2018a; Denton & Fergus, 2018; Babaeizadeh et al., 2017).
Predicting pixels is difﬁcult so several methods leverage auxiliary information like back/fore-ground (Villegas et al., 2017a;
Tulyakov et al., 2017; Vondrick et al., 2016), optical ﬂow (Walker et al., 2016; Liu et al., 2017), appearance transformation (Jia et al., 2016; Finn et al., 2016; Chen et al., 2017; Xue et al., 2016), etc.
These inductive biases help in a short interval but do not capture long-range behavior as needed in several scenarios, like playing billiards, due to lack of explicit reasoning.
Some approaches can scale to relative longer term but are domain-speciﬁc, e.g., pre-deﬁned human-pose space (Villegas et al., 2017b; Walker et al., 2017).
However, our goal is to model long-term interactions not only for prediction but also to facilitate planning for downstream tasks.
Learning Dynamics Models.
Unlike video prediction, dynamics models take actions into account for predicting the future, also known as forward models (Jordan & Rumelhart, 1992).
Learning these forward dynamics models from images has recently become popular in robotics for both speciﬁc tasks (Wahlström et al., 2015;
Agrawal et al., 2016;
Oh et al., 2015; Finn et al., 2016) and exploration (Pathak et al., 2017; Burda et al., 2019).
In contrast to these methods where a deep network directly predicts the whole outcome, we leverage our proposed region-proposal interaction module to capture each object trajectories explicitly to learn long-range forward dynamics as well as video prediction models.
Planning via Learned Models.
Leveraging models to plan is the standard approach in control for obtaining task-speciﬁc behavior.
Common approach is to re-plan after each action via Model
Predictive Control (Allgöwer & Zheng, 2012; Camacho & Alba, 2013; Deisenroth & Rasmussen, 2011).
Scaling the models and planning in a high dimensional space is a challenging problem.
With deep learning, several approaches shown promising results on real-world robotic tasks (Finn et al., 2016; Finn & Levine, 2017;
Agrawal et al., 2016; Pathak et al., 2018).
However, the horizon of these approaches is still very short, and replanning in long-term drifts away in practice.
Some methods try to alleviate this issue via object modeling (Janner et al., 2019; Li et al., 2019) or skip connections (Ebert et al., 2018a) but assume the models are trained with state-action pairs.
In contrast to prior works where a short-range dynamic model is unrolled in time, we learn our long-range models from passive data and then couple them with short-range forward models to infer actions during planning.
R EGION P ROPOSAL I NTERACTION N ETWORKS
Our model takes N video frames and the corresponding object bounding boxes at each frame as inputs, and outputs the objects’ bounding boxes and masks for the future T timesteps.
The overall model structure is illustrated in Figure 2.
For each frame, we ﬁrst extract the image features using a ConvNet.
Then we apply RoIPooling (Girshick, 2015; He et al., 2017) to obtain the object-centric visual features.
These features are then
𝐹(𝑥 !" ) 𝐹(𝑥 !"
#$ )
𝐹(𝑥 !" 𝐹(𝑥 !"
#& ) 𝐹(𝑥 !"
Object Location Prediction 𝐹(𝑥 !"
#( ) 𝑥 !
𝑥 " 𝑥 # 𝑧 !
𝑧 " 𝑧 # 𝑓(𝑥 !
,𝑥 " )
𝑓(𝑥 !
,𝑥 # ) 𝑓(𝑥 # ,𝑥 ! )
𝑓(𝑥 # ,𝑥 " )
Interaction Module ObjectFeatures ConvNet
RoIPool
ConvNet
RoIPool
ConvNet
RoIPool 𝐼 " 𝐼 "#$ 𝐼 "#%
ConvNet
RoIPool 𝐹(𝑥 !"
𝐹(𝑥 !"
#* ) 𝐼 "#& forwarded to our Convolutional Interaction Networks (CIN) to perform objects’ interaction reasoning and used to predict future object bounding boxes and masks.
The whole pipeline is trained end-to-end by minimizing the loss between the predicted outputs and the ground-truth.
3. O BJECT -C ENTRIC R EPRESENTATION
We apply the houglass network (Newell et al., 2016) to extract the image features.
Given an input RGB image I ∈ R 3 × H × W , the hourglass network ﬁrstly downsample the input via one 2-strided convolution and 2-strided max pooling layers, and then reﬁne the representation by a U-Net-like encoder-decoder modules (Ronneberger et al., 2015).
The hourglass network provides features with a large receptive ﬁeld and a ﬁne spatial resolution, both of which are crucial to accurately model object-object and object-environment interactions.
We denote the number of output channels of the feature map as d .
On top of this feature map, we use RoIPooling to extract a d × h × w object-centric features.
RoIPooling takes the feature map and a object bounding box as input.
The region corresponding to the bounding box on the feature map is cropped and resize to a ﬁxed spatial size (denoted as h × w ).
We use x ti to represent the feature at t -th timestep for the i -th object.
The key difference from most of the existing methods is that our object feature representation is a 3-dimensional feature map rather than a single vector representation.
Such feature with spatial dimension can represent the objects’ shapes while the vector representation cannot because the spatial dimension is ﬂattened.
As we will show in Section 5, such a representation with spatial information is the key to accurately predict object location and shape changes.
3. C ONVOLUTIONAL I NTERACTION N ETWORKS
To better utilize the spatial information of our object feature map, we propose Convolutional Interaction Networks, an extension of Interaction Network operators on 3-dimensional tensors.
In this section, we ﬁrst brieﬂy review the original Interaction Network (IN) (Battaglia et al., 2016; Watters et al., 2017) and introduce the proposed Convolutional Interaction Networks (CIN).
The original interaction network is a general-purpose model to learn and predict future physical dynamics.
It takes the feature representation of m objects at timestep t : X = { x t 1 , x t 2 , ..., x tm | x ti ∈ R d } and performs object reasoning f
O as well as relational reasoning f
R on these features.
Speciﬁcally, the updated rule of object features can be described as:
[Equation]
In the above equation, f A is the function to calculate the effect of both of object reasoning and relational reasoning results.
And f Z is used to combine the original object state and the reasoning effect.
Finally, f P is used to do future state predictions based on one or more previous object states.
In IN, f O,R,A,Z,P are instantiated by a fully-connected layer with learnable weight.
Convolutional Interaction Networks.
The Convolutional Interaction Module is shown in the gray box on the bottom right of Figure 2.
The input of CIN is m object features at timestep t :
X = { x t 1 , x t 2 , ..., x tm | x ti ∈ R d × h × w } .
The high-level update rule is the same as IN, but the key difference is that we use convolution to instantiate f O,R,A,Z,P .
Such instantiation is crucial to utilize the spatial information encoded in our object feature map and to effectively reason future object states.
Speciﬁcally, we have
[Equation]
One can plug the functions in Equation 1 for better understanding the operations.
In the above equations, ∗ denotes the convolution operator, and [ · , · ] denotes concatenation along the channel dimension.
W R , W Z ∈ R d × 2 d × 3 × 3 , W O , W A ∈ R d × d × 3 × 3 , W P ∈ R d × ( kd ) × 3 × 3 are learnable weights of the convolution kernels with kernel size 3 × 3 .
We also add ReLU activation after each of the convolutional operators.
3. L EARNING R EGION P ROPOSAL I NTERACTION N ETWORKS (RPIN)
Our model predicts the future bounding box and (optionally) masks of each object.
Given the predicted feature x t +1 i , we use a simple two layer MLP decoder to estimate its bounding boxes coordinates and masks.
The bounding box decoder takes a ﬂattened object feature map as input.
It ﬁrstly projects it to d dimensional vector, and outputs 4-d vector, representing the center and size of the box.
The mask decoder is of the same architecture but has 21 × 21 output channels, representing a 21 × 21 binary masks inside the corresponding bounding boxes.
We use the (cid:96) 2 loss for bounding box predictions.
For mask prediction, we use spatial cross-entropy loss which sums the cross-entropy values of a 21 × 21 predicted positions.
The objective can be written as:
[Equation]
We use discounted loss during training (Watters et al., 2017) to mitigate the effect of inaccurate prediction at early training stage and λ t is the discounted factor.
E XPERIMENTAL S ETUP
Datasets.
We evaluate our method’s prediction performance on four different datasets, and demonstrate the ability to perform downstream physical reasoning and planning tasks on two of them.
We brieﬂy introduce the four datasets below.
The full dataset details are in the appendix.
Our code and pretrained models will also be made available.
PHYRE: We use the BALL-tier of the PHYRE benchmark (Bakhtin et al., 2019).
In this dataset, we set T = 5 .
We treat all of the moving balls or jars as objects and other static bodies as background.
The benchmark provides two evaluation settings: 1) within task generalization (PHYRE-W), where the testing environments contain the same object category but different sizes and positions; 2) cross task generalization (PHYRE-C), where environments containing objects and context never present during training.
We report prediction and physical reasoning task performance on both of the settings.
ShapeStacks (SS) :
This dataset contains multiple stacked objects (cubes, cylinders, or balls)
(Ye et al., 2019).
In this dataset, we set T = 15 .
We evaluate all baselines and our methods follow the protocol of (Ye et al., 2019) with uncertainty estimation Incorporated (see Appendix for detail).
Real World Billiards (RealB) :
We collect “Three-cushion Billiards” videos from professional games with different viewpoints downloaded from YouTube.
There are 62 training videos with 18 ,frames, and 5 testing videos with 1 , 995 frames.
The bounding box annotations are from an offthe-shelf
ResNet-101 FPN detector (Lin et al., 2017) pretrained on COCO (Lin et al., 2014) and ﬁne-tuned on a subset of 30 images from our dataset.
We manually ﬁltered out wrong detections.
In this dataset, we set T = 20 . [Figure]
Simulation Billiards (SimB) :
We create a simulated billiard environment with three different colored balls with a radius 2 are randomly placed in a 64 × 64 image.
At starting point, one ball is moving with a randomly sampled velocity.
We generate 1,000 video sequences for training and 1,000 video sequences for testing, with 100 frames per sequence.
We will also evaluate the ability to generalize to more balls and different sized balls in the experiment section.
In this dataset, we set T = 20 .
For PHYRE and SS, it is possible to infer the future from just the initial conﬁguration.
So we set N = 1 in these two datasets. For SimB and RealB, the objects are moving in a ﬂat table so we cannot infer the future trajectories based on a single image. Therefore in this setting, we set N = 4 to infer the velocity/acceleration of objects.
We set k = N in all the dataset because we only have access to N features when we make prediction at t = N + 1 . We predict object masks for PHYRE and ShapeStacks, and only predict bounding boxes for Billiard datasets.
Baselines.
There are a large amount of work studying how to estimate objects’ states and predict their dynamics from raw image inputs.
They fall into the following categories:
VIN: Instead of using object-centric spatial pooling to extract object features, it use a ConvNet to globally encode an image to a ﬁxed d × m dimensional vector.
Different channels of the vector is assigned to different objects (Kipf et al., 2020; Watters et al., 2017).
This approach requires specifying a ﬁxed number of objects and a ﬁxed mapping between feature channels and object identity, which make it impossible to generalize to different number of objects and different appearances.
Object Masking (OM): This approach takes one image and m object bounding boxes or masks as input (Wu et al., 2017;
Veerapaneni et al., 2019; Janner et al., 2019).
For each proposal, only the pixels inside object proposals are kept while others are set to 0 , leading to m masked images.
This approach assumes no background information is needed thus fails to predict accurate trajectories in complex environments such as PHYRE.
And it also cost m times computational resources.
CVP:
The object feature is extracted by cropping the object image patch and forwarding it to an encoder (Ye et al., 2019;
Yi et al., 2020).
Since the object features are directly extracted from the raw image patches, the context information is also ignored.
We re-implement CVP’s feature extraction method within our framework.
We show we can reproduce their results in Section A.2.
E VALUATION R ESULTS : P REDICTION , G ENERALIZATION , AND P LANNING
Figure 3 shows some qualitative prediction results.
More results are in the supplementary material.
We organize this section and analyze our results by discussing four scientiﬁc questions related to the prediction quality, generalization ability across time & environment conﬁgurations, and the ability to plan actions for downstream tasks.
5. H OW ACCURATE IS THE PREDICTED DYNAMICS ?
To evaluate how well the world dynamics is modeled, we ﬁrst report the average prediction errors on the test split, over the same time-horizon as which model is trained on, i.e., t ∈ [0 , T train ] .
The prediction error is calculated by the squared (cid:96) 2 distance between predicted object center location and the ground-truth object centers.
The results are shown in Table 1 (left half). [Table] [Table]
is scaled by generalizes much than
Firstly, we show the effectiveness of our proposed RoI Feature by comparing Table 1 VIN, OM, CVP, and Ours (IN).
These four entries use the same backbone network and interaction network modules.
Among them, VIN cannot even be trained on PHYRE since it cannot handle varying number of objects.
The OM method performs slightly better than other baselines since it also explicitly models objects by instance masking.
For CVP, it cannot produce reasonable results on all of the datasets except on the SS dataset.
The reason is that in PHYRE and billiard environments, cropped images are unaware of environment information and the relative position and pose of different objects, making it impossible to make accurate predictions.
In SS, since the object size is large and objects are very close to each other, the cropped image regions already provide enough context.
Our results are signiﬁcantly better than all of the baselines.
In the very challenging PHYRE dataset, the prediction error is only 1 / 4 of the best baseline.
In the other three easier datasets, the gap is not as large but our method still achieves more than 10% improvements.
This demonstrates the advantage of using rich state representations.
Secondly, we show that the effectiveness of our proposed Convolutional Interaction Network by comparing Table 1 Ours (IN) and Ours (CIN).
With every other components the same, changing the vector-form representation to spatial feature maps and use convolution to model the interactions can further improve the performance by 10% ∼ 20%.
This result shows our convolutional interaction network could better utilize the spatial information encoded in the object feature map.
5. D OES LEARNED MODEL GENERALIZE TO LONGER HORIZON THAN TRAINING ?
Generalize to longer horizons is a challenge task.
In this section we compare the performance of predicting trajectories longer than training time.
In Table 1 (right half), we report the prediction error for t ∈ [ T train , 2 × T train ] .
The results in this setting are consistent with what we found in
Section 5.1.
Our method still achieves the best performance against all baselines.
Speciﬁcally, for all datasets except SimB, we reduce the error by more than 30% percent.
In SimB, the improvement is not as signiﬁcant because there is no environment information needed to infer future dynamics (see
Figure 3).
Meanwhile, changing IN to CIN further improve the performance.
This again validates our hypothesis that the key to making accurate long-term feature prediction is the rich state representation extracted from an image. [Table]
We show that our method achieves state-ofthe-art on both within-task generalization as well as cross-task generalization.
[Table]
[Table]
5. D OES LEARNED MODE GENERALIZE TO UNSEEN CONFIGURATIONS ?
The general applicability of RoI Feature has been extensively veriﬁed in the computer vision community.
As one of the beneﬁts, our method can generalize to novel environments conﬁgurations without any modiﬁcations or online training, thanks to the effective object-centric representations.
We test such a claim by testing on several novel environments unseen during training.
Speciﬁcally, we construct 1) simulation billiard dataset contains 5 balls with radius 2 (SimB-5); 2) PHYRE-C where the test tasks are not seen during training; 3)
ShapeStacks with 4 stacked blocks (SS-4).
The results are shown in Table 2.
Since VIN needs a ﬁxed number of objects as input, it cannot generalize to a different number of objects, thus we don’t report its performance on SimB-5, PHYRE-C, and SS-4.
In the SimBand PHYRE-C setting, where generalization ability to different numbers and different appearances is required, our method reduce the prediction error by 75%.
In SS-4, the improvement is not as signiﬁcant as the previous two because cropped image may be enough on this simpler dataset as mentioned above.
5. H OW WELL CAN THE LEARNED MODEL BE USED FOR PLANNING ACTIONS ?
The advantage of using a general purpose task-independent prediction model is that it can be used to do downstream planning tasks without any adaptation.
We evaluate our prediction model in PHYRE benchmark and simulation billiards planning tasks.
PHYRE.
In this task, we need to place a red ball to solve a speciﬁc goal for each environment (see ﬁgure 1 right for an example).
The action space contains the position and size of the red ball.
We use the ﬁrst 10 , 000 actions from pre-computed actions as our candidate sets and predict future object locations and masks for each action.
After that, we score each action based on the predicted trajectory.
To do that, we train a classifer to classify whether the trajectory will lead to a successful action.
Inspired by Bakhtin et al.
(2019), our classiﬁer is a ResNet-18 network whose input is the concatenation of initial image and ﬁnal predicted images.
The ﬁnal predicted image is rendered according to the predicted boxes and masks.
The full classiﬁer details will be included in the appendix.
We report the AUCCESS metric on the ofﬁcial 10 folds of train/test splits in Table 3.
Our prediction method with classiﬁer achieves 1 points improvement over the strong DQN baseline (Bakhtin et al., 2019) in within task generalization.
On the other hand, our method is 4.7 points higher than DQN.
SimB Planning.
We consider two tasks in the SimB environment: 1) Billiard Target State.
Given an initial and ﬁnal conﬁguration after 40 timesteps, the goal is to ﬁnd one action that will lead to the target conﬁguration.
We report the smallest distances between the trajectory between timestep 35and the ﬁnal position.
2) Billiard Hitting.
Given the initial conﬁgurations, the goal is to ﬁnd an action that can hit the other two balls within 50 timesteps.
We ﬁrstly train a forward model taken image and action as input, to predict the ﬁrst 4 object positions and render it to image.
After that the rendered images are passed in our prediction model.
We score each action according to the similarity between the generated trajectory and the goal state.
Then the action with the highest score is selected.
The results are shown in Table 4.
Our results achieves best performance on all tasks.
The full planning algorithm and implementation details are included in the appendix.
C ONCLUSIONS
In this paper, we leverage the modern computer vision techniques to propose Region Proposal
Interaction Networks for physical interaction reasoning with visual inputs.
We show that our general, yet simple method achieves a signiﬁcant improvement and can generalize across both simulation and real-world environments for long-range prediction and planning.
We believe this method may serve as a good benchmark for developing future methods in the ﬁeld of learning intuitive physics, as well as their application to real-world robotics.
A I MPLEMENTATION D ETAILS A. N ETWORK A RCHITECTURE D ETAILS
Backbone Networks: We use the same backbone network for our method and all the baselines (VIN, OM, CVP). We choose the hourglass network (Newell et al., 2016) as the image feature extractor.
Given an input image, the hourglass network ﬁrstly applies a 7 × 7 stride-2 convolution, three residual blocks with channel dimension d/ 4 , and a stride-2 max pooling on it.
Then this intermediate feature representation is fed into one hourglass modules.
In the hourglass module, the feature maps are downsampled with 3 stride-2 residual blocks and then up-sampled with nearest neighbor interpolation.
The dimensions of both the input channel and the output channel of each residual block are d .
For
SimB, we use d = 64 since the visual information in this environment is relatively simple.
For RealB, PHYRE, and ShapeStacks, d = 256 .
We use batch normalization before each convolutional layer in the backbone network.
The resulting feature map size is d × H 4 × W 4 , where H , W is the input image size and 4 is the spatial feature stride of the hourglass network.
The output features are transformed to object-centric representations differently for our method and each of the baseline methods:
VIN The resulting feature map is forwarded to 4 convolutional layers with stride 2 and a spatial global average pooling layer to get a d dimensional feature vector. This feature vector is transformed to d × m by one fully-connected layer. This feature will be reshaped to m vectors with d channel dimensional to represent each object.
This representation is reﬁned by two d × d fully-connected layers and passed to the (convolutional) interaction network.
OM and CVP:
The output feature map is of shape m × d × H 4 × W 4 because this method produce m masked (or cropped) images as input.
The resulting feature map is forwarded to 4 convolutional layers with stride 2 and a spatial global average pooling layer to get a d dimensional feature vector and the output feature is of size m × d , representing the features of m objects.
This representation is reﬁned by two d × d fully-connected layers and passed to the (convolutional) interaction network.
Only in the shapestack datasets, we also incorporate uncertainty estimation follows (Ye et al., 2019), by modeling the latent distribution using a variational auto-encoder (Kingma & Welling, 2014).
For the complete details, we refer the reader to (Ye et al., 2019).
Here we only give a summary: we build an encoder h which takes the image feature from ﬁrst F 0 and last frame F T of a video sequence as the input.
The output of h is a distribution parameter, denoted by h ( u |F 0 , F T ) .
Given a particular sample from such distribution, we recover the latent variable by feeding them into a one-layer LSTM and merge into the object feature x ti .
In this case, our pipeline is trained with an additional loss that minimize the KL divergence between the predicted distribution and normal distribution (Kingma &
Welling, 2014).
During inference, we sample 100 trajectories for each test image and report the minimum of them.
This setting is the same as the original CVP paper Ye et al.
Our implementation gets 5 .
28 × 10 − 3 squared (cid:96) 2 distance for T ∈ [0 , T train ] while the original paper report 6 .
69 × 10 − 3 .
A. D ATASET D ETAILS
SimB: To get the initial velocity, the magnitude (number of pixels moved per timestep) is sampled from [2 , 3 , 4 , 5 , 6] and the direction is sampled from { 6 iπ, i = 0 , 1 , . . .
RealB: We found that the bounding box prediction results are accurate enough to serve as the groundtruth.
After running the detector, we also manually go through the dataset and ﬁlter out images with incorrect detections.
ShapeStacks: There are 1,320 training videos and 296 testing videos, with 32 frames per video.
Only objects’ center positions provided.
Following (Ye et al., 2019), we assume the object bounding box is square and of size 70 × 70.
PHYRE: For within task generalization (PHYRE-W), the training set contains 80 templates from each task.
The testing set contains the remaining 20 templates from each task.
For cross task generalization (PHYRE-C), the training set contains 100 templates from 20 tasks while the test set contains 100 templates from the remaining 5 tasks.
For each template, we randomly sample a maximum 10 success and 20 fail actions to collect the trajectories to train our model.
The image sequence is temporally downsampled by 60 .
A. H YPERPARAMETERS
We use Adam optimizer Kingma & Ba (2014) with cosine decay Loshchilov & Hutter (2016) to train our networks.
The default input frames is N = 4 except N = 1 for ShapeStacks and PHYRE.
We set d to be 256 except for simulation billiard d is 64.
During training, T (denoted as T train ) is set to be 20 for SimB and RealB, 5 for PHYRE, and 15 for fair comparison with Ye et al.
The discounted factor λ t is set to be ( current _ iter max _ iter ) t .
Simulation Billiards.
The image size is 64 × 64.
We train the model for 100K iterations with a learning rate 2 × 10 − 3 and batch size 200.
Real World Billiards.
The image is resized to 192 × 64.
We train the model for 240K iterations with a learning rate 1 × 10 − 4 and batch size 20.
PHYRE.
The image is resized to 128 × 128.
We train the model for 150K iterations with a learning rate 2 × 10 − 4 and batch size 20.
ShapeStacks.
The image is resized to 224 × 224.
We train the model for 25K iterations with a learning rate 2 × 10 − 4 and batch size 40.
In this dataset, we apply uncertainty modeling as described in section 3.2.
The loss weight of KL-divergence is 3 × 10 − 5 .
During inference, following Ye et al.
(2019), we randomly sample 100 outputs from our model, and select the best (in terms of the distance to ground-truth) of them as our model’s output.
B P LANNING D ETAILS
Given an initial state (represented by an image) and a goal, we aim to produce an action that can lead to the goal from the initial state.
Our planning algorithm works in a similar way as visual imagination Fragkiadaki et al.
(2015): Firstly, we select a candidate action a from a candidate action set A .
Then we generate the input images I .
For SimB, we train a forward model take the initial image and the action embedding to generate object positions for the next 3 steps.
Then we use the simulator to generate the 3 images.
For PHYRE, we convert the action to the red ball using the simulator and get the initial image.
After that, we can forward the images to our prediction model to get future object trajectories for each object i and each timestep t : { ˆ p ti } t =1 ,...,T i =1 ,...,m .
The score of each action can be calculated by a score function designed for each task.
We then select the action that can maximize the score.
We introduce the action set for each task in section B.1, and how to design distance function in B.2.
A summary of our algorithm is in Algorithm 1.
B. C ANDIDATE A CTION S ETS
For simulation billiard, the action is 3 dimensional.
The ﬁrst two dimensions stand for the direction of the force.
The last dimension stands for the magnitude of the force.
During doing planning, we enumerate over 5 different magnitudes and 12 different angles, leading to 60 possible actions.
All of the initial condition is guaranteed to have a solution.
For PHYRE, the action is also 3 dimensional.
The ﬁrst two dimensions stand for the location placing the red ball.
The last dimension stands for the radius of the ball.
Following (Bakhtin et al., 2019), we use the ﬁrst 10 k actions provided by the benchmark.
B. D ISTANCE F UNCTION
Init-End State Error.
Denote the given target location of m objects as y ∈ R m × 2 .
We use the following distance function, which measures the distance between the ﬁnal rollout location and the target location:
[Equation]
Hitting Accuracy.
Denote the given initial location of m objects as x ∈ R m × 2 .
We apply force at the object i (cid:48) .
We use the following distance function, which prefer the larger moving distance for objects other than i (cid:48) :
[Equation]
PHYRE task.
In this task, we train a classiﬁer to classify whether the predicted trajectory will lead to a successful solution of the current task.
The classiﬁer has a the same structure as the DQN baseline in (Bakhtin et al., 2019).
In the training set, we sample maximum 50 positive and 200 negative actions for each task and then generate the trajectories using our prediction model.
The last predicted boxes and masks are rendered back using the simulator to get the last image.
We then use the concatenation of ﬁrst and last image as the input to our classiﬁer.
The network is trained by binary cross entropy loss.
After we have such model, during testing time we can use it to classify whether the current action will lead to a successful solution.
The distance function is just the negative output score.
B. P LANNING A LGORITHM
Algorithm
if
; end end
C Q UALITATIVE R ESULTS
Qualitative Results are shown in our website: https://sites.google.com/view/iclr21-rpin.