R EASONING A BOUT P HYSICAL I NTERACTIONS WITH O BJECT -O RIENTED P REDICTION AND P LANNING
A BSTRACT
Object based factorizations provide a useful level of abstraction for interacting with the world.
Building explicit object representations, however, often requires supervisory signals that are difﬁcult to obtain in practice.
We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties.
Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects and a rendering function to map objects back to pixels.
For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics.
After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training
I NTRODUCTION
Consider the castle made out of toy blocks in Figure 1a.
Can you imagine how each block was placed, one-by-one, to build this structure?
Humans possess a natural physical intuition that aids in the performance of everyday tasks.
This physical intuition can be acquired, and reﬁned, through experience.
Despite being a core focus of the earliest days of artiﬁcial intelligence and computer vision research (Roberts, 1963; Winston, 1970), a similar level of physical scene understanding remains elusive for machines.
Cognitive scientists argue that humans’ ability to interpret the physical world derives from a richly structured apparatus.
In particular, the perceptual grouping of the world into objects and their relations constitutes core knowledge in cognition (Spelke & Kinzler, 2007).
While it is appealing to apply such an insight to contemporary machine learning methods, it is not straightforward to do so.
A fundamental challenge is the design of an interface between the raw, often high-dimensional observation space and a structured, object-factorized representation.
Existing works that have investigated the beneﬁt of using objects have either assumed that an interface to an idealized object space already exists or that supervision is available to learn a mapping between raw inputs and relevant object properties (for instance, category, position, and orientation).
Assuming access to training labels for all object properties is prohibitive for at least two reasons.
The most apparent concern is that curating supervision for all object properties of interest is difﬁcult to scale for even a modest number of properties.
More subtly, a representation based on semantic
[Figure] [Figure]
Perception
Physics
Rendering attributes can be limiting or even ill-deﬁned. For example, while the size of an object in absolute terms is unambiguous, its orientation must be deﬁned with respect to a canonical, class-speciﬁc orientation. Object categorization poses another problem, as treating object identity as a classiﬁcation problem inherently limits a system to a predeﬁned vocabulary.
In this paper, we propose Object-Oriented Prediction and Planning (O2P2), in which we train an object representation suitable for physical interactions without supervision of object attributes.
Instead of direct supervision, we demonstrate that segments or proposal regions in video frames, without correspondence between frames, are sufﬁcient supervision to allow a model to reason effectively about intuitive physics.
We jointly train a perception module, an object-factorized physics engine, and a neural renderer on a physics prediction task with pixel generation objective.
We evaluate our learned model not only on the quality of its predictions, but also on its ability to use the learned representations for tasks that demand a sophisticated physical understanding.
O BJECT -O RIENTED P REDICTION AND P LANNING (O2P2)
In this section, we describe a method for learning object-based representations suitable for planning in physical reasoning tasks.
As opposed to much prior work on object-factorized scene representations (Section 4), we do not supervise the content of the object representations directly by way of labeled attributes (such as position, velocity, or orientation).
Instead, we assume access only to segments or region proposals for individual video frames.
Since we do not have labels for the object representations, we must have a means for converting back and forth between images and object representations for training.
O2P2 consists of three components, which are trained jointly:
• A perception module that maps from an image to an object encoding.
The perception module is applied to each object segment independently.
• A physics module to predict the time evolution of a set of objects.
We formulate the engine as a sum of binary object interactions plus a unary transition function.
• A rendering engine that produces an image prediction from a variable number of objects.
We ﬁrst predict an image and single-channel heatmap for each object.
We then combine all of the object images according to the weights in their heatmaps at every pixel location to produce a single composite image.
A high-level overview of the model is shown in Figure 2c.
Below, we give details for the design of each component and their subsequent use in a model-based planning setting.
2. P ERCEPTION M ODULE
The perception module is a four-layer convolutional encoder that maps an image observation to object representation vectors O = { o k } k =1 ...N .
We assume access to a segmentation of the input image S
= { s k } k
=1 ...N and apply the encoder individually to each segment.
The perception module is not supervised directly to predict semantically meaningful properties such as position or orientation; instead, its outputs are used by the physics and rendering modules to make image predictions.
In this way, the perception module must be trained jointly with the other modules.
2. P HYSICS M ODULE
The physics module predicts the effects of simulating a collection of object representations O forward in time.
As in Chang et al.
; Watters et al.
(2017), we consider the interactions of all pairs of object vectors.
The physics engine contains two learned subcomponents: a unary transition function f trans applied to each object representation independently, and a binary interaction function f interact applied to all pairs of object representations.
Letting ¯O = { ¯ o k } k =1 ...
N denote the output of the physics predictor, the k th object is given by ¯ o k
= f trans ( o k ) + (cid:80) j (cid:54) =
k f interact ( o k , o j )
+ o k , where both f trans and f interact are instantiated as two-layer MLPs.
Much prior work has focused on learning to model physical interactions as an end goal.
In contrast, we rely on physics predictions only insofar as they affect action planning.
To that end, it is more important to know the resultant effects of an action than to make predictions at a ﬁxed time interval.
We therefore only need to make a single prediction, ¯O =
f physics ( O ) , to estimate the steady-state conﬁguration of objects as a result of simulating physics indeﬁnitely.
This simpliﬁcation avoids the complications of long-horizon sequential prediction while retaining the information relevant to planning under physical laws and constraints.
2. R ENDERING E NGINE
Because our only supervision occurs at the pixel level, to train our model we learn to map all objectvector predictions back to images.
A challenge here lies in designing a function which constructs a single image from an entire collection of objects.
The learned renderer consists of two networks, both instantiated as convolutional decoders.
The ﬁrst network predicts an image independently for each input object vector.
Composing these images into a single reconstruction amounts to selecting which object is visible at every pixel location.
In a traditional graphics engine, this would be accomplished by calculating a depth pass at each location and rendering the nearest object.
To incorporate this structure into our learned renderer, we use the second decoder network to produce a single-channel heatmap for each object.
The composite scene image is a weighted average of all of the object-speciﬁc renderings, where the weights come from the negative of the predicted heatmaps.
In effect, objects with lower heatmap predictions at a given pixel location will be more visible than objects with higher heatmap values.
This encourages lower heatmap values for nearer objects.
Although this structure is reminiscent of a depth pass in a traditional renderer, the comparison should not be taken literally; the model is only supervised by composite images and no true depth maps are provided during training.
2. L EARNING OBJECT REPRESENTATIONS
We train the perception, physics, and rendering modules jointly on an image reconstruction and prediction task.
Our training data consists of image pairs ( I 0 , I 1 ) depicting a collection of objects on a platform before and after a new object has been dropped.
( I 0 shows one object mid-air, as if being held in place before being released.
We refer to Section 3 for details about the generation of training data.)
We assume access to a segmentation S 0 for the initial image I 0 .
Given the observed segmented image S 0 , we predict object representations using the perception module O = f percept ( S 0 ) and their time-evolution using the physics module ¯O =
f physics ( O ) .
The rendering engine then predicts an image from each of the object representations: ˆ I
= f render ( O ) , ˆ I
= f render ( ¯O ) .
We compare each image prediction ˆ I t to its ground-truth counterpart using both L 2 distance and a perceptual loss L VGG .
As in Johnson et al.
, we use L 2 distance in the feature space of a f f Observe Concatenate Compute pretrained VGG network (Simonyan & Zisserman, 2014) as a perceptual loss function.
The perception module is supervised by the reconstruction of I 0 , the physics engine is supervised by the reconstruction of I 1 , and the rendering engine is supervised by the reconstruction of both images.
Speciﬁcally, L percept ( · ) =
L 2 (ˆ I 0 , I 0 )
+ L VGG (ˆ I 0 , I 0 ) , L physics ( · ) =
L 2 (ˆ I 1 , I 1 ) + L VGG (ˆ I 1 , I 1 ) , and
L render ( · ) =
L percept ( · ) + L physics ( · ) .
2. P LANNING WITH L EARNED M ODELS
We now describe the use of our perception, physics, and rendering modules in the representative planning task depicted in Figure 1, in which the goal is to build a block tower to match an observed image.
Here, matching a tower does not refer simply to producing an image from the rendering engine that looks like the observation.
Instead, we consider the scenario where the model must output a sequence of actions to construct the conﬁguration.
This setting is much more challenging because there is an implicit sequential ordering to building such a tower.
For example, the bottom cubes must be placed before the topmost triangle.
O2P2 was trained solely on a pixel-prediction task, in which it was never shown such valid action orderings (or any actions at all).
However, these orderings are essentially constraints on the physical stability of intermediate towers, and should be derivable from a model with sufﬁcient understanding of physical interactions.
Although we train a rendering function as part of our model, we guide the planning procedure for constructing towers solely through errors in the learned object representation space.
The planning procedure, described in detail in Algorithm 1, can be described at a high level in four components:
The perception module encodes the segmented goal image into a set of object representations O goal .
We sample actions of the form (shape, position, orientation, color) , where shape is categorical and describes the type of block, and the remainder of the action space is continuous and describes the block’s appearance and where it should be dropped.
We evaluate the samples by likewise encoding them as object vectors and comparing them with O goal .
We view action sample a m as an image segment s m (analogous to observing a block held in place before dropping it) and use the perception module to produce object vectors O m .
Because the actions selected should produce a stable tower, we run these object representations through the physics engine to yield ¯ O m before comparing with O goal .
The cost is the L 2 distance between the object ¯o ∈ ¯O m corresponding to the most recent action and the goal object in O goal that minimizes this distance.
Using the action sampler and evaluation metric, we select the sampled action that minimizes L 2 distance.
We then execute that action in MuJoCo (Todorov et al., 2012).
We continue this procedure, iteratively re-planning and executing actions, until there are as many actions in the Reconstructions Simulated t = 1 t =t = 1 t = Reconstructions Simulated t = 1 t =t = 1 t = executed sequence as there are objects in the goal image.
In the simplest case, the distribution from which actions are sampled may be uniform, as in Algorithm 1.
Alternatively, the crossentropy method (CEM) (Rubinstein & Kroese, 2004) may be used, repeating the sampling loop multiple times and ﬁtting a Gaussian distribution to the lowest-cost samples.
In practice, we used
CEM starting from a uniform distribution with ﬁve iterations, 1000 samples per iteration, and used the top 10% of samples to ﬁt the subsequent iteration’s sampling distribution.
E XPERIMENTAL E VALUATION
In our experimental evaluation, we aim to answer the following questions, (1) After training solely on physics prediction tasks, can O2P2 reason about physical interactions in an actionable and useful way?
(2) Does the implicit object factorization imposed by O2P2’s structure provide a beneﬁt over an object-agnostic black-box video prediction approach?
(3) Is an object factorization still useful even without supervision for object representations?
3. I MAGE R ECONSTRUCTION AND P REDICTION
We trained O2P2 to reconstruct observed objects and predict their conﬁguration after simulating physics, as described in Section 2.4.
To generate training data, we simulated dropping a block on top of a platform containing up to four other blocks.
We varied the position, color, and orientation of three block varieties (cubes, rectangular cuboids, and triangles).
In total, we collected 60, training images using the MuJoCo simulator.
Since our physics engine did not make predictions at every timestep (Section 2.2), we only recorded the initial and ﬁnal frame of a simulation.
For this synthetic data, we used ground truth segmentations corresponding to visible portions of objects.
Representative predictions of our model for image reconstruction (without physics) and prediction (with physics) on held-out random conﬁgurations are shown in Figure 3.
Even when the model’s predictions differed from the ground truth image, such as in the last row of the ﬁgure, the physics engine produced a plausible steady-state conﬁguration of the observed scene.
3. B UILDING T OWERS
After training O2P2 on the random conﬁgurations of blocks, we ﬁxed its parameters and employed the planning procedure as described in Section 2.5 to build tower conﬁgurations observed in images.
We also evaluated the following models as comparisons:
• No physics is an ablation of our model that does not run the learned physics engine, but instead simply sets ¯O
= O • Stochastic adversarial video prediction ( SAVP ), a block-box video prediction model which does not employ an object factorization Lee et al.
The cost function of samples is evaluated directly on pixels.
The sampling-based planning routine is otherwise the same as in ours.
[Figure] [Table] • Oracle (pixels) uses the MuJoCo simulator to evaluate samples instead of our learned physics and graphics engines.
The cost of a block conﬁguration is evaluated directly in pixel space using L 2 distance.
• Oracle (objects) also uses MuJoCo, but has access to segmentation masks on input images while evaluating the cost of proposals.
Constraining proposed actions to account for only a single object in the observation resolves some of the inherent difﬁculties of using pixel-wise loss functions.
Qualitative results of all models are shown in Figure 4 and a quantitative evaluation is shown in
Table 1.
We evaluated tower stacking success by greedily matching the built conﬁguration to the ground-truth state of the goal tower, and comparing the maximum object error (deﬁned on its position, identity, and color) to a predetermined threshold.
Although the threshold is arbitrary in the sense that it can be chosen low enough such that all builds are incorrect, the relative ordering of the models is robust to changes in this value.
All objects must be of the correct shape for a built tower to be considered correct, meaning that our third row prediction in Figure 4 was incorrect because a green cube was mistaken for a green rectangular cuboid.
While SAVP made accurate predictions on the training data, it did not generalize well to these more complicated conﬁgurations with more objects per frame.
As such, its stacking success was low.
Physics simulation was crucial to our model, as our No-physics ablation failed to stack any towers correctly.
We explored the role of physics simulation in the stacking task in Section 3.3.
The ‘oracle’ model with access to the ground-truth physics simulator was hampered when making comparisons in pixel space.
A common failure mode of this model was to drop a single large block on the ﬁrst step to cover the visual area of multiple smaller blocks in the goal image.
This scenario was depicted by the blue rectangular cuboid in the ﬁrst row of Figure 4 in the Oracle (pixels) column.
3. T HE I MPORTANCE OF U NDERSTANDING P HYSICS
Figure 5 depicts the entire planning and execution procedure for O2P2 on a pyramid of six blocks.
At each step, we visualize the process by which our model selects an action by showing a heatmap of [Figure] [Figure]
O2P2’s scores reﬂect that the objects resting directly on the platform must be dropped ﬁrst, and that they may be dropped from any height because they will fall to the ground. The No-physics ablation, on the other hand, does not implicitly represent that the blocks need to be dropped in a stable sequence of actions because it does not predict the blocks moving after being released. scores (negative MSE) for each action sample according to the sample’s ( x, y ) position (Figure 5a).
Although the model is never trained to produce valid action decisions, the planning procedure selects a physically stable sequence of actions.
For example, at the ﬁrst timestep, the model scores three x -locations highly, corresponding to the three blocks at the bottom of the pyramid.
It correctly determines that the height at which it releases a block at any of these locations does not particularly matter, since the block will drop to the correct height after running the physics engine.
Figure 5b shows the selected action at each step, and Figure 5c shows the model’s predictions about the conﬁguration after releasing the sampled block.
Similar heatmaps of scored samples are shown for the No-physics ablation of our model in Figure 6.
Because this ablation does not simulate the effect of dropping a block, its highly-scored action samples correspond almost exactly to the actual locations of the objects in the goal image.
Further, without physics simulation it does not implicitly select for stable action sequences; there is nothing to prevent the model from selecting the topmost block of the tower as the ﬁrst action.
Planning for alternate goals.
By implicitly learning the underlying physics of a domain, our model can be used for various tasks besides matching towers.
In Figure 7a, we show our model’s representations being used to plan a sequence of actions to maximize the height of a tower.
There is no observation for this task, and the action scores are calculated based on the highest non-zero pixels after rendering samples with the learned renderer.
In Figure 7b, we consider a similar sampling procedure as in the tower-matching experiments, except here only a single unstable block is shown.
Matching a free-ﬂoating block requires planning with O2P2 for multiple steps at once.
[Figure]
3. T RANSFER TO R OBOTIC A RM
We evaluated O2P2 on a Sawyer robotic arm using real image inputs.
We deployed the same perception, physics, and rendering modules used on synthetic data with minor changes to the planning procedure to make real-world evaluation tractable.
Instead of evaluating a sampled action by moving an appropriate block to the speciﬁed position and inferring object representations with the perception module, we trained a separate two-layer MLP to map directly from actions to object representations.
We refer to this module as the embedder: o m = f embedder ( a m ) .
Mapping actions to object representations removed the need to manually move every sampled block in front of the camera, which would have been prohibitively slow on a real robot.
The embedder was supervised by the predicted object representations of the perception module on real image inputs; we collected a small dataset of the Sawyer gripper holding each object at one hundred positions and recorded the ground truth position of the gripper along with the output of the perception module for the current observation.
The embedder took the place of lines 6-8 of Algorithm 1.
We also augmented the objective used to select actions in line 11.
In addition to L 2 distance between goal and sampled object representations, we used a pixelwise L 2 distance between the observed and rendered object segments and between the rendered object segments before and after use of the physics module.
The latter loss is useful in a real setting because the physical interactions are less predictable than their simulated counterparts, so by penalizing any predicted movement we preferentially placed blocks directly in a stable position.
By using end-effector position control on the Sawyer gripper, we could retain the same action space as in synthetic experiments.
Because the position component of the sampled actions referred to the block placement location, we automated the picking motion to select the sampled block based on the shape and color components of an action.
Real-world evaluation used colored wooden cubes and rectangular cuboids.
Real image object segments were estimated by applying a simple color ﬁlter and ﬁnding connected components of sufﬁcient size.
To account for shading and specularity differences, we replaced all pixels within an object segment by the average color within the segment.
To account for noisy segment masks, we replaced each mask with its nearest neighbor (in terms of pixel MSE) in our
MuJoCo-rendered training set.
We tested O2P2 on twenty-ﬁve goal conﬁgurations total, of which our model correctly built seventeen.
Ten goal images, along with the result of our model’s executed action sequence, are shown in
Figure 8.
The remainder of the conﬁgurations are included in Appendix B.
R ELATED W ORK
Our work is situated at the intersection of two distinct paradigms.
In the ﬁrst, a rigid notion of object representation is enforced via supervision of object properties (such as size, position, and identity).
In the second, scene representations are not factorized at all, so no extra supervision is required.
These two approaches have been explored in a variety of domains.
Image and video understanding.
The insight that static observations are physically stable conﬁgurations of objects has been leveraged to improve 3D scene understanding algorithms.
For example,
Zheng et al.
; Gupta et al. (2010); Shao et al.
; Jia et al.
(2015) build physically-plausible scene representations using such stability constraints.
We consider a scenario in which the physical representations are learned from data instead of taking on a predetermined form.
Wu et al.
(2017b;a) encode scenes in a markup-style representation suitable for consumption by off-the-shelf rendering engines and physics simulators.
In contrast, we do not assume access to supervision of object properties (only object segments) for training a perception module to map into a markup language.
There has also been much attention on inferring object-factorized, or otherwise disentangled, representations of images (Eslami et al., 2016; Greff et al., 2017; van Steenkiste et al., 2018).
In contrast to works which aim to discover objects in a completely unsupervised manner, we focus on using object representations learned with minimal supervision, in the form of segmentation masks, for downstream tasks.
Object-centric scene decompositions have also been considered as a potential state representation in reinforcement learning (Diuk et al., 2008; Scholz et al., 2014; Devin et al., 2017; Goel et al., 2018; Keramati et al., 2018).
We are speciﬁcally concerned with the problem of predicting and reasoning about physical phenomena, and show that a model capable of this can also be employed for decision making.
Learning and inferring physics.
Fragkiadaki et al.
; Watters et al.
; Chang et al.
(2016) have shown approaches to learning a physical interaction engine from data.
Hamrick et al.
(2011) use a traditional physics engine, performing inference over object parameters, and show that such a model can account for humans’ physical understanding judgments.
We consider a similar physics formulation, whereby update rules are composed of sums of pairwise object-interaction functions, and incorporate it into a training routine that does not have access to ground truth supervision in the form of object parameters (such as position or velocity).
An alternative to using a traditional physics engine (or a learned object-factorized function trained to approximate one) is to treat physics prediction as an image-to-image translation or classiﬁcation problem.
In contrast to these prior methods, we consider not only the accuracy of the predictions of our model, but also its utility for downstream tasks that are intentionally constructed to evaluate its ability to acquire an actionable representation of intuitive physics.
Comparing with representative video prediction (Lee et al., 2018; Babaeizadeh et al., 2018) and physical prediction (Ehrhardt et al., 2017; Mottaghi et al., 2016;
Li et al., 2017; Lerer et al., 2016) methods, our approach achieves substantially better results at tasks that require building structures out of blocks.
C ONCLUSION
We introduced a method of learning object-centric representations suitable for physical interactions.
These representations did not assume the usual supervision of object properties in the form of position, orientation, velocity, or shape labels.
Instead, we relied only on segment proposals and a factorized structure in a learned physics engine to guide the training of such representations.
We demonstrated that this approach is appropriate for a standard physics prediction task.
More impor- tantly , we showed that this method gives rise to object representations that can be used for difﬁcult planning problems, in which object conﬁgurations differ from those seen during training, without further adaptation.
We evaluated our model on a block tower matching task and found that it outperformed object-agnostic approaches that made comparisons in pixel-space directly
A CKNOWLEDGMENTS
We thank Michael Chang for insightful discussion and anonymous reviewers for feedback on an early draft of this paper.
This work was supported by the National Science Foundation Graduation
Research Fellowship and the Open Philanthropy Project AI Fellowship.
A I MPLEMENTATIONS D ETAILS
Objects were represented as 256-dimensional vectors.
The perception module had four convolutional layers of { 32, 64, 128, 256 } channels, a kernel size of and a stride of 2 followed by a single fully-connected layer with output size matching the object representation dimension.
Both
MLPs in the physics engine had two hidden layers each of size 512.
The rendering networks had convolutional layers with { 128, 64, 32, 3 } channels (or 1 output channel in the case of the heatmap predictor), kernel sizes of { 5, 5, 6, 6 } , and strides of 2.
We used the Adam optimizer (Kingma &
Ba, 2015) with a learning rate of 1e-3.
B S AWYER RESULTS