C O P HY : C OUNTERFACTUAL L EARNING OF P HYSICAL D YNAMICS
A BSTRACT
Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world
This work poses a new problem of counterfactual learning of object mechanics from visual input.
We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting.
Having observed a mechanical experiment that involves, for example a falling tower of blocks a set of bouncing balls or colliding objects we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions such as displacing one of the objects in the scene.
The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision of confounders.
We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment which results in signiﬁcantly more accurate predictions at the level of super human performance
I NTRODUCTION
Reasoning is an essential ability of intelligent agents that enables them to understand complex relationships between observations, detect affordances, interpret knowledge and beliefs, and to leverage this understanding to anticipate future events and act accordingly.
The capacity for observational discovery of causal effects in physical reality and making sense of fundamental physical concepts, such as mass , velocity , friction , etc., may be one of differentiating properties of human intelligence that ensures our ability to robustly generalize to new scenarios (Martin-Ordas et al., 2008).
One way to express causality is based on the concept of counterfactual reasoning , that deals with a problem containing an if statement, which is untrue or unrealized.
Predicting the effect of the interventions based on the given observations without explicitly observing the effect of the intervention on data is a hard task and requires modeling of the causal relationships between the variable on which the intervention is performed and the variable whose alternative future should be predicted (Balke &
Pearl, 1994).
Using counterfactuals has been shown to be a way to perform reasoning over causal relationships between the variables of low dimensional spaces and has been an unexplored direction for high dimensional signals such as videos.
In this work, we develop the Co unterfactual Phy sics benchmark ( CoPhy ) and propose a framework for causal learning of dynamics in mechanical systems with multiple degrees of freedom, as illustrated in Fig.
For a number of scenarios, such as tower of blocks falling , balls bouncing against walls or objects colliding , we are given the starting frame A
= X 0 and a sequence of following frames
B = X 1: τ , where τ covers the range of 6 sec.
The observed sequences B , conditioned on the initial state A , are direct effects of the physical principles (such as inertia , gravity or friction ) applied to the closed system, that cause the objects change their positions and 3D poses over time.
The task is formulated as follows: having observed the tuple ( A , B ) , we wish to predict positions and poses of all objects in the scene at time t
= τ , if we had changed the initial frame X 0 by performing an intervention .
The intervention is formalized by the do-operator introduced by Pearl et al .
(Pearl, 2009; Pearl & McKenzie, 2018) for dealing with causal inference (Spirtes, 2010).
In our case, it implies modiﬁcation of the variable A to C , deﬁned as C
= do ( X
=
¯ X 0 ) .
Accordingly, for each experiment in the CoPhy benchmark, we provide pairs of original sequences X 0:
τ and their modiﬁed counterparts ¯ X 0: τ sharing the same values of all confounders.
We note the fundamental difference between this problem of counterfactual future forecasting and the conventional setup of feedforward future forecasting , like video prediction (Mathieu et al., 2016).
The latter involves learning spatio-temporal regularities and thereby predicting future frames X 1 ...τ from one or several past frame(s) X 0 (the causal chain of this problem is shown in Fig. 2a).
On the other hand, counterfactual forecasting beneﬁts from additional observations in the form of the original outcome X 1: τ before the do-operator .
This adds a confounder variable U into the causal chain (Fig. 2b), which provides information not observable in frame X 0 .
For instance, in the case of the CoPhy benchmark, observing the pair ( A , B ) might give us information on the masses, velocities or friction coefﬁcients of the objects in the scene, which otherwise cannot be inferred from frame ¯ Xalone.
Therefore, predicting the alternative outcome after performing counterfactual intervention then involves using the estimate of the confounder U together with the modiﬁed past do ( X 0 = ¯ X 0 )
Overall, we employ the idea of counterfactual intervention in predictive models and argue that counterfactual reasoning is an important step towards human-like reasoning and general intelligence.
More speciﬁcally, key contributions of this work include:
• a new task of counterfactual prediction of physical dynamics from high-dimensional visual input, as a way to access capacity of intelligent agents for causal discovery; • a large-scale CoPhy benchmark with three physical scenarios and 300k synthetic experiments including rendered sequences of frames, metadata ( object positions , angles , sizes ) and values of confounders ( masses , frictions , gravity ).
This benchmark was speciﬁcally designed in bias-free fashion to make the counter-factual reasoning task challenging by optimizing the impact of the confounders on the outcome of the experiment.
The dataset is publicly available 1 .
• a counterfactual neural model predicting an alternative outcome of a physical experiment given an intervention, by estimating the latent representation of the confounders.
The model outperforms state-of-the-art solutions implementing feedforward video prediction, successfully generalizes to unseen initial states and does not require supervision on the confounders.
We provide extensive ablations on the effects of key design choices and compare results with human performance, that show that the task is hard for humans to solve. The code will be made publicly available.
R ELATED WORK
This work is inspired by a signiﬁcant number of prior studies from several subﬁelds, including visual reasoning, learning intuitive physics and perceived causality.
[Table]
[Table]
null
[Table]
null
[Table]
<
[Table]
null
[Table]
(null
[Table]
<
[Table]
latexit
[Table]
latexit sha
[Table]
="
[Table]
null
[Table]
(null)</latexit> <latexit sha1_
[Table]
="(null)
[Table]
>(null)<
[Table]
latexit
[Table]
<latexit
[Table]
null
[Table]
null
[Table]
null
[Table]
null
[Table]
null
[Table]
null
[Table]
null
[Table]
latexit> <latexit sha1_base64="(null)">(null)</latexit>
D : ¯ X t =1 ...
⌧ <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
C : do ( X 0 = ¯ X 0 ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> Visual reasoning. The most recent works on visual reasoning approach the problem in the setting of visual question answering (Hu et al., 2017; Hudson & Manning, 2018; Johnson et al., 2017; Mao et al., 2019; Perez et al., 2018; Santoro et al., 2017), embodied AI (Wijmans et al., 2019), as well as learning intuitive physics (Lerer et al., 2016; Riochet et al., 2018).
(Santoro et al., 2017) introduced Relation
Networks (RN), a fully-differentiable trainable layer for reasoning in deep networks.
Following the same trend, (Baradel et al., 2018) estimate object relations from semantically well deﬁned entities using instance segmentation predictions for video understanding.
(Santoro et al., 2018) build a challenging dataset for solving the problem of abstract reasoning on the visual domain with some tasks such as interpolation or extrapolation.
External memory (Graves et al., 2016; Jaeger, 2016) extends known recurrent neural mechanisms by decoupling the size of a representation from the controller capacity and introduces the separation between long-term and short-term reasoning.
(Reed et al., 2015) propose to learn analogies in a fully supervised way.
Our work builds upon this literature and extends the idea of visual reasoning to the counterfactual setting.
Intuitive physics.
Fundamental studies on cognitive psychology have shown that humans perform poorly when asked to reason about expected outcomes of a physics based event, demonstrating striking deviations from Newtonian physics in their intuitions (McCloskey & Kohl, 1983; McClooskey et al., 1980; 1983; Kubricht et al., 2017).
The questions of approximating these mechanisms, learning from noisy observed and non-observed physical quantities (such as sizes or velocities vs masses or gravity), as well as justifying importance of explicit physical concepts vs cognitive constructs in intelligent agents have been raised and explored in recent works on deep learning (Wu et al., 2015).
(Lerer et al., 2016; Groth et al., 2018) follow this direction by training networks to predict stability of block towers.
(Ye et al., 2018) build an interpretable intuitive physical model from visual signals using full supervision on the physical properties of each object.
On similar tasks, (Wu et al., 2017) propose to learn physics by interpreting and reconstructing the visual information stream leading to inverting physics or a graphical engine.
(Zheng et al., 2018) propose to solve this task by ﬁrst extracting a visual perception of the world state and then predict the future.
(Battaglia et al., 2016) introduce a fully-differentiable network physics engine called Interaction Network (IN), which learns to predict physical systems such as gravitational ones, rigid body dynamics, and mass-spring systems.
Similarly, (van Steenkiste et al., 2018) discover objects and their interactions in a unsupervised manner from a virtual environment.
In (Veliˇckovi´c et al., 2018), attention and relational modules are combined on a graph structure.
Recent approaches (Chang et al., 2017; Janner et al., 2019; Battaglia et al., 2018) based on Graph Convolution Networks (Kipf & Welling, 2017) have shown promising results on learning physics but are restricted to setups where physical properties need to be fully observable, which is not the case of our approach.
The most similar to ours is work by (Ehrhardt et al., 2019) on unsupervised learning of intuitive physics from unpaired past experiences.
Other physics benchmarks and simulators.
The main objective for the creation of our benchmark is (a) to focus speciﬁcally on evaluating capabilities of state of the art models for performing counterfactual reasoning, (b) to be unbiased in terms of distributions of parameters to be estimated and balanced with respect to possible outcomes, and (c) to have sufﬁcient variety in terms of scenarios and latent physical characteristics of the scene that are not visually observed and therefore can act as confounders.
To the best of our knowledge, none of existing intuitive physics benchmarks have these properties.
IntPhys (Riochet et al., 2018) focuses on a high level task of estimating physical plausibility in a black box fashion and modeling out of distribution events at test time.
CATER (Girdhar & Ramanan, 2019) introduces a video dataset requiring spatiotemporal understanding in order to solve the tasks such as action recognition, compositional action recognition, and adversarial target tracking.
Phyre (Bakhtin et al., 2019) is an environment for solving physics based puzzles, where achieving sample efﬁciency may implicitly require counterfactual reasoning, but this component is not explicitly evaluated, construction of parallel data with several alternative outcomes is not straightforward, and the trivial baseline performance levels are not easy to estimate.
Adapting these benchmarks to counterfactual reasoning would require signiﬁcant refactoring and changing the logic of the data sampling.
CLEVRER (Yi et al., 2019) is a diagnostic video dataset for systematic evaluation of models on a wide range of reasoning tasks including counterfactual questions.
They cast the task as a classiﬁcation problem where the model has to choose between a set of possible answers, whereas our benchmark requires the predicting the dynamics of each object.
Perceptual causality.
In the ML community, causal reasoning gained mainstream attention relatively recently (Lopez-Paz et al., 2017; Lopez-Paz & Oquab, 2017; Kocaoglu et al., 2018; Rojas-Carulla et al., 2018; Mooij et al., 2016; Sch¨olkopf et al., 2012), due to limitations of statistical learning becoming increasingly apparent (Pearl, 2018;
Lake et al., 2017).
The concept of perceived causality has been however explored in cognitive psychology (Michotte, 1963), where human subjects have be shown to consistently report causal impressions not aligned with underlying physical principles of the events (Gerstenberg et al., 2015; Kubricht et al., 2017).
Exploiting the colliding objects scenario as a standard testbed for these studies led to discovery of a number of cognitive biases, e.g. Motor
Object Bias (i.e. false perceived association of object’s velocity with its mass).
In this work, we bring the domains of visual reasoning, intuitive physics and perceived causality together in a single framework to tackle the new problem of counterfactual learning of physical dynamics.
Following prior literature (Battaglia et al., 2013), we also compare counterfactual learning with human performance and expect that, similarly to learning intuitive vs Newtonian physics, modeling perceived vs true causality will get more attention from the ML community in the future.
C O P HY : C OUNTERFACTUAL P HYSICS BENCHMARK SUITE
In this paper we investigate visual reasoning problems involving a set of K physical objects and their interactions, while considering a speciﬁc setting of learning counterfactual prediction with the objective of estimating objects’ alternative 3D positions from images after do-intervention .
We introduce the Co unterfactual Phy sics benchmark suite ( CoPhy ) for counterfactual reasoning of physical dynamics from raw visual input.
It is composed of three tasks based on three physical scenarios: BlocktowerCF , BallsCF and CollisionCF , deﬁned similarly to existing state-ofthe-art environments for learning intuitive physics: Shape Stack (Groth et al., 2018), Bouncing balls environment (Chang et al., 2017) and Collision (Ye et al., 2018) respectively.
This was done to ensure natural continuity between the prior art in the ﬁeld and the proposed counterfactual formulation.
Each scenario includes training and test samples, that we call experiments .
Each experiment is represented by two sequences of τ synthetic RGB images (covering the time span of 6 sec at 5 fps):
• an observed sequence X =
{ X 0 , . .
, X τ } demonstrates evolution of the dynamic system under the inﬂuence of laws of physics (gravity, friction, etc.), from its initial state X 0 to its ﬁnal state X τ .
For simplicity, we denote A the initial state X 0 and B the observed outcome X 0 , . . .
, X ; • a counterfactual sequence ¯ X
= { ¯ X 0 , . .
, ¯ X τ } , where ¯ X 0 ( C ) corresponds to the initial state Xafter the do-intervention , and ¯ X 1 , . . .
, ¯ X τ ( D ) correspond to the counterfactual outcome.
A do-intervention is a visually observable change introduced to the initial physical setup x 0 (such as, for instance, object displacement or removal).
Finally, the physical world in each experiment is parameterized by a set of visually unobservable quantities, or confounders (such as object masses, friction coefﬁcients, direction and magnitude of gravitational forces), that cannot be uniquely estimated from a single time step.
Our dataset provides ground truth values of all confounders for evaluation purposes.
However, we do not assume access to this information during training or inference, and do not encourage it.
Each of the three scenarios in the CoPhy benchmark is deﬁned as follows (see Fig.
1 for illustrations).
BlocktowerCF — Each experiment involves K =3 or K =4 stacked cubes, which are initially at resting (but potentially unstable) positions.
We deﬁne three different confounder variables: masses , m ∈{ 1 , 10 } and friction coefﬁcients , µ ∈{ 0 . 5 , 1 } , for each block, as well as gravity components in X and Y direction, g x,y ∈{− 1 , 0 , 1 } .
The do-interventions include block displacement or removal.
This set contains 146k sample experiments corresponding to 73k different geometric block conﬁgurations.
BallsCF — Experiments show K bouncing balls ( K =2 ... 6 ).
Each ball has an initial random velocity.
The confounder variables are the masses , m ∈{ 1 , 10 } , and the friction coefﬁcients , µ ∈{ 0 . 5 , 1 } , of each ball.
There are two do-operators : block displacement or removal.
There are in total 100k experiments corresponding to 50k different initial geometric conﬁgurations.
CollisionCF — This set is about moving objects colliding with static objects (balls or cylinders).
The confounder variables are the masses , m ∈{ 1 , 10 } , and the friction coefﬁcients , µ ∈{ 0 . 5 , 1 } , of each object.
The do-interventions are limited to object displacement.
This scenario includes 40k experiments with 20k unique geometric object conﬁgurations.
Given this data, the problem can be formalized as follows.
During training , we are given the quadruplets of visual observations A , B , C , D (through sequences X and ¯ X , including GT object positions for supervision), but do not not have access to the values of the confounders.
During testing , the objective is to reason on new visual data unobserved at training time and to predict the counterfactual outcome D , having observed the ﬁrst sequence ( A , B ) and the modiﬁed initial state
C after the do-intervention, which is known.
The CoPhy benchmark is by construction balanced and bias free w.r.t.
(1) global statistics of all confounder values within each scenario, (2) distribution of possible outcomes of each experiment over the whole set of possible confounder values (for a given do-intervention).
We make sure that the data does not degenerate to simple regularities which are solvable by conventional methods predicting the future from the past.
In particular, for each experimental setup, we enforce existence of at least two different confounder conﬁgurations resulting in signiﬁcantly different object trajectories.
This guarantees that estimating the confounder variable is necessary for visual reasoning on this dataset .
More speciﬁcally, we ensure that for each experiment the set of possible counterfactual outcomes is balanced w.r.t.
(1) tower stability for BlocktowerCF and (2) distribution of object trajectories for BallsCF and CollisionCF .
As a result, the BlocktowerCF set, for example, has 50 ± 5% of stable and unstable counterfactual conﬁgurations.
The exact distribution of stable/unstable examples for each confounder in this scenario is shown in Fig.
All images for this benchmark have been rendered into the visual space (RGB, depth and instance segmentation) at a resolution of 448 × 448 px with PyBullet (only RGB images are used in this work).
We ensure diversity in visual appearance between experiments by rendering the pairs of sequences over a set of randomized backgrounds.
The ground truth physical properties of each object (3D pose, 4D quaternion angles, velocities) are sampled at a higher frame rate (20 fps) and also stored.
The training / validation / test split is deﬁned as 0 .
1 for each of the three scenarios.
[Figure]
[Table]
[Figure]
[Table] [Figure] Our model learns counterfactual reasoning in a weakly supervised way: while we supervise the do-operator, we do not supervise the confounder variables (masses, frictions, gravity). Input images of the original past ( A ) and the original outcome ( B ) are de-rendered into latent representations which are converted into fully-connected attributed graphs.
A Graph Network updates node features to augment them with contextual information, which is integrated temporally with a set of RNNs, one for each object, running over time.
The last hidden RNN state is taken as an estimate of the confounder
C OUNTERFACTUAL L EARNING OF P HYSICS
The task as described in Section 3 requires reasoning from visual inputs.
We propose a single neural model which can be trained end-to-end, as shown in Fig.
We address this problem by adding strong inductive biases to a deep neural network, structuring it in a way to favor counterfactual reasoning.
More precisely, we add structure for (i) estimation of physical properties from images, (ii) modelling interactions between objects through graph convolutions (GCN), (iii) estimating latent representations of the confounder variables, and (iv) exploiting these representations for predictions of the output object positions.
At this point we would like to stress again, that the representation of the confounders U is latent and discovered from data without supervision.
4. U NSUPERVISED ESTIMATION OF THE CONFOUNDERS
While our method is capable of handling raw RGB frames as input, its internal reasoning is done on estimated representations in object-centric viewpoints.
We train a convolutional neural network to detect the K objects and their 3D position in the scene, denoted as O = { o k } , k =0 ...
K − 1 where o k corresponds to the 3D position of object k .
The de-rendering module is explained in the appendix.
Predicting the future of a given block k requires modelling its interactions (through friction and collisions) with the other blocks in the scene, which we do with Graph Convolution Networks (GCN) (Kipf & Welling, 2017; Battaglia et al., 2018).
The set of K objects in the scene is represented as a graph G =( V , E ) where the nodes V are associated to objects { o k } , and the object interactions to edges ( o k , o j ) ∈ E in the fully-connected graph.
Object embeddings { o k } are updated classically and as follows, resulting in new embeddings
[Equation]
where Ω k is the set of neighboring objects of o k .
f ( . ) and g ( . ) are non-linear mappings (MLPs), and their inputs are by default concatenated.
For simplicity, in what follows we will denote the update of an object o k with GCN given a graph with set of nodes and embeddings O by ˜ o k = GCN
( o k , O ) .
As mentioned above, we want to infer a latent representation U of the confounding quantities for each object k given the input sequences X 1: τ (the original past A and the original outcome B ), without any supervision.
This latent representation U is trained end-to-end by optimizing the counterfactual prediction loss.
To this end, we pass the updated object states ˜ o k through a recurrent network to model the temporal evolution of this representation.
In particular, we run a dedicated RNN for each object, each object maintaining its own hidden state
h k :
[Equation]
where we index objects and states with subscript t indicating time, and φ is a gated recurrent unit (GRU) (gate equations have been omitted for simplicity).
The recurrent network parameters are shared over objects k , which results in a model which is invariant to the number of objects present in the set.
This allows to use do-operators which change the number of objects in the scene (removal).
We set the latent representation of the confounders to be the set U
= { u k } , where u k (cid:44) h kτ is the temporally last hidden state of the recurrent network.
4. T RAJECTORY PREDICTION GATED BY STABILITY
We predict the counterfactual outcome D , i.e. the 3D positions of all objects of the sequence ¯ X 1: τ , with a recurrent network, which takes into account the confounders U .
We cast this problem as a sequential prediction task, at each time step t predicting the residual position ∆ kt w.r.t.
to position t − 1 , i.e. the velocity vector.
As in the rest of the model, this prediction is obtained object-wise, albeit with explicit modelling of the inter-object relationships through a graph network.
More precisely,
[Equation]
where r kt is the hidden state of the GRU network denoted by ψ , and W is the weight matrix of a linear output layer.
GCN is a graph convolutional network as described in eq.
(1) and thereafter.
At each moment of time, each object can either remain stationary or move under the inﬂuence of external physical forces or by inertia.
The ﬁrst task for the model is therefore to detect which objects are moving (i.e. affected by the environment) and then estimate parameters of the motion if it occurs.
This is aligned well with the concepts of whether-causation and how-causation deﬁned in the ﬁeld of perceived causality (Gerstenberg et al., 2015).
In our work, the whether-cause is estimated in the form of a binary stability indicator s kt described below (for each object, updated at each time step) that is then leveraged to gate the object position predictor ( how-cause estimator):
[Equation]
where σ ( . ) is the sigmoid function and λ is a sparsifying temperature term.
Counterfactual estimation of stability — estimation of object stability s kt is a counterfactual problem, as stability depends on the physical properties, and therefore on the latent confounder representation u k .
We combine the confounders U
= { u k } with the past after do-intervention (C), encoded in object states denoted as ¯ O t = { ¯ o kt } at time step t = t .
In particular, for each node we concatenate its object features with its confounder representation and we update the resulting object state with a graph network to take into account inter-object relationships:
[Equation]
where s kt corresponds to the logits of stability of object k at time t and V is the weight matrix of a linear layer (for simplifying notations we omit bias here and in the rest of the paper).
Training — The full counterfactual prediction model is trained end-to-end in graph space only (i.e. not including the de-rendering engine) with the following losses:
[Equation]
where L ce is the binary cross entropy loss between the stability prediction of object k at time t and its ground truth value (calculated by thresholding applied to ground truth speed vectors), and L mse is mean squared error between the predicted positions ¯ o kt and GT positions ¯ o k ∗ t in the 3D space. A detailed description of the overall architecture is given in the Appendix. [Table] with
[Table] ICCV# ICCV#
[Figure]
[Figure]
as with 4 blocks.
And the set of possible masses is ﬁxed to
M = { 1 , 10 } .
Each sequence is of length 5 seconds.
We render the physical world into the visual space (RGB, depth and segmentation) every 0.5 second at a resolution of 448 ⇥ 448 .
RGB, deth and segmentation images are encoded as png ﬁles.
The full dataset is composed of 2 millions of frames.
We record the physical properties of each object (3D pose, 4D quaternion angles, velocities) at frame rate of 20 fps.
We split the dataset such that the training set if composed of 200K samples.
The validation set and test set are of size 50K each.
Table 6 gives an overview of the dataset.
The full set of data, including synthetic images, is 326GB.
This data will be made available publicly after acceptance of this paper.
We use Pybullet as physical engine.
We sample 200K frames (20K unstable sequences) for training the de-rendering and rendering part.
We split into a train, val and test sets (120K, 40K, 40K) There are roughly the same number of towers with 3 and 4 blocks.
Training details All models were implemented in PyTorch and trained on a cluster of Titan-X GPUs.
We used the XXX optimizer and a learning rate of XXX.
Training a full model until convergence takes XXXh.
Qualitative evaluation Figure ??
illustrates several problem instances and predictions by our model.
Human performance We measured human performance on this challenging dataset by ...
Natalia: NOTES.
100 workers, 20 assignments each, both in counterfactual and non-counterfactual settings.
Same experiment in the counterfactual setting, but limiting the time when the ﬁrst sequence A !
B is observed to 5 seconds.
Conclusion: CF setup is slightly better in terms of mean error, but it looks like it generally boils down to simple indictive biases, such as ”observed (un)stability” ! ”
predict (un)stability” [prove empirically by clustering trajectories / calculating correlations].
This is shown in Figure ?
: variance after having observed a stable sequence is decreased (ﬁrst row), after having observed a falling case - increased (second row).
Overall, variance in predictions is slightly higher.
Humans are doing much worse than copying baselines.
Performance and comparisons we evaluate the counterfactual prediction performance against various baselines:
• assuming stability (absence of motion) and copying the past after do-intervention, denoted as C !
D ; • assuming no do-intervention and copying the (observed) original outcome, denoted as B !
D ; • Network Physics Engine (NPE) [4], a non counterfactual baseline, which predicts the future from the past after do-intervention without taking into account confounders.
The performances are given in table ??
for various splits between training and validatio scenarios.
Data dependence Table XXX provides evaluations as
Fabien: Compute the LIPSP metric - ”The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”, Zang et al, CVPR
7. Conclusion
E XPERIMENTS
Training details.
All models were implemented in PyTorch.
We used the Adam optimizer (Kingma & Ba, 2015) and a learning rate of 0.001.
For training the de-rendering pipeline 200k frames were sampled for each of the three scenarios (see the appendix for more details).
Human performance.
We empirically measured human performance in the BlockTowerCF scenario with crowdsourcing (Amazon Mechanical Turk/AMT).
For this study, we have collected predictions from 100 participants, where each subject was given 20 assignments in both non-counterfactual (Fig. 5) and counterfactual (Fig. 6) settings.
The human subjects were given 10 sec to click on the ﬁnal positions of each block in the image C after the tower has fallen (or remained stable).
The obtained quantitative results for both settings are reported in Table 1.
We compare against copying baselines (i.e. predicting block positions in the frame D by either copying them from C or from B ).
In conclusion, we observe that humans perform slightly better in the counterfactual setup after having observed the ﬁrst dynamic sequence ( A , B ) together with C compared to the classical prediction where only C is shown.
This behavior has also been previously observed in experiments on intuitive physics in cognitive psychology (Kubricht et al., 2017) that revealed poor human abilities to extrapolate physical dynamics from a single image.
Similar human studies have also been conducted in (Battaglia et al., 2013) in a more simplistic setup of predicting the direction of falling, where the authors also reported that the task appeared to be challenging for human subjects.
The empirical results indicate that the participants decisions may have been however driven by simple inductive biases, e.g. “observed (in)stability in ( A , B ) ” → ”predict (in)stability in ( C , D ) ”.
The evidence for this approach is demonstrated qualitatively in Fig.
6: the variance in predictions after having observed a stable sequence is decreased (ﬁrst row), after having observed a falling case – increased (second row).
In all cases, human performance remains inferior w.r.t.
the copying baselines.
The last part of Table 1 shows results of our model (denoted by CoPhyNet in the rest of the discussion) after projecting the estimated 3D positions of all objects back into the 2D image space.
CoPhyNet signiﬁcantly outperforms both human subjects and copying baselines.
Performance and comparisons.
We evaluate the counterfactual prediction performance of the proposed CoPhyNet model against various baselines (shown in Tables 2-4 separately for each of the three scenarios of the CoPhy benchmark).
The evaluated Network Physics Engine (NPE) (Chang et al., 2017) and Interaction Network (IN) (Battaglia et al., 2016), are both non-counterfactual baselines, [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
> <latexit sha1_base64="(null)">(null)</latexit> { µ 0: m } 0 <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> { µ 0: m } 1 <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> that predict future block coordinates from past coordinates after do-intervention without taking the confounders into account.
More details for IN and NPE are given in the appendix A.3.
Our method consistently outperforms NPE and IN by a large margin in all scenarios.
The CoPhyNet model also usually (but not always) outperforms the augmented variants of these methods that include the GT confounder quantities as input (a not comparable setting).
Fig.
7 illustrates several randomly sampled experimental setups and corresponding counterfactual predictions by the CoPhyNet model in the BlocktowerCF scenario.
Generalization.
We evaluate the ability of the CoPhyNet model to generalize to new physical setups which were not observed in the training data.
In Table 2 we show model performance on unseen confounder combinations and on unseen number of blocks in the BlocktowerCF scenario (lines marked with † ).
Our proposed solution generalizes well under unseen settings compared to other methods.
In Table 3 we also demonstrate that our method outperforms the baselines by a large margin on unseen numbers of balls in the BallsCF setup.
Finally, in the CollisionCF scenario (Table 4) we train on one type of moving objects and test on another type (spheres vs cylinders).
In this case we also show that our method is able to generalize to the new object types even when it has not seen such a combination of < moving-object, static-object > before.
Our method is able to estimate the object properties when an object is moving or initially stable.
Impact of the confounder estimate.
Our model does not rely on any supervision of the confounders; we do, however, explore what effect supervision could have on performance, as shown in Table(Middle).
Adding the supervision increases the performance of the model for K =
but the difference seems marginal (0.004 for K =3 and 0.020 for K =4 ).
Directly feeding the confounder quantities as input leads to better performance, which is expected (but not comparable).
Model architecture.
All design choices of CoPhyNet are ablated in Table 6 (Left) to fully illustrate the impact of each submodule.
Estimating the stability once for the whole sequence D decreases the performance by 0.020 for K =3 and 0.018 for K =4 compared to predicting the stability per object at each time step.
Replacing the GCN by a MLP (i.e. concatenating the object representation) hurts the performance of the overall system by increasing the MSE by 0.286 when tested in the K =4 setting.
Finally we compare our approach against a single-step counterfactual prediction.
Non-surprisingly, predicting the future autoregressively in a step-by-step fashion turns out to be more effective than predicting the whole sequence at once.
Confounder estimation.
After training for predicting the target CF sequences, we evaluate the quality of the learned latent representation.
In this experiment, we predict the confounder quantities of each object (mass, friction coefﬁcient) from their latent representation by training a simple linear classiﬁer, freezing the weights of the whole network.
The obtained results are shown in Table 5.
A
[Table]
[Table]
[Table]
prediction is correct if both the mass and the friction coefﬁcients are correctly predicted.
Our model outperforms the random baseline by a large margin suggesting that the confounder quantities are correctly encoded into the latent representation of each object during training.
Stability prediction.
We studied the performance of the stability estimation module in the BlockTowerCF scenario and compared it to several baselines, as shown in Table 6 (Right).
Our method predicts stability of each block from the confounder estimate U and the frame C .
It outperforms the baselines estimating stability from a single input C or from the sequence ( A , B ) by a large margin, further indicating the efﬁciency of the confounder estimation and the complimentarity of this non-visual information w.r.t.
the visual observation C .
C ONCLUSION
We formulated a new task of counterfactual reasoning for learning intuitive physics from images, developed a large-scale benchmark suite and proposed a practical approach for this problem.
The task requires to predict alternative outcomes of a physical problem given the original past and outcome and an alternative past after do-intervention.
Our challenging benchmarks cannot be solved by classical methods predicting by extrapolation, as the alternative future depends on confounder variables, which are unobservable from a single image of the alternative past.
We train a neural model by supervising the do-operator, but not the confounders.
Our experiments show that the CF setting outperforms conventional forecasting, and that the latent representation is related to the GT confounder quantities.
We report human performance on this task, show its challenging nature and importance of CF reasoning.
We believe that counterfactual reasoning in high-dimensional spaces is a key component of AI and hope that our task will spawn new research in this area and thus contribute to bridging the gap between causal reasoning and deep learning.
We also expect the benchmark to become a testbed in model based RL, which employs predictive models of an environment for learning agent behavior.
Forward models are classically used in this context, but we conjecture that CF reasoning will contribute to disentangling representations and inferring causal relationships between different factors of variation. Acknowledgements.
This work was funded by grant Deepvision (ANR-15-CE23-0029, STPGP479356-15), a joint French/Canadian call by ANR & NSERC.
A A PPENDIX A. N EURAL DE - RENDERING
We train a convolutional neural network (CNN) to detect the K blocks and estimate their positions in 3D, denoted as O = { o k } , k =0 ...
K − 1 where o k corresponds to the position of object k .
The network is inspired by classical region-based methods for object detection (He et al., 2017) and takes as an input an RGB image of resolution 224 × 224 .
We deﬁne a double convolution module as a stack of a convolutional layer, batch normalization and
ReLu activation, repeated two times.
The resulting CNN includes three such modules with 64,and 256 channels respectively, separated by 2 × 2 max pooling, which produces output feature maps of size 256 × 56 × 56 .
We design K different heads by splitting the ﬁnal feature maps channel-wise into K feature maps and perform a double convolution module with 1 output channel.
Each head outputs a feature map of size 1 × 56 × 56 which is transformed into a vector to regress the object positions.
We ﬁrst pre-train the de-rendering module alone without the model parts responsible for reasoning in the graph space.
In particular, we de-render images ˆ X randomly sampled from A , B , C and D into its object representations O = { o k } and train with the following supervised loss:
[Equation]
where ˆ o k ∗ are the ground truth 3D positions and L mse corresponds to the mean square error.
The rest of the model is trained end-to-end as described in section 4.2, paragraph “ Training ”.
A. O THER ARCHITECTURES
Below are the descriptions of other networks used in our pipeline: f, g are implemented as MLPs with 4 and 2 layers respectively, with hidden layers of size 32 and
ReLu activations.
φ , ψ are implemented as GRU modules with 2 layers and a hidden state of dimension 32.
Confounders (mass and friction coefﬁcients) are predicted with a single fully connected layer on top of the “confounder” representation of each object denoted u k .
A. F EEDFORWARD BASELINES
We compare our approach against two recent feedforward approaches, namely IN (Battaglia et al., 2016) and NPE (Chang et al., 2017).
Both methods assume that GT object positions are available as input at training and test time, so they directly work on GT positions.
They both predict the next position of each object using a Graph Convolution Network.
IN is modeling object pairwise interaction between all objects in the scene, while NPE is taking into account only neighbouring objects for estimating the object interactions.
A. T RAINING FROM ESTIMATED POSITIONS
In Table 7 we report the impact of the de-rendering module on performance.
In particular, we compare performance of our model (CoPhyNet w/o GT, as described in the main part of the paper) with a version where we use ground-truth positions (CoPhyNet GT) for training.
During training time, GT positions are fed to the main model.
For testing, we do, however, use positions estimated by the de-rendering module in both versions.
We can see that training using the GT positions gives slighly better performance than training from estimated positions, which is expected.
[Table]