F ILTERED -C O P HY : U NSUPERVISED L EARNING OF C OUNTERFACTUAL P HYSICS IN P IXEL S PACE
A BSTRACT
Learning causal relationships in high-dimensional data (images, videos) is a hard task, as they are often deﬁned on low-dimensional manifolds and must be extracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data.
We present a method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions.
Going beyond the identiﬁcation of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons.
Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties.
Our model learns and acts on a suitable hybrid latent representation based on a combination of dense features, sets of 2D keypoints and an additional latent vector per keypoint.
We show that this better captures the dynamics of physical processes than purely dense or sparse representations.
We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.
I NTRODUCTION
Reasoning on complex, multi-modal and high-dimensional data is a natural ability of humans and other intelligent agents (Martin-Ordas et al., 2008) and one of the most important and difﬁcult challenges of AI.
While machine learning is well suited for capturing regularities in high-dimensional signals, in particular by using high-capacity deep networks, some applications also require an accurate modeling of causal relationships.
This is particularly relevant in physics, where causation is considered as a fundamental axiom.
In the context of machine learning, correctly capturing or modeling causal relationships can also lead to more robust predictions, in particular better generalization to out-of-distribution samples, indicating that a model has overcome the exploitation of biases and shortcuts in the training data.
In recent literature on physics-inspired machine learning, causality has often been forced through the addition of prior knowledge about the physical laws that govern the studied phenomena, e.g. (Yin et al., 2021).
A similar idea lies behind structured causal models, widely used in the causal inference community, where domain experts model these relationships directly in a graphical notation.
This particular line of work allows to perform predictions beyond statistical forecasting, for instance by predicting unobserved counterfactuals, the impact of unobserved interventions (Balke & Pearl, 1994) — “ What alternative outcome would have happened, if the observed event X had been replaced with an event Y (after an intervention) ”.
Counterfactuals are interesting, as causality intervenes through the effective modiﬁcation of an outcome.
As an example, taken from (Sch¨olkopf et al., 2021), an agent can identify the direction of a causal relationship between an umbrella and rain from the fact that removing an umbrella will not affect the weather.
We focus on counterfactual reasoning on high-dimensional signals, in particular videos of complex physical processes.
Learning such causal interactions from data is a challenging task, as spurious correlations are naturally and easily picked up by trained models.
Previous work in this direction was restricted to discrete outcomes, as in CLEVRER (Yi et al., 2020), or to the prediction of 3D trajectories, as in CoPhy (Baradel et al., 2020), which also requires supervision of object positions.
In this work, we address the hard problem of predicting the alternative (counterfactual) outcomes of physical processes in pixel space, i.e. we forecast sequences of 2D projective views of the 3D scene, requiring the prediction over long horizons (150 frames corresponding to ∼ 6 seconds).
We conjecture that causal relationships can be modeled on a low dimensional manifold of the data, and propose a suitable latent representation for the causal model, in particular for the estimation of the confounders and the dynamic model itself.
Similar to V-CDN (Kulkarni et al., 2019; Li et al., 2020), our latent representation is based on the unsupervised discovery of keypoints, complemented by additional information in our case.
Indeed, while keypoint-based representations can easily be encoded from visual input, as stable mappings from images to points arise naturally, we claim that they are not the most suitable representation for dynamic models.
We identiﬁed and addressed two principal problems: (i) the individual points of a given set are discriminated through their 2D positions only, therefore shape, geometry and relationships between multiple moving objects need to be encoded through the relative positions of points to each other, and (ii) the optimal representation for a physical dynamic model is not necessarily a 2D keypoint space, where the underlying object dynamics has also been subject to the imaging process (projective geometry).
We propose a new counterfactual model, which learns a sparse representation of visual input in the form of 2D keypoints coupled with a (small) set of coefﬁcients per point modeling complementary shape and appearance information.
Confounders (object masses and initial velocities) in the studied problem are extracted from this representation, and a learned dynamic model forecasts the entire trajectory of these keypoints from a single (counterfactual) observation.
Building on recent work in data-driven analysis of dynamic systems (Janny et al., 2021; Peralez & Nadri, 2021), the dynamic model is presented in a higher-dimensional state space, where dynamics are less complex.
We show, that these design choices are key to the performance of our model, and that they signiﬁcantly improve the capability to perform long-term predictions.
Our proposed model outperforms strong baselines for physics-informed learning of video prediction.
We introduce a new challenging dataset for this problem, which builds on CoPhy , a recent counterfactual physics benchmark (Baradel et al., 2020).
We go beyond the prediction of sequences of 3D positions and propose a counterfactual task for predictions in pixel space after interventions on initial conditions (displacing, re-orienting or removing objects).
In contrast to the literature, our benchmark also better controls for the identiﬁability of causal relationships and counterfactual variables and provides more accurate physics simulation.
R ELATED W ORK
Counterfactual (CF) reasoning — and learning of causal relationships in ML was made popular by works of J. Pearl, e.g. (Pearl, 2000), which motivate and introduce mathematical tools detailing the principles of do-calculus , i.e. study of unobserved interventions on data.
A more recent survey links these concepts to the literature in ML (Sch¨olkopf et al., 2021).
The last years have seen the emergence of several benchmarks for CF reasoning in physics.
CLEVRER (Yi et al., 2020) is a visual question answering dataset, where an agent is required to answer a CF question after observing a video showing 3D objects moving and colliding.
Li et al.
(2020) introduce a CF benchmark with two tasks: a scenario where balls interact with each other according to unknown interaction laws (such as gravity or elasticity), and a scenario where clothes are folded by the wind.
The agent needs to identify CF variables and causal relationships between objects, and to predict future frames.
CoPhy (Baradel et al., 2020) clearly dissociates the observed experiment from the CF one, and contains three complex 3D scenarios involving rigid body dynamics.
However, the proposed method relies on the supervision of 3D object positions, while our work does not require any meta data.
Physics-inspired ML — and learning visual dynamics has been dealt early on with recurrent models (Srivastava et al., 2015;
Finn et al., 2016; Lu et al., 2017), or GANs (Vondrick et al., 2016;
Mathieu et al., 2016).
Kwon & Park (2019) adopt a Cycle-GAN with two discriminator heads, in charge of identifying false images and false sequences in order to improve the temporal consistency of the model in long term prediction.
Nonetheless, the integration of causal reasoning and prior knowledge in these models is not straightforward.
Typical work in physics-informed models relies on disentanglement between physics-informed features and residual features (Villegas et al., 2017a;
Denton & Birodkar, 2017) and may incorporate additional information based on the available priors on the scene (Villegas et al., 2017b; Walker et al., 2017).
PhyDNet Le Guen & Thome (2020) explicitly disentangles visual features from dynamical features, which are supposed to follow a PDE.
It achieves SOTA performance on Human3.6M (Ionescu et al., 2014) and Sea Surface Temperature (de Bezenac et al., 2018), but we show that it fails on our challenging benchmark.
Keypoint detection — is a well researched problem in vision with widely used handcrafted baselines (Lowe, 1999).
New unsupervised variants emerged recently and have been shown to provide a suitable object-centric representation, close to attention models, which simplify the use of physical and/or geometric priors (Locatello et al., 2020; Veerapaneni et al., 2020).
They are of interest in robotics and reinforcement learning, where a physical agent has to interact with objects (Kulkarni et al., 2019; Manuelli et al., 2020; 2019).
KeypointsNet (Suwajanakorn et al., 2018) is a geometric reasoning framework, which discovers meaningful keypoints in 3D through spatial coherence between viewpoints.
Close to our work, (Minderer et al., 2019) proposes to learn a keypoints-based stochastic dynamic model.
However, the model is not suited for CF reasoning in physics and may suffer from inconsistency in the prediction of dynamics over long horizons.
T HE F ILTERED -C O P HY BENCHMARK
We build on CoPhy (Baradel et al., 2020), retaining its strengths, but explicitly focusing on a counterfactual scenario in pixel space and eliminating the ill-posedness of tasks we identiﬁed in the existing work.
Each data sample is called an experiment , represented as a pair of trajectories: an observed one with initial condition X
=
A and outcome X t
=1 ..T = B (a sequence), and a counter- factual one ¯ X 0 = C and ¯ X t
=1 ..T = D (a sequence).
Throughout this paper we will use the letters
A , B , C and D to distinguish the different parts of each experiment.
The initial conditions A and C are linked through a do-operator do ( X 0 = C ) , which modiﬁes the initial condition (Pearl, 2018).
Experiments are parameterized by a set of intrinsic physical parameters z which are not observable from a single initial image A .
We refer to these as confounders .
As in CoPhy , in our benchmark the do-operator is observed during training, but confounders are not — they have been used to generate the data, but are not used during training or testing.
Following (Pearl, 2018), the counterfactual task consists in inferring the counterfactual outcome D given the observed trajectory AB and the counterfactual initial state C , following a three-step process:
(cid:192) Abduction : use the observed data AB to compute the counterfactual variables, i.e. physical parameters, which are not affected by the do-operation.
(cid:193) Action : update the causal model; keep the same identiﬁed confounders and apply the dooperator, i.e. replace the initial state A by C .
(cid:194) Prediction : Compute the counterfactual outcome D using the causal graph.
The benchmark contains three scenarios involving rigid body dynamics.
BlocktowerCF studies stable and unstable 3D cube towers, the confounders are masses.
BallsCF focuses on 2D collisions between moving spheres (confounders are masses and initial velocities).
CollisionCF is about collisions between a sphere and a cylinder (confounders are masses and initial velocities)
(Fig. 1).
Unlike CoPhy , our benchmark involves predictions in RGB pixel space only.
The do-operation consists in visually observable interventions on A , such as moving or removing an object.
The confounders cannot be identiﬁed from the single-frame observation A , identiﬁcation requires the analysis of the entire AB trajectory.
Identiﬁability of confounders — For an experiment ( AB , CD , z ) to be well-posed, the confounders z must be retrievable from AB .
For example, since the masses of a stable cube tower cannot be identiﬁed generally in all situations, it can be impossible to predict the counterfactual outcome of an unstable tower, as collisions are not resolvable without known masses.
In contrast to CoPhy , we ensure that each experiment ψ : ( X 0 , z ) (cid:55)→ X t =1 ..T , given initial condition X 0 and confounders z , is well posed and satisﬁes the following constraints:
Deﬁnition 1 (Identiﬁability, (Pearl, 2018))
The experiment ( AB , CD , z ) is identiﬁable if, for any set of confounders z (cid:48) :
[Equation]
[Equation]
[Table]
+BK3pAEvpqguB62SVjS00nW5ULtIR53FEY5xSvM8Qw2XaKBJ3j5e8Io35UpJlEflaZ6qZFLNIX4t5fkbAe6Tuw==</latexit>
[Figure]
<latexit sha1_base64="r+txWyvKiAB+FzMaAfcOIUpKQqo=">AAACzXicjVHLTsJAFD3UF+ILdemmkZi4agqFiDuiLtyJiYARiGnLAA19pZ2aEMStP+BWf8v4B/oX3hlKogui07Q9c+49Z+bea4WuE3Nd/8goS8srq2vZ9dzG5tb2Tn53rxkHSWSzhh24QXRrmTFzHZ81uMNddhtGzPQsl7Ws0bmItx5YFDuBf8PHIet65sB3+o5tcqLuOp7Jh1Z/cjG9zxd07bSqGxVdlaBilGfA0CtqUdPlKiBd9SD/jg56CGAjgQcGH5ywCxMxPW0UoSMkrosJcREhR8YZpsiRNqEsRhkmsSP6DmjXTlmf9sIzlmqbTnHpjUip4og0AeVFhMVpqown0lmwi7wn0lPcbUx/K/XyiOUYEvuXbp75X52ohaOPqqzBoZpCyYjq7NQlkV0RN1d/VMXJISRO4B7FI8K2VM77rEpNLGsXvTVl/FNmClbs7TQ3wZe4JQ14PkV1MWiWtKKhla7LhdpZOuosDnCIY5rnCWq4RB0N8vbxgle8KVdKojwqT7NUJZNq9vFrKc
[Figure] In an identiﬁable experiment there is no pair ( z, z (cid:48) ) that gives the same trajectory AB but different counterfactual outcomes CD .
Details on implementation and impact are in appendix A.
Counterfactuality — We enforce sufﬁcient difﬁculty of the problem through the meaningfulness of confounders.
We remove initial situations where the choice of confounder values has no signiﬁcant impact on the ﬁnal outcome:
Deﬁnition 2 (Counterfactuality).
Let z k be the set of confounders z , where the k th value has been modiﬁed.
The experiment ( AB , CD , z ) is counterfactual if and only if:
[Equation]
In other words, we impose the existence of an object of the scene for which the (unobserved) physical properties have a determining effect on the trajectory.
Details on how this constraint was enforced are given in appendix A.2.
Temporal resolution — the physical laws we target involve highly non-linear phenomena, in particular collision and resting contacts.
Collisions are difﬁcult to learn because their actions are both intense, brief, and highly non-linear, depending on the geometry of the objects in 3D space.
The temporal resolution of physical simulations is of prime importance.
A parallel can be made with Nyquist-Shannon frequency, as a trajectory sampled with too low frequency cannot be reconstructed with precision.
We simulate and record trajectories at 25 FPS, compared to 5 FPS chosen in CoPhy , justiﬁed with two experiments.
Firstly, Fig.
2 shows the trajectories of the center of masses of cubes in BlocktowerCF , colored dots are shown at 25 FPS and black dots at 5 FPS.
We can see that collisions with the ground fall below the sampling rate of 5 FPS, making it hard to infer physical laws from regularities in data at this frequency.
A second experiment involves learning a prediction model at different frequencies, conﬁrming the choice 25 FPS — details are given in appendix A.3.
U NSUPERVISED LEARNING OF COUNTERFACTUAL PHYSICS
We introduce a new model for counterfactual learning of physical processes capable of predicting visual sequences D in the image space over long horizons.
The method does not require any supervision other than videos of observed and counterfactual experiences.
The code is publicly available online at https://filteredcophy.github.io .
The model consists of three parts, learning the latent representation and its (counterfactual) dynamics:
• The encoder (De-Rendering module) learns a hybrid representation of an image in the form of a (i) dense feature map and (ii) 2D keypoints combined with (iii) a low-dimensional vector of coef- [Table]
<
[Table]
> decoder
[Table]
[Figure]
ﬁcients, see Fig. 3.
we
• The Counterfactual Dynamic (CoDy) model based on recurrent graph networks, in the lines of (Baradel et al., 2020).
It estimates a latent representation of the confounders z from the keypoint + coefﬁcient trajectories of AB provided by the encoder, and then predicts D in this same space.
• The decoder that uses the predicted keypoints to generate a pixel-space representation of D .
4. D ISENTANGLING VISUAL INFORMATION FROM DYNAMICS
The encoder takes an input image X and predicts a representation with three streams, sharing a common conv.
backbone, as shown in Fig.
We propose an unsupervised training objective, which favors the emergence of a latent representation disentangling static and dynamic information.
A dense feature map F = F ( X ) , which contains static information, such as the background.
A set of 2D keypoints k = K ( X ) , which carry positional information from moving objects.
A set of corresponding coefﬁcients c = C ( X ) , one vector c k of size C per keypoint k , which encodes orientation and appearance information.
The unsupervised objective is formulated on pairs of images ( X source , X target ) randomly sampled from same D sequences (see appendix D.1 for details on sampling).
Exploiting an assumption on the absence of camera motion 1 , the goal is to favor the emergence of disentangling static and dynamic information.
To this end, both images are encoded, and the reconstruction of the target image is predicted with a decoder D fusing the source dense feature map and the target keypoints and coefﬁcients.
This formulation requires the decoder to aggregate dense information from the source and sparse values from the target, naturally leading to motion being predicted by the latter.
On the decoder D side, we add inductive bias, which favors the usage of the 2D keypoint information in a spatial way.
The 2D coordinates k k for each keypoint k are encoded as Gaussian heatmaps
G ( k k ) , i.e. 2D Gaussian functions centered on the keypoint position.
The additional coefﬁcient information, carrying appearance information, is then used to deform the Gaussian mapping into an anisotropic shape using a ﬁxed ﬁlter bank H , as follows:
[Equation]
where R ( ... ) is a reﬁnement network performing trained upsampling with transposed convolutions, whose inputs are stacked channelwise.
G ik are Gaussian mappings produced from keypoint positions k , deformed by ﬁlters from bank H and weighted by coefﬁcients c ik .
The ﬁlters H i are deﬁned as ﬁxed horizontal, vertical and diagonal convolution kernels.
This choice is discussed in section 5.
The joint encoding and decoding pipeline is illustrated in Fig.
The model is trained to minimize the mean squared error (MSE) reconstruction loss, regularized with a loss on spatial gradients ∇ X weighted by hyper-parameters γ 1 , γ 2 ∈ R :
[Equation]
where ˆ X target =
D ( F ( X source ) , K ( X target ) , C ( X target )) is the reconstructed image, γ 1 , γ 2 are weights.
Related work – our unsupervised objective is somewhat related to Transporter (Kulkarni et al., 2019), which, as our model, computes visual feature vectors F source and F target as well as 2D keypoints K source and K target , modeled as a 2D vector via Gaussian mapping.
It leverages a handcrafted transport equation: ˆΨ target =
F source × (1 − K source ) ×
(1 − K target )
+ F target × K target .
As in our case, the target image is reconstructed through a reﬁner network ˆ X target =
R (ˆΨ target ) .
The transporter suffers from a major drawback when used for video prediction, as it requires parts of the target image to reconstruct the target image — the model was originally proposed in the context of RL and control, where reconstruction is not an objective.
It also does not use shape coefﬁcients, requiring shapes to be encoded by several keypoints, or abusively be carried through the dense features F target .
This typically leads to complex dynamics non representative of the dynamical objects.
We conducted an in-depth comparison between the Transporter and our representation in appendix C.2.
4. D YNAMIC MODEL AND CONFOUNDER ESTIMATION
Our counterfactual dynamic model (CoDy) leverages multiple graph network (GN)
based modules (Battaglia et al., 2016) that join forces to solve the counterfactual forecasting tasks of FilteredCoPhy .
Each one of these networks is a classical GN, abbreviated as GN ( x k ) , which contextualizes input node embeddings x k through incoming edge interactions e ik , providing output node embeddings ˆx k (parameters are not shared over the instances):
[Equation]
such that ˆx
[Equation]
i
[Equation]
with e ij = f
( x i , x j )
[Equation]
(5) where f is a message-passing function and g is an aggregation function.
We deﬁne the state of frame X t at time t as a stacked vector composed of keypoints and coefﬁcients computed by the encoder (the de-rendering module), i.e. s ( t ) =
[ s 1 ( t ) ...
s k ( t )] where s k ( t )
=
[ k k c 1 k ...
c
C
+
k ]( t ) .
In the lines of (Baradel et al., 2020), given the original initial condition and outcome AB , CoDy estimates an unsupervised representation u k of the latent confounder variables per keypoint k through the counterfactual estimator ( CF estimator in Fig. 4).
It ﬁrst contextualizes the sequence s AB ( t ) through a graph network GN (cid:0) s AB ( t ) (cid:1)
=
h AB ( t ) =
[ h 1 ( t ) ...
h K ( t )] .
We then model the temporal evolution of this representation with a gated recurrent unit (Cho et al., 2014) per keypoint, sharing parameters over keypoints, taking as input the sequence h k .
Its last hidden vector is taken as the confounder estimate u k .
Recent works on the Koopman Operator (Lusch et al., 2018) and Kazantzis-Kravaris-Luenberger
Observer (Janny et al., 2021; Peralez & Nadri, 2021) have theoretically shown that, under mild assumptions, there exists a latent space of higher dimension, where a dynamical system given as an EDP can have a simpler dynamics.
Inspired by this idea, we used an encoder-decoder structure within CoDy, which projects our dynamic system into a higher-dimensional state space, performs forecasting of the dynamics in this latent space, and then projects predictions back to the original keypoint space.
Note that this dynamics encoder/decoder is different from the encoder/decoder of the de-rendering/rendering modules discussed in section 4.1.
The state encoder E ( s ( t ))
=
σ ( t ) is modeled as a graph network GN , whose aggregation function projects into an output embedding space σ ( t ) of dimension 256.
The decoder ∆( σ ( t ))
=
s ( t ) temporally processes the individual contextualized states σ ( t ) with a GRU, followed by new contextualization with a graph network
GN .
Details on the full architecture are provided in appendix D.2.
The dynamic model CoDy performs forecasting in the higher-dimensional space σ ( t ) , computing a displacement vector δ ( t + 1) such that σ ( t + 1)
= σ ( t )
+ δ ( t + 1) .
It takes the projected state embeddings σ CD k
( t ) per keypoint k concatenated with the confounder representation u k and contextualizes them with a graph network, resulting in embeddings
h CD k
( t ) =
GN ([ σ CD k ( t ) , u k ]) , which are processed temporally by a GRU.
We compute the displacement vector at time t +
[Table]
/yk1cplnH9+W8fABf1KT5g==</latexit> F <latexit sha1_base64="ncQtksAl8WfAXclpYeQaOKUgjAg=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LIoiMsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZS8srq2v59cLG5tb2TnF3r5GEacxFnYdeGLcclgjPDURdutITrSgWzHc80XRGlyreHIs4ccPgVk4i0fXZIHD7LmeSqHbHZ3LImTe9mt0VS1bZ0stcBHYGSshWLSy+oIMeQnCk8CEQQBL2wJDQ04YNCxFxXUyJiwm5Oi4wQ4G8KakEKRixI/oOaNfO2ID2Kmei3ZxO8eiNyWniiDwh6WLC6jRTx1OdWbG/5Z7qnOpuE/o7WS6fWIkhsX/55sr/+lQtEn2c6xpcqinSjKqOZ1lS3RV1c/NLVZIyRMQp3KN4TJhr57zPpvYkunbVW6bjb1qpWLXnmTbFu7olDdj+Oc5F0KiU7ZNy5ea0VL3IRp3HAQ5xTPM8QxXXqKGuO/6IJzwbNWNszIz7T6mRyzz7+LaMhw9zbZPh</latexit> <latexit sha1_base64="pwjVXc1t6q1Jb883GpAaFSJ4954=">AAAB+XicbVBNS8NAEN3Ur1q/oh69BIvQU0l6UI8FLx4r2A9oQ9lsJu3SzSbsTool9J948aCIV/+JN/+N2zYHbX0w8Hhvhpl5QSq4Rtf9tkpb2zu7e+X9ysHh0fGJfXrW0UmmGLRZIhLVC6gGwSW0kaOAXqqAxoGAbjC5W
[Table]
[Table] [Table]
/oOaNfO2ID2Kmei3ZxO8eiNyWniiDwh6WLC6jRTx1OdWbG/5Z7qnOpuE/o7WS6fWIkhsX/55sr/+lQtEn2c6xpcqinSjKqOZ1lS3RV1c/NLVZIyRMQp3KN4TJhr57zPpvYkunbVW6bjb1qpWLXnmTbFu7olDdj+Oc5F0KiU7ZNy5ea0VL3IRp3HAQ5xTPM8QxXXqKGuO/6IJzwbNWNszIz7T6mRyzz7+LaMhw9zbZPh</latexit>
C <latexit sha1_base64="5quqgpaO6fbEC2su/WcpYM1UN0w=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLYjcsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZK6tr6xv5zcLW9s7uXnH/oJmEacxFg4deGLcdlgjPDURDutIT7SgWzHc80XLGNRVvTUScuGFwK6eR6PlsGLgDlzNJVKfrMznizJvV5nfFklW29DKXgZ2BErJVD4sv6KKPEBwpfAgEkIQ9MCT0dGDDQkRcDzPiYkKujgvMUSBvSipBCkbsmL5D2nUyNqC9yploN6dTPHpjcpo4IU9IupiwOs3U8VRnVuxvuWc6p7rblP5OlssnVmJE7F++hfK/PlWLxACXugaXaoo0o6rjWZZUd0Xd3PxSlaQMEXEK9ykeE+baueizqT2Jrl31lun4m1YqVu15pk3xrm5JA7Z/jnMZNCtl+6xcuTkvVa+yUedxhGOc0jwvUMU16mjojj/iCc9G3ZgYc+P+U2rkMs8hvi3j4QNsSpPe</latexit>
K <latexit sha1_base64="SfiPy99ecpVKC/2nKPcBUIO6hnA=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoRnBTwT6gLTKZTtvQvEgmhVKKW
[Table]
>
[Table]
/yz3VOdXdJvR3slw+sRJDYv/yzZX/9alaJPo41zW4VFOkGVUdz7Kkuivq5uaXqiRliIhTuEfxmDDXznmfTe1JdO2qt0zH37RSsWrPM22Kd3VLGrD9c5yLoFEp2yflys1pqXqRjTqPAxzimOZ5hiquUUNdd/wRT3g2asbYmBn3n1Ijl3n28W0ZDx9uq5Pf< [Table] latexit [Table]
[Table]
</latexit>
H <latexit
[Table]
=
[Table]
=
=<
[Table]
latexit>
[Table]
<
[Table]
sha
[Table]
=
[Table]
=
[Table]
QD3vZHFzreHvMkFTK6VpOY90JvGImBYJ4iqstkNJZBpvFtueJUHbPsReDmoIJ8NWT5BTfoQ4IhQwiOCIpwAA8pPV24cBAT18OUuISQMHGOGUrkzUjFSeERO6LvkHbdnI1or3Omxs3olIDehJw2jsgjSZcQ1qfZJp6ZzJr9LffU5NR3m9Dfz3OFxCrcEfuXb678r0/XojDAmalBUE2xYXR1LM+Sma7om9tfqlKUISZO4z7FE8LMOOd9to0nNbXr3nom/maUmtV7lmszvOtb0oDdn+NcBK1a1T2p1q5qlfp5PuoiDnCIY5rnKeq4RANN0/FHPOHZalhja2bdf0qtQu7Zx7dlPXwAKO6UKw==</latexit> G ik <latexit sha1_base64="oBbqLZ1o7Q7gzY4DftZY+PkvZSU=">AAAC0XicjVHLSsNAFD3GV62vqks3wSK4KkkVdFl0ocuKVoXWSjKdtkPzYjIRSimIW3/Arf6U+
[Table]
[Table]
+fec2buvX4SiFQ5zvOENTk1PTNbmCvOLywuLZdWVs/SOJOMN1gcxPLC91IeiIg3lFABv0gk90I/
[Table]
+4EDHz6+5TEUcnaphwi9DrxeJrmCeIqrdCj3V97ujw3FbXA2uSmWn4phl/wRuDsrIVz0uPaGFDmIwZAjBEUERDuAhpacJFw4S4i4xIk4SEibOMUaRtBllccrwiB3Qt0e7Zs5GtNeeqVEzOiWgV5LSxiZpYsqThPVptolnxlmzv3mPjKe+25D+fu4VEqvQJ/
[Table]
[Table] [Table] [Table] [Table]
+
[Table]
+vEGlo31u17qjWRa9bwZVl3b7dZlSk [Table]
[Table]
+PQKUprYJE1EeYKwOs3U8VQ7K/Y370x7qrsN6O/kXgGxEhfE/qUbZf5Xp2qR6GFf1+BRTbFmVHVu7pLqrqibm1+qkuQQE6fwOcUFYVcrR302tSbRtaveMh1/05mKVXs3z03xrm5JA7Z/jnMcNLer9k51+3i3UjvIR13EOjawRfPcQw1HqKNB3jd4xBOeja5xa9wZ95+pRiHXrOLbMh4+AMFemj0=</latexit>
[Table]
[Table]
> coecients c k <
[Table]
[Figure]
[Table] [Table]
baselines.
We report MSE × 10 − 3 on prediction of keypoints + coefﬁcients.
as a linear transformation from the hidden state of the GRU.
We apply the dynamic model in an auto-regressive way to forecast long-term trajectories in the projected space σ k , and apply the state decoder to obtain a prediction ˆs CD ( t ) .
The dynamic model is trained with a loss in keypoint space,
[Equation]
[Equation]
The ﬁrst term enforces the model to learn to predict the outcomes and the second term favors correct reconstruction of the state in keypoint space.
The terms are weighted with a scalar parameter γ 3 .
T RAINING
End-to-end training of all three modules jointly is challenging, as the same pipeline controls both the keypoint-based state representation and the dynamic module (CoDy), involving two adversarial objectives:
optimizing reconstruction pushes the keypoint encoding to be as representative as possible, but learning the dynamics favors a simple representation.
Faced with these two contradictory tasks, the model is numerically unstable and rapidly converges to regression to the mean.
As described above, we separately train the encoder+decoder pair without dynamic information on reconstruction only, c.f. Equation (4).
Then we freeze the parameters of the keypoint detector and train CoDy to forecast the keypoints from D minimizing the loss in Equation (6).
E XPERIMENTS
We compare the proposed model to three strong baselines for physics-inspired video prediction.
• PhyDNet
(Le Guen & Thome, 2020) is a non-counterfactual video prediction model that forecasts future frames using a decomposition between (a) a feature vector that temporally evolves via an LSTM and (b) a dynamic state that follows a PDE learned through speciﬁcally designed cells.
[Figure] [Figure]
[Table]
( = N ), 2 N and 4 N . • V-CDN (Li et al., 2020) is a counterfactual model based on keypoints, close to our work. It identiﬁes confounders from the beginning of a sequence and learns a keypoint predictor through auto-encoding using the Transporter equation (see discussion in Sect. 4.1).
As it is, it cannot be used for video prediction and is incomparable with our work, see details in appendix C.
We therefore replace the Transporter by our own de-rendering/rendering modules, from which we remove the additional coefﬁcients.
We refer to this model as UV-CDN (for Unsupervised V-CDN).
• PredRNN (Wang et al., 2017) is a ConvLSTM-based video prediction model that leverages spatial and temporal memories a through a spatiotemporal LSTM cell.
All models have been implemented in PyTorch, architectures are described in appendix D. For the baselines PhyDNet, UV-CDN and PredRNN, we used the ofﬁcial source code provided by the authors.
We evaluate on each scenario of Filtered-CoPhy on the counterfactual video prediction task.
For the two counterfactual models (Ours and UV-CDN), we evaluate on the tasks as intended: we provide the observed sequence AB and the CF initial condition C , and forecast the sequence D .
The non-CF baselines are required to predict the entire video from a single frame, in order to prevent them from leveraging shortcuts in a part of the video and bypass the need for physical reasoning.
We measure performance with time-averaged peak signal-to noise ratio ( PSNR ) that directly measures reconstruction quality.
However, this metric is mainly dominated by error on the static background, which is not our main interest.
We also introduce Localized PSNR ( L-PSNR ), which measures area error on the important regions near moving objects, computed on masked images.
We compute the masks using classical background subtraction techniques.
Comparison to the SOTA — We compare our model against UV-CDN, PhyDNet and PredRNN in Table 1, consistently and signiﬁcantly outperforming the baselines.
The gap with UV-CDN is particularly interesting, as it conﬁrms the choice of additional coefﬁcients to model the dynamics of moving objects.
PredRNN shows competitive performances, especially on collisionCF .
However, our localized PSNR tends to indicates that the baseline does not reconstruct accurately the foreground, favoring the reconstruction of the background to the detriment of the dynamics of the scene.
Fig.
6 visualizes the prediction on a single example, more can be found in appendix G. We also compare to trivial copying baselines in Table 2, namely Copy B, which assumes no intervention and outputs the B sequence, and Copy C, which assumes a stable tower.
We evaluate these models in keypoints space measuring MSE on keypoints and coefﬁcients averaged over time, as copying baselines are unbeatable in the regions of static background, making the PSNR metrics unusable.
We provide additional empirical results by comparing the models using Multi-object Tracking metrics and studies on the impact of the do-operations on PSNR in appendix E. We also compute an upper bound of our model using CoPhyNet baseline as described in Baradel et al.
Performance on real-world data — is reported in appendix F, showing experiments on 516 videos of real wooden blocks introduced in (Lerer et al., 2016).
Impact of appearance coefﬁcients — are reported in Table 3, comparing to the baseline using a keypoint-only representation.
The coefﬁcients have a signiﬁcant impact: even increasing the number of keypoints to compensate for the loss of information cannot overcome the advantage of disentangling positions and shapes, as done in our model.
We provide a deeper analysis of the
G r ound T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 8kp t s )
P hy DN e t
UV - C DN ( 8kp t s )
P r e d R NN
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
has a mild negative effect on the reconstruction de-rendering/rendering modules in appendix B, which includes visualizations of the navigation of the latent shape space in B.2.
Learning ﬁlters — does not have a positive impact on reconstruction performance compared to the choice of the handcrafted bank, as can be seen in table 5.
We conjecture that the additional degrees of freedom are redundant with parts of the ﬁlter kernels in the reﬁnement module R : this corresponds to jointly learning a multi-channel representation { G ik } i =
C k =1 ..K for shapes as well as the mapping which geometrically distorts them into the target object shapes.
Fixing the latent representation does not constrain the system, as the mapping R can adjust to it — see Fig. 5.
Impact of the high-dimensional dynamic space — We evaluate the impact of modeling object dynamics in high-dimensional space through the CoDy encoder in Table 4, comparing projection to 256 dimensions to the baseline reasoning directly in keypoint + coefﬁcient space.
The experiment conﬁrms this choice of KKL-like encoder (Janny et al., 2021).
C ONCLUSION
We introduced a new benchmark for counterfactual reasoning in physical processes requiring to perform video prediction, i.e. predicting raw pixel observations over a long horizon.
The benchmark has been carefully designed and generated imposing constraints on identiﬁability and counterfactuality.
We also propose a new method for counterfactual reasoning, which is based on a hybrid latent representation combining 2D keypoints and additional latent vectors encoding appearance and shape.
We introduce an unsupervised learning algorithm for this representation, which does not require any supervision on confounders or other object properties and processes raw video.
Counterfactual prediction of video frames remains a challenging task, and Filtered CoPhy still exhibits failures in maintaining rigid structures of objects over long prediction time-scales.
We hope that our benchmark will inspire further breakthroughs in this domain.
A F URTHER DETAILS ON DATASET GENERATION
Confounders in our setup are masses, which we discretize in { 1 , 10 } .
For BallsCF and
CollisionCF , we can also consider the continuous initial velocities of each object as confounders variables, since they have to be identiﬁed in AB to forecast CD .
We simulate all trajectories associated with the various possible combinations of masses from the same initial condition.
Do-interventions , however, depend on the task.
For BlocktowerCF and BallsCF , dointerventions consist in (a) removing the top cube or a ball, or (b) shifting a cube/ball on the horizontal plane.
In this case, for BlocktowerCF , we make sure that the cube does not move too far from the tower, in order to maintain contact.
For CollisionCF , the do-interventions are restricted to shifting operations, since there are only two objects (a ball and a cylinder).
It can consist of either a switch of the cylinder’s orientation between vertical or horizontal, or a shift of the position of the moving object relative to the resting one in one of the three canonical directions x, y and z .
A. E NFORCING THE I DENTIFIABILTY CONSTRAINT
The identiﬁabilty and counterfactuality constraints described in section 3 are imposed numerically, i.e. we ﬁrst sample and simulate trajectories with random parameters and then reject those that violate these constraints
As stated in section 3, an identiﬁable experiment guarantees that there is no pair ( z, z (cid:48) ) that gives the same trajectory AB but a different counterfactual outcome CD .
Otherwise, there will be no way to choose between z and z (cid:48) only from looking at AB , thus no way to correctly forecast the counterfactual experiment.
By enforcing this constraint, we make sure that there exists at least a set { z, z (cid:48) , ... } of confounders that give at the same time similar observed outcomes AB and similar counterfactual outcomes CD .
In practice, there exists a ﬁnite set of possible variables
z i , corresponding to every combination of masses for each object in the scene (masses take their value in { 1 , 10 } ).
During generation, we submit each candidate experiment ( AB , CD , z ) to a test ensuring that the candidate is identiﬁable.
Let ψ ( X 0 , z ) be the function that gives the trajectory of a system with initial condition X 0 and confounders z .
We simulate all possible trajectories ψ
( A, z i ) and ψ ( C, z i ) for every possible z i .
If there exists z (cid:48) (cid:54) =
z such that the experiment is not identiﬁable, the candidate is rejected.
This constraint requires to simulate the trajectory of each experiment several times by modifying physical properties of the objects.
Equalities in Deﬁnition 1 are relaxed by thresholding distances between trajectories.
We reject a candidate experiment if there exists a z (cid:48) such that
[Equation]
The choice of the threshold value ε is critical, in particular for the identiﬁability constraint:
• If the threshold is too high , all AB trajectories will be considered as equal, which results in acceptance of unidentiﬁable experiments.
• If the threshold is too low , all trajectories CD are considered equal.
Again, this leads to mistakenly accepting unidentiﬁable experiments.
There exists an optimal value for ε , which allows to correctly reject unidentiﬁable experiences.
To measure this optimal threshold, we generated a small instance of the BlocktowerCF dataset without constraining the experiments, i.e. trajectories can be unidentiﬁable and non-counterfactual.
We then plot the percentage of rejected experiments in this unﬁltered dataset against the threshold
[Figure]
P e r ce n t a g e
o f i d e n ti ﬁ a b l ee xp e r i m e n t
[Figure]
Thresholdgridsearchforidentiﬁabilityconstraint
[Figure]
Thresholdgridsearchforcounterfactualityconstraint
[Figure]
[Figure] [Figure]
FPS
[Figure] [Figure]
value (Fig. 7, left).
We chose the threshold ε = 100 which optimizes discrimination and rejects the highest number of “unidentiﬁable” trajectories.
To demonstrate importance of this we train a recurrent graph network on BlocktowerCF to predict the cube masses from ground-truth state trajectories AB , including pose and velocities, see
Fig. 8.
It predicts each cube’s mass by solving a binary classiﬁcation task.
We train this model on both BlocktowerCF and an alternative version of the scenario generated without the identiﬁability constraint.
The results are shown in Table 6a.
We are not aiming 100% accuracy, and this problem remains difﬁcult in a sense that the identiﬁability constraint ensures the identiﬁability of a set of confounder variables, while our sanity check tries to predict a unique z .
However, the addition of the identiﬁability constraint to the benchmark signiﬁcantly improves the model’s accuracy, which indicates that the property acts positively on the feasability of FilteredCoPhy .
The corrected accuracy metric focuses solely on the critical cubes, i.e. those cubes whose masses directly deﬁne the trajectory CD .
A. E NFORCING THE C OUNTERFACTUALITY CONSTRAINT
Let ( AB , CD , z ) be a candidate experiment, and z k be a combination of masses identical to z except for the k th value.
The counterfactuality constraint consists in checking that there exists at least one k such that ψ ( C, z ) (cid:54) =
ψ ( C, z k ) .
To do so, we simulate ψ ( C, z k ) for all k and measure the difference with the candidate trajectory ψ ( C, z ) .
Formally, we verify the existence of k such that:
[Equation]
A. A NALYZING TEMPORAL RESOLUTION
We analyzed the choice of temporal frequency for the benchmark with another sanity check.
We simulate a non-counterfactual dataset from BlocktowerCF where all cubes have equal masses.
A
[Table]
[Table]
GN <latexit sha1_base64="8LZdPDoHp8i1UUsxyTj2GjFetyM
[Table]
<latexit
[Table]
+TdcroV/60poNKzLWaKSk8U4f9l7k2Sk9J+1zyRFbpee7phKPa1bwS/qiYtRubi9XUeeHJpXney520qvZOvaFWszR1JI2eLhaH/UE2zOJIL4NRBwboxpbpf0KOKQwkGlQgaHjGJQQcP/sYIUPN3AFa5iwjFeuEBXrsbVhFrBDMHvM859V+x2peh0wX3ZL/UvJr2ZniMXsM6yzj8Lc01puYHNi/ZbcxM+ztlL9Fl1Ux63HE7L98F8r/9YVePGZ4FntQ3FMdmdCd7FKaeCph5+kvXXlOqJkLeMp1y1hG58U5p9HjYu/hbEWsf4vKwIa17LQNvodd8gWP/rzOy2BnPBytDcfb48H68+6qV/AQj/CE7/Mp1vECW5hw9hk+4xxfk7fJu+R98uGnNFnqPA/
[Table]
/latexit> recurrent graph network takes as input cube trajectories (poses and velocities) over a time interval of one second and predicts the rest of the trajectory over the following second.
We vary the sampling frequency; for example, at 5 FPS, the model receives 5 measurements, and predicts the next 5 time steps, which correspond to a one second rollout in the future.
Finally, we compare the error in 3D positions between the predictions and the GT on the last prediction.
Results are shown in Table 6b.
This check shows clearly that 25 FPS corresponds to the best trade-off between an accurate representation of the collision and the amount of training data.
Fig.
9 shows a practical example of the effect of time resolution on dynamical information in a trajectory.
A. S IMULATION DETAILS
We used Pybullet as physics engine to simulate Filtered-CoPhy .
Each experiment is designed to respects the balance between good coverage of confounder combinations and counterfactuality and identiﬁability constraints described above.
We generate the trajectories iteratively:
We sample a combination of masses and other physical characteristics of the given experiment, such as stability of the tower, or object motion in CollisionCF , or if the do-operation consists in removing an object.
This allows us to maintain a balance of confounder conﬁgurations.
Then we search for an initial conﬁguration A .
For BlocktowerCF , we make sure that this conﬁguration is unstable to ensure identiﬁability.
Then we simulate the trajectory B .
[Figure] [Figure]
We look for a valid do-operation such that identiﬁability and counterfactuality constraints are satisﬁed.
If no valid do-operation is found after a ﬁxed number of trials, we reject this experiment.
If a valid pair ( AB , CD ) is found, we add the sample to the dataset.
The trajectories were simulated with a sample time of 0 .
04 seconds.
The video resolution is 448 × 448 and represents 6 seconds for BlocktowerCF and BallsCF , and 3 seconds for CollisionCF .
We invite interested readers to look at our code for more details, such as dooperation sampling, or intrinsic camera parameters.
Fig.
10 shows the confounder distribution in the three tasks.
B P ERFORMANCE EVALUATION OF THE DE - RENDERING MODULE B. I MAGE RECONSTRUCTION
[Table]
We evaluate the reconstruction performance of the de-rendering module in the reconstruction task.
Note that there is trade-off between the reconstruction performance and the dynamic forecasting accuracy: a higher number of keypoints may lead to better reconstruction, but can hurt prediction performance, as the dynamic model is more difﬁcult to learn.
Reconstruction error – We ﬁrst investigate the impact of the number of keypoints in Tableby measuring the Peak Signal to Noise Ratio (PSNR) between the target image and its reconstruction.
We vary the number of keypoints among multiples of N , the maximum number of objects in the scene.
Increasing the number of keypoints increases reconstruction quality (PSNR) up to a certain point, but results in degradation in forecasting performance.
Furthermore, doubling the number of keypoints only slightly improves reconstruction accuracy.
This tends to indicate that our additional coefﬁcients are already sufﬁcient to model ﬁner-grained visual details.
Table 3 in the main paper measures the impact of the number of keypoints and the presence of the additional appearance coefﬁcients on the full pipeline including the dynamic model.
Table 8 illustrates the impact of the number of keypoints and the additional appearance coefﬁcient on the reconstruction performance alone.
As we can see, the addition of the coefﬁcient consistently improves PSNR for low numbers of keypoints (over 2 dB for N keypoints).
[Figure]
The improvement is less visible for larger numbers of keypoints, since 3D visual details could be encoded via keypoints position, hence coefﬁcients become less relevant.
Visualizations are shown in Fig.
[Figure]
[Table]
The coefﬁcients signiﬁcantly improve the reconstruction on low number of keypoints.
This is related to in the main which this
B. N AVIGATING THE LATENT COEFFICIENT MANIFOLD
We evaluate the inﬂuence of the additional appearance coefﬁcients on our de-rendering model by navigating its manifold.
To do so, we sample a random pair ( X source , X target ) from an experiment in BlocktowerCF and compute the corresponding source features and target keypoints and coefﬁcients.
Then, we vary each component of the target keypoints and coefﬁcients and observe the reconstructed image (ﬁg. 12).
We observed that the keypoints accurately control the position of the cube along both spatial axes.
The rendering module does infer some hints on 3D shape information from the vertical position of the cube, exploiting a shortcut in learning.
On the other hand, while not being supervised, the coefﬁcients naturally learn to encode different orientations in space and distance from the camera.
Interestingly, a form of disentanglement emerges.
For example, coefﬁcients n ◦ 1 and 2 control rotation around the z-axis, and coefﬁcient n ◦ 4 models rotation around the y-axis.
The last coefﬁcient represents both the size of the cube and its presence in the image.
K e ypo i n t X -0.
K e ypo i n t Y -0.
C o e f ﬁ c
i e n t0.
C o e f ﬁ c
i e n t0.
C o e f ﬁ c
i e n t0.
C o e f ﬁ c
i e n t0.
C o e f ﬁ c
i e n t0.
C C OMPARISON WITH THE T RANSPORTER BASELINE C. C OMPARISON WITH OUR DE - RENDERING MODEL
As described in section 4.1, the Transporter (Kulkarni et al., 2019) is a keypoint detection model somewhat close to our de-rendering module.
It leverages the transport equation to compute a reconstruction vector:
[Equation]
This equation allows to transmit information from the input by two means: the 2D position of the keypoints ( K target ) and the dense visual features of the target ( F target ).
In comparison, our derendering solely relies on the keypoints from the target image and does not require a dense vector to be computed on the target to reconstruct the target.
This makes the Transporter incomparable with our de-rendering module.
We nevertheless compare the performances of the two models in
Table 13, and provide visual examples in Fig.
Even though the two models are not comparable, as the Transporter uses additional information, our model still outperforms the Transporter for small numbers of keypoints.
Interestingly, for higher numbers of keypoints Transporter tends to discover keypoints far from the object.
We investigate this behavior in the following section, and show that this is actually a critical problem for learning causal reasoning on the discovered keypoints.
C. A NALYSIS OF T RANSPORTER ’ S BEHAVIOR
The original version of the V-CDN model (Li et al., 2020) is based on Transporter (Kulkarni et al., 2019).
We have already highlighted the fact that this model is not comparable with our task, as it requires not only the target keypoints K target but also a dense feature map F target , whose dynamics can hardly be learned due to its high dimensionality.
More precisely, the transport equation (Eq. 9) allows to pass information from the target by two means: the 2D position of the keypoints ( K target ) and the dense feature map of the target ( F target ).
The number of keypoints therefore becomes a highly sensible parameter, as the transporter can decide to preferably transfer information through the target features rather than through the keypoint locations.
When the number of keypoints is low, they act as a bottleneck, and the model has to carefully discover them to reconstruct the image.
[Table]
[Figure]
On the other hand, when we increase the number of keypoints, Transporter stops tracking objects in the scene and transfers visual information through the dense feature map, making the predicted keypoints unnecessary for image reconstruction, and therefore not representative of the dynamics.
To illustrate our hypothesis, we set up the following experiment.
Starting from a trained Transporter model, we ﬁxed the source image to be X
(the ﬁrst frame from the trajectory) during the evaluation step.
Then, we compute features and keypoints on the target frame X t regularly sampled in time.
We reconstruct the target image using the transport equation, but without updating the target keypoints .
Practically, this consists in computing ˆΨ target with Eq.
(9) substituting K target for K source .
Results are shown in Fig.
There is no dynamic forecasting involved in this ﬁgure, and the Transporter we used was trained in a regular way, we only change the transport equation on evaluation time.
Even though the keypoint positions have been ﬁxed, the Transporter manages to reconstruct a signiﬁcant part of the images, which indicates that a part of the dynamics has been encoded in the dense feature map.
In contrast, this issue does not arise from our de-rendering module, since our decoder solely relies on the target keypoints to reconstruct the image.
Note that this is absolutely not contradictory with the claim in Li et al.
(2020), since they do not evaluate V-CDN in pixel space.
A rational choice of the number of keypoints leads to satisfactory performance, allowing V-CDN to accurately forecast the trajectory in keypoints space, and retrieve the hidden confounders on their dataset.
C. T EMPORAL INCONSISTENCY ISSUES
Increasing the number of keypoints of the Transporter may lead to temporal inconsistency during the long-range reconstruction.
For example, a keypoint that tracks the edge of a cube in the ﬁrst
G r ound
T r u t h t= t= t= t= t= t=
T r a n s po r t e r
T r a n s po r t e r
T r a n s po r t e r frame may target a face of this same cube in the future, since dynamics does not intervene in the keypoint discovery process.
Our de-rendering directly addresses this through the usage of additional appearance coefﬁcients, which allows us to limit the number of keypoints to the number of objects in the scene, effectively alleviating the consistency issue.
Fig.
15 illustrates this phenomenon by plotting the discovered keypoint locations forward in time, as well as the 2D location of the center of mass of each object.
Note that the Transporter suffers from the temporal inconsistency issue with numbers of keypoints as low as 4 (green cube).
In contrast, our model manages to solve the problem and accurately tracks the centers of mass, even though they were never supervised.
D D ETAILS OF MODEL ARCHITECTURES D. D E - RENDERING MODULE
We call a “block” a 2D convolutional layer followed by a 2D batch norm layer and ReLU activation.
The exact architecture of each part of the encoder is described in Table 10a.
The decoder hyperparameters are described in Table 10b.
Dense feature map estimator F – We compute the feature vector from X source by applying a convolutional network F on the output of the common CNN of the encoder.
This produces the source feature vector F source of shape ( batch , 16 , 28 , 28) .
Keypoin detector K – The convolutional network K outputs a set of 2D heatmaps of shape ( batch , K, 28 , 28) , where K is the desired number of keypoints.
We apply a spatial softmax function on the two last dimensions, then we extract a pair of coordinates on each heatmap by looking for the location of the maximum, which gives us K target of shape ( batch , K, 2) .
T r a n s po r t e r ( 4kp t s ) t= t= t= t= t= t=
T r a n s po r t e r ( 8kp t s )
O u r s ( 4kp t s )
O u r s ( 8kp t s )
Coefﬁcient estimator C – We obtain the coefﬁcient by applying a third convolutional network
C to the output of the common encoder CNN, which again results in a set of 2D vectors of shape ( batch , K, 28 , 28) .
These vectors are ﬂattened channel-wise and provide a tensor of shape ( batch , K, 28 × 28) ) fed to an MLP (see Table 10a for the exact architecture) that estimates the coefﬁcients C target of shape ( batch , K, C + 1) .
Gaussian mapping G – The keypoint vector K target is mapped to a 2D vector through a Gaussian mapping process :
[Equation]
where G ( k ) ∈ R 28 × 28 is the Gaussian mapping of the keypoint k
=
[ k x k y ] .
We deform these
Gaussian mappings by applying convolutions with ﬁlters H i controlled by the coefﬁcients c ik .
The ﬁlters from H are 5 × 5 kernels that elongate the Gaussian in a speciﬁc direction.
Practically, we obtain the ﬁlter H i by drawing a line crossing the center of the kernel and with a slope angle of i πC where C is the number of coefﬁcients.
We then apply a 2D convolution :
[Equation]
Note that we also compute a supplementary coefﬁcient α C +
k used as a gate on the keypoints.
By setting this coefﬁcient to zero, the de-rendering module can de-activate a keypoint (which is redundant with deactivating the full set of coefﬁcients for this keypoint).
Reﬁner R – To reconstruct the target image, we channel-wise stack feature vectors from the source with the constructed ﬁlters and feed them to the decoder CNN R (Table 10b).
We trained the de-rendering module on pairs of images ( X source , X target ) randomly sampled from sequences D .
For a given sequence D , we take T − 25 ﬁrst frames of the trajectory as a source (where T is the number of frames in the video).
The last 25 frames are used as a target.
For evaluation, we take the 25 th frame as the source, and the 50 th frame as the target.
We use Adam optimizer with a learning rate of 10 − 3 , γ 1 = 10 4 and γ 2 = 10 − 1 to minimize equation 4.
[Table]
D. C O D Y
We describe the architectural choices made in CoDy.
Let
[Equation]
(12) be the state representation of an image X t , composed of the K keypoints 2D coordinates with their
C + 1 coefﬁcients.
The time derivative of each component of the state is computed via an implicit
Euler derivation scheme ˙k ( t ) =
k ( t ) − k
( t − 1) .
We use a subscript notation to distinguish the keypoints from AB and CD .
CF estimator – The latent representation of the confounders is discovered from s AB .
The graph neural network from this module implements the message passing function f
() and the aggregation function g () (see equation 5) by an MLP with 3 hidden layers of 64 neurons and ReLU activation unit.
The resulting nodes embeddings h AB ( t ) =
GN ( s AB ( t )) belong to R 128 .
We then apply a gated recurrent unit with 2 layers and a hidden vector of size 32 to each node in h AB ( t ) (sharing parameters between nodes).
The last hidden vector is used as the latent representation of the confounders u k .
State encoder-decoder – The state encoder is modeled as a GN where the message passing function and the aggregation function are MLPs with one hidden layer of 32 units.
The encoded state σ CD =
E ( s CD ) lies in R 256 .
We perform dynamical prediction in this σ space, and then project back the forecasting in the keypoint space using a decoder.
The decoder ∆( σ ( t )) ﬁrst applies a shared GRU with one layer and a hidden vector size of 256 to each keypoint σ k ( t ) , followed by a graph neural network with the same structure as the state encoder.
Dynamic system – Our dynamic system forecasts the future state ˆ σ ( t + 1) from the current estimation ˆ σ ( t ) and the confounders u
= [ u 1 ... u K ] .
It ﬁrst applies a graph neural network to the concatenated vector [ ˆ σ ( t ) u ] .
The message passing function f and the aggregation function g are
MLPs with 3 hidden layers of 64 neurons and ReLU activation function.
The resulting nodes embeddings GN ( ( σ )( t )) belong to R 64 and are fed to a GRU sharing weights among each nodes with
2 layers and a hidden vector of size 64.
This GRU updates the hidden vector v CD ( t ) =
[ v 1 ... v K ] , that is then used to compute a displacement vector with a linear transformation:
[Equation]
CoDy is trained using Adam to minimize equation 6 (learning rate 10 − 4 , γ 3 = 1 ).
We train each CoDy instance by providing it with ﬁxed keypoints states s AB ( t ) and the initial condition s CD ( t = 0) computed by our trained de-rendering module.
CoDy ﬁrst computes the latent confounder representation u k , and then projects the initial condition into the latent dynamic space σ ( t = 0) =
E ( s CD ( t = 0)) .
We apply the dynamic model multiple times in order to recursively forecast T time steps from CD .
We then apply the decoder ∆( ˆ σ ( t )) to compute the trajectory in the keypoint space.
E A DDITIONAL QUANTITATIVE EVALUATION E. M ULTI - OBJECT TRACKING METRICS
When the number of keypoints matches the number of objects in the scene, the keypoint detector naturally and in an unsupervised manner places keypoints near the center of mass of each object (see
Figure 15).
Leveraging this emerged property, we provide additional empirical demonstration of the accuracy of our model by computing classical Multi-Object Tracking (MOT) metrics.
In particular, we computed the Multi-Object Tracking Precision (MOTP) and the Multi-Object Tracking (MOTA) as described in Bernardin & Stiefelhagen.
• MOTA requires to compute the number of missed objects (i e. not tracked by a keypoints) and the number of false positives (i
e. keypoints that do not represent an actual object).
MOTA takes values in [ − 1 , 1] , where 1 represents perfect tracking:
[Equation]
where m t is the number of missed objects at time t , f t is the number of false positives at time t , s t is the number of swaps at time t , and g t is the number of objects at time t .
• MOTP is a measurement of the distance between the keypoints and the ground-truth centers of mass conditioned on the pairing process:
[Equation]
where c t is the number of accurately tracked objects at time t and d it is the distance between the keypoint and the center of mass of the i th association { keypoints+center of mass } .
Note that these metrics are related: low MOTP indicates that the tracked objects are tracked precisely, and low MOTA indicates that many objects are missed.
Thus, to be efﬁcient, a model needs to achieve both low MOTP and high MOTA.
We also reported the performances of CoPhyNet (Baradel et al., 2020) that predicts counterfactual outcomes in euclidian space using the ground-truth 3D space.
As it uses GT object positions during training, it is not comparable and should be considered as a soft upper bound of our method.
We present our results in Table 11.
This conﬁrms the superiority of our method over UV-CDN in keypoint space.
The upper bound CoPhyNet takes advantage of the non-ambiguous 3D representation modeled by the ground-truth state of the object of the scene.
Our method also outperforms CoPhyNet on the ballsCF task, probably due to two phenomena.
First, ballsCF is the only 2D task of FilteredCoPhy.
Thus, CoPhyNet does not have an advantage of using ground-truth 3D positions.
Second, the state-encoder in CoDy projects the 2D position of each sphere in a space where the dynamics is easier to learn, probably by breaking the non-linearity of collisions.
[Table]
[Figure]
E. I MPACT OF THE DO - OPERATIONS
We also measure the impact of the do-operation types on the video forecasting.
Fig.
16 (left) is obtained by computing PSNR for each example of the training set and reporting the result on a 2D graph, depending on the amplitude of the displacement that characterizes the do-operation.
We applied the same method to obtain Fig.16 (right) that focuses on the type of do-operation, that is moving, removing or rotating an object.
These ﬁgures are computed using the 2N keypoints models.
Our method generalizes well across different do-operations, including both the type of the operation, and the amplitude.
A key to this success it the careful design of the dataset (balanced with respect to the types of do-operations), and a reasonable representation (our set of keypoints and coefﬁcients) able to detect and model each do-operation from images.
F E XPERIMENTS ON REAL - WORLD DATA
Our contributions are focused on the discovery of causality in physics through counterfactual reasoning.
We designed our model in order to solve the new benchmark and provided empirical evidence that our method is well suited for modeling rigid-body physics and counterfactual reasoning.
The following section aims to demonstrate that our approach can also be extended to a real-world dataset.
We provide qualitative results obtained on a derivative of BlocktowerCF using real cubes tower (Lerer et al., 2016). [Figure] T r u t h
[Figure]
P r e d i c ti
[Figure]
G r ound T r u t h
[Figure]
P r e d i c ti on
G r ound T r u t h
[Figure]
P r e d i c ti on manages to
We refer to this dataset as Blocktower IRL .
It is composed of 516 videos of wooden blocks stacked in a stable or unstable manner.
The amount of cubes in a tower varies from 2 to 4.
We aim to predict the dynamics of the tower in pixel space.
This is highly related with our task BlocktowerCF (which inspired by the seminal work from (Lerer et al., 2016)) with three main differences: (1) the dataset shows real cube towers, (2) the problem is not counterfactual, i.e. every cube has the same mass and (3) the dataset contains only few videos.
To cope with the lack of data, we exploit our pre-trained models on BlocktowerCF and ﬁne-tune on Blocktower IRL.
The adaptation of the de-rendering module is straightforward: we choose the 4 keypoints-5 coefﬁcients conﬁguration and train the module for image reconstruction after loading the weights from previous training on our simulated task.
CoDy, on the other hand, requires careful tuning to preserve the learned regularities from BlocktowerCF and prevent over-ﬁtting.
Since
Blocktower IRL is not counterfactual, we de-activate the confounder estimator and set u k to vectors of ones.
We also freeze the weights of the last layers of the MLPs in the dynamic model.
To the best of our knowledge, we are the ﬁrst to use this dataset for video prediction.
(Lerer et al., 2016) and Wu et al.
(2017) leverage the video for stability prediction but actual trajectory forecasting was not the main objective.
To quantitatively evaluate our method, we predict 20 frames in the future from a single image sampled in the trajectory.
We measured an average PSNR of 26.27 dB, which is of the same order of magnitude compared to the results obtained in simulation.
Figure 17 provides visual example of the output.
G Q UALITATIVE EVALUATION : MORE VISUAL EXAMPLES
More qualitative results produced by our model on different tasks from our datasets are given below.
G r ound
T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 8kp t s )
P hy DN e t
UV - C DN ( 8kp t s )
P r e d R NN
G r ound
T r u t h t= t= t=
t= t
[Figure]
t= t= t=
F ilt e r e d C o P hy ( 8kp t s )
P h
y DN e t
UV - C DN ( 8kp t s )
P r e d R NN
G r ound
T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 8kp t s )
P hy DN e t
UV - C DN ( 8kp t s )
P r e d R NN
G r ound
T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 8kp t s )
P h
y DN e t
UV - C DN ( 8kp t s )
P r e d R NN
G r ound
T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 4kp t s )
P hy DN e t
UV - C DN ( 4kp t s )
P r e d R NN
G r ound
T r u t h t= t= t= t= t= t= t= t=
F ilt e r e d C o P hy ( 4kp t s )
P h
y DN e t
UV - C DN ( 4kp t s )
P r e d R NN