G ROUNDING P HYSICAL C ONCEPTS OF O BJECTS AND E VENTS T HROUGH D YNAMIC V ISUAL R EASONING
MIT BCS, CBMM, CSAIL
A BSTRACT
We study the problem of dynamic visual reasoning on raw videos.
This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical to obtain in real life.
In this paper, we present the Dynamic Concept Learner (DCL), a uniﬁed framework that grounds physical objects and events from dynamic scenes and language.
DCL ﬁrst adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector.
Building upon this object-centric representation, DCL learns to approximate the dynamic interaction among objects using graph networks.
DCL further incorporates a semantic parser to parse question into semantic programs and, ﬁnally, a program executor to run the program to answer the question, levering the learned dynamics model.
After training, DCL can detect and associate objects across the frames, ground visual properties and physical events, understand the causal relationship between events, make future and counterfactual predictions, and leverage these extracted presentations for answering queries.
DCL achieves state-of-the-art performance on CLEVRER, a challenging causal video reasoning dataset, even without using ground-truth attributes and collision labels from simulations for training.
We further test DCL on a newly proposed video-retrieval and event localization dataset derived from CLEVRER, showing its strong generalization capacity.
I NTRODUCTION
Visual reasoning in dynamic scenes involves both the understanding of compositional properties, relationships and events of objects, and the inference and prediction of their temporal and causal structures.
As depicted in Fig.
1, to answer the question “What will happen next?”
based on the observed video frames, one needs to detect the object trajectories, predict their dynamics, analyze the temporal structures, and ground visual objects and events to get the answer “The blue sphere and the yellow object collide” .
Recently, various end-to-end neural network-based approaches have been proposed for joint understanding of video and language (Lei et al., 2018; Fan et al., 2019).
While these methods have shown great success in learning to recognize visually complex concepts, such as human activities (Xu et al., 2017; Ye et al., 2017), they typically fail on benchmarks that require the understanding of compositional and causal structures in the videos and text (Yi et al., 2020).
Another line of research has been focusing on building modular neural networks that can represent the compositional structures in scenes and questions, such as object-centric scene structures and multi-hop reasoning (Andreas et al., 2016; Johnson et al., 2017b; Hudson & Manning, 2019).
However, these methods are designed for static images and do not handle the temporal and causal structure in dynamic scenes well, leading to inferior performance on video causal reasoning benchmark CLEVRER (Yi et al., 2020).
To model the temporal and causal structures in dynamic scenes, Yi et al.
(2020) proposed an oracle model to combine symbolic representation with video dynamics modeling and achieved state-ofthe-art performance on CLEVRER.
However, this model requires videos with dense annotations for visual attributes and physical events, which are impractical or extremely labor-intensive in real scenes.
[Figure]
Question: What will happen next?
Answer: The blue sphere and the yellow object collide.
We argue that such dense explicit video annotations are unnecessary for video reasoning, since they are naturally encoded in the question answer pairs associated with the videos.
For example, the question answer pair and the video in Fig. 1 can implicitly inform a model what the concepts “ sphere ”, “ blue ”, “ yellow ” and “ collide ” really mean.
However, a video may contain multiple fast-moving occluded objects and complex object interactions, and the questions and answers have diverse forms.
It remains an open and challenging problem to simultaneously represent objects over time, train an accurate dynamic model from raw videos, and align objects with visual properties and events for accurate temporal and causal reasoning, using vision and language as the only supervision.
Our main ideas are to factorize video perception and reasoning into several modules: object tracking, object and event concept grounding, and dynamics prediction.
We ﬁrst detect objects in the video, associating them into object tracks across the frames.
We can then ground various object and event concepts from language, train a dynamic model on top of object tracks for future and counterfactual predictions, analyze relationships between events, and answer queries based on these extracted representations.
All these modules can be trained jointly by watching videos and reading paired questions and answers.
To achieve this goal, we introduce Dynamic Concept Learner (DCL), a uniﬁed neural-symbolic framework for recognizing objects and events in videos and analyzing their temporal and causal structures, without explicit annotations on visual attributes and physical events such as collisions during training.
To facilitate model training, a multi-step training paradigm has been proposed.
We ﬁrst run an object detector on individual frames and associate objects across frames based on a motion-based correspondence.
Next, our model learns concepts about object properties, relationships, and events by reading paired questions and answers that describe or explain the events in the video.
Then, we leverage the acquired visual concepts in the previous steps to reﬁne the object association across frames.
Finally, we train a dynamics prediction network (Li et al., 2019b) based on the reﬁned object trajectories and optimize it jointly with other learning parts in this uniﬁed framework.
Such a training paradigm ensures that all neural modules share the same latent space for representing concepts and they can bootstrap the learning of each other.
We evaluate DCL’s performance on CLEVRER, a video reasoning benchmark that includes descriptive, explanatory, predictive, and counterfactual reasoning with a uniform language interface.
DCL achieves state-of-the-art performance on all question categories and requires no scene supervision such as object properties and collision events.
To further examine the grounding accuracy and transferability of the acquired concepts, we introduce two new benchmarks for video-text retrieval and spatial-temporal grounding and localization on the CLEVRER videos, namely CLEVRERRetrieval and CLEVRER-Grounding.
Without any further training, our model generalizes well to these benchmarks, surpassing the baseline by a noticeable margin.
R ELATED W ORK
Our work is related to reasoning and answering questions about visual content.
Early studies like (Wu et al., 2016; Zhu et al., 2016; Gan et al., 2017) typically adopted monolithic network architectures and mainly focused on visual understanding.
To perform deeper visual reasoning, neural module networks were extensively studied in recent works (Johnson et al., 2017a;
Hu et al., 2018; Hudson &
Manning, 2018; Amizadeh et al., 2020), where they represent symbolic operations with small neural networks and perform multi-hop reasoning.
Some previous research has also attempted to learn
[Figure]
Trajectories {" !" } !
#$% {" !$ } !
#$% {" !& } !
#$% {" !' } !
#$%
Question: Which of the following will not happen if the yellow cube is removed?
Choice:
[Figure]
collision between the sphere
[Figure]
the cylinder.
[Figure]
{" !$ ! } !
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Stationary
[Figure] SequenceFeature [Figure] visual concepts through visual question answering (Mao et al., 2019).
However, it mainly focused on learning static concepts in images, while our DCL aims at learning dynamic concepts like moving and collision in videos and at making use of these concepts for temporal and causal reasoning.
Later, visual reasoning was extended to more complex dynamic videos (Lei et al., 2018;
Fan et al., 2019; Li et al., 2020; 2019a; Huang et al., 2020).
Recently, Yi et al.
(2020) proposed CLEVRER, a new video reasoning benchmark for evaluating computational models’ comprehension of the causal structure behind physical object interaction.
They also developed an oracle model, combining the neuro-symbolic visual question-answering model (Yi et al., 2018) with the dynamics prediction model (Li et al., 2019b), showing competitive performance.
However, this model requires explicit labels for object attributes, masks, and spatio-temporal localization of events during training.
Instead, our DCL has no reliance on any labels for objects and events and can learn these concepts through natural supervision (i.e., videos and question-answer pairs).
Our work is also related to temporal and relational reasoning in videos via neural networks (Wang &
Gupta, 2018; Materzynska et al., 2020; Ji et al., 2020).
These works typically rely on speciﬁc action annotations, while our DCL learns to ground object and event concepts and analyze their temporal relations through question answering.
Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning in physical scenes.
However, these datasets mainly target at pure video understanding and do not contain natural language question answering.
Much research has been studying dynamic modeling for physical scenes (Lerer et al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016;
Finn et al., 2016;
Shao et al., 2014; Fire & Zhu, 2015; Ye et al., 2018; Li et al., 2019b).
We adopt PropNet (Li et al., 2019b) for dynamics prediction and feed the predicted scenes to the video feature extractor and the neuro-symbolic executor for event prediction and question answering.
While many works (Zhou et al., 2019; 2018; Gan et al., 2015) have been studying on the problems of understanding human actions and activities (e.g., running, cooking, cleaning) in videos, our work’s primary goal is to design a uniﬁed framework for learning physical object and event concepts (e.g., collision, falling, stability).
These tasks are of great importance in practical applications such as industrial robot manipulation which requires AI systems with human-like physical common sense.
D YNAMIC C ONCEPT L EARNER
In this section, we introduce a new video reasoning model, Dynamic Concept Learner (DCL), which learns to recognize video attributes, events, and dynamics and to analyze their temporal and causal structures, all through watching videos and answering corresponding questions.
DCL contains ﬁve modules, 1) an object trajectory detector, 2) video feature extractor, 3) a dynamic predictor, 4) a language program parser, and 5) a neural symbolic executor.
As shown in Fig.
2, given an input video, the trajectory detector detects objects in each frame and associates them into trajectories; the feature extractor then represents them as latent feature vectors.
After that, DCL quantizes the objects’ static concepts (i.e., color, shape, and material) by matching the latent object features with the corresponding concept embeddings in the executor.
As these static concepts are motion-independent, they can be adopted as an additional criteria to reﬁne the object trajectories.
Based on the reﬁned trajectories, the dynamics predictor predicts the objects’ movement and interactions in future and counterfactual scenes.
The language parser parses the question and choices into functional programs, which are executed by the program executor on the latent representation space to get answers.
The object and event concept embeddings and the object-centric representation share the same latent space; answering questions associated with videos can directly optimize them through backpropagation.
The object trajectories and dynamics can be reﬁned by the object static attributes predicted by DCL.
Our framework enjoys the advantages of both transparency and efﬁciency, since it enables step-by-step investigations of the whole reasoning process and has no requirements for explicit annotations of visual attributes, events, and object masks.
3. M ODEL D ETAILS
Object Detection and Tracking.
Given a video, the object trajectory detector detects object proposals in each frame and connects them into object trajectories O
= { o n } N n =1 , where o n = { b nt } Tt =
and N is the number of objects in the video.
b t =
[ x nt , y nt , w nt , h nt ] is an object proposal at frame t and T is the frame number, where ( x nt , y nt ) denotes the normalized proposal coordinate center and w nt and h nt denote the normalized width and height, respectively.
The object detector ﬁrst uses a pre-trained region proposal network (Ren et al., 2015) to generate object proposals in all frames, which are further linked across connective frames to get all objects’ trajectories.
Let { b it } Ni =1 and { b jt +1 } Nj =1 to be two sets of proposals in two connective frames.
Inspired by Gkioxari & Malik (2015), we deﬁne a connection score s l between b it and b jt +1 to be
[Equation]
where s c ( b it ) is the conﬁdence score of the proposal
b it , IoU is the intersection over union and λis a scalar.
Gkioxari & Malik (2015) adopts a greedy algorithm to connect the proposals without global optimization.
Instead, we assign boxes { b jt +1 } Nj =1 at the t + 1 frame to { b it } Ni =1 by a linear sum assignment.
Video Feature Extraction.
Given an input video and its detected object trajectories, we extract three kinds of latent features for grounding object and event concepts.
It includes 1) the average visual feature f v ∈ R N × D 1 for static attribute prediction, 2) temporal sequence feature f s ∈ R N × 4 T for dynamic attribute and unary event prediction, and 3) interactive feature f c ∈ R K × N × N × D 2 for collision event prediction, where D 1 and D 2 denote dimensions of the features and K is the number of sampled frames.
We give more details on how to extract these features in Appendix B.
Grounding Object and Event Concepts.
Video Reasoning requires a model to ground object and event concepts in videos.
DCL achieves this by matching object and event representation with object and event embeddings in the symbolic executor.
Speciﬁcally, DCL calculates the conﬁdence score that the n -th object is moving by (cid:2) cos( s moving , m da ( f sn ))
− δ (cid:3) /λ , where f sn denotes the temporal sequence feature for the n -th object, s moving denotes a vector embedding for concept moving , and m da denotes a linear transformation, mapping f
sn into the dynamic concept representation space.
δ and λ are the shifting and scaling scalars, and cos() calculates the cosine similarity between two vectors.
DCL grounds static attributes and the collision event similarly, matching average visual features and interactive features with their corresponding concept embeddings in the latent space.
We give more details on the concept and event quantization in Appendix E.
Trajectory Reﬁnement.
The connection score in Eq.
1 ensures the continuity of the detected object trajectories.
However, it does not consider the objects’ visual appearance; therefore, it may fail to track the objects and may connect inconsistent objects when different objects are close to each other and moving rapidly.
To detect better object trajectories and to ensure the consistency of visual appearance along the track, we add a new term to Eq.
1 and re-deﬁne the connection score to be
[Equation]
where f appear ( { b im } tm =0 , b jt +1 ) measures the attribute similarity between the newly added proposal
b jt +1 and all proposals ( { b im } tm =0 in previous frames.
We deﬁne f appear as f appear ( { b im } tm =0 , b jt +1 )
= 3 × t (cid:88) attr t (cid:88) m =
f attr ( b im , b jt +1 ) , (3) where attr ∈ { color , material , shape } .
f attr ( b m , b t +1 ) equals to 1 when b im and b jt +1 have the same attribute, and
0 otherwise.
In Eq. 2, f appear ensures that the detected trajectories have consistent visual appearance and helps to distinguish the correct object when different objects are close to each other in the same frame.
These additional static attributes, including color, material, and shape, are extracted without explicit annotation during training.
Speciﬁcally, we quantize the attributes by choosing the concept whose concept embedding has the best cosine similarity with the object feature.
We iteratively connect proposals at the t + 1 frame to proposals at the t frame and get a set of object trajectories O
= { o n } Nn =1 , where o n = { b nt } Tt =1 .
Dynamic Prediction.
Given an input video and the reﬁned trajectories of objects, we predict the locations and RGB patches of the objects in future or counterfactual scenes with a Propagation
Network (Li et al., 2019b).
We then generate the predicted scenes by pasting RGB patches into the predicted locations.
The generated scenes are fed to the feature extractor to extract the corresponding features.
Such a design enables the question answer pairs associated with the predicted scenes to optimize the concept embeddings and requires no explicit labels for collision prediction, leading to better optimization.
This is different from Yi et al.
(2020), which requires dense collision event labels to train a collision classiﬁer.
To predict the locations and RGB patches, the dynamic predictor maintains a directed graph (cid:104) V, D (cid:105)
= (cid:68) { v n } Nn =1 , { d n 1 ,n 2 } N,Nn 1 =1 ,n 2 =1 (cid:69) .
The n -th vertex v n is represented by a concatenation of tuple (cid:104)
b nt , p nt (cid:105) over a small time window w , where b nt =
[ x nt , y nt , w nt , h nt ] is the n -th object’s normalized coordinates and p nt is a cropped RGB patch centering at ( x nt , y nt ) .
The edge d n 1 ,n 2 denotes the relation between the n 1 -th and n 2 -th objects and is represented by the concatenation of the normalized coordinate difference b n 1 t − b n 2 t .
The dynamic predictor performs multi-step message passing to simulate instantaneous propagation effects.
During inference, the dynamics predictor predicts the locations and patches at frame k + 1 using the features of the last w observed frames in the original video.
We get the predictions at frame k +by autoregressively feeding the predicted results at frame k
+ 1 as the input to the predictor.
To get the counterfactual scenes where the n -th object is removed, we remove the n -th vertex and its associated edges from the input to predict counterfactual dynamics.
Iteratively, we get the predicted normalized coordinates { ˆ b nk (cid:48) } N,K (cid:48) n =1 ,k
cid:48)
=1 and RGB patches { ˆ p nk (cid:48) } N,Kn =1 ,k (cid:48)
=1 at all predicted K (cid:48) frames.
We give more details on the dynamic predictor at Appendix C.
Language Program Parsing.
The language program parser aims to translate the questions and choices into executable symbolic programs.
Each executable program consists of a series of operations like selecting objects with certain properties, ﬁltering events happening at a speciﬁc moment, ﬁnding the causes of an event, and eventually enabling transparent and step-by-step visual reasoning.
Moreover, these operations are compositional and can be combined to represent questions with various compositionality and complexity.
We adopt a seq2seq model (Bahdanau et al., 2015) with an attention mechanism to translate word sequences into a set of symbolic programs and treat questions and choices, separately.
We give detailed implementation of the program parser in Appendix D.
Symbolic Execution.
Given a parsed program, the symbolic executor explicitly runs it on the latent features extracted from the observed and predicted scenes to answer the question.
The executor consists of a series of functional modules to realize the operators in symbolic programs.
The last operator’s output is the answer to the question.
Similar to Mao et al.
(2019), we represent all object states, events, and results of all operators in a probabilistic manner during training.
This makes the whole execution process differential w.r.t.
the latent representations from the observed and predicted scenes.
It enables the optimization of the feature extractor and concept embeddings in the symbolic executor.
We provide the implementation of all the operators in Appendix E.
3. T RAINING AND INFERENCE
Training.
We follow a multi-step training paradigm to optimize the model: 1) We ﬁrst extract object trajectories with the scoring function in Eq. 1 and optimize the video feature extractor and
[Table]
concept embeddings in the symbolic executor with only descriptive and explanatory questions; 2)
We quantize the static attributes for all objects with the feature extractor and the concept embeddings learned in Step 1) and reﬁne object trajectories with the scoring function Eq. 2; 3) Based on the reﬁned trajectories, we train the dynamic predictor and predict dynamics for future and counterfactual scenes; 4) We train the full DCL with all the question answer pairs and get the ﬁnal model.
The program executor is fully differentiable w.r.t.
the feature extractor and concept embeddings.
We use cross-entropy loss to supervise open-ended questions and use mean square error loss to supervise counting questions.
We provide speciﬁc loss functions for each module in Appendix H.
Inference.
During inference, given an input video and a question, we ﬁrst detect the object trajectories and predict their motions and interactions in future and counterfactual scenes.
We then extract object and event features for both the observed and predicted scenes with the feature extractor.
We parse the questions and choices into executable symbolic programs.
We ﬁnally execute the programs on the latent feature space and get the answer to the question.
E XPERIMENTS
To show the proposed DCL’s advantages, we conduct extensive experiments on the video reasoning benchmark CLEVRER.
Existing other video datasets either ask questions about the complex visual context (Tapaswi et al., 2016;
Lei et al., 2018) or study dynamics and reasoning without question answering (Girdhar & Ramanan, 2020; Baradel et al., 2020).
Thus, they are unsuitable for evaluating video causal reasoning and learning object and event concepts through question answering.
We ﬁrst show its strong performance on video causal reasoning.
Then, we show DCL’s ability on concept learning, predicting object visual attributes and events happening in videos.
We show
DCL’s generalization capacity to new applications, including CLEVRER-Grounding and CLEVRERRetrieval.
We ﬁnally extend DCL to a real block tower video dataset (Lerer et al., 2016).
4. I MPLEMENTATION D ETAILS
Following the experimental setting in Yi et al.
(2020), we train the language program parser with 1000 programs for all question types.
We train all our models without attribute and event labels.
Our models for video question answering are trained on the training set, tuned on the validation set, and evaluated in the test set.
To show DCL’s generalization capacity, we build CLEVRER-Grounding and CLEVRER-Retrieval datasets from the original CLEVRER videos and their associated video annotations.
We provide more implementation details in Appendix A.
4. C OMPARISONS ON T EMPORAL AND C AUSAL R EASONING
We compare our DCL with previous methods on CLEVRER, including Memory (Fan et al., 2019),
IEP (Johnson et al., 2017b), TbD-net (Mascharka et al., 2018), TVQA+ (Lei et al., 2018), NS-
[Table]
DR (Yi et al., 2020), MAC (V) (Hudson & Manning, 2018) and its attribute-aware variant, MAC (V+) .
We refer interested readers to CLEVRER (Yi et al., 2020) for more details.
Additionally, we also include a recent state-of-the-art VQA model HCRN (Le et al., 2020) for performance comparison, which adopts a conditional relation network for representation and reasoning over videos.
To provide more extensive analysis, we introduce DCL-Oracle by adding object attribute and collision supervisions into DCL’s training.
We summarize their requirement for visual labels and language programs in the second and third columns of Table 1.
According to the results in Table 1, we have the following observations.
Although HCRN achieves state-of-the-art performance on human-centric action datasets (Jang et al., 2017; Xu et al., 2017; 2016), it only performs slightly better than Memory and much worse than NS-DR on CLEVRER.
We believe the reason is that HCRN mainly focuses on motion modeling across frames while CLEVRER requires models to perform dynamic visual reasoning on videos and analyze its temporal and causal structures.
NS-DR performs best among all the baseline models, showing the power of combining symbolic representation with dynamics modeling.
Our model achieves the state-of-the-art question answering performance on all kinds of questions even without visual attributes and event labels from simulations during training, showing its effectiveness and label-efﬁciency.
Compared with NS-DR, our model achieves more signiﬁcant gains on predictive and counterfactual questions than that on the descriptive questions.
This shows DCL’s effectiveness in modeling for temporal and causal reasoning.
Unlike NS-DR, which directly predicts collision event labels with its dynamic model,
DCL quantizes concepts and executes symbolic programs in an end-to-end training manner, leading to better predictions for dynamic concepts.
DCL-Oracle shows the upper-bound performance of the proposed model to ground physical object and event concepts through question answering.
4. E VALUATION OF O BJECT AND E VENT C ONCEPT G ROUNDING IN V IDEOS
Previous methods like MAC (V) and TbD-net (V) did not learn explicit concepts during training, and NS-DR required intrinsic attribute and event labels as input.
Instead, DCL can directly quantize video concepts, including static visual attributes (i.e. color , shape , and material ), dynamic attributes (i.e. moving and stationary ) and events (i.e. in , out , and collision ).
Speciﬁcally, DCL quantizes the concepts by mapping the latent object features into the concept space by linear transformation and calculating their cosine similarities with the concept embeddings in the neural-symbolic executor.
We predict the static attributes of each object by averaging the visual object features at each sampled frame.
We regard an object to be moving if it moves at any frame, and otherwise stationary .
We consider there is a collision happening between a pair of objects if they collide at any frame of the video.
We get the ground-truth labels from the provided video annotation and report the accuracy in table 2 on the validation set.
We observe that DCL can learn to recognize different kinds of concepts without explicit concept labels during training.
This shows DCL’s effectiveness to learn object and event concepts through natural question answering.
We also ﬁnd that DCL recognizes static attributes and events better than dynamic attributes.
We further ﬁnd that DCL may misclassify objects to be “stationary” if they are missing for most frames and only move slowly at speciﬁc frames.
We suspect the reason is that we only learn the dynamic attributes through question answering and question answering pairs for such slow-moving objects rarely appear in the training set.
4. G ENERALIZATION
We further apply DCL to two new applications, including CLEVRER-Grounding , spatio-temporal localization of objects or events in a video, and CLEVRER-Retrieval , ﬁnding semantic-related videos for the query expressions and vice versa.
We ﬁrst build datasets for video grounding and video-text retrieval by synthesizing language expressions for videos in CLEVRER.
We generate the expressions by ﬁlling visual contents from the video annotations into a set of pre-deﬁned templates.
For example, given the text template, “The < static attribute > that is < dynamic attribute > < time identiﬁer > ”, we can ﬁll it and generate “ The
[Figure]
[Table] [Table]
metal cube that is moving when the video ends. ”.
Fig.
3 shows examples for the generated datasets, and we provide more statistics and examples in Appendix G. We transform the grounding and retrieval expressions into executable programs by training new language parsers on the expressions of the synthetic training set.
To provide more extensive comparisons, we adopt the representative video grounding/ retrieval model WSSTG
(Chen et al., 2019) as a baseline.
We provide more details of the baseline implementation in Appendix A.
CLEVRER-Grounding.
CLEVRER-Grounding contains object grounding and event grounding.
For video object grounding, we localize each described object’s whole trajectory and compute the mean intersection over union (IoU) with the ground-truth trajectory.
For event grounding, including collision , in and out , we temporally localize the frame that the event happens at and calculate the frame difference with the ground-truth frames.
For collision event, we also spatially localize the collided objects’ the union box and compute it’s IoU with the ground-truth.
We don’t perform spatial localization for in and out events since the target object usually appears to be too small to localize at the frame it enters or leaves the scene.
Table 3 lists the results.
From the table, we can ﬁnd that our proposed DCL transforms to the new CLEVRER-Grounding task well and achieves high accuracy for spatial localization and low frame differences for temporal localization.
On the contrary, the traditional video grounding method
WSSTG performs much worse, since it mainly aligns simple visual concepts between text and images and has difﬁculties in modeling temporal structures and understanding the complex logic.
CLEVRER-Retrieval.
For CLEVRER-Retrieval, an expression-video pair is considered as a positive pair if the video contains the objects and events described by the expression and otherwise negative.
Given a video, we deﬁne its matching similarity with the query expression to be the maximal similarity between the query expression and all the object or event proposals.
Additionally, we also introduce a recent state-of-the-art video-text retrieval model HGR (Chen et al., 2020) for performance comparison, which decomposes video-text matching into global-to-local levels and performs cross-modal matching with attention-based graph reasoning.
We densely compare every possible expression-video pair and use mean average precision (mAP) as the retrieval metric.
We report the retrieval mAP in Table 4.
Compared with CLEVRER-Grounding, CLEVRER-Retrieval is more challenging since it contains many more distracting objects, events and expressions.
WSSTG performs worse on the retrieval setting because it does not model temporal structures and understand its logic.
HGR achieves better performance than the previous baseline WSSTG since it performs hierarchical modeling for events, actions and entities.
However, it performs worse than DCL since it doesn’t explicitly model the temporal structures and the complex logic behind the video-text pairs in
CLEVRER-Retrieval.
On the other hand, DCL is much more robust since it can explicitly ground object and event concepts, analyze their relations and perform step-by-step visual reasoning.
[Table] [Table]
[Table]
4. E XTENSION TO REAL VIDEOS AND THE NEW CONCEPT
We further conduct experiments on a real block tower video dataset (Lerer et al., 2016) to learn the new physical concept “ falling ”.
The block tower dataset has 493 videos and each video contains a stable or falling block tower.
Since the original dataset aims to study physical intuition and doesn’t contain question-answer pairs, we manually synthesize question-answer pairs in a similar way to CLEVRER (Yi et al., 2020).
We show examples of the new dataset in Fig 4.
We train models on randomly-selected 393 videos and their associated question-answer pairs and evaluate their performance on the rest 100 videos.
Similar to the setting in CLEVRER
[Table]
we use the average visual feature from ResNet-34 for static attribute prediction and temporal sequence feature for the prediction of the new dynamic concept “ falling ”.
Additionally, we train a visual reasoning baseline MAC (V) (Hudson & Manning, 2018) for performance comparison.
Table 5 lists the results.
Our model achieves better question-answering performance on the block tower dataset especially on the counting questions like “ How many objects are falling? ”.
We believe the reason is that counting questions require a model to estimate the states of each object.
MAC (V) just simply adopts an MLP classiﬁer to predict each answer’s probability and doesn’t model the object states.
Differently, DCL answers the counting questions by accumulating the probabilities of each object and is more transparent and accurate.
We also show the accuracy of color and “ falling ” concept prediction on the validation set in Table 6.
Our DCL can naturally learn to ground the new dynamic concept “ falling ” in the real videos through question answering.
This shows DCL’s effectiveness and strong generalization capacity.
D ISCUSSION AND FUTURE WORK
We present a uniﬁed neural symbolic framework, named Dynamic Concept Learner (DCL), to study temporal and causal reasoning in videos.
DCL, learned by watching videos and reading questionanswers, is able to track objects across different frames, ground physical object and event concepts, understand the causal relationship, make future and counterfactual predictions and combine all these abilities to perform temporal and causal reasoning.
DCL achieves state-of-the-art performance on the video reasoning benchmark CLEVRER.
Based on the learned object and event concepts, DCL generalizes well to spatial-temporal object and event grounding and video-text retrieval.
We also extend DCL to real videos to learn new physical concepts.
Our DCL suggests several future research directions.
First, it still requires further exploration for dynamic models with stronger long-term dynamic prediction capability to handle some counterfactual questions.
Second, it will be interesting to extend our DCL to more general videos to build a stronger model for learning both physical concepts and human-centric action concepts.
Acknowledgement This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216), the Samsung
Global Research Outreach (GRO) Program, Autodesk, and IBM Research.
A I MPLEMENTATION D ETAILS
DCL Implementation Details.
Since it’s extremely computation-intensive to predict the object states and events at every frame, we evenly sample 32 frames for each video.
All models are trained using Adam (Kingma & Ba, 2014) for 20 epochs and the learning rate is set to 10 − 4 .
We adopt a two-stage training strategy for training the dynamic predictor.
For the dynamic predictor, we set the time window size w , the propogation step L and dimension of hidden states to be 3, 2 and 512, respectively.
Following the sample rate at the observed frames, we sample a frame for prediction every 4 frames.
We ﬁrst train the dynamic model with only location prediction and then train it with both location and RGB patch prediction.
Experimentally, we ﬁnd this training strategy provides a more stable prediction.
We train the language parser with the same training strategy as Yi et al.
(2018) for fair comparison.
Baseline Implementation.
We implement the baselines HCRN (Le et al., 2020), HGR Chen et al.
(2020) and WSSTG (Chen et al., 2019) carefully based on the public source code.
WSSTG ﬁrst generate a set of object or event candidates and match them with the query sentence.
We choose the proposal candidate with the best similarity as the grounding result.
For object grounding, we use the same tube trajectory candidates as we use for implementing DCL.
For grounding event concepts in and out , we treat each object at each sampled frame as a potential candidate for selection.
For grounding event concept collision , we treat the union regions of any object pairs as candidates.
For
CLEVRER-Retrieval, we treat the proposal candidate with the best similarity as the similarity score between the video and the query sentence.
We train WSSTG with a synthetic training set generated from the videos of CLEVRER-VQA training set.
A fully-supervised triplet loss is adopted to optimize the model.
B F EATURE E XTRACTION
We evenly sample K frames for each video and use a ResNet-34 (He et al., 2016) to extract visual features.
For the n -th object in the video, we deﬁne its average visual feature to be f vn = 1 K (cid:80)
Kk
=1 f nk , where f nk is the concatenation of the regional feature and the global context feature at the k -th frame.
We deﬁne its temporal sequence feature f sn to be the contenation of [ x nt , y nt , w nt , h nt ] at all
T frames, where ( x nt , y nt ) denotes the normalized object coordinate centre and w nt and h nt denote the normalized width and height, respectively.
For the collision feature between the n 1 -th object and n 2 -th objet at the k -th frame, we deﬁne it to be f cn 1 ,n 2 ,k = f un 1 ,n 2 ,k
|| f locn 1 ,n 2 ,k , where f un 1 ,n 2 ,k is the ResNet feature of the union region of the n 1 -th and n 2 -th objects at the k -th frame and f locn 1 ,n 2 ,k is a spatial embedding for correlations between bounding box trajectories.
We deﬁne f locn 1 ,n 2 ,k = IoU ( s n 1 , s n 2 ) ||
( s n 1 − s n 2 ) ||
( s n 1 × s n 2 ) , which is the concatenation of the intersection over union (IoU), difference ( − ) and multiplication ( × ) of the normalized trajectory coordinates for the n 1 -th and n 2 -th objects centering at the k -th frame.
We padding f un 1 ,n 2 , k with a zero vector if either the n 1 -th or the n 2 -th objects doesn’t appear at the k -th frame.
C D YNAMIC P REDICTOR
To predict the locations and RGB patches, the dynamic predictor maintains a directed graph (cid:104) V, D (cid:105)
= (cid:68) { v n } Nn =1 , { d n 1 ,n 2 } N,Nn 1 =1 ,n 2 =1 (cid:69) .
The n -th vertex o n is represented by the concatenation of its normalized coordinates b nt =
[ x nt , y nt , w nt , h nt ] and RGB patches p nt .
The edge d n 1 ,n 2 is represented by the concatenation of the normalized coordinate difference b n 1 t − b n 2 t .
To capture the object dynamics, we concatenate the features over a small history window.
To predict the dynamics at the k + 1 frame, we ﬁrst encode the vertexes and edges
[Equation] [Equation]
where || indicates concatenation, w is the history window size, f encO and f encR are CNN-based encoders for objects and relations.
w is set to 3.
We then update the object inﬂuences { h l n,k } N n =1 and relation inﬂuences { e ln 1 ,n 2 ,k } N,Nn 1 =1 ,n 2 =1 through L propagation steps.
Speciﬁcally, we have
[Equation]
where l ∈ [1 , L ] , denoting the l -th step, f O and f R denote the object propagator and relation propagator, respectively.
We initialize h on,t = 0 .
We ﬁnally predict the states of objects and relations at the k +1 frame to be
[Equation]
where f predOand f predOare predictors for the normalized object coordinates and RGB patches at the next frame.
We optimize this dynamic predictor by mimizing the L 2 distance between the predicted ˆ b
nk +1 , ˆ p nk +
and the real future locations b nk +1 and extracted patches p
nk +1 .
During inference, the dynamics predictor predicts the locations and patches at k +1 frames by using the features of the last w observed frames in the original video.
We get the predictions at the k +2 frames by feeding the predicted results at the k +1 frame to the encoder in Eq. 4.
To get the counterfactual scenes where the n -th object is removed, we use the ﬁrst w frames of the original video as the start point and remove the n -th vertex and its associated edges of the input to predict counterfactual dynamics.
Iteratively, we get the predicted normalized coordinates { ˆ b n k
[Equation]
} N,
[Equation]
=
[Equation] [Equation]
=
[Equation]
and RGB patches { ˆ p nk (cid:48) } N,Kn =1 ,k (cid:48)
=1 at all predicted K (cid:48) frames.
D P ROGRAM P ARSER
Following Yi et al.
(2020), we use a seq2seq model (Bahdanau et al., 2015) with attention mechanism to word sequences into a set of symbolic programs and treat questions and choices, separately.
The model consists of a Bi-LSTM (Graves et al., 2005) to encode the word sequences into hidden states and a decoder to attentively aggregate the important words to decode the target program.
Speciﬁcally, to encode the word embeddings { w i } Ii =1 into the hidden states, we have
[Equation]
where I is the number of words and f encw is an encoder for word embeddings.
To decode the encoded vectors { e i } Ii =1 into symbolic programs { p j } Jj =1 , we have
[Equation]
where e i =
−→ e i ||←−
e i and J is the number of programs.
The dimension of the word embedding and all the hidden states is set to 300 and 256, respectively.
E CLEVRER O PERATIONS AND P ROGRAM E XECUTION
We list all the available data types and operations for CLEVRER VQA (Yi et al., 2020) in Tableand Table 7.
In this section, we ﬁrst introduce how we represent the objects, events and moments in the video.
Then, we describe how we quantize the static and dynamic concepts and perform temporal and causal reasoning.
Finally, we summarize the detailed implementation of all operations in Table 9.
Representation for Objects, Events and Time.
We consider a video with N objects and T frames and we sample K frames for collision prediction.
The objects in Table 8 can be represented by a vector objects of length N , where objects n ∈ [0 , 1] represents the probability of the n -h object being referred to.
Similarly, we use a vector events in of length N to representing the probability of objects coming into the visible scene.
We additionally store frame indexes t in for event in , where t inn indicates the moment when the n -th object ﬁrst appear in the visual scene.
We represent event out in a similar way as we represent event in .
For event collision , we represent it with a matrix events col ∈ R N × N × K , where events coln 1 ,n 2 ,k represents the n 1 -th and n 2 -th objects collide at the k -th frame.
Since CLEVRER requires temporal relations of events , we also maintain a time mask
M ∈ R T to annotate valid time steps, where M t = 1 indicates the t -th is valid at the current step and M t =
indicating invalid.
In CLEVRER, Unique also involves transformation from objects (object set) to object (a single object).
We achieve by selecting the object with the largest probability.
We perform in a similar way to transform events to event .
Object and Event Concept Quantization.
We ﬁrst introduce how DCL quantizes different concepts by showing an example how DCL quantizes the static object concept cube .
Let f vn denote the latent visual feature for the n -th object in the video, SA denotes the set of all static attributes.
The concept cube is represented by a semantic vector s Cube and an indication vector i cube .
i cube is of length | SA | and L-1 normalized, indicating concept Cube belongs to the static attribute Shape .
We compute the conﬁdence scores that an object is a Cube by
[Equation]
where δ and λ denotes the shifting and scaling scalars and are set to 0.15 and 0.2, respectively.
cos () calculates the cosine similarity between two vectors and m sa denotes a linear transformation, mapping object features into the concept representation space.
We get a vector of length N by applying this concept ﬁlter to all objects, denoted as ObjFilter ( cube ) .
We perform similar quantization to temporal dynamic concepts.
For event in and out , we simply replace f vn with temporal sequence features f sn ∈ R 4 T to get events inn .
For event collision , we replace f vn with f cn 1 ,n 2 ,k to predict the conﬁdence that the n 1 -th and the n 2 -th objects collide at the k -frame and get events outn 1 ,n 2 ,k .
For moment-speciﬁc dynamic concepts moving and stationary , we adopt frame-speciﬁc feature f
sn,t ∗ ∈ R 4 T for concept prediction.
We denote the ﬁlter result on all objects as ObjFilter ( moving, t ∗ ) .
Speciﬁcally, we generate the sequence feature f
sn,t ∗ at the t ∗ -th frame by only concatenating [ x nt , y nt , w nt , h nt ] from t ∗ − τ to t ∗ + τ frames and padding other dimensions with 0 .
Temporal and causal Reasoning.
One unique feature for CLEVRER is that it requires a model to reason over temporal and causal structures of the video to get the answer.
We handle
Filter before and Filter after by updating the valid time mask M .
For example, to ﬁlter events happening after a target event.
We ﬁrst get the frame t ∗ that the target event happens at and update valid time mask M by setting M t
= 1 if t > t ∗ else M t = 0 .
We then ignore the events happening at the invalid frames and update the temporal sequence features to be f s (cid:48) n = f sn ◦ M exp , where ◦ denotes the component-wise multiplication and M exp =
[ M ; M ; M ; M ] ∈ R 4 T .
For Filter order of events type , we ﬁrst ﬁlter all the valid events by ﬁnd events who event type > η .
η is simply set to 0 and type ∈ { in, out, collision } .
We then sort all the remain events based on t type to ﬁnd the target event.
For Filter ancestor of a collision event , we ﬁrst predict valid events by ﬁnding events type > η .
We then return all valid events that are in the causal graphs of the given collision event.
We summarize the implementation of all operations in Table 9.
F T RAJECTORY P ERFORMANCE E VALUATION .
In this section, we compare different kinds of methods for generating object trajectory proposals.
Greedy+IoU denotes the method used in (Gkioxari & Malik, 2015), which adopts a greedy
Viterbi algorithm to generate trajectories based on IoUs of image proposals in connective frames.
Greedy+IoU+Attr. denotes the method adopts the greedy algorithm to generate trajectory proposals based on the IoUs and predicted static attributes.
LSM+IoU denotes the method that we use linear sum assignment to connect the image proposals based on IoUs.
LSM+IoU+Attr.
denotes the method we use linear sum assignment to connect image proposals based on IoUs and predicted static attributes.
LSM+IoU+Attr.+KF denotes the method that we apply additional Kalman ﬁltering (Kalman, 1960; Bewley et al., 2016;
Wojke et al., 2017) to LSM+IoU+Attr.
We evaluate the performance of different methods by compute the IoU between the generated trajectory proposals and the ground-truth trajectories.
We consider it a “correct” trajectory proposal if the IoU between the proposal and the ground-truth is larger than a threshold.
Speciﬁcally, two metrics are used evaluation, precision = N correct N p and recall =
N correct N gt , where N correct , N p and N gt denotes the number of correct proposals, the number of proposals and the number of ground-truth objects, respectively.
Table 10 list the performance of different thresholds.
We can see that Greedy+IoU achieve bad performance when the IoU threshold is high while our method based on linear sum assignment and static attributes are more robust.
Empirically, we ﬁnd that linear sum assignment and static attributes can help distinguish close object proposals and make the correct image proposal assignments.
Similar to normal object tracking algorithms (Bewley et al., 2016; Wojke et al., 2017)
, we also ﬁnd that adding additional Kalman ﬁlter can further slightly improve the trajectory quality.
G S TATISTICS FOR CLEVRER-G ROUNDING AND CLEVRER-R ETRIEVAL
We simply use the videos from original CLEVRER training set as the training videos for
CLEVRER-Grounding and CLEVRER-Retrieval and evaluate their performance on the validation set.
CLEVERER-Grounding contains 10.2 expressions for each video on average.
CLEVERER-Retrieval contains 7.4 expressions for each video in the training set.
We for evaluating the video retrieval task on the validation set.
We evaluate the performance of CLEVRER-Grounding task on all 5, videos from the original CLEVRER validation set.
For CLEVERER-Retrieval, We additionally generate 1,129 unique expressions from the validation set as query and treat the ﬁrst 1,000 videos from CLEVRER validation set as the gallery.
We provide more examples for CLEVRER-Grounding and CLEVRER-Retrieval datasets in Fig.
5, Fig. 6 and Fig.
It can be seen from the examples that the newly proposed CLEVRER-Grounding and CLEVRER-Retrieval datasets contain delicate and compositional expressions for objects and physical events.
It can evaluate models’ ability to perform compositional temporal and causal reasoning.
H T RAINING O BJECTIVES
In this section, we provide the explicit training objectives for each module.
We optimize the feature extractor and the concept embeddings in the executors by question answering.
We treat each option of a multiple-choice question as an independent boolean question during training and we use different loss functions for different question types.
Speciﬁcally, we use cross-entropy loss to supervise open-ended questions and use mean square error loss to supervise counting questions.
Formally, for open-ended questions, we have
[Equation]
where C is the size of the pre-deﬁned answer set, p c is the probability for the c -th answer and y a is the ground-truth answer label.
For counting questions, we have
[Equation]
where z is the predicted number and y a is the ground-truth number label.
We train the program parser with program labels using cross-entropy loss,
[Equation]
where J is the size of the pre-deﬁned program set, p j is the probability for the j -th program and y p is the ground-truth program label.
We optimize the dynamic predictor with mean square error loss.
Mathematically, we have
[Equation]
where b n is the object coordinates for the n -th object, p ni 1 ,i 2 is the pixel value of the n -th object’s cropped patch at ( i 1 , i 2 ) , and N p is the cropped size.
ˆ b n and ˆ p ni 1 ,i 2 are the dynamic predictor’s predictions for b n and p ni 1 ,i 2 .
Type
Operation
Signature
InputModules
Objects
[Table]
Returns all
[Table]
in the video
Events
[Table]
events Returns all
[Table]
in the video
UnseenEvents ()
[Table]
events Returns all future events happening in the video
Start () → event Returns the special “start” event end () → event Returns the special “end” event
ObjectFilterModules
Filter static concept (
[Table]
Select objects from the
[Table]
list with the input static concept
Filter dynamic concept ( objects , concept
[Table]
[Table]
objects Selects objects in the
[Table]
frame with the dynamic concept
EventFilterModules
Filter in ( events , objects ) → events Select incoming events of the input objects Filter out ( events , objects ) → events Select existing events of the input objects
Filter collision ( events , objects ) → events Select all collisions that involve an of the input objects
Get col partner ( event , object ) → object Return the collision partner of the input object
Filter before ( events , events ) → events Select all events before the target event
Filter after ( events , events ) → events Select all events after the target event
Filter order ( events , order ) → event Select the event at the speciﬁc time order
Filter ancestor ( event , events ) → events Select all ancestors of the input event in the causal graph
Get frame ( event ) → frame Return the frame of the input event in the video
OutputModules
Query Attribute ( object ) → concept Returns the query attribute of the input object
Count ( objects ) → int Returns the number of the input objects/ events ( events ) → int Exist ( objects ) → bool Returns “yes” if the input objects is not empty
Belong to ( event , events ) → bool Returns “yes” if the input event belongs to the input event sets
Negate ( bool ) → bool Returns the negation of the input boolean
Unique (
[Table]
→
[Table]
Return the only event /object in the input list ( objects ) → object
[Table]
A single object in the video.
[Table]
A set of objects in the video
[Table]
A single event
in the video
[Table]
Type
Operation/ Signature
Implementation
InputModules
Objects objects =()
[Table]
objects Events events type for type ∈ { in, out, col. } () → events
UnseenEvents events col (cid:48) and events out (cid:48) () → events Start
M t
= 1 if t < 5 else M t = 0 () → M end
M t
=
if t > ( T − 5) else M t = 0 ()
→ M
ObjectFilterModules
Filter static concept min( objs, ObjFilter( sa ) )
( objs: objects , sa: concept ) → objects Filter dynamic concept min( objects, ObjFilter( da ,t) )
( objs: objects , da: concept , t: frame ) → objects
EventFilterModules
Filter in min( objs, events in ) ( events in : events , objs: objects ) → events
Filter out min( objs, events out ) ( events out : events , objs: objects ) → events
Filter collision min( objs exp , events col. )
( events col : events , objs: objects ) → events
Get col partner max k ∈ [1 ,K ] ( events coln,k ) ( events col : events , obj n : object ) → objects
Filter before events inn = − 1 if t inn > t event( events inn : events , event1: event )
Filter after events inn = −
if t inn < t event( events inn : events , event1: event )
→ events
Filter order events inn > 0 if order inn = or ( events inn : events , or: order ) → event Filter ancestor { event1 n > 0 and events1 n ( event1: event , events1: events ) → events in the causal graph of event1 } Get frame t event( event1: event )
→ frame
OutputModules
Query Attribute
P op =
ObjFilter ( op ) · i opa (cid:80) op (cid:48)
ObjFilter ( op (cid:48) )
· i op (cid:48) a ( obj
[Table]
[Table] [Table]
[Figure]
video that contains a collision that happens after the yellow metal cylinder enters the scene.
[Figure]
A video that contains an object that collides with the brown metal cube. A video that contains an object that collides with the gray metal sphere.
A video that contains an object to collide with the brown metal cube. A video that contains an object to collide with the gray metal sphere.
A video that contains a collision that happens after the yellow metal cube enters the scene. A video that contains a collision that happens after the brown metal cube enters the scene.
A video that contains a collision that happens before the yellow metal cube enters the scene.