  We study the problem of dynamic visual reasoning on raw videos. This is a
challenging problem; currently, state-of-the-art models often require dense
supervision on physical object properties and events from simulation, which are
impractical to obtain in real life. In this paper, we present the Dynamic
Concept Learner (DCL), a unified framework that grounds physical objects and
events from video and language. DCL first adopts a trajectory extractor to
track each object over time and to represent it as a latent, object-centric
feature vector. Building upon this object-centric representation, DCL learns to
approximate the dynamic interaction among objects using graph networks. DCL
further incorporates a semantic parser to parse questions into semantic
programs and, finally, a program executor to run the program to answer the
question, levering the learned dynamics model. After training, DCL can detect
and associate objects across the frames, ground visual properties, and physical
events, understand the causal relationship between events, make future and
counterfactual predictions, and leverage these extracted presentations for
answering queries. DCL achieves state-of-the-art performance on CLEVRER, a
challenging causal video reasoning dataset, even without using ground-truth
attributes and collision labels from simulations for training. We further test
DCL on a newly proposed video-retrieval and event localization dataset derived
from CLEVRER, showing its strong generalization capacity.
