Original: CLEVRER: C OLLISION E VENTS FOR V IDEO R EPRESENTATION AND R EASONING
Revised: CLEVRER: C OLLISION E VENTS FOR V IDEO R EPRESENTATION AND R EASONING

Original: Anonymous authors
Revised: Yunzhu

Original: Paper under double-blind review
Revised: Pushmeet Kohli DeepMind BCS CBMM

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability to reason about temporal and causal events from videos lies at the core of human intelligence.
Revised: The ability to reason about temporal and causal events from videos lies at the core of human intelligence.

Original: Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure.
Revised: Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure.

Original: We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance.
Revised: We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance.

Original: To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks.
Revised: To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks.

Original: Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g., ‘what color’), explanatory (‘what’s responsible for’), predictive (‘what will happen next’), and counterfactual (‘what if’).
Revised: Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g., ‘what color’), explanatory (‘what’s responsible for’), predictive (‘what will happen next’), and counterfactual (‘what if’).

Original: We evaluate various state-of-the-art models for visual reasoning on our benchmark.
Revised: We evaluate various state-of-the-art models for visual reasoning on our benchmark.

Original: While these models thrive on the perceptionbased task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.
Revised: While these models thrive on the perceptionbased task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.

Original: We also study an oracle model that explicitly combines these components via symbolic representations.
Revised: We also study an oracle model that explicitly combines these components via symbolic representations.

Original: CLEVRER will be made publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to recognize objects and reason about their behaviors in physical events from videos lies at the core of human cognitive development (Spelke, 2000).
Revised: The ability to recognize objects and reason about their behaviors in physical events from videos lies at the core of human cognitive development (Spelke, 2000).

Original: Humans, even young infants, group segments into objects based on motion, and use concepts of object permanence, solidity, and continuity to explain what has happened, infer what is about to happen, and imagine what would happen in counterfactual situations.
Revised: Humans, even young infants, group segments into objects based on motion, and use concepts of object permanence, solidity, and continuity to explain what has happened, infer what is about to happen, and imagine what would happen in counterfactual situations.

Original: The problem of complex visual reasoning has been widely studied in artiﬁcial intelligence and computer vision, driven by the introduction of various datasets on both static images (Antol et al., 2015; Zhu et al., 2016; Hudson & Manning, 2019) and videos (Jang et al., 2017; Tapaswi et al., 2016; Zadeh et al., 2019).
Revised: The problem of complex visual reasoning has been widely studied in artiﬁcial intelligence and computer vision, driven by the introduction of various datasets on both static images (Antol et al., 2015; Zhu et al., 2016; Hudson & Manning, 2019) and videos (Jang et al., 2017; Tapaswi et al., 2016; Zadeh et al., 2019).

Original: However, despite the complexity and variety of the visual context covered by these datasets, the underlying logic, temporal and causal structure behind the reasoning process is less explored.
Revised: However, despite the complexity and variety of the visual context covered by these datasets, the underlying logic, temporal and causal structure behind the reasoning process is less explored.

Original: In this paper, we study the problem of temporal and causal reasoning in videos from a complementary perspective: inspired by a recent visual reasoning dataset, CLEVR (Johnson et al., 2017a), we simplify the problem of visual recognition, but emphasize the complex temporal and causal structure behind the interacting objects.
Revised: In this paper, we study the problem of temporal and causal reasoning in videos from a complementary perspective: inspired by a recent visual reasoning dataset, CLEVR (Johnson et al., 2017a), we simplify the problem of visual recognition, but emphasize the complex temporal and causal structure behind the interacting objects.

Original: We introduce a video reasoning benchmark for this problem, drawing inspirations from developmental psychology (Gerstenberg et al., 2015; Ullman, 2015).
Revised: We introduce a video reasoning benchmark for this problem, drawing inspirations from developmental psychology (Gerstenberg et al., 2015; Ullman, 2015).

Original: We also evaluate and assess limitations of various current visual reasoning models on the benchmark.
Revised: We also evaluate and assess limitations of various current visual reasoning models on the benchmark.

Original: Our benchmark, named CoLlision Events for Video REpresentation and Reasoning (CLEVRER), is a diagnostic video dataset for temporal and causal reasoning under a fully controlled environment.
Revised: Our benchmark, named CoLlision Events for Video REpresentation and Reasoning (CLEVRER), is a diagnostic video dataset for temporal and causal reasoning under a fully controlled environment.

Original: The design of CLEVRER follows two guidelines: ﬁrst, the posted tasks should focus on logic reasoning in the temporal and causal domain while staying simple and exhibiting minimal biases on visual scenes and language; second, the dataset should be fully controlled and well-annotated in order to host the  complex reasoning tasks and provide effective diagnostics for models on those tasks.
Revised: The design of CLEVRER follows two guidelines: ﬁrst, the posted tasks should focus on logic reasoning in the temporal and causal domain while staying simple and exhibiting minimal biases on visual scenes and language; second, the dataset should be fully controlled and well-annotated in order to host the [Figure] complex reasoning tasks and provide effective diagnostics for models on those tasks.

Original: CLEVRER includes 20,000 synthetic videos of colliding objects and more than 300,000 questions and answers (Figure 1).
Revised: CLEVRER includes 20,000 synthetic videos of colliding objects and more than 300,000 questions and answers (Figure 1).

Original: We focus on four speciﬁc elements of complex logical reasoning on videos: descriptive (e.g., ‘what color’), explanatory (‘what’s responsible for’), predictive (‘what will happen next’), and counterfactual (‘what if’).
Revised: We focus on four speciﬁc elements of complex logical reasoning on videos: descriptive (e g., ‘what color’), explanatory (‘what’s responsible for’), predictive (‘what will happen next’) and counterfactual (‘what if’).

Original: CLEVRER comes with ground-truth motion traces and event histories[Figure] of each object in the videos.
Revised: CLEVRER comes with ground-truth motion traces and event histories of each object in the videos.

Original: Each question is paired with a functional program representing its underlying logic.
Revised: Each question is paired with a functional program representing its underlying logic.

Original: As summarized in table 1, CLEVRER complements existing visual reasoning benchmarks on various aspects and introduces several novel tasks.
Revised: As summarized in table 1, CLEVRER complements existing visual reasoning benchmarks on various aspects and introduces several novel tasks.

Original: We also present analysis of various state-of-the-art visual reasoning models on CLEVRER.
Revised: We also present analysis of various state-of-the-art visual reasoning models on CLEVRER.

Original: While these models perform well on descriptive questions, they lack the ability to perform causal reasoning and struggle on the explanatory, predictive [Figure] and counterfactual questions.
Revised: While these models perform well on descriptive questions, they lack the ability to perform causal reasoning and struggle on the explanatory, predictive, and counterfactual questions.

Original: We therefore identify three key elements that are essential to the task: recognition of the objects and events in the videos; modeling the dynamics and causal relations between the objects and events; and understanding of the symbolic logic behind the questions.
Revised: We therefore identify three key elements that are essential to the task: recognition of the objects and events in the videos; modeling the dynamics and causal relations between the objects and events; and understanding of the symbolic logic behind the questions.

Original: As a ﬁrst-step exploration of this principle, we study an oracle model, Neuro-Symbolic Dynamic Reasoning (NS-DR), that explicitly joins these components via a symbolic video representation, and assess its performance and limitations.
Revised: As a ﬁrst-step exploration of this principle, we study an oracle model, Neuro-Symbolic Dynamic Reasoning (NS-DR), that explicitly joins these components via a symbolic video representation, and assess its performance and limitations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work can be uniquely positioned in the context of three recent research directions: video understanding, visual question answering, and physical and causal reasoning.
Revised: Our work can be uniquely positioned in the context of three recent research directions: video understanding, visual question answering, and physical and causal reasoning.

Original: Video understanding.
Revised: Video understanding.

Original: With the availability of large-scale video datasets (Caba Heilbron et al., 2015; Kay et al., 2017), joint video and language understanding tasks have received much interest.
Revised: With the availability of large-scale video datasets (Caba Heilbron et al., 2015; Kay et al., 2017), joint video and language understanding tasks have received much interest.

Original: This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.
Revised: This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.

Original: In particular, recent papers have explored different approaches to acquire and ground various reasoning tasks to videos.
Revised: In particular, recent papers have explored different approaches to acquire and ground various reasoning tasks to videos.

Original: Among those, MovieQA (Tapaswi et al., 2016), TGIF-QA (Jang et al., 2017), TVQA (Lei et al., 2018) are based on real-world videos and human-generated questions.
Revised: Among those, MovieQA (Tapaswi et al., 2016), TGIF-QA (Jang et al., 2017), TVQA (Lei et al., 2018) are based on real-world videos and human-generated questions.

Original: Social-IQ (Zadeh et al., 2019) discusses causal relations in human social interactions based on real videos.
Revised: Social-IQ (Zadeh et al., 2019) discusses causal relations in human social interactions based on real videos.

Original: COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks.
Revised: COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks.

Original: Compared to them, CLEVRER focuses on the causal relations grounded in object dynamics and physical interactions, and introduces a wide range of tasks including description, explanation, prediction and counterfactuals.
Revised: Compared to them, CLEVRER focuses on the causal relations grounded in object dynamics and physical interactions, and introduces a wide range of tasks including description, explanation, prediction and counterfactuals.

Original: CLEVRER also emphasizes compositionality in the visual and logic context.
Revised: CLEVRER also emphasizes compositionality in the visual and logic context.

Original: Visual question answering.
Revised: Visual question answering.

Original: Many benchmark tasks have been introduced in the domain of visual question answering.
Revised: Many benchmark tasks have been introduced in the domain of visual question answering.

Original: The Visual Question Answering (VQA) dataset (Antol et al., 2015) marks an important milestone towards top-down visual reasoning, based on large-scale cloud-sourced real images and human-generated questions.
Revised: The Visual Question Answering (VQA) dataset (Antol et al., 2015) marks an important milestone towards top-down visual reasoning, based on large-scale cloud-sourced real images and human-generated questions.

Original: The CLEVR dataset (Johnson et al., 2017a) follows a bottomup approach by deﬁning the tasks under a controlled close-domain setup of synthetic images and questions with compositional attributes and logic traces.
Revised: The CLEVR dataset (Johnson et al., 2017a) follows a bottomup approach by deﬁning the tasks under a controlled close-domain setup of synthetic images and questions with compositional attributes and logic traces.

Original: More recently, the GQA dataset (Hudson &
Revised: More recently, the GQA dataset (Hudson &

Original: Manning, 2019) applies synthetic compositional questions to real images.
Revised: Manning, 2019) applies synthetic compositional questions to real images.

Original: The VCR dataset (Zellers  et al., 2019) discusses explanations and hypothesis judgements based on common sense.
Revised: The VCR dataset (Zellers [Table] et al., 2019) discusses explanations and hypothesis judgements based on common sense.

Original: There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017;
Revised: There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017;

Original: Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018;
Revised: Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018;

Original: Aditya et al., 2018).
Revised: Aditya et al., 2018).

Original: Here we brieﬂy review a few.
Revised: Here we brieﬂy review a few.

Original: The[Table] stacked attention networks (SAN) (Yang et al., 2016) introduce a hierarchical attention mechanism for end-to-end VQA models.
Revised: The stacked attention networks (SAN) (Yang et al., 2016) introduce a hierarchical attention mechanism for end-to-end VQA models.

Original: The MAC network (Hudson & Manning, 2018) combines visual and language attention for compositional visual reasoning.
Revised: The MAC network (Hudson & Manning, 2018) combines visual and language attention for compositional visual reasoning.

Original: The IEP model (Johnson et al., 2017b) proposes to answer questions via neural program execution.
Revised: The IEP model (Johnson et al., 2017b) proposes to answer questions via neural program execution.

Original: The NS-VQA model (Yi et al., 2018) disentangles perception and logic reasoning by combining an object-based abstract representation of the image with symbolic program execution.
Revised: The NS-VQA model (Yi et al., 2018) disentangles perception and logic reasoning by combining an object-based abstract representation of the image with symbolic program execution.

Original: In this work we study a complementary problem of causal reasoning and assess the strengths and limitations of these baseline methods.
Revised: In this work we study a complementary problem of causal reasoning and assess the strengths and limitations of these baseline methods.

Original: Physical and causal reasoning.
Revised: Physical and causal reasoning.

Original: Our work is also related to research on learning scene dynamics for physical and causal reasoning (Lerer et al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016;
Revised: Our work is also related to research on learning scene dynamics for physical and causal reasoning (Lerer et al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016;

Original: Fragkiadaki et al., 2016; Battaglia et al., 2016;
Revised: Fragkiadaki et al., 2016; Battaglia et al., 2016;

Original: Chang et al., 2017; Agrawal et al., 2016; Finn et al., 2016;
Revised: Chang et al., 2017; Agrawal et al., 2016; Finn et al., 2016;

Original: Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016;
Revised: Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016;

Original: Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the environment (Battaglia et al., 2016; Chang et al., 2017).
Revised: Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the environment (Battaglia et al., 2016; Chang et al., 2017).

Revised: Concurrent to our work, CoPhy (Baradel et al., 2020) studies physical dynamics prediction in a counterfatual setting.

Revised: CATER (Girdhar & Ramanan, 2020) introduces a synthetic video dataset for temporal reasoning associated with compositional actions.

Original: In this work, we investigate how the incorporation of a learned dynamics model contributes to causal  reasoning on CLEVRER.
Revised: CLEVRER complements these works by incorporating dynamics modeling with compositional causal reasoning, and grounding the reasoning tasks to the language domain.

Original:  T HE CLEVRER D ATASET
Revised:  T HE CLEVRER D ATASET

Original: The CLEVRER dataset studies temporal and causal reasoning on videos.
Revised: The CLEVRER dataset studies temporal and causal reasoning on videos.

Original: It is carefully designed in a fully-controlled synthetic environment, enabling complex reasoning tasks, providing effective diagnostics for models while simplifying video recognition and language understanding.
Revised: It is carefully designed in a fully-controlled synthetic environment, enabling complex reasoning tasks, providing effective diagnostics for models while simplifying video recognition and language understanding.

Original: The videos describe motion and collisions of objects on a ﬂat tabletop (as shown in Figure 1) simulated by a physics engine, and are associated with the ground-truth motion traces and histories of all objects and events.
Revised: The videos describe motion and collisions of objects on a ﬂat tabletop (as shown in Figure 1) simulated by a physics engine, and are associated with the ground-truth motion traces and histories of all objects and events.

Original: Each video comes with four types of questions generated by machine, including descriptive (‘what color’, ‘how many’), explanatory (‘What is responsible for’), predictive (‘What will happen next’), and counterfactual (‘what if’).
Revised: Each video comes with four types of questions generated by machine, including descriptive (‘what color’, ‘how many’), explanatory (‘What is responsible for’), predictive (‘What will happen next’), and counterfactual (‘what if’).

Original: Each question is paired with a functional program.
Revised: Each question is paired with a functional program.

Original: 3. V IDEOS
Revised: 3. V IDEOS

Original: CLEVRER includes 10,000 videos for training, 5,000 for validation, and 5,000 for testing.
Revised: CLEVRER includes 10,000 videos for training, 5,000 for validation, and 5,000 for testing.

Original: All videos last for 5 seconds.
Revised: All videos last for 5 seconds.

Original: The videos are generated by a physics engine that simulates object motion plus a graphs engine that renders the frames.
Revised: The videos are generated by a physics engine that simulates object motion plus a graphs engine that renders the frames.

Original: Extra examples from the dataset can be found in supplementary material C.
Revised: Extra examples from the dataset can be found in supplementary material C.

Original: Objects and events.
Revised: Objects and events.

Original: Objects in CLEVRER videos adopt similar compositional intrinsic attributes as in CLEVR (Johnson et al., 2017a), including three shapes (cube, sphere, and cylinder), two materials (metal and rubber), and eight colors (gray, red, blue, green, brown, cyan, purple, and yellow).
Revised: Objects in CLEVRER videos adopt similar compositional intrinsic attributes as in CLEVR (Johnson et al., 2017a), including three shapes (cube, sphere, and cylinder), two materials (metal and rubber), and eight colors (gray, red, blue, green, brown, cyan, purple, and [Figure] The cube collides [Figure]

Revised: the

Revised: [Figure]

Original:  All objects have the same size so no vertical bouncing occurs during collision.
Revised: object [Figure] yellow). All objects have the same size so no vertical bouncing occurs during collision.

Original: In each video, we prohibit identical objects, such that each combination of the three attributes uniquely identiﬁes one object.
Revised: In each video, we prohibit identical objects, such that each combination of the three attributes uniquely identiﬁes one object.

Original: Under this constraint, all intrinsic attributes for each object are sampled randomly.
Revised: Under this constraint, all intrinsic attributes for each object are sampled randomly.

Original: We further introduce three types of events: enter , exit and collision , each of which contains a ﬁxed number of object participants: 2 for collision and 1 for enter and exit.
Revised: We further introduce three types of events: enter , exit and collision , each of which contains a ﬁxed number of object participants: 2 for collision and 1 for enter and exit.

Original: The objects and events form an abstract representation of the video.
Revised: The objects and events form an abstract representation of the video.

Original: These ground-truth annotations, together with the object motion traces, enable model diagnostics, one of the key advantages offered by a fully controlled environment.[Figure]
Revised: These ground-truth annotations, together with the object motion traces, enable model diagnostics, one of the key advantages offered by a fully controlled environment.

Original: Causal structure.
Revised: Causal structure.

Original: Objects and events in CLEVRER videos exhibit rich causal structures.
Revised: Objects and events in CLEVRER videos exhibit rich causal structures.

Original: An event can be either caused by an object if the event is the ﬁrst one participated by the object, or another event if the cause event happens right before the outcome event on the same object.
Revised: An event can be either caused by an object if the event is the ﬁrst one participated by the object, or another event if the cause event happens right before the outcome event on the same object.

Original: For example, if a sphere collides with a cube and then a cylinder, then the ﬁrst collision and the cube jointly “cause” the second collision.
Revised: For example, if a sphere collides with a cube and then a cylinder, then the ﬁrst collision and the cube jointly “cause” the second collision.

Original: The object motion traces with complex causal structures are generated by the following recursive process.
Revised: The object motion traces with complex causal structures are generated by the following recursive process.

Original: We start with one randomly initialized moving object, and then add another object whose initial position and velocity are set such that it will collide with the ﬁrst object.
Revised: We start with one randomly initialized moving object, and then add another object whose initial position and velocity are set such that it will collide with the ﬁrst object.

Original: The same process is then repeated to add more objects and collisions to the scene.
Revised: The same process is then repeated to add more objects and collisions to the scene.

Original: All collision pairs and motion trajectories are randomly chosen.
Revised: All collision pairs and motion trajectories are randomly chosen.

Original: We discard simulations with repetitive collisions between the same pair of objects.
Revised: We discard simulations with repetitive collisions between the same pair of objects.

Original: Video generation.
Revised: Video generation.

Original: CLEVRER videos are generated from the simulated motion traces, including each object’s position and pose at each time step.
Revised: CLEVRER videos are generated from the simulated motion traces, including each object’s position and pose at each time step.

Original: We use the Bullet (Coumans, 2010) physics engine for motion simulation.
Revised: We use the Bullet (Coumans, 2010) physics engine for motion simulation.

Original: Each simulation lasts for seven seconds.
Revised: Each simulation lasts for seven seconds.

Original: The motion traces are ﬁrst down-sampled to ﬁt the frame rate of the output video (25 frames per second).
Revised: The motion traces are ﬁrst down-sampled to ﬁt the frame rate of the output video (25 frames per second).

Original: Then the motion of the ﬁrst ﬁve seconds are sent to Blender (Blender Online Community, 2016) to render realistic video frames of object motion and collision.
Revised: Then the motion of the ﬁrst ﬁve seconds are sent to Blender (Blender Online Community, 2016) to render realistic video frames of object motion and collision.

Original: The remaining two seconds are held-out for predictive tasks.
Revised: The remaining two seconds are held-out for predictive tasks.

Original: We further note CLEVRER adopts the same software and parameters for rendering as CLEVR.
Revised: We further note CLEVRER adopts the same software and parameters for rendering as CLEVR.

Original: 3. Q UESTIONS
Revised: 3. Q UESTIONS

Original: Count

Original: Exist

Original: QueryShape

Original: [Figure]
Revised: [Figure]

Original: We pair each video with machine-generated questions for descriptive, explanatory, predictive, and counterfactual reasoning.
Revised: We pair each video with machine-generated questions for descriptive, explanatory, predictive, and counterfactual reasoning.

Original: Sample questions of the four types can be found in Figure 1.
Revised: Sample questions of the four types can be found in Figure 1.

Original: Each question is paired with a functional program executable on the video’s dynamical scene.
Revised: Each question is paired with a functional program executable on the video’s dynamical scene.

Original: Unlike CLEVR (Johnson et al., 2017a), our questions focus on the temporal and causal aspects of the objects and events in the videos.
Revised: Unlike CLEVR (Johnson et al., 2017a), our questions focus on the temporal and causal aspects of the objects and events in the videos.

Original: We exclude all questions on static object properties, which can be answered by looking at a single frame.
Revised: We exclude all questions on static object properties, which can be answered by looking at a single frame.

Original: CLEVRER consists of 219,918 descriptive questions, 33,811 explanatory questions, 14,298 predictive questions and 37,253 counterfactual questions.
Revised: CLEVRER consists of 219,918 descriptive questions, 33,811 explanatory questions, 14,298 predictive questions and 37,253 counterfactual questions.

Original: Detailed distribution and split of the questions can be found in Figure 3 and supplementary material A.
Revised: Detailed distribution and split of the questions can be found in Figure 3 and supplementary material A.

Original: Descriptive.
Revised: Descriptive.

Original: Descriptive questions evaluate a model’s capability to understand and reason about a video’s dynamical content and temporal relation.
Revised: Descriptive questions evaluate a model’s capability to understand and reason about a video’s dynamical content and temporal relation.

Original: The reasoning tasks are grounded to the compositional space of both object and event properties, including intrinsic attributes (color, material, shape), motion, collision, and temporal order.
Revised: The reasoning tasks are grounded to the compositional space of both object and event properties, including intrinsic attributes (color, material, shape), motion, collision, and temporal order.

Original: All descriptive questions are ‘open-ended’ and can be answered by a single word.
Revised: All descriptive questions are ‘open-ended’ and can be answered by a single word.

Original: Descriptive questions contain multiple sub-types including count , exist , query color , query material , and query shape .
Revised: Descriptive questions contain multiple sub-types including count , exist , query color , query material , and query shape .

Original: Distribution of the sub-types is shown in Figure 3.
Revised: Distribution of the sub-types is shown in Figure 3.

Original: We evenly sample the answers within each sub-type to reduce answer bias.
Revised: We evenly sample the answers within each sub-type to reduce answer bias.

Original: Explanatory.
Revised: Explanatory.

Original: Explanatory questions query the causal structure of a video by asking whether an object or event is responsible for another event.
Revised: Explanatory questions query the causal structure of a video by asking whether an object or event is responsible for another event.

Original: Event A is responsible for event B if A is among
Revised: Event A is responsible for event B if A is among

Original: B ’s ancestors in the causal graph.
Revised: B ’s ancestors in the causal graph.

Original: Similarly, object O is responsible for event
Revised: Similarly, object O is responsible for event

Original: A
Revised: A

Original: if O participates in
Revised: if O participates in

Original: A or any other event responsible for A .
Revised: A or any other event responsible for A .

Original: Explanatory questions are multiple choice questions with at most four options, each representing an event or object in the video.
Revised: Explanatory questions are multiple choice questions with at most four options, each representing an event or object in the video.

Original: Models need to select all options that match the question’s description.
Revised: Models need to select all options that match the question’s description.

Original: There can be multiple correct options for each question.
Revised: There can be multiple correct options for each question.

Original: We sample the options to balance the number of correct and wrong ones, and minimize text-only biases.
Revised: We sample the options to balance the number of correct and wrong ones, and minimize text-only biases.

Original: Predictive.
Revised: Predictive.

Original: Predictive questions test a model’s capability of predicting possible occurrences of future events after the video ends.
Revised: Predictive questions test a model’s capability of predicting possible occurrences of future events after the video ends.

Original: Similar to explanatory questions, predictive questions are multiplechoice, whose options represent candidate events that will or will not happen.
Revised: Similar to explanatory questions, predictive questions are multiplechoice, whose options represent candidate events that will or will not happen.

Original: Because post-video events are sparse, we provide two options for each predictive question to reduce bias.
Revised: Because post-video events are sparse, we provide two options for each predictive question to reduce bias.

Original: Counterfactual.
Revised: Counterfactual.

Original: Counterfactual questions query the outcome of the video under certain hypothetical conditions (e.g. removing one of the objects).
Revised: Counterfactual questions query the outcome of the video under certain hypothetical conditions (e.g. removing one of the objects).

Original: Models need to select the events that would or would not happen under the designated condition.
Revised: Models need to select the events that would or would not happen under the designated condition.

Original: There are at most four options for each question.
Revised: There are at most four options for each question.

Original: The numbers of correct and incorrect options are balanced.
Revised: The numbers of correct and incorrect options are balanced.

Original: Both predictive and counterfactual questions require knowledge of object dynamics underlying the videos and the ability to imagine and reason about unobserved events.
Revised: Both predictive and counterfactual questions require knowledge of object dynamics underlying the videos and the ability to imagine and reason about unobserved events.

Original: Program representation.
Revised: Program representation.

Original: In CLEVRER, each question is represented by a tree-structured functional program, as shown in Figure 2.
Revised: In CLEVRER, each question is represented by a tree-structured functional program, as shown in Figure 2.

Original: A program begins with a list of objects or events from the video.
Revised: A program begins with a list of objects or events from the video.

Original: The list is then passed through a sequence of ﬁlter modules, which select entries from the list and join the tree branches to output a set of target objects and events.
Revised: The list is then passed through a sequence of ﬁlter modules, which select entries from the list and join the tree branches to output a set of target objects and events.

Original: Finally, an output module is called to query a designated property of the target outputs.
Revised: Finally, an output module is called to query a designated property of the target outputs.

Original: For multiple choice questions, each question and option correspond to separate programs, which can be jointly executed to output a yes/no token that indicates if the choice is correct for the question.
Revised: For multiple choice questions, each question and option correspond to separate programs, which can be jointly executed to output a yes/no token that indicates if the choice is correct for the question.

Original: A list of all program modules can be found in the supplementary material B.
Revised: A list of all program modules can be found in the supplementary material B.

Original: Question generation.
Revised: Question generation.

Original: Questions in CLEVRER are generated by a multi-step procedure.
Revised: Questions in CLEVRER are generated by a multi-step procedure.

Original: For each question type, a pre-deﬁned logic template is chosen.
Revised: For each question type, a pre-deﬁned logic template is chosen.

Original: The logic template can be further populated by attributes that are associated with the context of the video (i.e. the color, material, shape that identiﬁes the object to be queried) and then turned into natural language.
Revised: The logic template can be further populated by attributes that are associated with the context of the video (i.e. the color, material, shape that identiﬁes the object to be queried) and then turned into natural language.

Original: We ﬁrst generate an exhaustive list of all possible questions for each video.
Revised: We ﬁrst generate an exhaustive list of all possible questions for each video.

Original: To avoid language bias, we sample from that list to maintain a balanced answer distribution for each question type and minimize the correlation between the questions and answers over the entire dataset.
Revised: To avoid language bias, we sample from that list to maintain a balanced answer distribution for each question type and minimize the correlation between the questions and answers over the entire dataset.

Original:  B ASELINE E VALUATION
Revised:  B ASELINE E VALUATION

Original: In this section, we evaluate and analyse the performances of a wide range of baseline models for video reasoning on CLEVRER.
Revised: In this section, we evaluate and analyse the performances of a wide range of baseline models for video reasoning on CLEVRER.

Original: For descriptive questions, the models treat each question as a multi-class classiﬁcation problem over all possible answers.
Revised: For descriptive questions, the models treat each question as a multi-class classiﬁcation problem over all possible answers.

Original: For multiple choice questions, each question-choice pair is treated as a binary classiﬁcation problem indicating the correctness of the choice.
Revised: For multiple choice questions, each question-choice pair is treated as a binary classiﬁcation problem indicating the correctness of the choice.

Original: 4. M ODEL D ETAILS
Revised: 4. M ODEL D ETAILS

Original: The baseline models we evaluate fall into three families: language-only models, models for video question answering, and models for compositional visual reasoning.
Revised: The baseline models we evaluate fall into three families: language-only models, models for video question answering, and models for compositional visual reasoning.

Original: Language-only models.
Revised: Language-only models.

Original: This model family includes weak baselines that only relies on question input to assess language biases in CLEVRER.
Revised: This model family includes weak baselines that only relies on question input to assess language biases in CLEVRER.

Original: Q-type (random) uniformly samples an answer from the answer space or randomly select each choice for multiple-choice questions.
Revised: Q-type (random) uniformly samples an answer from the answer space or randomly select each choice for multiple-choice questions.

Original: Q-type (frequent) chooses the most frequent answer in the training set for each question type.
Revised: Q-type (frequent) chooses the most frequent answer in the training set for each question type.

Original: LSTM uses a pretrained word embedding trained on the Google News corpus (Mikolov et al., 2013) to encode the input question and processes the sequence with a LSTM (Hochreiter & Schmidhuber, 1997).
Revised: LSTM uses a pretrained word embedding trained on the Google News corpus (Mikolov et al., 2013) to encode the input question and processes the sequence with a LSTM (Hochreiter & Schmidhuber, 1997).

Original: A MLP is then applied to the ﬁnal hidden state to predict a distribution over the answers.
Revised: A MLP is then applied to the ﬁnal hidden state to predict a distribution over the answers.

Original: Video question answering.
Revised: Video question answering.

Original: We also evaluate the following models that relies on both video and language inputs.
Revised: We also evaluate the following models that relies on both video and language inputs.

Original: CNN+MLP extracts features from the input video via a convolutional neural network (CNN) and encodes the question by taking the average of the pretrained word embeddings (Mikolov et al., 2013).
Revised: CNN+MLP extracts features from the input video via a convolutional neural network (CNN) and encodes the question by taking the average of the pretrained word embeddings (Mikolov et al., 2013).

Original: The video and language features are then jointly sent to a MLP for answer prediction.
Revised: The video and language features are then jointly sent to a MLP for answer prediction.

Original: CNN+LSTM relies on the same architecture for video feature extraction but uses the ﬁnal state of a LSTM for answer prediction.
Revised: CNN+LSTM relies on the same architecture for video feature extraction but uses the ﬁnal state of a LSTM for answer prediction.

Original: TVQA (Lei et al., 2018) introduces a multi-stream end-to-end neural model that sets the state of the art for video question answering.
Revised: TVQA (Lei et al., 2018) introduces a multi-stream end-to-end neural model that sets the state of the art for video question answering.

Original: We apply attribute-aware object-centric features acquired by a video frame parser ( TVQA+ ).
Revised: We apply attribute-aware object-centric features acquired by a video frame parser ( TVQA+ ).

Original: We also include a recent model that incorporates heterogeneous memory with multimodal attention work ( Memory )
Revised: We also include a recent model that incorporates heterogeneous memory with multimodal attention work ( Memory )

Original: (Fan et al., 2019) that achieves superior performance on several datasets. 
Revised: (Fan et al., 2019) that achieves superior performance on several datasets. [Table]

Original: Compositional visual reasoning.
Revised: Compositional visual reasoning.

Original: The CLEVR dataset (Johnson et al., 2017a) opened up a new direction of compositional visual reasoning, which emphasizes complexity and compositionality in the logic and visual context.
Revised: The CLEVR dataset (Johnson et al., 2017a) opened up a new direction of compositional visual reasoning, which emphasizes complexity and compositionality in the logic and visual context.

Original: We modify several best-performing models and apply them to our[Table] video benchmark.
Revised: We modify several best-performing models and apply them to our video benchmark.

Original: The IEP model (Johnson et al., 2017b) applies neural program execution for visual reasoning on images.
Revised: The IEP model (Johnson et al., 2017b) applies neural program execution for visual reasoning on images.

Original: We apply the same approach to our program-based video reasoning task ( IEP (V) ) by substituting the program primitives by the ones from CLEVRER, and applying the execution modules on the video features extracted by a convolutional LSTM (Shi et al., 2015).
Revised: We apply the same approach to our program-based video reasoning task ( IEP (V) ) by substituting the program primitives by the ones from CLEVRER, and applying the execution modules on the video features extracted by a convolutional LSTM (Shi et al., 2015).

Original: TbD-net (Mascharka et al., 2018) follows a similar approach by parsing the input question into a program, which is then assembled into a neural network that acts on the attention map over the image features.
Revised: TbD-net (Mascharka et al., 2018) follows a similar approach by parsing the input question into a program, which is then assembled into a neural network that acts on the attention map over the image features.

Original: The ﬁnal attended image feature is then sent to an output layer for classiﬁcation.
Revised: The ﬁnal attended image feature is then sent to an output layer for classiﬁcation.

Original: We adopt the same approach through spatial-temporal attention over the video feature space ( TbD-net (V) ).
Revised: We adopt the same approach through spatial-temporal attention over the video feature space ( TbD-net (V) ).

Original: MAC (Hudson & Manning, 2018) incorporates a joint attention mechanism on both the image feature map and the question, which leads to strong performance on CLEVR without program supervision.
Revised: MAC (Hudson & Manning, 2018) incorporates a joint attention mechanism on both the image feature map and the question, which leads to strong performance on CLEVR without program supervision.

Original: We modify the model by applying a temporal attention unit across the video frames to generate a latent encoding for the video ( MAC (V) ).
Revised: We modify the model by applying a temporal attention unit across the video frames to generate a latent encoding for the video ( MAC (V) ).

Original: The video feature is then input to the MAC network to output an answer distribution.
Revised: The video feature is then input to the MAC network to output an answer distribution.

Original: We study an augmented approach: we construct object-aware video features by adding the segmentation masks of all objects in the frames and labeling them by the values of their intrinsic attributes ( MAC (V+) ).
Revised: We study an augmented approach: we construct object-aware video features by adding the segmentation masks of all objects in the frames and labeling them by the values of their intrinsic attributes ( MAC (V+) ).

Original: Implementation details.
Revised: Implementation details.

Original: We use a pre-trained ResNet-50 (He et al., 2016) to extract features from the video frames.
Revised: We use a pre-trained ResNet-50 (He et al., 2016) to extract features from the video frames.

Original: We use the 2,048-dimensional pool5 layer output for CNN-based methods, and the 14 × 14 feature maps for MAC, IEP and TbD-net.
Revised: We use the 2,048-dimensional pool5 layer output for CNN-based methods, and the 14 × 14 feature maps for MAC, IEP and TbD-net.

Original: The program generators of IEP and TbD-net are trained on 1000 programs.
Revised: The program generators of IEP and TbD-net are trained on 1000 programs.

Original: We uniformly sample 25 frames for each video as input.
Revised: We uniformly sample 25 frames for each video as input.

Original: Object segmentation masks and attributes are obtained by a video parser consisted of an object detector and an attribute network.
Revised: Object segmentation masks and attributes are obtained by a video parser consisted of an object detector and an attribute network.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: We summarize the performances of all baseline models in Table 2.
Revised: We summarize the performances of all baseline models in Table 2.

Original: The fact that these models achieve different performances over the wide spectrum of tasks suggest that CLEVRER offers powerful assessment to the models’ strength and limitations on various domains.
Revised: The fact that these models achieve different performances over the wide spectrum of tasks suggest that CLEVRER offers powerful assessment to the models’ strength and limitations on various domains.

Original: All models are trained on the training set until convergence, tuned on the validation set and evaluated on the test set.
Revised: All models are trained on the training set until convergence, tuned on the validation set and evaluated on the test set.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For descriptive questions, we calculate the accuracy by comparing the predicted answer token to the ground-truth.
Revised: For descriptive questions, we calculate the accuracy by comparing the predicted answer token to the ground-truth.

Original: For multiple choice questions, we adopt two metrics: per-option accuracy measures the model’s overall correctness on single options across all questions; per-question accuracy measures the correctness of the full question, requiring all choices to be selected correctly.
Revised: For multiple choice questions, we adopt two metrics: per-option accuracy measures the model’s overall correctness on single options across all questions; per-question accuracy measures the correctness of the full question, requiring all choices to be selected correctly.

Original: Descriptive reasoning.
Revised: Descriptive reasoning.

Original: Descriptive questions query the content of the video from various aspects.
Revised: Descriptive questions query the content of the video from various aspects.

Original: In order to do well on this question type, a model needs to both accurately recognize the objects and events that happen in the video, as well as understanding the compositional logic pattern behind the questions.
Revised: In order to do well on this question type, a model needs to both accurately recognize the objects and events that happen in the video, as well as understanding the compositional logic pattern behind the questions.

Original: In other words, descriptive questions require strong perception and logic operations on both visual and language signals.
Revised: In other words, descriptive questions require strong perception and logic operations on both visual and language signals.

Original: As shown in Table 2, the LSTM baseline that relies only on question input performs poorly on the descriptive questions, only outperforming the random baselinesby a small margin.
Revised: As shown in Table 2, the LSTM baseline that relies only on question input performs poorly on the descriptive questions, only outperforming the random baselines

Original: This suggests that CLEVRER has very small bias on the questions.

Original: Video QA models, including the state of the art model TVQA+ (Lei et al., 2018) achieve better performances. But because of their limited capability of handling the compositionality in the question logic and What shape is the second object to collide with the gray object? Video
Revised: [Table]

Original: LSTMEncoder
Revised: LSTMEncoder

Original: LSTM LSTM
Revised: LSTM LSTM

Original: LSTM
Revised: LSTM

Original: Objects
Revised: Objects

Original: Filter_color (gray) ...
Revised: Filter_color (gray) [Table]

Original: Query_shape .
Revised: MaskR-CNN

Original: [Table]
Revised: [Table]

Original: I. Video Frame Parser III. Question Parser IV. Program Executor II. Dynamics Predictor
Revised: I. Video Frame Parser III. Question Parser IV. Program Executor II. Dynamics Predictor

Original: [Table]
Revised: [Table]

Original: /diZZ4EPFV/
Revised: /diZZ4EPFV/

Original: T2Qk0noUhaYzIjDQ895E/M/rpNC78DMukxSYpLNFvVRgiPEkItzlilEQI0MIVdz8FdMBUYSCCbJkQnDnT14kzdOqa/jtWbl2mcdRRAfoEB0jF52jGrpGddRAFD2iZ/SK3qwn68V6tz5mrQUrn9lHf2B9/gDoPp6k</latexit>
Revised: T2Qk0noUhaYzIjDQ895E/M/rpNC78DMukxSYpLNFvVRgiPEkItzlilEQI0MIVdz8FdMBUYSCCbJkQnDnT14kzdOqa/jtWbl2mcdRRAfoEB0jF52jGrpGddRAFD2iZ/SK3qwn68V6tz5mrQUrn9lHf2B9/gDoPp6k</latexit>

Original: Answer  visual context, these models are still unable to thrive on the task.
Revised: Answer by a small margin. This suggests that CLEVRER has very small bias on the questions. Video QA models, including the state of the art model TVQA+ (Lei et al., 2018) achieve better performances. But because of their limited capability of handling the compositionality in the question logic and visual context, these models are still unable to thrive on the task.

Original: In contrast, models designed for compositional reasoning, including TbD-net that operates on neural program execution and MAC that introduces a joint attention mechanism, are able to achieve more competitive performances.
Revised: In contrast, models designed for compositional reasoning, including TbD-net that operates on neural program execution and MAC that introduces a joint attention mechanism, are able to achieve more competitive performances.

Original: Causal reasoning.
Revised: Causal reasoning.

Original: Results on the descriptive questions demonstrate the power of models that combine visual and language perception with compositional logic operations.
Revised: Results on the descriptive questions demonstrate the power of models that combine visual and language perception with compositional logic operations.

Original: However, the causal reasoning tasks (explanatory, predictive, counterfactual) require further understanding beyond perception.
Revised: However, the causal reasoning tasks (explanatory, predictive, counterfactual) require further understanding beyond perception.

Original: Our evaluation results (Table 2) show poor performance of most baseline models on these questions.
Revised: Our evaluation results (Table 2) show poor performance of most baseline models on these questions.

Original: The compositional reasoning models that performs well on the descriptive questions (MAC (V) and TbD-net (V)) only achieve marginal improvements over the random and language-only baselines on the causal tasks.
Revised: The compositional reasoning models that performs well on the descriptive questions (MAC (V) and TbD-net (V)) only achieve marginal improvements over the random and language-only baselines on the causal tasks.

Original: However, we do notice a reasonable gain in performance on models that inputs object-aware representations: TVQA+ achieves high accuracy on the predictive questions, and MAC (V+) improves upon MAC (V) on all tasks.
Revised: However, we do notice a reasonable gain in performance on models that inputs object-aware representations: TVQA+ achieves high accuracy on the predictive questions, and MAC (V+) improves upon MAC (V) on all tasks.

Original: We identify the following messages suggested by our evaluation results.
Revised: We identify the following messages suggested by our evaluation results.

Original: First, object-centric representations are essential for the causal reasoning tasks.
Revised: First, object-centric representations are essential for the causal reasoning tasks.

Original: This is supported by the large improvement on
Revised: This is supported by the large improvement on

Original: MAC (V+) over MAC (V) after using features that are aware of object instances and attributes, as well as the strong performance of TVQA+ on the predictive questions.
Revised: MAC (V+) over MAC (V) after using features that are aware of object instances and attributes, as well as the strong performance of TVQA+ on the predictive questions.

Original: Second, all baseline models lack a component to explicitly model the dynamics of the objects and the causal relations between the collision events.
Revised: Second, all baseline models lack a component to explicitly model the dynamics of the objects and the causal relations between the collision events.

Original: As a result, they struggle on the tasks involving unobserved scenes and in particular performs poorly on the counterfactual tasks.
Revised: As a result, they struggle on the tasks involving unobserved scenes and in particular performs poorly on the counterfactual tasks.

Original: The combination of object-centric representation and dynamics modeling therefore suggests a promising direction for approaching the causal tasks.
Revised: The combination of object-centric representation and dynamics modeling therefore suggests a promising direction for approaching the causal tasks.

Original:  N EURO -S YMBOLIC D YNAMIC R EASONING
Revised:  N EURO -S YMBOLIC D YNAMIC R EASONING

Original: Baseline evaluations on CLEVRER have revealed two key elements that are essential to causal reasoning: an object-centric video representation that is aware of the temporal and causal relations between the objects and events; and a dynamics model able to predict the object dynamics under unobserved or counterfactual scenarios.
Revised: Baseline evaluations on CLEVRER have revealed two key elements that are essential to causal reasoning: an object-centric video representation that is aware of the temporal and causal relations between the objects and events; and a dynamics model able to predict the object dynamics under unobserved or counterfactual scenarios.

Original: However, unifying these elements with video and language understanding posts the following challenges: ﬁrst, all the disjoint model components should operate on a common set of representations of the video, question, dynamics and causal relations; second, the representation should be aware of the compositional relations between the objects and events.
Revised: However, unifying these elements with video and language understanding posts the following challenges: ﬁrst, all the disjoint model components should operate on a common set of representations of the video, question, dynamics and causal relations; second, the representation should be aware of the compositional relations between the objects and events.

Original: We draw inspirations from Yi et al.
Revised: We draw inspirations from Yi et al.

Original: and study an oracle model that operates on a symbolic representation to join video perception, language understanding with dynamics modeling.
Revised: and study an oracle model that operates on a symbolic representation to join video perception, language understanding with dynamics modeling.

Original: Our model
Revised: Our model

Original: Neuro-Symbolic Dynamic Reasoning (NS-DR) combines neural nets for pattern recognition and dynamics prediction, and symbolic logic for causal reasoning.
Revised: Neuro-Symbolic Dynamic Reasoning (NS-DR) combines neural nets for pattern recognition and dynamics prediction, and symbolic logic for causal reasoning.

Original: As shown in Figure 4, NS-DR consists of a video frame parser (Figure 4-I), a neural dynamics predictor (Figure 4-II), a question parser (Figure 4-III), and a program executor (Figure 4-IV).
Revised: As shown in Figure 4, NS-DR consists of a video frame parser (Figure 4-I), a neural dynamics predictor (Figure 4-II), a question parser (Figure 4-III), and a program executor (Figure 4-IV).

Original: We present details of the model components below and in supplementary material B.
Revised: We present details of the model components below and in supplementary material B.

Original: Video frame parser.
Revised: Video frame parser.

Original: The video frame parser serves as a perception module that is disentangled from other components of the model, from which we obtain an object-centric representation of the video frames.
Revised: The video frame parser serves as a perception module that is disentangled from other components of the model, from which we obtain an object-centric representation of the video frames.

Original: The parser is a Mask R-CNN (He et al., 2017) that performs object detection and scene de-rendering on each video frame (Wu et al., 2017).
Revised: The parser is a Mask R-CNN (He et al., 2017) that performs object detection and scene de-rendering on each video frame (Wu et al., 2017).

Original: We use a ResNet-50 FPN (Lin et al., ; He et al., 2016) as the backbone.
Revised: We use a ResNet-50 FPN (Lin et al., 2017; He et al., 2016) as the backbone.

Original: For each input video frame, the network outputs the object mask proposals, a label that corresponds to the intrinsic attributes (color, material, shape) and a score representing the conﬁdence level of the proposal used for ﬁltering.
Revised: For each input video frame, the network outputs the object mask proposals, a label that corresponds to the intrinsic attributes (color, material, shape) and a score representing the conﬁdence level of the proposal used for ﬁltering.

Original: Please refer to He et al.
Revised: Please refer to He et al.

Original: (He et al., 2017) for more details.
Revised: (He et al., 2017) for more details.

Original: The video parser is trained on 4000 video frames randomly sampled from the training set with object masks and attribute annotations.
Revised: The video parser is trained on 4000 video frames randomly sampled from the training set with object masks and attribute annotations.

Original: Neural dynamics predictor.
Revised: Neural dynamics predictor.

Original: We apply the Propagation Network (PropNet) (Li et al., 2019) for dynamics modeling, which is a learnable physics engine that, extending from the interaction networks (Battaglia et al., 2016), performs object- and relation-centric updates on a dynamic scene.
Revised: We apply the Propagation Network (PropNet) (Li et al., 2019) for dynamics modeling, which is a learnable physics engine that, extending from the interaction networks (Battaglia et al., 2016), performs object- and relation-centric updates on a dynamic scene.

Original: The model inputs the object proposals from the video parser, and learns the dynamics of the objects across the frames for predicting motion traces and collision events.
Revised: The model inputs the object proposals from the video parser, and learns the dynamics of the objects across the frames for predicting motion traces and collision events.

Original: PropNet represents the dynamical system as a directed graph, G = (cid:104)
Revised: PropNet represents the dynamical system as a directed graph, G = (cid:104)

Original: O, R (cid:105) , where the vertices O =
Revised: O, R (cid:105) , where the vertices O =

Original: { o i } represent objects and edges
Revised: { o i } represent objects and edges

Original: R = { r k } represent relations.
Revised: R = { r k } represent relations.

Original: Each object (vertex) o i and relation (edge) r k can be further written as o i = (cid:104) s i , a oi (cid:105) , r k = (cid:104)
Revised: Each object (vertex) o i and relation (edge) r k can be further written as o i = (cid:104) s i , a oi (cid:105) , r k = (cid:104)

Original: u k , v k , a rk (cid:105) , where s i is the state of object i ; a oi denotes its intrinsic attributes; u k , v k are integers denoting the index of the receiver and sender vertices joined by edge r k ; a rk represents the state of edge r k , indicating whether there is collision between the two objects.
Revised: u k , v k , a rk (cid:105) , where s i is the state of object i ; a oi denotes its intrinsic attributes; u k , v k are integers denoting the index of the receiver and sender vertices joined by edge r k ; a r k represents the state of edge r k , indicating whether there is collision between the two objects.

Original: In our case, s i is a concatenation of tuple (cid:104)
Revised: In our case, s i is a concatenation of tuple (cid:104)

Original: c
Revised: c

Original: i , m i , p i (cid:105) over a small history window to encode motion history, where c i and m i are the corresponding image and mask patches cropped at p i , which is the x - y position of the mask in the original image (please see the cyan metal cube in Figure 4 for an example).
Revised: i , m i , p i (cid:105) over a small history window to encode motion history, where c i and m i are the corresponding image and mask patches cropped at p i , which is the x - y position of the mask in the original image (please see the cyan metal cube in Figure 4 for an example).

Original: PropNet handles the instantaneous propagation of effects via multi-step message passing.
Revised: PropNet handles the instantaneous propagation of effects via multi-step message passing.

Original: Please refer to supplementary material B for more details.
Revised: Please refer to supplementary material B for more details.

Original: Question parser.
Revised: Question parser.

Original: We use an attention-based seq2seq model (Bahdanau et al., 2015) to parse the input questions into their corresponding functional programs.
Revised: We use an attention-based seq2seq model (Bahdanau et al., 2015) to parse the input questions into their corresponding functional programs.

Original: The model consists of a bidirectional
Revised: The model consists of a bidirectional

Original: LSTM encoder plus an LSTM decoder with attention, similar to the question parser in (Yi et al., 2018).
Revised: LSTM encoder plus an LSTM decoder with attention, similar to the question parser in (Yi et al., 2018).

Original: For multiple choice questions, we use two networks to parse the questions and choices separately.
Revised: For multiple choice questions, we use two networks to parse the questions and choices separately.

Original: Given an input word sequence ( x 1 , x 2 , ...x I ) , the encoder ﬁrst generates a bi-directional latent encoding at each step:
Revised: Given an input word sequence ( x 1 , x 2 , ...x I ) , the encoder ﬁrst generates a bi-directional latent encoding at each step:

Original: [Equation]
Revised: [Equation]

Original: The decoder then generates a sequence of program tokens ( y 1 , y 2 , ..., y J ) from the latent encodings using attention: 
Revised: The decoder then generates a sequence of program tokens ( y 1 , y 2 , ..., y J ) from the latent encodings using attention: [Equation]

Original: [Equation]
Revised: [Equation]

Original: At training time, the input program tokens { y j } are used for generating the predicted labels y j − 1 → ˆ y j at each step j = 1 , 2 , 3 , ..., J .
Revised: At training time, the input program tokens { y j } are used for generating the predicted labels y j − 1 → ˆ y j at each step j = 1 , 2 , 3 , ..., J .

Original: The generated label ˆ y j is then compared with y j to compute a loss for training.
Revised: The generated label ˆ y j is then compared with y j to compute a loss for training.

Original: At test time, the decoder rolls out by feeding the sampled prediction at the previous step to the input of the current step ˆ y j = y j .
Revised: At test time, the decoder rolls out by feeding the sampled prediction at the previous step to the input of the current step ˆ y j = y j .

Original: The roll-out stops when the end token appears or when the sequence reaches certain maximal length.
Revised: The roll-out stops when the end token appears or when the sequence reaches certain maximal length.

Original: For both the encoder and decoder LSTM, we use the same parameter setup of two hidden layers with 256 units, and a 300-dimensional word embedding vector.
Revised: For both the encoder and decoder LSTM, we use the same parameter setup of two hidden layers with 256 units, and a 300-dimensional word embedding vector.

Original: Program executor.
Revised: Program executor.

Original: The program executor explicitly executes the program on the motion and event traces extracted by the dynamics predictor and output an answer to the question.
Revised: The program executor explicitly executes the program on the motion and event traces extracted by the dynamics predictor and output an answer to the question.

Original: It consists of a collection of functional modules implemented in Python.
Revised: It consists of a collection of functional modules implemented in Python.

Original: Given an input program, the executor ﬁrst assembles the modules and then iterate through the program tree.
Revised: Given an input program, the executor ﬁrst assembles the modules and then iterate through the program tree.

Original: The output of the ﬁnal module is the answer to the target question.
Revised: The output of the ﬁnal module is the answer to the target question.

Original: There are three types of program modules: input module, ﬁlter module, and output module.
Revised: There are three types of program modules: input module, ﬁlter module, and output module.

Original: The input modules are the entry points of the program trees and can directly query the output of the dynamics predictor.
Revised: The input modules are the entry points of the program trees and can directly query the output of the dynamics predictor.

Original: For example, the Events and Objects modules that commonly appear in the programs of descriptive questions indicate inputting all the observed events and objects from a video.
Revised: For example, the Events and Objects modules that commonly appear in the programs of descriptive questions indicate inputting all the observed events and objects from a video.

Original: The ﬁlter modules perform logic operations on the input objects/events  based on a designated intrinsic attribute, motion state, temporal order, or causal relation.
Revised: The ﬁlter modules perform logic operations on the input objects/events [Table] based on a designated intrinsic attribute, motion state, temporal order, or causal relation.

Original: The output modules return the answer label.
Revised: The output modules return the answer label.

Original: For open-ended descriptive questions, the executor will return a token from the answer space.
Revised: For open-ended descriptive questions, the executor will return a token from the answer space.

Original: For multiple choice questions, the executor will ﬁrst execute the choice program and then send the result to the question program to output a yes/no token.
Revised: For multiple choice questions, the executor will ﬁrst execute the choice program and then send the result to the question program to output a yes/no token.

Original: A comprehensive list of all modules in the NS-DR program executor is presented in supplementary material B.[Table]
Revised: A comprehensive list of all modules in the NS-DR program executor is presented in supplementary material B.

Original: Results.
Revised: Results.

Original: We summarize the performance of NS-DR on CLEVRER in Table 3.
Revised: We summarize the performance of NS-DR on CLEVRER in Table 3.

Original: On descriptive questions, our model achieves an 88.1% accuracy when the question parser is trained under 1, programs, outperforming other baseline methods.
Revised: On descriptive questions, our model achieves an 88.1% accuracy when the question parser is trained under 1, programs, outperforming other baseline methods.

Original: On explanatory, predictive, and counterfactual questions, our model achieves a more signiﬁcant gain.
Revised: On explanatory, predictive, and counterfactual questions, our model achieves a more signiﬁcant gain.

Original: We also study a variation of our model,
Revised: We also study a variation of our model,

Original: NS-DR (No Events/NE), which uses the dynamics predictor to generate only the predictive and counterfactual motion traces, and collision events are identiﬁed by the velocity change of the colliding objects.
Revised: NS-DR (No Events/NE), which uses the dynamics predictor to generate only the predictive and counterfactual motion traces, and collision events are identiﬁed by the velocity change of the colliding objects.

Original: NS-DR (NE) performs comparably to the full model, showing that our disentangled system is adaptable to alternative methods for dynamics modeling.
Revised: NS-DR (NE) performs comparably to the full model, showing that our disentangled system is adaptable to alternative methods for dynamics modeling.

Original: We also conduct ablation study on the number of programs for training the question parser.
Revised: We also conduct ablation study on the number of programs for training the question parser.

Original: As shown in Figure 5, NS-DR reaches full capability at 1,000 programs for all question types.
Revised: As shown in Figure 5, NS-DR reaches full capability at 1,000 programs for all question types.

Original: We highlight the following contributions of the model.
Revised: We highlight the following contributions of the model.

Original: First, NS-DR incorporates a dynamics planner into the visual reasoning task, which directly enables predictions of unobserved motion and events, and enables the model for the predictive and counterfactual tasks.
Revised: First, NS-DR incorporates a dynamics planner into the visual reasoning task, which directly enables predictions of unobserved motion and events, and enables the model for the predictive and counterfactual tasks.

Original: This suggests that dynamics planning has great potential for language-grounded visual reasoning tasks and NS-DR takes a preliminary step towards this direction.
Revised: This suggests that dynamics planning has great potential for language-grounded visual reasoning tasks and NS-DR takes a preliminary step towards this direction.

Original: Second, symbolic representation provides a powerful common ground for vision, language, dynamics and causality.
Revised: Second, symbolic representation provides a powerful common ground for vision, language, dynamics and causality.

Original: By design, it empowers the model to explicitly capture the compositionality behind the video’s causal structure and the question logic. 
Revised: By design, it empowers the model to explicitly capture the compositionality behind the video’s causal structure and the question logic. [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We further discuss limitations of NS-DR and suggest possible directions for future research.
Revised: We further discuss limitations of NS-DR and suggest possible directions for future research.

Original: First, training of the video and question parser relies on extra supervisions such as object masks, attributes, and question programs.
Revised: First, training of the video and question parser relies on extra supervisions such as object masks, attributes, and question programs.

Original: Even though the amount of data required for training is minimal compared to end-to-end approaches (i.e. thousands of annotated frames and programs), these data is hard to acquire in real-world applications.
Revised: Even though the amount of data required for training is minimal compared to end-to-end approaches (i.e. thousands of annotated frames and programs), these data is hard to acquire in real-world applications.

Original: This constraint could be relaxed by applying unsupervised/weaklysupervised methods for scene decomposition and concept discovery (Burgess et al., 2019; Mao et al., 2019).
Revised: This constraint could be relaxed by applying unsupervised/weaklysupervised methods for scene decomposition and concept discovery (Burgess et al., 2019; Mao et al., 2019).

Original: Second, our model performance decreases on tasks that require long-term dynamics prediction such as the counterfactual questions.
Revised: Second, our model performance decreases on tasks that require long-term dynamics prediction such as the counterfactual questions.

Original: This suggests that we need a better dynamics model capable of generating more stable and accurate trajectories.
Revised: This suggests that we need a better dynamics model capable of generating more stable and accurate trajectories.

Original: CLEVRER provides a benchmark for assessing the predictive power of such dynamics models.
Revised: CLEVRER provides a benchmark for assessing the predictive power of such dynamics models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a systematic study of temporal and causal reasoning in videos.
Revised: We present a systematic study of temporal and causal reasoning in videos.

Original: This profound and challenging problem deeply rooted to the fundamentals of human intelligence has just begun to be studied with ‘modern’ AI tools.
Revised: This profound and challenging problem deeply rooted to the fundamentals of human intelligence has just begun to be studied with ‘modern’ AI tools.

Original: We introduce a set of benchmark tasks to better facilitate the research in this area.
Revised: We introduce a set of benchmark tasks to better facilitate the research in this area.

Original: We also believe video understanding and reasoning should go beyond passive knowledge extraction, and focus on building an internal understanding of the dynamics and causal relations, which is essential for practical applications such as dynamic robot manipulation under complex causal conditions.
Revised: We also believe video understanding and reasoning should go beyond passive knowledge extraction, and focus on building an internal understanding of the dynamics and causal relations, which is essential for practical applications such as dynamic robot manipulation under complex causal conditions.

Original: Our newly introduced CLEVRER dataset and the NS-DR model are preliminary steps toward this direction.
Revised: Our newly introduced CLEVRER dataset and the NS-DR model are preliminary steps toward this direction.

Original: We hope that with recent advances in graph networks, visual predictive models, and neuro-symbolic algorithms, the deep learning community can now revisit this classic problem in more realistic setups in the future, capturing true intelligence beyond pattern recognition. 
Revised: We hope that with recent advances in graph networks, visual predictive models, and neuro-symbolic algorithms, the deep learning community can now revisit this classic problem in more realistic setups in the future, capturing true intelligence beyond pattern recognition. Acknowledgment.

Revised: We thank Jiayuan Mao for helpful discussions and suggestions.

Revised: This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds and Machines (CBMM, funded by NSF STC award CCF-1231216), and

Revised: IBM Research

Original: S UPPLEMENTARY M ATERIALS A D ESCRIPTIVE Q UESTION S UB - TYPES AND S TATISTICS
Revised: S UPPLEMENTARY M ATERIALS A D ESCRIPTIVE Q UESTION S UB - TYPES AND S TATISTICS

Original: Descriptive questions in CLEVRER consists of ﬁve sub-types.
Revised: Descriptive questions in CLEVRER consists of ﬁve sub-types.

Original: When generating the questions, we balance the frequency of the answers within each question sub-type to reduce bias.
Revised: When generating the questions, we balance the frequency of the answers within each question sub-type to reduce bias.

Original: The sub-types and their answer space distribution are shown in Figure 6.
Revised: The sub-types and their answer space distribution are shown in Figure 6.

Original: [Figure]
Revised: [Figure]

Original: Query material
Revised: Query material

Original: [Figure]
Revised: [Figure]

Original: Query color gray , brown , green , red , blue , purple , yellow , cyan
Revised: Query color gray , brown , green , red , blue , purple , yellow , cyan

Original: [Figure]
Revised: [Figure]

Original: B NS-DR M ODEL D ETAILS AND T RAINING P ARADIGM
Revised: B NS-DR M ODEL D ETAILS AND T RAINING P ARADIGM

Original: Here we present further details of the NS-DR model and the training paradigm.
Revised: Here we present further details of the NS-DR model and the training paradigm.

Original: A running example of the model can be found in Figure 7.
Revised: A running example of the model can be found in Figure 7.

Original: Video frame parser.
Revised: Video frame parser.

Original: Our frame parser is trained on 4,000 frames randomly selected from the training videos plus ground-truth masks and attribute annotations of each object in the frames.
Revised: Our frame parser is trained on 4,000 frames randomly selected from the training videos plus ground-truth masks and attribute annotations of each object in the frames.

Original: We train the model for 30,000 iterations with stochastic gradient decent, using a batch size of 6 and learning rate 0.001.
Revised: We train the model for 30,000 iterations with stochastic gradient decent, using a batch size of 6 and learning rate 0.001.

Original: At test time, we keep the object proposals with a conﬁdence score of > 0 .
Revised: At test time, we keep the object proposals with a conﬁdence score of > 0 .

Original: 9 and use those to form the object-level abstract representation of the video.
Revised: 9 and use those to form the object-level abstract representation of the video.

Original: Neural dynamics predictor.
Revised: Neural dynamics predictor.

Original: The object encoder f enc O and the relation encoder f enc R in PropNet are instantiated as convolutional neural networks and output a D enc -dim vector as the representation.
Revised: The object encoder f enc O and the relation encoder f enc R in PropNet are instantiated as convolutional neural networks and output a D enc -dim vector as the representation.

Original: We add skip connections between the object encoder f enc O and the predictor f pred O for each object to generate higher quality images.
Revised: We add skip connections between the object encoder f enc O and the predictor f pred O for each object to generate higher quality images.

Original: We also include a relation predictor f pred R to determine whether two objects will collide or not in the next time step.
Revised: We also include a relation predictor f pred R to determine whether two objects will collide or not in the next time step.

Original: At time t , we ﬁrst encodes the objects and relations c oi,t = f enc O
Revised: At time t , we ﬁrst encodes the objects and relations c oi,t = f enc O

Original: ( o i,t ) , c rk,t = f enc R ( o u k ,t , o v k ,t , a rk ) , where o i,t denotes object i at time t .
Revised: ( o i,t ) , c rk,t = f enc R ( o u k ,t , o v k ,t , a rk ) , where o i,t denotes object i at time t .

Original: We then denotes the propagating effect from relation k at propagation step l as e lk,t , and the effect from object i as h li,t .
Revised: We then denotes the propagating effect from relation k at propagation step l as e lk,t , and the effect from object i as h li,t .

Original: For step 1 ≤ l ≤ L , the propagation procedure can be described as
Revised: For step 1 ≤ l ≤ L , the propagation procedure can be described as

Original: [Equation]
Revised: [Equation]

Original: where f O denotes the object propagator, f R denotes the relation propagator, and N i denotes the relation set where object i is the receiver.
Revised: where f O denotes the object propagator, f R denotes the relation propagator, and N i denotes the relation set where object i is the receiver.

Original: The neural dynamics predictor is trained by minimizing the
Revised: The neural dynamics predictor is trained by minimizing the

Original: L 2 distance between the predicted ˆ r k,t +1 , ˆ o i,t +1 and the real future observation r k,t +1 , o i,t +1 using stochastic gradient descent.
Revised: L 2 distance between the predicted ˆ r k,t +1 , ˆ o i,t +1 and the real future observation r k,t +1 , o i,t +1 using stochastic gradient descent.

Original: [Figure]
Revised: [Figure]

Revised: Gray

Revised: Yellow

Original: [Figure]
Revised: Red Red Material Rubber Metal

Revised: Rubber

Revised: Rubber

Original: [Figure] 
Revised: [Figure] Shape Cylinder Cylinder

Revised: Sphere

Revised: Sphere

Revised: Sphere

Revised: Events Mode

Revised: Observation

Revised: Pred.

Revised: CF.

Revised: Frame

Original: [Figure]
Revised: [Figure]

Revised: Collision

Revised: Enter

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: Question: Which event will happen next?
Revised: Question: Which event will happen next?

Original: Choice: The red rubber sphere collides with the metal sphere
Revised: Choice: The red rubber sphere collides with the metal sphere

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (filter_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ), Sphere )),

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (filter_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ), Sphere )), UnseenEvents )
Revised: )), UnseenEvents )

Original: Answer:
Revised: Answer:

Original: Question: What shape is the first object to collide with the cyan object?
Revised: Question: What shape is the first object to collide with the cyan object?

Original: Program: query_
Revised: Program: query_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: _col_
Revised: _col_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ( Objects
Revised: ( Objects

Original: [Figure]
Revised: [Figure]

Original: Cyan ))
Revised: Cyan ))

Original: [Figure]
Revised: [Figure]

Revised: First )

Revised: [Figure]

Revised: ( Objects

Original: [Figure]
Revised: [Figure]

Revised: Cyan )))

Original: Answer: Sphere
Revised: Answer: Sphere

Original: Question: Which event will happen without the cyan
Revised: Question: Which event will happen without the cyan

Original: [Figure]
Revised: [Figure]

Original: Choice: The red 
Revised: Choice: The red rubber sphere collides with the yellow sphere

Original: [Figure]
Revised: [Figure]

Original: sphere collides with the yellow sphere
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Answer:
Revised: Answer:

Original: The output of the neural dynamics predictor is (cid:104){ ˆ O t } t =1 ...T , { ˆ R t } t =1 ...T (cid:105) , the collection of object states and relations across all the observed and rollout frames.
Revised: The output of the neural dynamics predictor is (cid:104){ ˆ O t } t =1 ...T , { ˆ R t } t =1 ...T (cid:105) , the collection of object states and relations across all the observed and rollout frames.

Original: From this representation, one can recover the full motion and event traces of the objects under different rollout conditions and generate a dynamic scene representation of the video.
Revised: From this representation, one can recover the full motion and event traces of the objects under different rollout conditions and generate a dynamic scene representation of the video.

Original: For example, if we want to know what will happen if an object is removed from the scene, we just need to erase the corresponding vertex and associated edges from the graph and rollout using the learned dynamics to obtain the traces.
Revised: For example, if we want to know what will happen if an object is removed from the scene, we just need to erase the corresponding vertex and associated edges from the graph and rollout using the learned dynamics to obtain the traces.

Original: Our neural dynamics predictor is trained on the proposed object masks and attributes from the frame parser.
Revised: Our neural dynamics predictor is trained on the proposed object masks and attributes from the frame parser.

Original: We ﬁlter out inconsistent object proposals by matching the object intrinsic attributes across different frames and keeping the ones that appear in more than 10 frames.
Revised: We ﬁlter out inconsistent object proposals by matching the object intrinsic attributes across different frames and keeping the ones that appear in more than 10 frames.

Original: For a video of length 125, we will sample 5 rollouts, where each rollout contains 25 frames that are uniformly sampled from the original video.
Revised: For a video of length 125, we will sample 5 rollouts, where each rollout contains 25 frames that are uniformly sampled from the original video.

Original: We normalize the input data to the range of [ − 1 , 1] and concatenate them over a time window of size 3.
Revised: We normalize the input data to the range of [ − 1 , 1] and concatenate them over a time window of size 3.

Original: We set the propagation step L to 2, and the dimension of the propagating effects
Revised: We set the propagation step L to 2, and the dimension of the propagating effects

Original: D enc to 512.
Revised: D enc to 512.

Original: We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 10 − 4 , and a decay factor of 0.3 per 3 epochs.
Revised: We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 10 − 4 , and a decay factor of 0.3 per 3 epochs.

Original: The model is trained for 9 epochs with batch size 2.
Revised: The model is trained for 9 epochs with batch size 2.

Original: Question parser.
Revised: Question parser.

Original: For open-ended descriptive questions, the question parser is trained on various numbers of randomly selected question-program pairs.
Revised: For open-ended descriptive questions, the question parser is trained on various numbers of randomly selected question-program pairs.

Original: For multiple choice questions, we separately train the question parser and the choice parser.
Revised: For multiple choice questions, we separately train the question parser and the choice parser.

Original: Training with n samples means training the question parser with n questions randomly sampled from the multiple choice questions (all three types together), and then training the choice parser with 4 n choices sampled from the same pool.
Revised: Training with n samples means training the question parser with n questions randomly sampled from the multiple choice questions (all three types together), and then training the choice parser with 4 n choices sampled from the same pool.

Original: All models are trained using Adam (Kingma & Ba, 2015) for 30,000 iterations with batch size 64 and learning rate 7 × 10 − 4 .
Revised: All models are trained using Adam (Kingma & Ba, 2015) for 30,000 iterations with batch size 64 and learning rate 7 × 10 − 4 .

Original: Program executor.
Revised: Program executor.

Original: A comprehensive list of all modules in the NS-DR program executor is summarized in Table 4 and Table 5.
Revised: A comprehensive list of all modules in the NS-DR program executor is summarized in Table 4 and Table 5.

Original: The input and output data types of each module are summarized in
Revised: The input and output data types of each module are summarized in

Original: Table 6.
Revised: Table 6.

Original: C E XTRA E XAMPLES FROM CLEVRER
Revised: C E XTRA E XAMPLES FROM CLEVRER

Original: We show extra examples from CLEVRER in Figure 8.
Revised: We show extra examples from CLEVRER in Figure 8.

Original: We also present visualizations of question/choice programs of different question types in Figure 9 through Figure 11.
Revised: We also present visualizations of question/choice programs of different question types in Figure 9 through Figure 11.

Original: The full datasetwill be
Revised: The full dataset

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Filter color(
Revised: Filter color

Original: [Table]

Original: , color ) objects

Original: Selects

Original: [Table]
Revised: [Table]

Original: color Filter material
Revised: Filter material

Original: [Table]
Revised: [Table]

Original: material
Revised: Selects

Original: [Table]
Revised: [Table]

Original: objects
Revised: Object

Original: Selects objects from the input list with the input material Object

Original: Filter shape ( objects , shape ) objects
Revised: Filter shape ( objects , shape ) objects

Original: Filter
Revised: Filter

Original: Selects objects from the input list with the input shape Modules
Revised: Selects objects from the [Table] list with the [Table]

Revised: shape Modules

Original: Filter move ( objects , frame ) objects
Revised: Filter moving ( objects , frame ) objects

Original: Selects all moving objects in the input frame Filter stationary ( objects , frame ) objects
Revised: Selects all moving objects in the [Table] frame or the entire video (when [Table]

Original:  Selects all stationary objects in the given frame 
Revised: frame is [Table] Filter stationary ( objects , frame ) objects Selects all stationary objects in the input frame or the entire video (when

Revised: [Table]

Revised: frame is “null”)

Original: Filter in ( events , objects ) events
Revised: Filter in ( events , objects ) events

Original: Selects all incoming events of the input objects Filter out ( events , objects ) events 
Revised: Selects all incoming events of the input objects Filter out ( events , objects ) events Selects all exiting events of the

Revised: [Table]

Original: Selects all exiting events of the input objects Filter collision ( events , objects ) events Selects all collisions that involve any of the input objects Filter before ( events , event ) events
Revised: objects Filter collision ( events , objects ) events Selects all collisions that involve any of the input objects Filter before ( events , event ) events

Original: Selects all events before the target event Filter after ( events , event ) events
Revised: Selects all events before the target event Filter after ( events , event ) events

Original: Selects all events after the target event Event
Revised: Selects all events after the target event Event

Original: Filter order ( events , order ) event
Revised: Filter order ( events , order ) event

Original: Filter
Revised: Filter

Original: Selects the event at the speciﬁc time order Modules
Revised: Selects the event at the speciﬁc time order Modules

Original: Filter ancestor  event  eventsReturns all ancestors of the [Table]
Revised: Filter ancestor ( events , event ) events

Original: in the causal graph Get frame event frameReturns the frame of the
Revised: Selects all causal ancestors of the input event Get frame event frame

Original: [Table] in the video Get counterfact object events Returns all eventsafter removing the
Revised: Returns the frame of the input event in the video Get counterfact ( events , object ) events

Original: [Table]

Original:  object  Get col partner ( event , object ) objectReturns the collision partner of the
Revised: Selects all events that would happen if the object is removed Get col partner ( event , object ) object

Original: [Table] object (the [Table] event must be a collision)
Revised: Returns the collision partner of the input object (the input event must be a collision)

Original: Get object event object
Revised: Get object event object

Original: Returns the

Original: [Table]

Original: that participates in the

Original: [Table] event (the [Table] must be a incoming / outgoing event)
Revised: Returns the object that participates in the input event (the input event must be a incoming / outgoing event)

Original: Unique events / event /
Revised: Unique events / event /

Original: Returns the only event / object in the input list objects object
Revised: Returns the only event / object in the input list objects object

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ),Filter_color(Objects, "Green") ) )
Revised: ),Filter_color(Objects, "Green") ) )

Original: Question: "Are there any stationary spheres when the metal sphere enters the scene?"
Revised: Question: "Are there any stationary spheres when the metal sphere enters the scene?"

Original: Exist(Filter_stationary(
Revised: Exist(Filter_stationary(

Original: Filter_shape(Objects, "Sphere"), Query_frame(
Revised: Filter_shape(Objects, "Sphere"), Query_frame(

Original: Filter_in(Events,
Revised: Filter_in(Events,

Original: Filter_shape(Filter_material(Objects, "Metal"), "Sphere" ) ) ) ) )
Revised: Filter_shape(Filter_material(Objects, "Metal"), "Sphere" ) ) ) ) )

Original: Question: "Which of the following is not responsible for the metal sphere's colliding with the rubber sphere?"
Revised: Question: "Which of the following is not responsible for the metal sphere's colliding with the rubber sphere?"

Original: Negate(Belong_to(
Revised: Negate(Belong_to(

Original: Event, // Output of the Choice program Filter_ancestor(
Revised: Event, // Output of the Choice program Filter_ancestor(

Original: Filter_collision(Filter_collision(
Revised: Filter_collision(Filter_collision(

Original: Events,Filter_shape(
Revised: Events,Filter_shape(

Original: Filter_material(Objects, "Metal"), "Sphere" ) ),
Revised: Filter_material(Objects, "Metal"), "Sphere" ) ),

Original: Filter_shape(Filter_material(Objects, "Rubber"), "Sphere" ) ) ) ) )
Revised: Filter_shape(Filter_material(Objects, "Rubber"), "Sphere" ) ) ) ) )

Original: Choice: "The collision between the green cube and the rubber sphere"
Revised: Choice: "The collision between the green cube and the rubber sphere"

Original: Filter_collision(Filter_collision(
Revised: Filter_collision(Filter_collision(

Original: Events,Filter_shape(
Revised: Events,Filter_shape(

Original: Filter [Figure] (Objects, "Cube"), "Cube" ) ),
Revised: Filter_color(Objects, "Cube"), "Cube" ) ),

Original: Filter_shape(Filter_material(Objects, "Rubber"), "Sphere" ) )
Revised: Filter_shape(Filter_material(Objects, "Rubber"), "Sphere" ) )

Original: Choice: "
Revised: Choice: "

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: cube's entering the scene"
Revised: cube's entering the scene"

Original: [Figure]
Revised: [Figure]

Original: in(

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (Objects, "
Revised: (Objects, "

Original: [Figure]
Revised: [Figure]

Original: "), "Cube"

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Counterfactual question: "Which event will happen if the rubber sphere is removed?"
Revised: Counterfactual question: "Which event will happen if the rubber sphere is removed?"

Original: Belong_to(Event, //
Revised: Belong_to(Event, //

Original: Output of the choice program
Revised: Output of the choice program

Original: Get_counterfact(Filter_shape(
Revised: Get_counterfact(Filter_shape(

Original: Filter_material(Objects, "Rubber"), "Sphere" ) ) )
Revised: Filter_material(Objects, "Rubber"), "Sphere" ) ) )

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Choice: "The metal sphere and the cylinder collide" [Figure] [Figure]
Revised: Choice: "The metal sphere and the cylinder collide" Filter_collision(

Original: [Figure] [Figure] Filter_shape(
Revised: Filter_collision(AllEvents,

Original: [Figure] _material(Objects, "Metal"), "Sphere" ) ), [Figure] shape(Objects, "Cylinder") )
Revised: Filter_shape(Filter_material(Objects, "Metal"), "Sphere" ) ),Filter_shape(Objects, "Cylinder") )

Original: Choice: "The cube collides with the metal sphere" Filter_collision(
Revised: Choice: "The cube collides with the metal sphere" Filter_collision(

Original: Filter_collision(AllEvents,
Revised: Filter_collision(AllEvents,

Original: Filter_shape(Objects, "Cube") ), Filter_shape(Filter_material(Objects, "Metal"), "Sphere" ) )
Revised: Filter_shape(Objects, "Cube") ), Filter_shape(Filter_material(Objects, "Metal"), "Sphere" ) )