CRIPP - VQA : Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering (2022)

Abstract: Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce CRIPP-VQA, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that involve counterfactual reasoning about the effect of actions, questions about planning in order to reach a goal, and descriptive questions about visible properties of objects. The CRIPP-VQA test set enables evaluation under several out-of-distribution settings – videos with objects with masses, coefficients of friction, and initial velocities that are not observed in the training distribution. Our experiments reveal a surprising and significant performance gap in terms of answering questions about implicit properties (the focus of this paper) and explicit properties of objects (the focus of prior work).

## introduction visual grounding seeks to link images or videos with natural language. towards this goal, many tasks such as referring expressions @xcite , captioning @xcite ), text-based retrieval @xcite , and visual question answering @xcite have been studied for both images and videos. videos often contain objects which can be identified in terms of their visible properties such as their shapes, sizes, colors, textures, and categories. these visible properties can be estimated by using computer vision algorithms for object recognition, detection, color recognition, figure 1 : the cripp-vqa dataset contains questions about the future effect of actions (such as removing, adding, or replacing objects) as well as planning-based questions. frames from an example video are shown above with the red highlighted area depicting the objects on which actions (remove, replace, add) are performed. collisions between objects, however, do offer visual cues about mass and friction. when objects collide, their resulting velocities and directions of motion depend upon their physical properties, and are governed by fundamental laws of physics such as conservation of momentum and energy. by observing the change in velocities and directions, it is possible to reason about the relative physical properties of colliding objects. in many cases, when humans watch objects in motion and under collision, we do not accurately know the masses, friction, or other properties of objects. yet, when we interact with these objects, for example in sports such as billiards, carrom, or curling, we can reason about the effect of actions such as hitting one ball with another, removing an object, replacing an object with a different one, or adding an object to the scene. in this paper, we consider the task of reasoning about such implicit properties of objects, via the use of language, without having annotations for the true values of mass and friction of objects. we propose a video question answering dataset called cripp-vqa, short for counterfactual reasoning about implicit physical properties. each video contains several objects with at least one object in motion. the object in motion causes collisions and changes the spatial configuration of the scene. the consequences of these collision are directly impacted by the physical properties of objects. cripp-vqa contains videos annotated with question-answer pairs, where the questions are about the consequences of actions and collisions, as illustrated in figure 1 . these questions require an understanding of the current configuration as well as counterfactual situations, i.e. the effect of actions such as removing, adding, and replacing objects. the dataset also contains questions that require the ability to plan in order to achieve certain configurations, for example producing or avoiding particular collisions. it is important to note that both tasks can not be performed without an understanding of the relative mass. for example, the "replace" action can lead to a change in mass inside the reference video, which can drastically change the consequences (i.e., set of collisions). we benchmark existing state-of-the-art video question-answering models on the new cripp-vqa dataset. our key finding is that compared to performance on questions about visible properties ("descriptive" questions), the performance on counterfactual and planning questions is significantly low. this reveals a large gap in under-standing the physical properties of objects from video and language supervision. detailed analysis reveals that models can answer questions about the first collision with higher accuracy compared to questions about subsequent future collisions. aloe @xcite is a strong baseline for video qa tasks and has improved the state of the art on many previous video qa benchmarks such as clevrer @xcite and cater @xcite . however on cripp-vqa, we discovered that the object identification module from aloe failed to recognize objects in our videos, which we believe is due to the presence of complex textures, reflections, and shadows in our dataset. to mitigate these failures, we modified aloe by adapting the mask-rcnn @xcite as the object segmentation module. we also found that using pre-trained bert-based word embeddings significantly improves the performance over our modified aloe (aloe * ), serving as the strongest model on cripp-vqa. cripp-vqa also allows us to evaluate trained models on out-of-distribution (ood) test sets, where the videos vary in terms of objects having previously unobserved physical properties. there are four ood test sets in cripp-vqa such that one physical property varies at test time -objects with a new mass, zero friction coefficient, increased initial velocity, and two moving objects at initialization. this ood evaluation reveals a further degradation in performance and a close-to-random accuracy for most state-of-the-art models. out of all ood scenarios the results show that the most challenging scenario is the one where there are two objects initially moving. ## analysis in this section, we raise several important questions and derive the insights accordingly. performance for true vs. false collision detection. consider the example with three objects (a,b,c), where only object a collides with b. in this case, we categorize the collision between a & b as the actual collision ( i.e., prediction label true), and we categorize the collision between b & c and a & c as an absent collision (i.e., prediction label false). following this rule, we independently check the performance of detecting all occurring collisions and the collisions that never happened. ## conclusion in this work, we present a new video question answering benchmark: cripp-vqa, for reasoning we evaluate state-of-the-art models on this benchmark and observe a significant performance gap between descriptive questions about visible properties and counterfactual and planning questions about implicit properties. we also show that models can learn the initial dynamics of object trajectories but they fail to detect subsequent collisions, which requires an understanding of relative mass. this result is positioned as a challenge for the v&l community for building robust video understanding systems that can interact with language.