CRAFT : A Benchmark for Causal Reasoning About Forces and in T eractions (2022)

Abstract: Humans are able to perceive, understand and reason about causal events. Developing models with similar physical and causal understanding capabilities is a long-standing goal of artificial intelligence. As a step towards this direction, we introduce CRAFT, a new video question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories in CRAFT include previously studied descriptive and counterfactual questions. Additionally, inspired by the Force Dynamics Theory in cognitive linguistics, we introduce a new causal question category that involves understanding the causal interactions between objects through notions like cause, enable, and prevent. Our results show that even though the questions in CRAFT are easy for humans, the tested baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark.

## introduction causal reasoning is a key cognitive capability that involves making predictions about physical objects and their interactions. cognitive scientists have mainly studied causal reasoning as simple causes or chains of events @xcite @xcite , rather than processing of complex causal scenes, see @xcite . referring to the interactions of multiple forces, the force dynamics theory emphasizes the processing and reasoning of complex scenes, and how causal language defines the patterns of forces in causal events @xcite . in the past decade, though artificial learning systems have shown astonishing progress in natural language and image understanding, there are some tasks in which these systems are still significantly below human performance. one such challenging research area includes reasoning about physical actions of objects in complex causal scenes. in this paper, we explore how language and vision interact with each other in making plausible projections about causal reasoning, and analyze how well the existing neural models understand and reason about physical and causal relationships between dynamic objects in a scene through images and text. we propose a new video question answering dataset, named craft (causal reasoning about forces and interactions), which is designed to be complex for artificial models and simple for humans. our dataset contains synthetically generated videos of 2d scenes with accompanying questions. its most prominent features are that it contains video clips with complex physical interactions between objects, and questions that test strong reasoning capabilities. answering our causal questions needs comprehending what is being asked, identifying objects in the scene, tracking their states in relation to other objects, which in turn can be attributed to different semantic categories of causes (cause, enable or prevent) that highlight unique patterns of causal forces in events -in line with the force dynamics theory. in craft, there are also some descriptive and counterfactual questions, the latter requiring understanding what would have happened after an intervention, i.e. a slight change in the scene @xcite . figure 1 shows sample questions from different question types, which are explained in detail in the subsequent sections. ## the craft dataset craft is built to evaluate temporal and causal reasoning capabilities of existing algorithms on video clips of 2d simulations and related questions. the dataset has approximately 57k question and video pairs, which are created from 10k videos. it is split into train, validation, and test sets with a 60:20:20 ratio per video basis, meaning that video clips in the training set are not seen in the validation or test set. moreover, we have two different settings, an easy setting and a hard setting. they differ from each other in the way how the test split is chosen. in the hard setting, we deliberately use scene layouts that are not seen during training in picking the video and question pairs. the easy setting does not have this constraint. in the easy setting, there are 35k, 12k, and 11k question and video pairs in the train, validation and test splits, whereas in the hard setting these numbers are 35k, 11k and 12k, respectively. we provide an example set of questions from craft in figure 1 . video generation. we use box2d physics simulator @xcite to create our virtual scenes. there are 20 distinct scene layouts from which 10 seconds of video clips are collected with a spatial resolution of 256 × 256 pixels. besides generating original simulation video, craft scripts also generate variation videos by removing each object of the same video from the scene. these variation videos help question generation script to provide answer for certain types of questions, as explained later. objects. each scene is composed of both static scene elements and dynamic objects, containing variable number of and different type of these elements and objects. there are 7 static scene elements (ramp, platform, button, basket, left wall, right wall, ground). these elements are all drawn in black color in order to differentiate them from the dynamic objects. their attributes such as position or orientation are decided at the beginning of a simulation and then they are kept fixed throughout the video sequence. the values of these attributes are assigned randomly from sets of different intervals which are predefined for each type of scene as in figure 2 . the set of the dynamic objects contains 3 shapes (cube, triangle, circle), 2 sizes (small, large), and 8 colors (gray, red, blue, green, brown, purple, cyan, yellow). attributes of dynamic objects, in contrast, are in continuous change throughout the sequence due to the gravity or the interactions that they are subject to, until they rest. events. to formally represent the dynamical interactions in the simulations, we extract different types of events: start, end, collision, touch start, touch end, and enter basket. start and end events represent the start and the end of the simulations, respectively. although we mainly question collision events in our tasks, we want models to understand the difference between a collision and rolling on a ramp or a platform or two objects moving together. therefore, we also extract touch start, touch end events. finally, enter basket event is triggered if the object enters the basket in the scene. all events happening a simulation are represented as a causal graph, which is also key for the question generator to extract causal relationships in an easy manner. causal graph is a directed graph where events are represented as nodes. each edge represents a cause relation where the source event is considered as the cause of target event because of the shared objects between them. we demonstrate the causal graph of a sample simulation in figure 3 . ## experimental analysis in this section, we evaluate the performances of a wide range of baseline models on craft. we also analyze how these performances relate with that of humans in understanding physical interactions between the objects and the environment. ## conclusion we have presented craft, a new vqa dataset to test causal reasoning capabilities of the current models. motivated by the force dynamics theory, which highlights distinct causative verbs, craft requires models to perform temporal and causal reasoning and even to imagine alternative versions of the events occurring in videos. our results demonstrate that, while human adults can reason about the physical interactions between objects, these questions cannot be solved reliably by current models. at present, there is substantial room for improvement compared to humans. in our experiments, we did not report the results of recent neuro-symbolic models, e.g. ns-dr @xcite . such ap-proaches are very compelling and worth pursuing, but they currently require extra object-level annotations. another exciting direction is to test objectcentric methods other than g-swm. however, it seems that they might require extra pretraining or self-supervised objectives, as explored by @xcite . we believe that developing more effective models for craft is an exciting research direction for video qa systems to mimic humans in causal reasoning about forces and interactions.