ACQUIRED : A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023)

Abstract: Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems. It involves predicting the outcomes of hypothetical circumstances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only a few datasets targeting the counterfactual reasoning abilities of multimodal models. Among them, they only cover reasoning over synthetic environments or specific types of events (e.g. traffic collisions), making them hard to reliably benchmark the model generalization ability in diverse real-world scenarios and reasoning dimensions. To overcome these limitations, we develop a video question answering dataset, ACQUIRED: it consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints, which ensures a focus on real-world diversity. In addition, each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal, which can comprehensively evaluate the model counterfactual abilities along multiple aspects. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap (>13%) between models and humans. The findings suggest that multimodal counterfactual reasoning remains an open challenge and ACQUIRED is a comprehensive and reliable benchmark for inspiring future research in this direction.

## introduction multimodal counterfactual reasoning refers to the ability to imagine and reason about what might have happened if certain conditions were different * the authors contribute equally. if he tested the weight of the dolly before pushing it, it wouldn't have dragged him down the ladder. if he tested the weight of the dolly before pushing it, he would have pushed it harder down the ladder. what if the man tested the weight of the dolly before pushing it? figure 1 : the acquired dataset is a video question answering (qa) dataset that specifically focuses on counterfactual reasoning on diverse real-world events. our dataset concerns three types of commonsense reasoning dimensions: physical, social, and temporal, and encompasses videos from both third-person (upper) and first-person (lower) viewpoints. each question is curated with a correct and a distractor answer. each answer is by itself individually judgeable, and hence our dataset can be approached in either binary true/false or multiple-choice setting. from what actually occurred based on vision and language inputs. it involves mentally simulating alternative scenarios and evaluating their potential outcomes. this cognitive process plays a crucial role in human intelligence, as it allows us to understand causality, make predictions, and learn from past experiences. for ai models, developing the capacity for counterfactual reasoning is a significant area of research and a challenging task. by enabling ai models to engage in counterfactual reasoning, we can enhance their understanding of causal relationships and their ability to assess the impact of interventions or changes in conditions. however, despite the significance of counterfactual reasoning, it remains a relatively unexplored area of research. to assess the overall reasoning capabilities of models, several visual question answering datasets have been proposed on both images @xcite and videos @xcite . these datasets require reasoning skills such as commonsense reasoning, extracting human/object-to-object relations, and inferring physical properties. one specific dataset in the realm of counterfactual reasoning is clevrer @xcite , which generates synthetic videos and associated questions in a controlled environment, featuring simulated object motion and rendered video frames. this dataset allows for evaluating models using descriptive, explanatory, predictive, and counterfactual questions, covering a wide range of reasoning scenarios. however, the data generation process in clevrer is overly synthetic, limiting its effectiveness to assess models' counterfactual reasoning abilities in realistic contexts. to address this limitation, trafficqa @xcite focuses on real-world traffic event cognition and reasoning in videos, specifically targeting scenarios like traffic accidents. it leverages crowdsourcing to gather diverse types of questions, including fundamental comprehension, counterfactual inference, and event forecasting. nevertheless, because trafficqa concentrates solely on traffic events, it fails to encompass other real-life events, resulting in a substantial domain gap between trafficqa and general video datasets such as kinetics @xcite and youtube @xcite . in this paper, we construct a benchmark that can evaluate the counterfactual reasoning abilities of visual models on various kinds of real-world events. we introduce acquired 1 that covers multiple dimensions of counterfactual reasoning and includes videos of both egocentric and exocentric views. specifically, based on videos in both oops @xcite and ego4d @xcite , we crowd-source 11k questions over 3.7k videos targeting physical, temporal, and social counterfactual reasoning. both the oops and ego4d datasets consist of human activities and interactions in nu-1 abbreviation of: answering counterfactual questions in real-life videos merous settings, making them ideal sources for curating video question answering datasets. in addition, many videos contain unintentional human actions (e.g., the person accidentally falling down the ladder in figure 1 ), which naturally enables people to come up with diverse what-if questions. inspired by @xcite , we adopt a similar methodology for gathering counterfactual questions. each question consists of a pair of answers, with one being the correct response and the other serving as a distractor. importantly, the distractor answer represents a minimal contrastive counterpart to the correct answer. as we can see from examples in figure 1 , the design of using complementary pairs requires the model to understand the subtle differences between different options, which ensures that the model exhibits an intuitive grasp of counterfactual reasoning. in addition, having one distractor for each question allows for testing models in either true/false or multiple-choice setting. we extensively evaluate numerous strong language models such as gpt-4, as well as state-ofthe-art video-language models such as valor on our acquired dataset. the experimental results suggest that models struggle to effectively utilize the video contexts and perform counterfactual reasoning, with multimodal models achieving only comparable and sometimes inferior performance than language-only models. moreover, the significant gap between the human and model (>13%) performance highlights the challenging nature of our task and room for improvements in visual counterfactual reasoning. ## related work we will overview three lines of relevant research to this work: visual question answering, visual understanding models, and counterfactual reasoning. ## benchmarking models we benchmark our dataset with both state-ofthe-art language-only and vision-language models. specifically, we perform experiments with de-berta @xcite , unifiedqa @xcite , violet @xcite , valor @xcite , and vl-adapter @xcite on our dataset. language-only models. while acquired is a multimodal dataset that has both vision and language inputs, previous works @xcite have pointed out that unimodal models can sometimes achieve surprisingly strong performance because of the annotation bias. therefore, we evaluate both deberta-v3 @xcite and the unifiedqa model family @xcite (state-of-the-art question answering models based on the t5 architecture @xcite ) on our dataset, which can reflect the dataset biases and provide an important reference point for multimodal models. the language-only models answer the textual questions without looking at the videos. inspired by the superior performance of the recent large language models, i.e., the gpt model from openai, we also evaluate its zero-shot performance on the textual parts of our dataset. specifically, we consider both chatgpt (openai, 2023a) and gpt-4 (openai, 2023b). in addition, we further include a version of gpt models that can condition on pre-annotated descriptions describing the general contents of the videos, to serve as the pseudo visual (and situated) contexts of the questions. details on how we prompt the gpt family are in append. sec. c.1. violet @xcite . violet is a videolanguage model that has three components, including a video encoder (swin transformer-base @xcite ), a language encoder (bert-base (de-vlin et al., 2019)), and a cross-modal transformer module that performs cross-modal fusion. the video and language encoders extract features from the video and language inputs respectively, and the extracted features are then fed into the crossmodal transformer for cross-modal interactions. violet is pretrained on large-scale video-text data with masked language modeling that predicts the original word tokens given the masked inputs, masked visual-token modeling (mvm) that recovers the masked video patches conditioned on the unmasked video and language inputs, visual-text matching that aims to align the paired video-text inputs between video and text modality. valor @xcite . valor is a recently proposed multimodal model that can take video, language, as well as audio as inputs. similar to violet, valor also first encodes vision, audio, and text inputs separately, and the encoded features are then fed into a multimodal decoder for text generation. valor demonstrates strong performance across a wide range of tasks, including video retrieval, video captioning, video question answering, audio-visual captioning, text-to-audio retrieval, and audio captioning. vl-adapter @xcite . vl-adapter uses a pretrained vision encoder (e.g. clip @xcite ) to extract vision features and feed the vision features as well as text tokens to a pretrained language model (e.g. t5 @xcite ) so that the model can take both vision and language information. when adapting the model for downstream tasks, because it can be costly to finetune all the model parameters, vl-adapter investigates different adapter-based parameter-efficient finetuning strategies and demonstrates that training the adapter allows them to only update a rather small portion (e.g. 4%) of total parameters and match the performance of finetuning the entire model. because vl-adapter supports different combinations of pretrained vision and language encoders, we employ different versions of clip-vit-b/16 and unifiedqa-large as the vision and text encoders. ## conclusions in this work, we present a novel counterfactualreasoning-focused video question answering dataset, named acquired. the dataset provides questions about counterfactual hypotheses over visual events (videos). we collect a correct and a distractor answer for three commonsense reasoning dimensions: physical, social, and temporal. we benchmarked various state-of-the-art language models (including llms like gpt) and videolanguage models on the collected dataset, where the results demonstrate algorithm performance well below human performance (>13% accuracy). we hope our studies and the collected acquired dataset can spur relevant future research, specifically on testing multimodal models' capabilities in counterfactual reasoning, devising assistive ai for remedial and/or cause estimation of observed failures, and more sophisticated visual event understanding and reasoning.