C o SI m: Commonsense Reasoning for Counterfactual Scene Imagination (2022)

Abstract: As humans, we can modify our assumptions about a scene by imagining alternative objects or concepts in our minds. For example, we can easily anticipate the implications of the sun being overcast by rain clouds (e.g., the street will get wet) and accordingly prepare for that. In this paper, we introduce a new dataset called Commonsense Reasoning for Counterfactual Scene Imagination (CoSIm) which is designed to evaluate the ability of AI systems to reason about scene change imagination. To be specific, in this multimodal task/dataset, models are given an image and an initial question-response pair about the image. Next, a counterfactual imagined scene change (in textual form) is applied, and the model has to predict the new response to the initial question based on this scene change. We collect 3.5K high-quality and challenging data instances, with each instance consisting of an image, a commonsense question with a response, a description of a counterfactual change, a new response to the question, and three distractor responses. Our dataset contains various complex scene change types (such as object addition/removal/state change, event description, environment change, etc.) that require models to imagine many different scenarios and reason about the changed scenes. We present a baseline model based on a vision-language Transformer (i.e., LXMERT) and ablation studies. Through human evaluation, we demonstrate a large human-model performance gap, suggesting room for promising future work on this challenging, counterfactual multimodal task.

## introduction anticipating what would happen when there is a condition change in a situation is an important ability as it allows preparation for the implications of the change. for example, when swimming in the sea on a clear day, you might feel safe. however, if someone told you a storm warning has been issued and dark clouds are coming in soon, you would know that it is no longer safe to swim and return to land. it will be also very useful to have ai systems that could reason about the implications of such scenario changes and provide appropriate guidance/warnings; however, current ai systems will have a hard time performing such counterfactual commonsense reasoning. many efforts have been made to teach machines how to reason about images @xcite @xcite and videos @xcite @xcite . this area has been built upon further by efforts to teach machines to use commonsense knowledge when analyzing visual scenes @xcite @xcite . through these efforts, many ai systems have reached near human-level performance on scene understanding tasks. on the other hand, more complex reasoning on scene 'changes' has been less explored. @xcite applies simple condition manipulations (e.g., "paint the small green ball with cyan color") on synthetic images in a visual question answering setup. however, this task is based on simple block objects that might not require complex implicit reasoning. thus, complicated counterfactual commonsense reasoning on scene changes on real-world images and situations remains widely unexplored. therefore, in this paper, we introduce a new dataset called commonsense reasoning for counterfactual scene imagination (cosim) to evaluate the commonsense reasoning ability of agents about counterfactual visual scenes imagined via textual descriptions. to be specific, one data instance in our cosim dataset consists of an image, an initial question-response pair, an imagined is the railway line safe? yes because there are safety lights and crosswalk signs add a train to the tracks. new response: no. although there are safety lights and crossing gates, they don't appear to be working and there is a train coming. distractor #1: yes. there are safety lights and crossing gates, they appear to be working and there is a train coming and it will stop. distractor #2: yes. although there are safety lights and crossing gates, they don't appear to be working and there is no train coming. distractor #3: no. although there are safety lights and crossing gates, there is a power outage and there is a train coming. visual scene change, and a new response with three distractors. the question is about commonsense which can be inferred from the image and the initial response includes a reasoning/justification for its answer. the imagined visual scene change is a textual description of what to modify in the scene to alter the conditions. the new response follows the same format as the initial response, but should be influenced by the imagined change (see figure 1 ). a model for this task needs to take this context information as input and try to predict the correct new response among other distractors. the distractors look similar to the correct new response but have subtle differences and are semantically different from the correct new response, thus a good model on this challenging new multimodal task cannot take shortcuts and needs to fully understand what each choice means based on the context. for example, as shown in figure 1 , given an image, the initial question-response pair ("is the railway line safe?" -"yes because there are safety lights and crosswalk signs"), and the scene change ("add a train to the tracks."), models should choose the correct new response ("no. although there are safety lights and crossing gates, they don't appear to be working and there is a train coming.") among other distractors ("no. although there are safety lights and crossing gates, there is a power outage and there is a train coming.", etc.). to solve this problem, models need to be able to understand the implications of an incoming train and how safety lights and gates operate at a railroad crossing. we collect 3.5k high-quality and challenging data instances for this new multimodal reasoning task via a crowd-sourcing annotation platform. to collect each data instance and to help reduce individual crowd-worker load, we break the task up into three separate phases: the question collection phase, the scene change collection phase, and the distractor collection phase. during the distractor collection phase, to help avoid unexpected biases such as text-only, we implement a modified version of human-and-model-in-the-loop enabled training (hamlet) adversarial data collection @xcite for the validation and test splits. we deploy the model trained on only the textual data and allow annotators to test their distractors against the model as they write (see figure 2 ). our cosim dataset features several diverse types of imagined scene changes (object addition/removal, object state changes, etc.; see sec. 5.2 for the full change type list and examples) which requires to deeply understand the contexts, making the task very challenging. for example, to understand the scene change of "add another person to the dock ...", the model should figure out what a dock is, where it is located in the image and be able to add one more person onto it via imagination. as a baseline model for this new multimodal reasoning task, we employ a vision-language transformer (based on lxmert (tan and bansal, 2019)) which computes vision and language feature matching scores via multi-head self-attention layers followed by cross-modal attention layers, and we report ablation studies on input modality and scene change types. we also show a large human-model performance gap allowing more effective future work from the community on this new challenging multimodal task on commonsense reasoning for imagined counterfactual scene changes. ## related work visual question answering. there have been many efforts to teach machines how to reason about images @xcite @xcite and videos @xcite @xcite , and in some of these tasks, machine performance is approaching human levels. although these tasks require a complicated reasoning process, they provide very explicit context to solve the problems and might not be enough to evaluate the ability to reason about implicit aspects (i.e., commonsense). visual commonsense reasoning. another actively explored line of study has been on visual commonsense reasoning @xcite @xcite . in addition to using the provided clues in the context, these tasks require commonsense knowledge to reason about given problems, making these tasks more challenging since machines should be equipped with prior or external information. however, these tasks handle static scene understanding for which contexts and conditions are not changed during the reasoning process. on the other hand, our proposed cosim introduces an additional dimension of difficulty by integrating imagined scene changes in the context. moreover, the changes in our cosim dataset are imagined (textually) and counterfactual, so imagination-based commonsense is required for the reasoning. ## dataset our cosim dataset is composed of 3.5k foot_1 images paired with a commonsense question-response pair, a description of an imagined counterfactual change to the image, a new response to the question based on the effect of the described change, and then three distractor responses to the question (all text is in english). we employ annotators from the crowd-sourcing platform amazon mechanical turk foot_2 . our data collection is broken into three separate phases (question, change, and distractor) in order to reduce the workload for each worker. in the question phase, workers are asked to select an image (from three random images) to use, write a commonsense question and then respond to it. in the change phase, they are asked to describe a counterfactual scene change for the image and then write a new response to the initial question. lastly, in the distractor phase, they are asked to write three distractor responses for the question. ## data analysis we collect 3.5k task instances (3.5k images, initial questions-response pairs, scene changes, new responses, and 10.5k distractors). ## models we employ a vision-language transformer as the base architecture of our baseline model for the cosim task. to be specific, we use lxmert @xcite to compute the score of each context-response pair given an image feature, and select one with the highest score among them. we employ faster r-cnn (ren et al., text encoder visual encoder self-attention / cross-attention matching layer … [cls] is this refrigerator new ? [sep] yes, ... red tape around the refrigerator indicating that it is brand new. ... a wooden crate. [sep] there is no red tape and not on wooden crate, but there is plastic cover ... [sep] yes, the fridge is fully covered by plastic cover, so it is new. [sep] question: is this refrigerator new ? original answer: yes, as you can see a red tape around the refrigerator indicating that it is brand new. also the product is on a wooden crate. change: there is no red tape and not on wooden crate , but there is plastic cover on the fridge everywhere . new answer: yes, the fridge is fully covered by plastic cover, so it is new. distractor #1: no, the fridge is fully covered by plastic cover because the owner is moving it to somewhere else. distractor #2: no, the fridge is fully covered by plastic cover because the owner wants to protect the paint that is falling off. distractor #3: yes, the fridge is fully covered by plastic cover and it is on wooden stack, so it is new. question: is this refrigerator new ? original answer: yes, as you can see a red tape around the refrigerator indicating that it is brand new. also the product is on a wooden crate. change: there is no red tape and not on wooden crate , but there is plastic cover on the fridge everywhere . new answer: yes, the fridge is fully covered by plastic cover, so it is new. distractor #1: no, the fridge is fully covered by plastic cover because the owner is moving it to somewhere else. distractor #2: no, the fridge is fully covered by plastic cover because the owner wants to protect the paint that is falling off. distractor #3: yes, the fridge is fully covered by plastic cover and it is on wooden stack, so it is new. question: is this refrigerator new ? original answer: yes, as you can see a red tape around the refrigerator indicating that it is brand new. also the product is on a wooden crate. change: there is no red tape and not on wooden crate , but there is plastic cover on the fridge everywhere . new answer: yes, the fridge is fully covered by plastic cover, so it is new. distractor #1: no, the fridge is fully covered by plastic cover because the owner is moving it to somewhere else. distractor #2: no, the fridge is fully covered by plastic cover because the owner wants to protect the paint that is falling off. distractor #3: yes, the fridge is fully covered by plastic cover and it is on wooden stack, so it is new. question: is this refrigerator new ? original answer: yes, as you can see a red tape around the refrigerator indicating that it is brand new. also the product is on a wooden crate. change: there is no red tape and not on wooden crate , but there is plastic cover on the fridge everywhere . new answer: yes, the fridge is fully covered by plastic cover, so it is new. distractor #1: no, the fridge is fully covered by plastic cover because the owner is moving it to somewhere else. distractor #2: no, the fridge is fully covered by plastic cover because the owner wants to protect the paint that is falling off. distractor #3: yes, the fridge is fully covered by plastic cover and it is on wooden stack, so it is new. we split the dataset into 1,924/800/800 (train/val/test). training details. we use 768 as the hidden size and use adam (kingma and ba, 2015) as the optimizer, setting the learning rate to 1 × 10 -5 . see appendix for more details. human upper bound evaluation setup. we conduct a human evaluation of our cosim task to estimate the upper bound that models can reach. we take 50 samples from the validation split and ask two experts to complete the task and average their scores. ## conclusion we introduced a challenging counterfactual commonsense reasoning task/dataset called cosim which features imagined counterfactual scene changes requiring models to imagine the changed ch: add a trolley in the foreground with many books replaced on it. add labels to the spines of all the books. answer: it seems from the trolley that this is a library. the books on display are probable for reading and maybe to be borrowed but they are likely not for sale. q: are the books for sale? ir: yes, as you can see the books are organized by names and authors on the shelves for the customers. figure 8 : the model focuses on a crucial part in the scene change to properly select the answer. shaoqing ren, kaiming he, ross b girshick, and jian sun. 2015. faster r-cnn: towards real-time object detection with region proposal networks. in nips. shailaja keyur sampat, akshay kumar, yezhou yang, and chitta baral. 2021. clevr_hyp: a challenge dataset and baselines for visual question answering with hypothetical actions over images. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 3692-3709. hao tan and mohit bansal. 2019. lxmert: learning cross-modality encoder representations from transformers. in proceedings of the 2019 conference on empirical methods in natural language processing. makarand tapaswi, yukun zhu, rainer stiefelhagen, antonio torralba, raquel urtasun, and sanja fidler. 2016. movieqa: understanding stories in movies through question-answering. in proceedings of the ieee conference on computer vision and pattern recognition, pages 4631-4640. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in proceedings of the 31st international conference on neural information processing systems. misha wagner, hector basevi, rakshith shetty, wenbin li, mateusz malinowski, mario fritz, and ales leonardis. 2018. answering visual what-if questions: from actions to predicted scene descriptions. in proceedings of the european conference on computer vision (eccv) workshops, pages 0-0. jianwei yang, jiasen lu, dhruv batra, and devi parikh. 2017. a faster pytorch implementation of faster rcnn. https://github.com/jwyang/faster-rcnn.pytorch . rowan zellers, yonatan bisk, ali farhadi, and yejin choi. 2019. from recognition to cognition: visual commonsense reasoning. in the ieee conference on computer vision and pattern recognition (cvpr). linchao zhu, zhongwen xu, yi yang, and alexander g hauptmann. 2017. uncovering the temporal context for video question answering. international journal of computer vision, 124(3):409-421. yuke zhu, oliver groth, michael bernstein, and li fei-fei. 2016. visual7w: grounded question answering in images. in proceedings of the ieee conference on computer vision and pattern recognition, pages 4995-5004.