Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios (2022)

Abstract: The possible consequences for the same context may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). Through a comparison with existing datasets, we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that require counterfactual reasoning, as well as those requiring readers’ reactions and fictional information, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning.

## introduction commonsense reasoning, inclusive of counterfactual, abductive, and monotonic reasoning, is a core element of language understanding. researchers are interested in whether these abilities can be learned in systems, and several benchmarks have been proposed to investigate machine commonsense reasoning @xcite @xcite . recent pretrained models have shown competitive results @xcite . commonsense reasoning has often been framed as a task to infer whether candidate answers are plausible, such as in the multiple-choice format @xcite . the ## the possible stories dataset context passages to collect story texts, we use rocstories @xcite , a corpus of five-sentence stories. the first to fourth sentences describe the context, and the final sentence, the ending, concludes the story. we choose roc-stories because each of its stories has a clear beginning and ending, while being generic enough to come up with different endings. the details on our story selection criteria are provided in appendix a. the following tasks are carried out by the crowdworkers in amazon mechanical turk (mturk) who perform above certain levels during our worker recruitment phase, which is designed to be fairer than the conventional qualifications used in mturk. the details of the worker recruitment are provided in appendix b. the instructions and task interface presented to the workers are also provided in appendix i. ## analysis 6.1 human-model performance gap to investigate the relative difficulty of our dataset among multiple-choice qa datasets, we compare the accuracy gap between humans and models in an unsupervised setting with existing datasets, including cosmosqa (cosmos; we report the valida- ## discussion circumscribing commonsense reasoning from simple heuristics has been a long-standing problem in the field of artificial intelligence @xcite . although the answer options in our dataset are free from annotation artifacts, our ablation analysis in section 5 also shows that the questions and answer options may still involve some clues that the models can exploit. further research is needed to explain how commonsense reasoning is distinguished from a set of simple heuristics in machines' situational understanding. in the question-writing task, one of the workers addresses the difficulty of creating questions that cannot be answered without reading the passages and that it is even practically impossible unless asking the questions to small children. this illustrates that humans may also use a small amount of information available to draw inferences. this issue arises possibly because of our task formulation, answering which option is more plausible than the others. it can be argued that modifying this task to a generative task @xcite is one way to directly assess machines' commonsense reasoning ability, but it should be noted that this would entail some difficulties in the evaluation of generated answers. in addition, exploring what kind of conditions narrow down the possibilities of consequences is important to effectively evaluate machines' situational understanding. although several studies have captured the dynamics of conditions in moral and immoral settings @xcite , many other factors come into play in decision-making in our daily lives, such as feelings, personal beliefs, expectations from others, or even unconscious biases. ## conclusion this paper proposes a new dataset, possible stories, consisting of 4.5k crowdsourced questions with 1.3k story passages to investigate whether machines can infer the most plausible ending among four possible endings under certain situations postulated by questions. we discover that current strong pretrained language models struggle to answer questions consistently, showing a large accuracy gap compared with humans. a comparison with existing multiple-choice datasets demonstrates that our questions contain minimal annotation artifacts in the answer options and require counterfactual reasoning as well as an understanding of characters' motivations and emotions, suggesting that our dataset can serve as a challenging benchmark for future commonsense reasoning studies.