UN commonsense Reasoning: Abductive Reasoning about Uncommon Situations (2024)

Abstract: Language technologies that accurately model the dynamics of events must perform commonsense reasoning. Existing work evaluating commonsense reasoning focuses on making inferences about common, everyday situations. To instead investigate the ability to model unusual, unexpected, and unlikely situations, we explore the task of uncommonsense abductive reasoning. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation that makes the unexpected outcome more likely in the context. To this end, we curate and release a new English language corpus called UNcommonsense. We characterize the performance differences between human explainers and the best-performing large language models, finding that model-enhanced human-written explanations achieve the highest quality by trading off between specificity and diversity. Finally, we experiment with several imitation learning algorithms to train open and accessible language models on this task. When compared with the vanilla supervised fine-tuning approach, these methods consistently reduce lose rates on both common and uncommonsense abductive reasoning judged by human evaluators.

## introduction the ability to perform commonsense reasoning is crucial for understanding the dynamics of everyday events, both for humans and for natural language processing systems. however, most existing commonsense reasoning benchmarks focus on the ability to model common events (sap et al., 2019; talmor et al., 2019; lin et al., 2020b), i.e., given a commonly encountered situation, what commonsense inferences can be made? comparatively less effort has been devoted to evaluating a different class of inputs: unusual scenarios, improbable situations, and implausible events. despite disliking the taste of sushi, cameron decided to stay and eat more sushi plates to avoid disappointing his partner, who was excited about sharing… figure 1 : given a context and an uncommon outcome, uncommonsense abductive reasoning aims to produce an explanation so that the unlikely outcome becomes likely. the explanation needs to follow the three rules noted with the check marks. understanding and reasoning about these situations is crucial for the fairness and reliability of language technologies. for example, most llms are trained on english data. they are accustomed to western cultural norms, and therefore non-english culture could be considered uncommon in current llm-based nlp systems, e.g., wearing shoes indoors is normal in western culture, but is often viewed as disrespectful in asian households. being able to reason about uncommon situations helps llms serve individuals from diverse cultural backgrounds more effectively. uncommon situations could also be associated with important and highrisk scenarios (weidinger et al., 2022). consider a situation where an individual tries out a massage chair and subsequently develops small, itchy, and red welts on their back. one explanation may be that this person is allergic to vibrations, a rare yet real medical condition called vibratory urticaria. while this is an uncommon situation, an nlp system that incorrectly interprets or handles this situation could lead to severe consequences, for example a misdiagnosis of a more common condition unrelated to the chair. to bridge this gap, we introduce uncommon-sense, a benchmark that explicitly challenges models to reason about implausible, yet still possible, events. uncommonsense is an englishlanguage corpus consisting of 20k unique contexts paired with explicitly uncommon outcomes. we source uncommon outcomes from the incorrect answers in several multiple choice commonsense reasoning benchmarks, which were designed to challenge models to identify the most likely outcome among multiple candidates, given a context. given these contexts and uncommon outcomes, we crowdsource 41k abductive explanations, which provide a plausible explanation of how an uncommon outcome could have arisen, given an input context. see figure 1 for an example. uncommonsense complements existing commonsense reasoning corpora (e.g., mostafazadeh et al., 2016a; bhagavatula et al., 2020; rudinger et al., 2020) that focus on reasoning about common events. @xcite we examine the gap between human and model performance in generating abductive uncommonsense explanations, finding subtle differences in explanation quality. given a few demonstrations, the top-performing llm gpt-4 (openai, 2023) produces more specific explanations than those acquired through crowdsourcing; however, these explanations are less diverse. while their explanations often lack sufficient details to connect contexts to outcomes, workers recruited through crowdsourcing excel at creating a broader picture of possible intermediate events. to combine the creativity of human authors and the specificity of llmgenerated explanations, we experiment with using an llm to refine crowd-authored explanations by filling in more details. though llm-generated explanations are generally preferred over the original crowd-written explanations, we find that llmrefined crowd-written explanations hold a notable advantage over those generated only by an llm. generating abductive explanations for uncommon outcomes without conditioning on a humanwritten starting point remains a challenge, particularly for publicly available models. specifically, we find that the purely offline learning approach of supervised fine-tuned models suffer from compounding errors during generation. this is particularly problematic for our task, which generally requires lengthy explanations that bridge the gap between a context and an uncommon outcome. to this end, we experiment with two online imitation learning methods to improve the performance of open and accessible language models on abductive reasoning. when compared with supervised fine-tuning, these methods show an absolute 10% increase in win rates against the strong gpt-4 baseline when evaluated by workers on both commonsense and uncommonsense abductive reasoning. ## uncommonsense abductive reasoning given a natural language context x and outcome y, the task of abductive reasoning requires generating a natural language explanation z that augments the context, making the outcome more probable (bhagavatula et al., 2020). in uncommonsense abductive reasoning, we focus on situations where an outcome y is very unlikely to happen in context x. for example, in figure 1 , our context "cameron tried sushi for the first time, and really disliked it." is paired with the unlikely outcome "cameron will want to stay and eat more sushi.". one possible abductive explanation of this outcome is that "... cameron decided to stay and eat more sushi plates to avoid disappointing his partner, who was excited about sharing...". when the context is augmented with this explanation, it becomes significantly more likely that the outcome will occur. to our knowledge, no existing datasets explicitly study abductive reasoning for uncommon situations. we fill this gap by collecting the un-commonsense dataset, which contains contexts paired with both uncommon outcomes and explanations that rationalize these uncommon outcomes. ## data analysis unlikely outcomes. we utilize gpt-4 prompting to quantify, on a scale from 1 to 5, how likely an outcome may occur given the context. ## imitation learning for abductive reasoning existing methods for abductive reasoning focus on performing supervised fine-tuning (sft) with a static dataset (bhagavatula et al., 2020; rudinger et al., 2020). training using static demonstration data is vulnerable to exposure bias: during training, the model learns to predict the next token in an explanation conditioned on a gold-standard prefix; however, when the model generates an entirely new explanation during inference, it is conditioned on its own previously generated tokens. this inconsistency between training and inference procedures leads to error propagation at inference time, and a reduction in the quality of explanations. to address this issue, we experiment with several on-policy imitation learning algorithms. ## conclusion we propose a new task, uncommonsense abductive reasoning, designed to assess the ability of nlp systems to reason about uncommon scenarios in abductive reasoning tasks. we explore two imitation learning methods to improve accessible language models on uncommonsense abductive reasoning. experiments show that access to expert behavior, particularly when using the expert as an oracle in online training, significantly improves the explanation quality of smaller models.