Generating Hypothetical Events for Abductive Inference (2021)

Abstract: Abductive reasoning starts from some observations and aims at finding the most plausible explanation for these observations. To perform abduction, humans often make use of temporal and causal inferences, and knowledge about how some hypothetical situation can result in different outcomes. This work offers the first study of how such knowledge impacts the Abductive NLI task – which consists in choosing the more likely explanation for given observations. We train a specialized language model LMI that is tasked to generate what could happen next from a hypothetical scenario that evolves from a given event. We then propose a multi-task model MTL to solve the Abductive NLI task, which predicts a plausible explanation by a) considering different possible events emerging from candidate hypotheses – events generated by LMI – and b) selecting the one that is most similar to the observed outcome. We show that our MTL model improves over prior vanilla pre-trained LMs fine-tuned on Abductive NLI. Our manual evaluation and analysis suggest that learning about possible next events from different hypothetical scenarios supports abductive inference.

## introduction abductive reasoning (ar) is inference to the best explanation. it typically starts from an incomplete set of observations about everyday situations and comes up with what can be considered the most likely possible explanation given these observations @xcite . one of the key characteristics that make abductive reasoning more challenging and distinct from other types of reasoning is its non-monotonic character @xcite i.e., even the most likely explanations are not necessarily correct. for example, in figure 1 , the most likely explanation for observation 1: "wet grass outside my house" is that "it has been raining". however, when a new piece of information (observation or evidence) becomes available, the explanation must possibly be retracted, showing the defeasible character of abduction. with the new observation ("the sprinkler was switched on") the most plausible explanation changes to "sprinkler caused the grass to be wet". humans, in such situations, could induce or validate such abductive inferences by performing hypothetical reasoning (such as "what would happen if the sprinkler was switched on?") to arrive at a plausible explanation for "wet grass outside my house". in this work, we focus on the αnli task @xcite , where given two observations (o 1 at time t 1 , o 2 at time t 2 , with t 1 < t 2 ) as an incomplete context, the task is to predict which of two given hypothesized events (h 1 or h 2 ) is more plausible to have happened between o 1 and o 2 . figure 2 illustrates this with an example: given observations o 1 :"priya decided to try a new restaurant." and o 2 : "priya thought her food was delicious.", the task is to predict whether h 1 or h 2 is the more plausible explanation given observations o 1 and o 2 . both h 1 and h 2 are different plausible hypothetical situations that can evolve from the same observation (premise) o 1 . in this paper, we hypothesize that learning how different hypothetical scenarios (h 1 and h 2 ) can result in different outcomes (e.g., o h j 2 , fig. 2 ) can help in performing abductive inference. in order to decide which h i , is more plausible given observa-o 1 : priya decided to try a new restaurant. o 2 : priya thought her food was delicious. we design a language model (lm i ) which, given observations and a hypothesis, generates a possible event that could happen next, given one hypothesis. in order to train this language model, we use the timetravel (tt) corpus @xcite (a subpart of the rocstories corpus 1 ). we utilize the lm i model to generate a possible next event for each hypothesis, given the observations. we then propose a multi-task learning model mt l that jointly chooses from the generated possible next events (o h 1 2 or o h 2 2 ) the one most similar to the observation o 2 and predicts the most plausible hypothesis (h 1 or h 2 ). our contributions are: i) to our best knowledge, we are the first to demonstrate that a model that learns to perform hypothetical reasoning can support and improve abductive tasks such as αnli. we show that ii) for αnli our multi-task model outperforms a strong bert baseline @xcite . our code is made publicly available. 2 ## learning about counterfactual scenarios the main idea is to learn to generate assumptions, in a given situation, about "what could have hap1 we ensure that αnli testing instances are held out. pened (next) if we had done x?" or "what could happen (next) if we do x?" @xcite . figure 3 (a) depicts the αnli task framework. we hypothesize that getting to know what will happen (next) if any of two hypotheses occurs, will help us verifying which of them is more plausible (see fig. 3(c )). therefore, we encourage the model to learn how different hypothetical events (including counterfactual events) evolving from the same premise (s 1 ) can lead to different or similar outcomes (see fig. 3(b) ). accordingly, we teach a pre-trained gpt-2 (radford et al., 2019) language model how to generate a sequence of possible subsequent events given different hypothetical situations in a narrative setting. training such a model on narrative texts encourages it to learn causal and temporal relations between events. we train a conditional language model, lm i , which generates a possible event that could happen next, given some counterfactual scenarios for a given story. we train this model on the timetravel (tt) dataset @xcite , by fine-tuning gpt-2 to learn about possible next events emerging from a situation in a story, given some alternative, counterfactual event. the tt dataset consists of fivesentence instances @xmath0,s 2 ,..,s 5 ) 3 from the roc-stories corpus 1 plus additional crowd-sourced sen-o 1 : dotty was being very grumpy. o 2 : she felt much better afterwards. h 1 : dotty ate something bad. h 2 : dotty call some close friends to chat. o h 1 2 : she started to feel sick. o h 2 2 : they all tried to make her happy. tences s 2:5 , where s 2 is counterfactual foot_1 to s 2 from the original story foot_2 . there are two reasons for using the tt dataset for our purposes: a) the domains on which gpt-2 was pretrained are broad foot_3 and different from the domain of rocstories, b) the model can see how alternative situations can occur starting from the same premise s 1 , resulting in similar or different outcomes. note that, although intermediate situations may be counterfactual to each other, the future outcome can still be similar to the original ending due to causal invariance foot_4 . concretely, the language model lm i reads the premise (s 1 ) and the alternative event(s) (s 2 or s 2 ), the masked token (serving as a placeholder for the missing possible next event(s) (s 3:i or s 3:i ), then the rest of the story (s i+1:5 or s i+1:5 ) and again the premise (s 1 ). we train the model to maximize the log-likelihood of the missing ground-truth sentence(s) (s 3:i ). ## generating hypothetical events to support the αnli task in this paper, we aim to investigate whether models perform better on the αnli task when explicitly learning about events that could follow other events in a hypothetical scenario. we do so by introducing two methods lm i + bertscore and lm i + mt l for unsupervised and supervised settings, respectively. we first apply the trained model lm i on the αnli task, where the given observations o 1 and o 2 , and alternative hypotheses h j are fed as shown in (2) below. foot_5 we generate a possible next event for each hypothetical event h j , i.e., o h 1 2 and o h 2 2 (or: what will happen if some hypothesis h j occurs given the observations), where @xmath1. ## results unlike the unsupervised lm i + bertscore, our supervised lm i + mt l model also improves over the bert large baseline, by +3.3 pp. we can attribute the improvement to the model having been jointly trained to assess the similarity and dissimilarity of possible next events o h j 2 and observations (o 2 ) along with the αnli task. one of the advantages of training our proposed multitask learning (mt l) model, instead of directly feeding the possible next events o h j 2 as knowledge inputs is that it adds an explainable component to the model. one can view the generated next events o h j 2 as natural language rationales and our multitask model explicitly chooses one of them. hence, the multi-task framework makes the model more expressive. finally, we compare, for the mt l model, our embedded generation model lm i to pre-trained gpt-2 and comet. ## manual evaluation since the automatic scores only account for wordlevel similarity between observations and generated possible next events, we conduct a manual evaluation study, to assess the quality of sentences generated by our lm i model. ## related work commonsense reasoning. there is growing interest in this research field, which led to the creation of several new resources on commonsense reasoning, in form of both datasets, such as so-cialiqa @xcite , commonsenseqa (talmor et al., 2019), cosmosqa @xcite and knowledge bases, e.g. conceptnet @xcite , atomic @xcite , or event2mind (rashkin et al., 2018). recently, many works proposed to utilize external static knowledge graphs (kgs) to address the bottleneck of obtaining relevant commonsense knowledge. lin et al. (2019) proposed to utilize knowledge graph embeddings to rank and select relevant knowledge triples or paths. paul and frank (2019) proposed to extract subgraphs from kgs using graph-based ranking methods and further paul et al. (2020) adopted the graph-based ranking method and proposed to dynamically extend the kg to combat sparsity. in concurrent work, paul and frank (2021) introduced a method to dynamically generate contextually relevant knowledge that guides a model while performing the narrative story completion task. both hypothetical reasoning and abductive reasoning are understudied problems in nlp. recently, tandon et al. (2019) proposed a first large-scale dataset of "what if..." questions over procedural text. they introduced the dataset to study the effect of perturbations in procedural text. related to our work, qin et al. (2019) investigated the capabilities of state-of-the-art lms to rewrite stories with counterfactual reasoning. in our work we utilize this dataset to model how to generate possible next events emerging from different hypothetical and counterfactual events. mostafazadeh et al. (2016) designed the narrative cloze task, a task to choose the correct ending of a story. foot_8 conversely, bhagavatula et al. ( 2020 ) proposed a task that requires reasoning about plausible explanations for narrative omissions. our research touches on the issue of hypothetical reasoning about alternative situations. we found that making language models learn how different hypothetical events can evolve from a premise and result in similar or different future events forming from a premise and how these events can result in similar or different future events helps models to perform better in abduction. explainability. despite the success of large pretrained language models, recent studies have raised some critical points such as: high accuracy scores do not necessarily reflect understanding @xcite , large pretrained models may exploit superficial clues and annotation artifacts @xcite . therefore, the ability of models to generate explanations has become desirable, as this enhances interpretability. recently, there has been substantial effort to build datasets with natural language explanations @xcite @xcite . there have also been numerous research works proposing models that are interpretable or explainable @xcite @xcite . our work sheds light in this direction, as our mt l model not only predicts the plausible hypothesis h j but also generates possible next events o h j 2 and chooses the one that is closer to the given context, thereby making our model more expressive. abductive reasoning. there has been longstanding work on theories of abductive reasoning @xcite @xcite . researchers have applied various frameworks, some focused on pure logical frameworks @xcite , some on probabilistic frameworks @xcite , and others on markov logics @xcite . recently, moving away from logic-based abductive reasoning, @xcite proposed to study languagebased abductive reasoning. they introduced two tasks: abductive natural language inference (αnli) and generation (αnlg). they establish baseline performance based on state-of-the-art language models and make use of inferential structured knowledge from atomic @xcite as background knowledge. @xcite proposed to use a learning-to-rank framework to address the abductive reasoning task. @xcite proposed a model grf that enables pre-trained models (gpt-2) with dynamic multi-hop reasoning on multi-relational paths extracted from the external conceptnet commonsense knowledge graph for the αnlg task. @xcite have proposed a multi-head knowledge attention method to incorporate commonsense knowledge to tackle the αnli task. unlike our previous work in @xcite , which focused on leveraging structured knowledge, in this work, we focus on learning about what will happen next from different counterfactual situations in a story context through language model fine-tuning. specifically, we study the impact of such forward inference on the αnli task in a multi-task learning framework and show how it can improve performance over a strong bert model. ## conclusion we have introduced a novel method for addressing the abductive reasoning task by explicitly learning what events could follow other events in a hypothetical scenario, and learning to generate such events, conditioned on a premise or hypothesis. we show how a language model -fine-tuned for this capability on a suitable narrative dataset -can be leveraged to support abductive reasoning in the αnli tasks, in two settings: an unsupervised setting in combination with bertscore, to select the proper hypothesis, and a supervised setting in a mt l setting. the relatively strong performance of our proposed models demonstrates that learning to choose from generated hypothetical next events the one that is most similar to the observation, supports the prediction of the most plausible hypothesis. our experiments show that our unsupervised lm i +bertscore model outperforms some of the strong supervised baseline systems on αnli. our research thus offers new perspectives for training generative models in different ways for various complex reasoning tasks.