COGEN : Abductive Commonsense Language Generation (2023)

Abstract: Reasoning is one of the most important elements in achieving Artificial General Intelligence (AGI), specifically when it comes to Abductive and counterfactual reasoning. In order to introduce these capabilities of reasoning in Natural Language Processing (NLP) models, there have been recent advances towards training NLP models to better perform on two main tasks - Abductive Natural Language Inference (alphaNLI) and Abductive Natural Language Generation Task (alphaNLG). This paper proposes CoGen, a model for both alphaNLI and alphaNLG tasks that employ a novel approach of combining the temporal commonsense reasoning for each observation (before and after a real hypothesis) from pre-trained models with contextual filtering for training. Additionally, we use state-of-the-art semantic entailment to filter out the contradictory hypothesis during the inference. Our experimental results show that CoGen outperforms current models and set a new state of the art in regards to alphaNLI and alphaNLG tasks. We make the source code of CoGen model publicly available for reproducibility and to facilitate relevant future research.

## introduction different kinds of reasoning can be categorized into three classes @xcite : deduction, induction, and abduction. in deduction, the truth of the conclusion is already provided in the premise, therefore, it is impossible that the premises are true and the conclusion is false. induction is the process of going from the truth of some premises to the conclusion. finally, abduction is the process of forming the most plausible hypothesis based on incomplete observations. the focus of this paper is on abductive reasoning. the abductive inference could be viewed as going backward from the conclusions of a valid deductive inference to the premises to find its plausible causes and effects. in terms of classical logic, this is a fallacy @xcite . abductive reasoning is defeasible (and also non-monotonic) which means the conclusions can be refuted in the light of new data. although abductive reasoning forms one of the core abilities of human cognition, its research in the area of nlp is still widely unexplored. recent work on large language models like gpt-3 @xcite and gpt-neo @xcite had impressive results on different nlp tasks but still struggled with abductive natural language inference (αnli ) tasks. these models embed a great deal of world knowledge @xcite , but their potential for commonsense reasoning (e.g. abductive reasoning) has not been fully harnessed. the task of abductive commonsense language generation can be defined as generating reasons given incomplete observations. abductive commonsense language generation can be formulated as a controlled language generation task. like other controllable language generation problems that involve maintaining fluency and relevance of the generated text conditioned on some property, such as sentiment @xcite , topic @xcite , and style @xcite , the abductive commonsense language generation can be viewed as a controllable language generation task that is conditioned on incomplete observations. in this paper, we introduce cogen 1 , a model for generating and inferring abductive reasons that are compatible with observations. this combines temporal commonsense reasoning for each observation (before and after the hypothesis) from pretrained models with contextual filtering for training. contextual filtering refers to the technique of refining temporal entailment during text generation to produce more coherent and contextually relevant output. we also use state-of-the-art semantic entail-figure 1 : cogen first uses temporal reasoner to produce before and after commonsense then with a cross-encoder it filters unrelated temporal commonsense based on the context. with gpt-2 the system takes both observations and contextual knowledge as inputs a set of hypotheses h will be generated that in semantic entailment will be cleaned up from contradictions by a bert model. the bold arrows indicate a set of inputs. ment to filter out contradictory hypotheses during the inference. our results show that cogen outperforms all previous models regarding αnli and αnlg tasks. our main contributions are the following: ## related work previous research on reasoning in nlp mainly focuses on monotonic reasoning, which is usually about finding the "entailment", "contradiction" or "neutral" relationships between a premise and a hypothesis. for example, snli @xcite and multinli @xcite are both datasets that focus on monotonic inference. there is a choice of plausible reasoning task with the copa dataset @xcite which is designed for causal reasoning. in @xcite , the authors introduced the timetravel dataset which contains over 28k counterfactual instances. the results show the current language models lack understanding of the reasoning behind the stories, sometimes even adding more samples will not improve the quality of the generation. @xcite proposes delorean, a new unsupervised decoding algorithm based on backpropagation that incorporates observations from the past and future to generate constrained text in between. they used the art dataset @xcite which contains 20k samples. the most relevant work to cogen is abductive commonsense reasoning (cometemb+gpt2) @xcite , which introduces art dataset consisting of 20k commonsense narrative contexts with 200k explanations. they also introduced two tasks: abductive nli (αnli) a multiplechoice task for choosing the best hypothesis and abductive nlg (αnlg) which generates an abductive hypothesis given the two before and after contextual observations. results showed that abductive nlg is much more challenging compared to (αnli) and needs further research. they also used gpt-2 and comet @xcite for commonsense reasoning to generate new abductive hypotheses. the human judgment results show that only 44.56 percent of these generated hypotheses make sense to evaluators. in @xcite , they consider possible events emerging from the candidate hypothesis and then select the one that is most similar to the observed outcome. their approach outperforms cometemb+gpt2 on the αnli task and achieves 72.2 on the test set. @xcite proposed grf, which is based on gpt-2 and dynamic multi-hop reasoning for multi-relational paths extracted from conceptnet for αnlg. reflective decoding @xcite is an unsupervised text generation algorithm for text infilling that uses two pre-trained forward and backward language models. this algorithm outperforms all unsupervised methods, but is still significantly behind the fine-tuned model of come-temb+gpt2 in abductive generation. ## method abductive reasoning can be formulated using a single observation as a premise and generating a hypothesis. however, following @xcite we formulate abductive commonsense language generation as the task of generating a hypothesis h given two observations, o 1 and o 2 that happen at times t 1 and t 2 , respectively, in which t 2 > t 1 . the hypothesis h happens between t 1 and t 2 . this shows abductive and temporal reasoning is closely related to each other @xcite . more specifically, abductive reasoning requires temporal reasoning about the consequences of events (what typically occurs after them) and the reasons behind them (what may happen prior to or trigger them). commonsense knowledge graphs (cskb) are knowledge graphs containing many commonsense facts about the world that help to understanding and reasoning about events, social interactions and physical entities. atomic2020 @xcite is the largest cskb having 1.33m tuples about entities and events of inferential knowledge and introduces 23 relation types. in this paper we focus on two classes of these relations: before relations and after relations. before relations are those that take place before the observation or trigger them, such as: isbefore, causes, xeffect, xreacts, xintents, and xwants. after relations are those that occur after the observation, such as: isafter, oreact, owant, oeffect, xreason. neural knowledge graphs are models trained on cskb tuples and are able to generate tails given the new heads. for instance, to predict the tail of the tuple (x votes for y, xintents ?) is to generate "to give support". we use the state-of-the-art pretrained bidirectional and autoregressive transformer (bart) @xcite named comet that is trained on atomic2020. for temporal commonsense augmentation, we generate n after relation facts for o 1 and n before relation facts for o 2 . the comet(o, r) is the function that generates the commonsense for observation o for the relation r. if r a and r b are the after and before relations, then the following commonsense responses are generated: however, not all after and before relations are relevant for every situation. the generated commonsense facts should be filtered out based on the context. for each commonsense relation, we chose the most likely fact based on the semantic similarity to the other observation. more specifically, the most likely after (before) fact for o 1 (o 2 ) based on the similarity to o 2 (o 1 ) is chosen: o 1 1 . . . t o 1 m }, o 2 = {t o 2 1 . . . t o 2 n } and hypothesis @xmath0. . . t h l } as a sequence of tokens, we can augment the input with the commonsense knowledge from the previous step @xmath1, c b } as a sequence of tokens @xmath2. . . t k q }. the abductive commonsense language generation can be formulated by minimizing the following negative log-likelihood: training: we trained three different models for αnlg -cogen lg , cogen m d and cogen sm by fine-tuning three gpt-2 models of sizes large, medium, and small, respectively. we used an embedding size of 512 for all models with a maximum token size of 128. the learning rate was set to 5e-4 with a weight decay of 0.01. we stopped training after 5 epochs before overfitting to the training set occurrs. we also propose the fine-tuned cogen rb model for αnli, which is based on the large roberta @xcite first 20% for the warm-up with the learning rate of 1e -5 and after that decrease it linearly by a ratio of 0.01. inference: for inference, we use beam search decoding with a beam size of 5. we chose this search as it works best with controllable language generation @xcite . for each pair of observations, multiple hypotheses are generated and then filtered out based on entailment. we use the pre-trained semantic entailment bert cross-encoder @xcite , trained on snli @xcite and multinli @xcite , to filter out each generated hypothesis h, if the o 1 → h or @xmath3. using this technique we can remove undesired hypotheses that are incompatible with the given observations. ## result we report bert-score @xcite , bleurt @xcite , bleu @xcite , ter @xcite , meteor @xcite and rouge @xcite for automatic evaluation of our model. the results in we assessed human evaluations on 100 randomly selected results from the test set. the evaluation was completed by five graduate students unrelated to our research, providing us with unbiased data. these evaluations, shown in finally, we show the results of αnli task from different models in ## conclusion we present cogen, a novel approach to generate abductive reasoning given incomplete observations in three different sizes. this integrates temporal reasoning, context filtering, and semantic entailment to complete the base gpt-2 model for better reasoning. both human and automatic evaluations assessed in this study show that cogen outperforms previous methods used for abductive reasoning. our approach sets a new state-of-the-art for αnli and αnlg tasks on art dataset.