Learning Event Graph Knowledge for Abductive Reasoning (2021)

Abstract: Abductive reasoning aims at inferring the most plausible explanation for observed events, which would play critical roles in various NLP applications, such as reading comprehension and question answering. To facilitate this task, a narrative text based abductive reasoning task \alpha NLI is proposed, together with explorations about building reasoning framework using pretrained language models. However, abundant event commonsense knowledge is not well exploited for this task. To fill this gap, we propose a variational autoencoder based model ege-RoBERTa, which employs a latent variable to capture the necessary commonsense knowledge from event graph for guiding the abductive reasoning task. Experimental results show that through learning the external event graph knowledge, our approach outperforms the baseline methods on the \alpha NLI task.

## introduction abductive reasoning aims at seeking for the best explanations for incomplete observations @xcite . for example, given observations forgot to close window when leaving home and the room was in a mess, human beings can generate a reasonable hypothesis for explaining the observations, such as a thief entered the room based on commonsense knowledge in their mind. however, due to the lack of commonsense knowledge and effective reasoning mechanism, this is still a challenging problem for today's cognitive intelligent systems @xcite @xcite . most previous works focus on conducting abductive reasoning based on formal logic @xcite @xcite . however, the rigidity of formal logic limits the application of abductive reasoning in nlp * corresponding author tasks, as it is hard to express the complex semantics of natural language in a formal logic system. to facilitate this, @xcite proposed a natural language based abductive reasoning task αnli. as shown in figure 1 (a), given two observed events o 1 and o 2 , the αnli task requires the prediction model to choose a more reasonable explanation from two candidate hypothesis events h 1 and h 2 . both observed events and hypothesis events are daily-life events, and are described in natural language. together with the αnli task, @xcite also explored conducting such reasoning using pretrained language models such as bert @xcite and roberta @xcite . however, despite pretrained language models could capture rich linguistic knowledge benefit for understanding the semantics of events, additional commonsense knowledge is still necessary for the abductive reasoning. for example, as illustrated in figure 1 (b), given observations o 1 and o 2 , to choose the more likely explanation h 1 : a thief entered the room and exclude h 2 : a breeze blew in the window, prediction model should have the commonsense knowledge that it is hardly possible for a breeze to mess up the room, whereas a thief may enter the room from the open window (i 1 ), then rummage through the room (i 2 ) and lead to a mess. these intermediary events (i 1 and i 2 ) can serve as necessary commonsense knowledge for understanding the relationship between observed events and hypothesis events. we notice that the observed events, hypothesis events, intermediary events and their relationships could be described using an event graph, which can be constructed based on an auxiliary dataset. the challenge is how to learn such commonsense knowledge from the constructed event graph. to address this issue, we propose an event graph enhanced roberta (ege-roberta) model, and a two-stage training procedure. specifically, as shown in figure 1 (c ), on the basis of the roberta framework, we additionally introduce a latent variable z to model the information about the intermediary events. in the pretraining stage, ege-roberta is trained upon an event-graph-based pseudo instance set to capture the commonsense knowledge using the latent variable z. in the finetuning stage, model adapts the commonsense knowledge captured by z to conduct the abductive reasoning. experimental results show that ege-roberta could effectively learn the commonsense knowledge from a well-designed event graph, and improve the model performance on the αnli task compared to the baseline methods. the code is released at https://github.com/sjcfr/ege-roberta . ## ege-roberta as a conditional variational autoencoder based reasoning framework in this paper, rather than directly predicts the relatedness score y based on the event sequence x, we propose to predict y based on both x and additional commonsense knowledge (i.e. posterior event sequence x and adjacency matrix a). to this end, we introduce a latent variable z to learn such knowledge from an event graph through a two stage training procedure. to effectively capture the event graph knowledge through z and conduct the abductive reasoning task based on z, we frame the ege-roberta model as a conditional variational autoencoder (cvae) @xcite . specifically, with regard to the latent variable z, ege-roberta characterizes the conditional distribution p (y |x) using three neural networks: a prior network p θ (z|x), a recognition network q φ (z|x , a) and a neural likelihood p θ (y |x, z), where θ and φ denote the parameters of networks. moreover, instead of directly maximize p (y |x), following cvae @xcite , ege-roberta proposes to maximize the evidence lower bound (elbo) of p (y |x) : note that, in the recognition network, the latent variable z is directly conditioned on x and a, where @xmath0, i1, hi, i2, o2} is the posterior event sequence, a is an adjacency matrix describing the relationship between events within x . this enables z to capture the event graph knowledge from x and a. through minimizing the kl term of elbo, we can teach the prior network p θ (z|x) to learn the event graph knowledge from the recognition network as much as possible. then in the neural likelihood p θ (y |x, z) the relatedness score y could be predicted based on x and z, which captures the event graph knowledge. however, the event graph knowledge is absent in the αnli dataset. to learn such knowledge, we design the following two-stage training procedure: pre-training stage: learning event graph knowledge from a pseudo instance set in this stage, ege-roberta is pretrained on a prebuilt event-graph-based pseudo instance set, which contains rich information about the intermediary events and the events relationships. as shown in figure 2 (a), the latent variable z is directly conditioned on x and a. therefore, z could be employed to learn the event graph knowledge. ## architecture of ege-roberta we introduce the specific implementation of ege-roberta. as illustrated in figure 3 , ege-roberta introduces four modules in addition to the roberta framework: (1) an aggregator providing representation for any event within x and x ; ## discussion in this paper, to involve the event graph knowledge, we formalize the posterior event sequence as @xmath1, i 1 , h i , i 2 , o 2 }. while our approach also allows other forms of posterior event sequences, such as @xmath2, h i , i 1 , o 2 }, @xmath3, i 1 , h i , o 2 }, or @xmath4, i 1 , i 2 , h i , o 2 }, etc. we also pretrained ege-roberta on pseudoinstance sets derived by these manners. the results are shown in ## conclusion in this paper, we propose a variational autoencoder based framework ege-roberta with a twostage training procedure for the abductive reasoning task. in the pretraining stage, ege-roberta is able to learn commonsense knowledge from an event graph through the latent variable, then in the following stage the learned event graph knowledge can be adapted to the abductive reasoning task. experimental results show improvement over the baselines on the αnli task.