Evidence-Aware Inferential Text Generation with Vector Quantised Variational A uto E ncoder (2020)

Abstract: Generating inferential texts about an event in different perspectives requires reasoning over different contexts that the event occurs. Existing works usually ignore the context that is not explicitly provided, resulting in a context-independent semantic representation that struggles to support the generation. To address this, we propose an approach that automatically finds evidence for an event from a large text corpus, and leverages the evidence to guide the generation of inferential texts. Our approach works in an encoderdecoder manner and is equipped with Vector Quantised-Variational Autoencoder, where the encoder outputs representations from a distribution over discrete variables. Such discrete representations enable automatically selecting relevant evidence, which not only facilitates evidence-aware generation, but also provides a natural way to uncover rationales behind the generation. Our approach provides state-of-the-art performance on both Event2mind and Atomic datasets. More importantly, we find that with discrete representations, our model selectively uses evidence to generate different inferential texts.

## introduction inferential text generation aims to understand dailylife events and generate texts about their underlying causes, effects, and mental states of event participants, which is crucial for automated commonsense reasoning. taking figure 1 as an example, given an event "personx reads persony's diary", the cause of the participant "personx" is to "obtain person y's secrets" and the mental state of "personx" is "guilty". standard approaches for inferential text generation @xcite @xcite typically only take the event as the input, while ignoring the background knowledge that provides crucial evidence to generate reasonable inferences. for example, if the background knowledge of this example is "per-sony invites personx to read his diary", the outputs should be different. in this paper, we present an evidence-aware generative model, which first retrieves relevant evidence from a large text corpus and then leverages retrieved evidence to guide the generation of inferential texts. our model is built upon transformerbased @xcite encoder-decoder architecture, and is equipped with vector quantised-variational autoencoder to map an event to a discrete latent representation @xcite . these discrete representations embody the latent semantic distribution of inferences given the event, thus supporting selection of relevant evidence as background knowledge to guide the generation in different perspectives. furthermore, our model has two attractive properties: (1) it avoids the problem of posterior collapse, caused by latent variables being ignored, in traditional variational autoencoder with continuous latent variables @xcite , and more importantly (2) it uncovers the rationale of a generation to some extent through tracing back the evidence that guides the generation and the selected discrete representation of the event. we evaluate our approach on event2mind @xcite and atomic @xcite datasets, both of which focus on reasoning about causes and effects of events and mental states of event participants. experimental results show that our approach achieves state-ofthe-art performances on both datasets. further analysis shows that our approach can equip the generation with an explicit control over the semantics of latent variables and selected evidence to generate inferential texts in different perspective. the source codes are available at https: //github.com/microsoft/ea-vq-vae. we conduct experiments on event2mind 2 (rashkin et al., 2018) and atomic 3 @xcite datasets. both datasets contain about 25,000 unique events extracted from multiple data sources and provide multiple inferences under different inference dimensions by crowd-sourcing on amazon mechanical turk. event2mind and atomic contain 2.6 and 3.6 inferences on average per example, respectively. event2mind focuses on three inference dimensions related to mental states of participants (i.e. intents and reactions of the events participants), while atomic has broader inference dimensions including mental states, probable preand post conditions of the event, and persona status. more details about the two datasets are provided in the appendix a. ## overview of the approach we present our approach in this section, which first retrieves relevant evidence from a large text corpus, and then utilizes retrieved evidence as background knowledge to generate inferences. figure 2 gives an overview of our approach. ## conclusion in this paper, we present an evidence-aware generative model based on vq-vae, which utilizes discrete semantic latent variables to select evidence as background knowledge to guide the generation. experimental results show that our approach achieves state-of-the-art performance on event2mind and atomic datasets. further analysis shows that our approach selectively uses evidence to generate different inferential texts from multiple perspectives. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.