Heterogeneous Graph Neural Networks to Predict What Happen Next (2020)

Abstract: Given an incomplete event chain, script learning aims to predict the missing event, which can support a series of NLP applications. Existing work cannot well represent the heterogeneous relations and capture the discontinuous event segments that are common in the event chain. To address these issues, we introduce a heterogeneous-event (HeterEvent) graph network. In particular, we employ each unique word and individual event as nodes in the graph, and explore three kinds of edges based on realistic relations (e.g., the relations of word-and-word, word-and-event, event-and-event). We also design a message passing process to realize information interactions among homo or heterogeneous nodes. And the discontinuous event segments could be explicitly modeled by finding the specific path between corresponding nodes in the graph. The experimental results on one-step and multi-step inference tasks demonstrate that our ensemble model HeterEvent[W+E] can outperform existing baselines.

## introduction event chain, also known as script @xcite , is a structural knowledge format that models stereotypical human activities in a given scenario, e.g., "dining at a restaurant" and "catching a thief" in fig. 1 . representing such knowledge in a machine-readable way can help machine understand the semantics of natural language and further perform human-like inferences. besides, event representation can also support a series of downstream applications, such as question answering @xcite , discourse understanding @xcite and information extraction @xcite ), text classification @xcite , etc. existing work on event representation mainly model event chain from three aspects, the intra-event based @xcite , the individual-event based @xcite and the event-segment @xcite based models. these methods concentrate on depicting the relations among homogeneous modeling objectives, e.g., the inter-event relations. however, the relations among heterogeneous ones, e.g., the subordinated relations between word and event, which are also critical to the event chain, have not yet been taken into account in previous work. besides, @xcite found the event segments, a set of individual events related to each other, were helpful to predict the missing event and event segments could be continuous or discontinuous (see fig. 1 ). although the self-attention mechanism can implicitly represent such discontinuous event segments by greedily making connections among all events @xcite , it inevitably introduces noises. in this paper, we attempt to deal with these issues by proposing a heterogeneous-event (heterevent) graph network . specifically, we define two different types of nodes in the heterevent graph, including word and event nodes, which respectively represent unique words and individual events in the event chain. then we construct three types of edges, namely word-word edges denoting the co-occurrence relations within word nodes, word-event edges denoting the subordinate relations between word and event nodes, and event-event edges denoting the order relations within event nodes. we also design a message x walk to restaurant x read menu x order food x eat food x make payment x steal money x chased by y x run to subway x blocked by door x got arrested by y continuous event segment discontinuous event segment figure 1: examples of continuous and discontinuous event segments passing process to realize information interactions among homo or heterogeneous nodes. furthermore, the heterevent graph can explicitly represent discontinuous event segments by finding a path between their corresponding nodes in the graph. we evaluate our proposal on one-step (granroth-wilding and clark, 2016) and multi-step (lee and goldwasser, 2018) inference tasks, and the experimental results prove that our proposals present a stronger inference ability than existing baselines in event prediction. our research contributions in this paper are in three folds: 1. to the best of our knowledge, we are the first to construct a heterogeneous graph network to model the event chain. 2. our model outperforms the existing baselines on one-step and multi-step inference tasks. 3. our proposed heterevent is a expandable framework that can be easily adapted to other granularities of information nodes, e.g., subwords or event scenario, which both are a part of the event chain. ## related work event chain models human understandings about the relevant causal relationships among events. an event chain can be used to infer how events will unfold in a given scenario @xcite . restricted to the manual acquisition, early work on event chain shows a slow progression until narrative event chains introduced by @xcite . @xcite assumed that although a narrative script had several participants, there was a central actor (i.e., protagonist) who characterized a narrative chain. in this assumption, probabilistic co-occurrence-based models combined with dependency parser can realize the automatic extraction of narrative event chains from raw text. they also casted narrative events as the @xmath0, dependency type >, where the predicate was a verb lemma and the dependency type denoted a grammatical dependency relation between the predicate and the protagonist, e.g., 'subj', 'obj' or 'iobj'. besides, @xcite explored a richer representation over multi-argument event format. from the perspective of modeling objectives, existing event representation works can be classified into three main types, i.e., the intra-event based, the individual-event based and the event-segment based models. firstly, intra-event based methods concentrate on the multiplicative interaction among intraevent elements. for instance, @xcite simply concatenated predicate and argument embeddings and fed them into a neural network to get the event representation. while @xcite used the tensor-network-based model to capture more subtle semantic interactions. secondly, individual-event based methods mainly investigate the complex and diverse relations between two individual events. @xcite utilized the lstm hidden states to integrate the chain order information into event model. @xcite extended the narrative event chains into the narrative event evolutionary graph to model the dense connections among events. while @xcite broadened the single relation (time-order relation) into the diverse ones based on the discourse relations from pdtb @xcite . thirdly, event-segment based methods focus on a set of semanticrelated events. @xcite developed self-attention mechanism to implicitly model relations in event segments. @xcite proposed a unified fine-tuning framework to integrate the training losses from different layers. besides, by jointly training the event representation model and external knowledge, some work intended to mine the potential connections between narrative event chains and external knowledge. @xcite introduced atomic @xcite to obtain the sentiment and intent information of event. different from the above methods, our work attempts to synthetically represent multi-granularity information and discontinuous event segments contained in a event chain by a heterogeneous graph network, which can provide strong inferring abilities on the event prediction it also worths noting that the het-erevent graph is a scalable framework that can be easily adjusted to fusion other grained information, e.g., subwords or event scenario. ## methodology in this paper, we aim at learning the event representation to predict the missing event. the research problem of event prediction in a narrative event chain is defined as follows. given an incomplete event chain {e 1 , e 2 , • • • , e n } and a set of candidate events {e c 1 , e c 2 , • • • , e cm }, our goal is to choose the correct one from candidate events for the missing event in the event chain. as fig. 2 shows, the overall architecture of heterevent can be divided into the following three components: an encoding layer, a graph layer and a prediction layer. (1) encoding layer aims to transform words and events into a distributed representation. (2) graph layer first performs the heterogeneous graph construction, where each individual word and event are defined as nodes, three types of relations including word-word, word-event and event-event, are extracted as edges. then the message passing layer is employed to realized information interactions among homo or heterogeneous nodes. (3) prediction layer calculates probabilities of candidate events as the missing event conditioned on the representation learned from the graph layer. ## conclusion and future work in this paper, we introduce a novel heterogeneous-event graph network (heterevent) for the event representation to predict the missing event. based on the characteristics of event chain, we explore two types of nodes (i.e., word and event nodes) and three kinds of edges (i.e., the relations of @xcite to construct a heterogeneous graph. experimental results on five inference datasets demonstrate that our graph-based model can effectively encode homo and heterogeneous relations as well as multi-hop connections, which help heterevent [w +e] to achieve the best performance compared to all discussed models. as to future work, on the one hand, we plan to investigate how to incorporate more granularities of nodes into the heterogeneous graph, e.g., subwords, event segments or event scenario. on the other hand, we plan to further extend and refine the type of edges, since multi types of inter-event relations have been proven effective in eventtranse @xcite .