Incorporating Circumstances into Narrative Event Prediction (2021)

Abstract: The narrative event prediction aims to predict what happens after a sequence of events, which is essential to modeling sophisticated real-world events. Existing studies focus on mining the inter-events relationships while ignoring how the events happened, which we called circumstances. With our observation, the event circumstances indicate what will happen next. To incorporate event circumstances into the narrative event prediction, we propose the CircEvent, which adopts the two multi-head attention to retrieve circumstances at the local and global levels. We also introduce a regularization of attention weights to leverage the alignment between events and local circumstances. The experimental results demonstrate our CircEvent outperforms existing baselines by 12.2%. The further analysis demonstrates the effectiveness of our multi-head attention modules and regularization.

## introduction the narrative event chain, which is similar to the classical notion of the script @xcite , is a structural knowledge that captures the relationships between event sequences and their participants in the given scenario. figure 1 describes a scenario of "going to the restaurant.". modeling the narrative event chain can help the ai systems to understand sophisticated real-world events and benefit many downstream applications @xcite , such as financial analysis @xcite . this paper focuses on modeling the narrative event chain and predicting what will happen next, which is called the multiple choice narrative cloze (mcnc) @xcite . as shown in figure 1 , the mcnc evaluation aims to choose the correct event from the candidate choices set given a sequence of historical events. early studies learn event representation with the rule-based @xcite , countbased @xcite , and deep learning @xcite method. recently, more and more studies attempt to incorporating external knowledge into event representation. @xcite builds the narrative event evolutionary graph (neeg), which describes event evolutionary principles and patterns. feel @xcite introduces a feature enriched event embedding. despite the subject, predicate, and object, feel also considers sentiment and animacy as the parts of events. @xcite regards event embedding learning as a multi-relational problem and captures different relations of events pairs, such as the cause and contrast. @xcite builds a heterogeneous event graph to mining subordinated relations between events and words. in addition to the previous events that have already happened, particular situations also affect the event evolution, which are defined by circumstances in this paper. the event circumstances include detailed descriptions of the event situation such as the weather, the place status, and the protagonist behavior. as the example shown in figure 1 , existing works tend to predict choice (c) or (d) based on previous events or historical knowledge for the event walk (peter, restaurant). given different circumstances shown in figure 2 , they could not make a different decision based on the restaurant's environment, such as described in circumstances 2 and 3. peter is more likely to get seated if the restaurant has a few customers. he will ask the waiter about the available seats if the restaurant is popular, meaning it is crowded. circumstances, such as the crowdedness of the restaurant, weather, or the protagonist action et al., will influence the event evolutionary, while existing works do not consider them. in this paper, we propose circevent to represent events together with their circumstances. following previous studies @xcite , events in this paper are also extracted from the unstructured text corpus, and each event belongs to one specific sentence in the text. the extracted event only contains the minimum information of an event, e.g., the subject, predicate, and object. the contextual information, environment description, and semantics, which we discussed as circumstances, are left in the original sentence. we attempt to collect event circumstance information from the unstructured text. however, the unstructured text contains so much information that not all contribute to the event evolution. to tackle this challenge, we develop two multihead attention-based networks to incorporate event representation and its circumstance into narrative event prediction at the local and global levels. at the local level, events come from a specific sentence, containing the most related circumstance. we develop the a multi-head attention to retrieve the local circumstances for events. moreover, the context local circumstances also contribute the event representation. we develop another multihead attention to get the global circumstances by aggregating the context local circumstances adaptively. after the circumstances retrieval, we adopt the transformer as backbone to encode the context events and circumstances. the transformer decoder is used to compute the similarity scores of candidate events. the candidate events are compared implicitly inner the transformer decoder benefited from its architecture. our contributions in this paper are three folds: ## model this section introduces our circevent neural network in four modules: the event representation, the circumstance representation, the event chain encoder and the prediction module. ## experiment in this section, we describe the dataset and the preprocessing pipeline. we evaluate our model in the mcnc task and report the accuracy score. ## conclusion this paper develops multi-head attention modules to capture the circumstances from event text at local and global levels. we utilize the transformer architecture, encode context events and circumstances. the standard evaluation shows that our model achieves the best accuracy score compared to other baselines. the visual analysis on the attention heat map shows the effectiveness of circumstances.