Integrating Deep Event-Level and Script-Level Information for Script Event Prediction (2021)

Abstract: Scripts are structured sequences of events together with the participants, which are extracted from the texts. Script event prediction aims to predict the subsequent event given the historical events in the script. Two kinds of information facilitate this task, namely, the event-level information and the script-level information. At the event level, existing studies view an event as a verb with its participants, while neglecting other useful properties, such as the state of the participants. At the script level, most existing studies only consider a single event sequence corresponding to one common protagonist. In this paper, we propose a Transformer-based model, called MCPredictor, which integrates deep event-level and script-level information for script event prediction. At the event level, MCPredictor utilizes the rich information in the text to obtain more comprehensive event semantic representations. At the script-level, it considers multiple event sequences corresponding to different participants of the subsequent event. The experimental results on the widely-used New York Times corpus demonstrate the effectiveness and superiority of the proposed model.

## introduction scripts, consisting of structured sequences of events, are a kind of knowledge that describes everyday scenarios @xcite . a typical script is the restaurant script, which describes the scenario of a person going to a restaurant. in this script, "customer enter restaurant", "customer order food", "customer eat food" and "customer pay bill" happen successively. this structured knowledge is helpful to downstream natural language processing tasks, such as anaphora resolution @xcite and story generation @xcite . script event prediction aims to predict the subsequent event based on the historical events in the historical events candidate events customer enter restaurant customer enter restaurant customer sit down customer sit down customer order food customer order food waiter offer food waiter offer food customer criticize waiter customer criticize waiter customer praise waiter customer praise waiter script. what is vital to this task is to understand the historical events comprehensively. therefore, two categories of information are essential, namely, the event-level information and the script-level information. the event-level information contains necessary elements to describe the events, such as the verbs and their participants, while the script-level information describes how the events are connected and structured, such as via the temporal order or a common participant. the existing studies only consider the verb with its participants (usually the headwords) at the event level. however, this event representation method suffers from a lack of necessary information to derive a more accurate prediction. there exists other important properties of the events in the original texts, such as the intention and the state of the participants. for instance, as shown in figure 1 , if the waiter's service is friendly, the customer will be more likely to praise the waiter. if the customer is irritable, he/she will be more likely to criticize the waiter. unfortunately, the current formulation does not consider these features. from the aspect of the script-level information, existing studies only model the events that share a specific protagonist. these events are organized into a sequence by temporal order, which is thus called the narra-tive event chain. however, an event may contain multiple participants, each of which has its own influence on the occurrence of the event. as shown in figure 1 , the two participants, i.e., the customer and the waiter, take their own actions and jointly determine whether the customer will criticize or praise the waiter. motivated by these observations, at the event level, we trace the events back to their original texts and consider all the constituents in the texts that describe the events. then, we utilize the informative constituents to obtain more comprehensive event semantic representations. with respect to the script level, we view the script as multiple narrative event chains derived from the participants of the subsequent event. via modeling the narrative event chains of multiple participants, we are able to capture their behavior trends and predict the occurrence of the subsequent event. thus, by integrating the above deep event-level and script-level information, we propose mcpredictor, a transformerbased script event prediction model. at the event level, mcpredictor contains an event encoding component and a text encoding component. via aggregating the output of the two components, it obtains more comprehensive event semantic representations. at the script level, mcpredictor contains a chain modeling component and a scoring component. the former integrates the temporal order information into event representations. the latter aggregates the influence of multiple narrative event chains through an attention mechanism to predict the subsequent event. in general, this paper has the following main contributions: ## related work recent studies on script event prediction start from @xcite , which proposes the basic structure of the narrative event chain with a specific participant (called the protagonist). each event is represented as a tuple of the verb and the dependency relation between the verb and the protagonist, i.e., verb, dependency . then, it uses pointwise mutual information (pmi) to measure the score of two events to be in the same narrative event chain. finally, it aggregates the pairwise scores to infer a candidate's probability of being the subsequent event of the narrative event chain. @xcite pointed out that the event representation method mentioned above may lose the co-occurrence information between a subject and its object. therefore, they represented an event as a subject, verb, object triple. pichotta and mooney (2014); granroth-wilding and clark (2016) further extended the event representation method by taking the indirect object into consideration. following studies on script event prediction are mainly based on this event representation method. the above symbolic representation methods may cause the sparsity problem. therefore, distributed event representation method is applied in more recent studies @xcite . in addition, early studies aggregate the scores of each event in the narrative event chain and the candidate event, ignoring the temporal order of the events. therefore, @xcite ; @xcite ; @xcite introduced lstm to integrate temporal order information foot_0 . @xcite further converted the narrative event chain to a narrative event evolutionary graph and used graph neural networks to model it. conventionally, an event contains a verb and several participants, where the headwords are used to represent the participants. this event representation method suffers from a lack of information since only a few words are considered. therefore, @xcite used if-then commonsense knowledge to enrich the event representation. @xcite ; @xcite ; @xcite labeled extra discourse relations be-tween the events according to the conjunctions between them. zheng et al. (2020); lv et al. (2020); lee et al. (2020) introduced pre-trained language models, such as bert @xcite , to script event prediction and achieved excellent performance. still, these studies lose some of the informative constituents in the texts that directly describe the events. the above models all focus on a single narrative event chain. the only study that considers the multiple chains is @xcite . however, our study is different from it in the following aspects: (1) when representing an event, it only considers the verb and the dependency relation between the verb and the protagonist, while we keep all the participants in an event; (2) it adopts the symbolic event representation method and a pair-based model (pmi), while we use distributed event representation method and consider the temporal order information. ## problem statement in this paper, an event @xmath0, a 0 , a 1 , a 2 , t consists of a verb v, three participants foot_2 (i.e., subject a 0 , object a 1 , and indirect object a 2 ), and its original sentence t where the event appears. here, @xmath1, w 1 , ...} consists of a sequence of words. the script event prediction task aims to predict the most probable subsequent event e * given the candidate event set s and the historical events h. here, @xmath2, e c 1 , ..., e c m-1 } consists of the m candidate events and @xmath3, e 1 , ...} consists of the happened events. note that, since the candidate events have not happened yet, there are no corresponding sentences for them. ## the mcpredictor model in this section, we will describe our mcpredictor model. as show in figure 2 , it consists of four main components: (1) event encoding, (2) text encoding, (3) chain modeling, and (4) scoring. we will describe the model in single-chain mode and then introduce how the results from the multiple chains (derived from different participants of the candidate event) are aggregated. finally, we will list the variants of the scoring component. ## experiments in this section, we compare mcpredictor with a number of baselines to validate its effectiveness. in addition, we investigate the variants of mcpredictor and the importance of different constituents. finally we conduct case studies on the model. ## conclusion and future work in this paper, we proposed the mcpredictor model to handle the script event prediction task, which integrates deep event-level and script-level information. at the event level, mcpredictor additionally utilizes the original sentence to obtain more informative event representation. at the script level, it considers multiple event sequences corresponding to different participants of the subsequent event. experimental results demonstrate its merits and superiority. however, currently, we still need to know the participants of the candidate events to extract multiple chains. when participants are unknown, we have to enumerate all possible combinations of entities in the script, which is time-consuming. moreover, extracting informative constituents of events besides the verb and their participants is still a challenge. in the future, we will study these problems.