A Graph Enhanced BERT Model for Event Prediction (2022)

Abstract: Predicting the subsequent event for an existing event context is an important but challenging task, as it requires understanding the underlying relationship between events. Previous methods propose to retrieve relational features from event graph to enhance the modeling of event correlation. However, the sparsity of event graph may restrict the acquisition of relevant graph information, and hence influence the model performance. To address this issue, we consider automatically building of event graph using a BERT model. To this end, we incorporate an additional structured variable into BERT to learn to predict the event connections in the training process. Hence, in the test process, the connection relationship for unseen events can be predicted by the structured variable. Results on two event prediction tasks: script event prediction and story ending prediction, show that our approach can outperform state-of-the-art baseline methods.

## introduction understanding the semantics of events and their underlying connections is a long-standing task in natural language processing @xcite . much research has been done on extracting script knowledge from narrative texts, and making use of such knowledge for predicting a likely subsequent event given a set of context events. a key issue to fulfilling such tasks is the modeling of event relation information. to this end, early work exploited event pair relations @xcite @xcite and temporal information @xcite . the former has been used for event prediction by using embedding methods, where the similarity between subsequent events and context events are measured and used for candidate ranking. the latter has been used * corresponding author figure 1 : (a) an example for event prediction. (b) given an event sequence, retrieval-based methods lookup structural information of events from event graph. however, in the test process, part of events may be not covered by the event graph, hence their connection information is unavailable. different from retrieval-based methods, graphbert is able to predict the connection strength between events. for neural network methods, where models such as lstms have been used to model a chain of context events. there has also been work integrating the two methods @xcite . despite achieving certain effectiveness, the above methods do not fully model the underlying connection between context events. as shown in figure 1 (a), given the facts that jason had been overstretched at work, he decided to change job and jason finds a new job, the subsequent event jason is satisfied with his new job is more likely than jason feels much stressed at his new job, which can be inferred by understanding the fact that the reason for his new job search is stress in his job. @xcite and @xcite consider such context structure by building event evolutionary graphs, and using network embedding models to extract relational features. for these methods, event graphs serve as a source of external structured knowledge, which are extracted from narrative texts and provide prior features for event correlation. one limitation of their methods is that the effectiveness of their methods heavily relies on the coverage of the event graph. as shown in figure 1 (b), @xcite and @xcite 's methods work by looking up the event tuples in the event graph to retrieve the connection information between events for predicting the output. this is done by the standard knowledge graph lookup operation. however, if the context events are not in the event graph, the method cannot find relevant information. figure 1 (b) shows an extreme case. in event sequence β, although the context events be starving and go for a meal are highly similar to the event graph content feel hungry and go for lunch, the retrieval-based methods can fail to match context events in the event graph and utilize the event graph knowledge. however, in practice, it is infeasible to construct an event graph that covers most of the possible events. as an event is the composition of multiple arguments, so the same event can correspond to various semantically equivalent expressions, such as "feel hungry" vs "be starving", or "hunger", etc. this would limit the performance of the retrieval-based systems. to address this issue, we consider automatically predicting the event links using a graph-enhanced bert model (graphbert). as shown in figure 1 (b), we collect event structure information into a bert model with graph structure extension. given a set of event contexts, we use the graph-bert model to construct an event graph structure by predicting connection strengths between context events, instead of retrieving them from a prebuilt event graph. specifically, we extend the bert model by introducing a structured variable, which captures the connection strengths between events. as shown in figure 2 , during training, both context events and external event graph information are used to train the structured variable. during testing, the structured variable which describes connection strengths between events is obtained using the context event only, which is used for finding the next event. subsequently, we encode the predicted link strength for making a prediction. experimental results on standard datasets show that our model outperforms baseline methods. further analysis demonstrates that graphbert can predict the connection strengths for unseen events and improve the prediction accuracy. the codes are publicly available at https://github.com/ sjcfr . ## background as shown in figure 1 (a), the task of event prediction @xcite can be defined as choosing the most reasonable subsequent event for an existing event context. formally, given a candidate event sequence @xmath0, . . . , xe t , xe c j }, where {xe 1 , . . . , xe t }are t context events and xe c j is the c j th candidate subsequent event, the prediction model is required to predict a relatedness score @xmath1, 1] for the candidate subsequent event given the event context. event graphs @xcite have been used to represent relationships between multiple events. formally, an event graph could be denoted as ## baseline system before formally introducing the graphbert framework, we first introduce a retrieval-based baseline system. as figure 2 (a) shows, given an event sequence @xmath2, . . . , xe t , xe c j }, the baseline system retrieves the corresponding structural information for each event within x from a prebuilt event graph g, and then integrates the retrieved structural information into the bert frame for predicting the relatedness score y . for an arbitrary event tuple (x e i , x e j ), if it is covered by the event graph g (i.e., both x e i and x e j are nodes of g), then we can retrieve the corresponding node embeddings e i and e j , together with the edge weight a ij by matching the event tuple in the event graph. the representation vector of the events within x further form into an embedding matrix e, and the edge weights form into an adjacency matrix a. to make use of the retrieved structural information for enhancing the prediction process, we first employ a graph neural network to combine the event representation matrix and the adjacency matrix: then the combined event graph knowledge can be merged into the frame of bert for enhancing the prediction process. to this end, we employ an attention operation to softly select relevant information from the updated event representations e (u ) , and then update the hidden states of bert. specifically, we take the hidden states of the s 1 th transformer layer of bert (denoted as h s 1 ) as the query, and take the updated event representation e (u ) as the key: h s 1 * contains both the node feature information and the connection information between events. by taking h s 1 * as the input of the subsequent (s 1 + 1)th transformer layers of bert, the event prediction process is enhanced with the predicted event graph knowledge. this retrieval-based baseline system can be regarded as the adaption of @xcite and @xcite 's retrieval-based methods on a pretrained model bert. ## graphbert a critical weakness of the retrieval-based baseline system is that it heavily relies on the coverage of the event graph. in other words, if an event is not covered by the event graph, then the structural information (i.e., node features and the adjacency matrix) would be absent from the constructed event graph, which further limits the model performance. in this paper, we propose a predictive-based framework graphbert. graphbert uses the transformer layers of bert as an encoder to obtain the representation for arbitrary events, and then learns to predict the link strength between events in the training process, so that the sparsity issues in the retrieval process can be avoided. to this end, as figure 2 (b) shows, in contrast to the retrieval-based baseline system, we introduce two more modules: (1) an aggregator to obtain event representations from the bert framework; ## experiments we evaluate our approach on two event prediction tasks: multiple choice narrative cloze task (mcnc) @xcite and story cloze test (sct) @xcite by constructing an event graph based on the training set of mcnc to train the graphbert model and then adapts the graphbert model trained on the mcnc dataset to the sct dataset to evaluate whether graphbert can predict the link strength between unseen events to enhance the prediction performance. ## discussion the graphbert model employs a structure variable â to capture the "is next event" relationship between events. by introducing more parallel structural variables { â1 , . . . , âk }, it can be extended to simultaneously learn multiple kinds of event relationships, such as temporal or causal relationship. furthermore, previous researches demonstrate that the graph-structured relationship extensively exist between other semantic units, such as sentences @xcite , or even paragraphs (sonawane and kulkarni, 2014). however, similar to the situation in event graph, it would be impractical to construct knowledge graphs that cover all possible connection relationships between all the sentences or paragraphs. this restricts the applicable of retrieval-based methods in these situations. on the contrary, our generative approach suggests a potential solution by learning the connection relationship from graph-structured knowledge base with limited size, then generalizing to the unseen cases. ## related work the investigation of scripts dates back to 1970's @xcite . the script event prediction task models the relationships between abstract events. previous studies propose to model the pair-wise relationship @xcite @xcite or event order information @xcite @xcite for predicting the subsequent event. @xcite and @xcite propose to leverage the rich connection between events using graph neural network and attention mechanism, respectively. different from script event prediction, the story cloze task @xcite focuses on concrete events. therefore, it requires prediction models to learn commonsense knowledge for un-derstanding the story plot and predicting the ending. to this end, @xcite and @xcite propose to combine context clues with external knowledge such as kgs. @xcite finetune pretrained language models to solve the task. compared to their works, our approach can use both the language knowledge enriched in bert to promote the comprehension of event context, and the structural information from event graph to enhance the modeling of event connections. a recent line of work has been engaged in combining the strength of transformer based models with graph structured data. to integrate kg with language representation model bert, @xcite encode kg with a graph embedding algorithm transe @xcite , and takes the representation of entities in kg as input of their model. however, this line of work only linearizes kgs to adapt the input of bert. graph structure is not substantially integrated with bert. @xcite and @xcite propose retrievalbased methods to leverage the structural information of kg. however, in the event prediction task, the diversity of event expression challenges the coverage of the event graph, and prevents us from simply retrieving events in the test instances from the event graph. we propose to integrate the graph structural information with bert through a predictive method. compared to retrieval-based methods, our approach is able to learn the structural information of the event graph and generate the structural information of events to avoid the unavailable of structural information in test instances. ## conclusion we devised a graph knowledge enhanced bert model for the event prediction task. in addition to the bert structure, graphbert introduces a structured variable to learn structural information from the event graph, and model the relationship between the event context and the candidate subsequent event. compared to retrieval-based methods, graphbert is able to predict the link strength between all events, thus avoiding the (inevitable) sparsity of event graph. experimental results on mcnc and sct task show that graphbert can improve the event prediction performances compared to state-of-the-art baseline methods. in addition, graphbert could also be adapted to other graph-structured data, such as knowledge graphs.