Generating Temporally-ordered Event Sequences via Event Optimal Transport (2022)

Abstract: Generating temporally-ordered event sequences in texts is important to natural language processing. Two emerging tasks in this direction are temporal event ordering (rearranging the set of events to correct order) and event infilling (generating an event at a specified position). To tackle the two related tasks, the existing method adopts a vanilla sequence-to-sequence model via maximum likelihood estimation (MLE). However, applying this approach to these tasks will cause two issues. One issue is that the MLE loss emphasizes strict local alignment and ignores the global semantics of the event. The other issue is that the model adopts a word-level objective to model events in texts, failing to evaluate the predicted results of the model from the perspective of event sequence. To alleviate these issues, we present a novel model to tackle the generation of temporally-ordered event sequences via Event Optimal Transport (EOT). First, we treat the events in the sequence as modeling units and explicitly extract the semantics of the events. Second, to provide event sequence-level evaluation of the predicted results of the model, we directly match events in sequences. Extensive experimental results show that our approach outperforms previous models on all evaluation datasets. In particular, the accuracy is improved by 7.7%, and the Macro F1 is improved by 7.2% on one of the datasets.

## introduction generating temporally-ordered event sequences in texts is crucial to many artificial intelligence applications, such as discourse understanding @xcite , dialog generation @xcite and stock prediction @xcite . temporal event ordering and event infilling are two challenging tasks in this line of work. the former refers to the rearrangement of an unordered sequence of events to an ordered sequence of events, and the latter refers to inferring missing events in an incomplete sequence of events. figure 1 shows an example of the temporal event ordering task and event infilling task in (a) and (b), respectively. for the temporal event ordering, given an unordered sequence (e 3 , e 2 , e 4 , e 1 ) as input, the output is ordered sequence (e 1 , e 2 , e 3 , e 4 ). for the event infilling, given an incomplete sequence (e 1 , e 2 , e 4 ) as input, the output is complete sequence (e 1 , e 2 , e 3 , e 4 ) with the infilled event e 3 . to handle these two related tasks, the currently existing method @xcite employs a unified generation model which is shown in figure 1 (c ). their model takes incomplete unordered events as input sequence and outputs a complete ordered event sequence, based on the sequence-to-sequence (seq2seq) model via maximum likelihood estimation (mle), which maximizes the likelihood of the next word conditioned on its previous ground-truth words. such an approach adopts cross-entropy loss as the objective, essentially measuring the word difference at each position of the target sequence and providing a word-level training loss. although their model has shown good performance in handling the two tasks, there are still two issues. first, the mle loss emphasizes strict local alignment and ignores the global semantics of the event. for example in figure 2 , the mle loss will give the predicted event a a high score and b a low score because all words except one in event a are aligned with the words in the answer event yet no word event b is aligned with the words in the answer event. however, the loss is expected to give event a a low score and event b a high score because from the perspective of event semantics, event a is completely different from the answer event and event b is the same as the answer event. therefore, we should consider the overall semantics of the event, rather than the strict alignment of words within the event. second, the goal of the model is to infer events in texts, but the sequence-to-sequence model uses a word-level objective so it fails to evaluate the result from the perspective of event sequence. for example in figure 1 , the correct predicted event sequence is (e 1 , e 2 , e 3 , e 4 ). if the model predicts an event sequence (e 4 , e 1 , e 2 , e 3 ), the mle loss will give it a low score because no event is predicted correctly compared to the answer event sequence. however, the sequence (e 4 , e 1 , e 2 , e 3 ) should not get a low score, because at least the orders of 3 events are predicted correctly. therefore, how to evaluate the predicted results of the model from the perspective of event sequence is a challenging problem. to tackle the above issues, we introduce a novel method for the generation of temporallyordered event sequences via event optimal transport (eot). specifically, for the first issue, we treat the events in the sequence as modeling units and explicitly extract the semantics of the events. for the second issue, we propose to use optimal transport to directly match events in sequences. the eot allows end-to-end supervised training and acts as an effective sequence-level regularization to the mle loss. in summary, our contributions can be summarized as follows: script event prediction given context event sequence, script event prediction aims at predicting the subsequent event from a candidate list. the task is first proposed by chambers and jurafsky (2008), and a statistical model is proposed to learn the cooccurrence between events. jans et al. (2012) leverages a bigram model to model the temporal order between events explicitly. the above two methods are count-based, and then researchers have proposed methods based on neural networks. granroth-wilding and clark (2016) proposes a neural network based model for simultaneously learning word embedding and composition function. in wang et al. (2017), they propose an lstm based model to integrate order information and event relation. li et al. (2018) treats event chain as a subgraph and leverages recurrent networks to better model relatedness between events in the candidate list with events in the graph. lv et al. (2019) proposes a model that integrates event-level and chainlevel attentions to better leverage information contained in event chain. zhou et al. (2021) proposes a multi-task self-supervised model to cope with the problem of lack of training data in script event prediction. to incorporate event circumstances into the narrative event prediction, wang et al. (2021) adopts the two multi-head attention to retrieve circumstances at the local and global levels. in this paper, we consider two related tasks of temporal event ordering and event infilling and leverage a single generation model to tackle these two tasks. optimal transport in nlp optimal transport has been applied in nlp for a variety of tasks recently. in kusner et al. (2015), the author proposes the word mover's distance (wmd), a bart encoder bart decoder ipot l eot output event semantics extraction event semantics extraction transportation matrix cost matrix event features word features input event optimal transport novel distance function between text documents that measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document. later in @xcite , a novel wasserstein method with a distillation mechanism is proposed, yielding joint learning of word embeddings and topics. the cross-lingual correspondence problem is cast as an optimal transport problem in @xcite , and the gromov-wasserstein distance is exploited for the alignment of word embedding spaces. a content-aware sparse attention module based on optimal transport is proposed in @xcite to deal with textual network embedding problem. @xcite proposes student-forcing optimal transport to tackle the exposure bias problem in text-generation models trained by maximum likelihood estimation. @xcite formulates the quest of vocabularization -finding the best token dictionary with a proper size -as an optimal transport problem and proposes a simple and efficient solution without trial training. a time-aware optimal transport distance is introduced @xcite for learning the model to compress the eventgraphs of news articles in an unsupervised manner. in this paper, we leverage optimal transport and propose event ot to directly match events in event sequences to provide sequence-level supervision of the predicted results. ## methodology the overall structure of our model is illustrated in figure 3 . the input of the model is a sequence of events @xmath0, ..., e n ), which is incomplete and unordered, and the output of the model is a complete and ordered sequence of events @xmath1, ..., e m ). we represent e in the event sequence as the concatenation of its predicate with all its arguments. ## conclusion in this paper, we consider a single generation model which can support inferences in the two related tasks. we introduce a novel method for the generation of temporally-ordered event sequences via event optimal transport (eot). compared with the mle-based seq2seq model, our approach has two advantages: (i) we treat the events in the sequence as modeling units and explicitly extract the semantics of the events; (ii) we directly match events in sequences to provide event sequence-level evaluation of the predicted results of the model. experimental results show the superiority of our model on all evaluation datasets. specifically, the accuracy is improved by 7.7%, and the macro f1 is improved by 7.2% on one of the datasets.