Conditional Generation of Temporally-ordered Event Sequences (2021)

Abstract: Models of narrative schema knowledge have proven useful for a range of event-related tasks, but they typically do not capture the temporal relationships between events. We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence. We use a BART-based conditional generation model that can capture both temporality and common event co-occurrence, meaning it can be flexibly applied to different tasks in this space. Our model is trained as a denoising autoencoder: we take temporally-ordered event sequences, shuffle them, delete some events, and then attempt to recover the original event sequence. This task teaches the model to make inferences given incomplete knowledge about the events in an underlying scenario. On the temporal ordering task, we show that our model is able to unscramble event sequences from existing datasets without access to explicitly labeled temporal training data, outperforming both a BERT-based pairwise model and a BERT-based pointer network. On event infilling, human evaluation shows that our model is able to generate events that fit better temporally into the input events when compared to GPT-2 story completion models.

## introduction this paper proposes a single model of events to support inferences in two seemingly different tasks: (1) temporal event ordering and (2) event infilling, or inferring unseen or unmentioned events occurring as part of a larger scenario. figure 1 shows an example illustrating these two goals. unlike prior approaches, we aim to address both with the same model architecture, rather than having to annotate data and build ad-hoc models for each task separately; our goal is to work towards models that cap- temporal bart i opened a present she gave me the present she bought the present complete ordered event sequence she bought the present i opened a present scrambled input events ture temporal event knowledge broadly and support a wide range of inferences. we thus need a suitably general modeling framework to capture temporal knowledge about events, which in our case will be a bart-based @xcite ) model we call temporalbart. note that classic temporal relation extraction models, which model temporal ordering in context for a particular document, may chiefly learn how to use local discourse cues rather than generalizable event knowledge @xcite . the goals in this work relate to past work on learning narrative schemas @xcite @xcite . our approach particularly follows a recent line of work using distributed representations of schemas @xcite , which support inferences about events without explicitly materializing a discrete schema library. the target tasks in this work are directly motivated by downstream applications of schema learning. text generation tasks like story completion rely on understanding what makes narratives plausible and what events might be likely to happen before, after, and between other events @xcite , motivating our event infilling task. answering questions about causes, effects, or what might happen next in a scenario requires knowing typical temporal orders of event sequences @xcite @xcite , motivating our temporal ordering task. prior work has not combined traditional event cooccurrence with event temporality as we do. we propose a conditional generation model to tackle temporal event ordering and event infilling, and train it as a denoising autoencoder over outof-context temporal event sequences. as shown in figure 1 , the encoder of our temporalbart model reads a temporally scrambled sequence of a subset of input events, obtained by corrupting a temporally-ordered sequence of events from a corpus. the decoder, which can be viewed as a conditional event language model @xcite @xcite , then reconstructs the complete, temporally-ordered event sequence. such denoising training has been successful exploited in many applications @xcite @xcite , and using seq2seq models to reorder and smooth inputs has been explored before @xcite , but to our knowledge we are the first to apply this in this temporal modeling setting. the conditional generation architecture of our model is flexible enough to address a variety of tasks, including our temporal ordering and event infilling tasks, by either sampling from the model or using it to score sequences. capitalizing on the success of recent pre-trained encoder-decoder transformers @xcite , our model itself is based on bart, consuming and producing predicate-argument structures rendered in surface order. gathering large-scale high-quality labeled data with temporal annotations is often expensive and requires specially designed annotation schemes @xcite @xcite . here, we instead turn to a narrative documents corpus, eventsnarratives @xcite and design an automatic method to extract the training data we need. in these documents, discourse order is loosely assumed to reflect temporal order, so events extracted from this text can directly provide training data for our models. this use of automatic annotation allows us to use broad-domain data, giving us a strong domain-independent temporal model @xcite . to evaluate how well our proposed models capture temporal knowledge and solve the two targeted tasks, we apply them on out-of domain test sets in a zero-shot manner. specifically, for event ordering, we first extract test temporal event sequences from the caters @xcite and mc-taco @xcite datasets, which include the annotations on temporal relations between events. we then compare the performance of our models with two baselines: a bert-based pairwise model and a bert-based pointer network. for event infilling, we use the test event sequences from caters and examine the ability of our models to order unseen events and generate infilled events in comparison with gpt-2 baselines from story generation. our bart-based models significantly outperform the baseline models on the ordering settings we consider, and human evaluation verifies that our models can generate infilled events that are better temporally-ordered with respect to the input. ## background and related work learning temporal knowledge to order events and generate new events as part of schemas or stories are two problems that have received significant attention, but in contrast to our work, previous work typically focuses on each in isolation. ## conclusion this work presents a bart-based conditional generation model and a denoising autoencoder framework to learn temporal event knowledge, and addresses both temporal ordering and event infilling tasks by pretraining on automatically collected data. our experiments demonstrate that our model is able to perform temporal ordering and infilling in a zeroshot manner, not fine-tuned on our target datasets, which suggests that it can also be applied to other settings requiring event schematic and temporal knowledge. nathanael chambers. 2013. event schema induction with a probabilistic entity-driven model. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1797-1807, seattle, washington, usa. association for computational linguistics. nathanael chambers, taylor cassidy, bill mcdowell, and steven bethard. 2014. dense event ordering with a multi-pass architecture. transactions of the association for computational linguistics, 2:273-284. nathanael chambers and dan jurafsky. 2008. unsupervised learning of narrative event chains. in proceedings of acl-08: hlt, pages 789-797, columbus, ohio. association for computational linguistics. nathanael chambers, shan wang, and dan jurafsky. 2007. classifying temporal relations between events. in proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 173-176, prague, czech republic. association for computational linguistics. fei cheng and yusuke miyao. 2017. classifying temporal relations by bidirectional lstm over dependency paths. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pages 1-6, vancouver, canada. association for computational linguistics. a scoring orderings with temporalbart-indexed (tags only) temporalbart-indexed (tags only) scores whether an output sequence y is temporally ordered by gathering the generation scores on the special tokens [ei] only as its final ordering score: p tag (y|x) = @xmath0, w y 1 , • • • , w y t-1 ) since many of the generation decisions here are copying event arguments, the prediction could be largely affected by the correlation of tokens within each argument.