Don’t Let Discourse Confine Your Model: Sequence Perturbations for Improved Event Language Models (2021)

Abstract: Event language models represent plausible sequences of events. Most existing approaches train autoregressive models on text, which successfully capture event co-occurrence but unfortunately constrain the model to follow the discourse order in which events are presented. Other domains may employ different discourse orders, and for many applications, we may care about different notions of ordering (e.g., temporal) or not care about ordering at all (e.g., when predicting related events in a schema). We propose a simple yet surprisingly effective strategy for improving event language models by perturbing event sequences so we can relax model dependence on text order. Despite generating completely synthetic event orderings, we show that this technique improves the performance of the event language models on both applications and out-of-domain events data.

## introduction event-level language models (lms) provide a way to reason about events, and to approximate schematic and script-like knowledge @xcite @xcite about them @xcite @xcite . these models aim to learn high-level representations of complex events (e.g., an arrest) and possibly their entity roles from raw text (e.g., a suspect). however, a major limitation is their reliance on the discourse order of event mentions when training the lm. although powerful, these event lms capture information we don't want in true world knowledge. for instance, a script of events may be weakly ordered in real life, but the system instead learns to strongly rely on the text order in which the events were described. figure 1 shows an example where discourse and actual temporal order are different: a model trained on newswire may learn the pattern on the left from obituaries, but will fail to generalize to biographical or other narrative descriptions of someone's life. in this paper, we aim to improve event-level lms in order to make them more suitable for general knowledge learning. while a range of possible modifications to the model can be imagined, such as set transformers @xcite , we want to leverage autoregressive pre-trained lms. we instead find that we can encode the necessary invariances via data augmentation: namely, we apply a set of event sequence perturbations to sequences in the training data to relax the model's dependence on discourse order. by considering the next event based on shuffled sequences of events, we encourage the model to treat the input more as a set of events rather than strictly as a discourse sequence. surprisingly, despite our disruption of discourse order, experiments show how perturbations can improve event language modeling of text, particularly when evaluating the model on other domains which present events in different orders (e.g., novels or blogs present data in more of a "narrative" fashion than news datasets common in nlp @xcite ). our experiments evaluate accuracy on the inverse narrative cloze task on in-domain newswire, as well as out-domain novels and blogs foot_0 . ## perturbing discourse sequences event language modeling tasks are typically defined over sequences of events as they appear in text. the events can be represented either as a sequence of words annotated with predicateargument structure (e.g., semantic roles @xcite , open ie tuples @xcite or with compositional embeddings @xcite . generative models are trained to predict subsequent events in a sequence conditioning on previously observed events. naturally, these models learn the order in which events appeared in text @xcite . however, relying on discourse order may not be necessary and can potentially limit generalization of event lms. for some event related tasks such as schema learning @xcite , the discourse order is not directly relevant. for other tasks such as event ordering @xcite @xcite , temporal or logical order of events is most criticaldiscourse order, at best, is a noisy proxy. in fact, the first systems for schema learning were noticeably not language models @xcite @xcite . we introduce three simple perturbation techniques shown in figure 2 that relax the reliance on discourse sequences. ## experimental setup data we train event language models on the annotated nyt corpus using open ie event tuples extracted by ollie @xcite . the dataset contains a total of around 1.8 million articles. after preprocessing steps, 1,467,366 articles are used as the training set, 6k articles as test set and 4k articles as the dev set. each event is a 4-tuple (v,s,o,p) containing the verb, subject, object and preposition. we follow the same preprocessing steps outlined in @xcite to create event sequences. the components of the events (the verb, subject, etc.) are all individual tokens, and are treated like normal text. for example, the events (truck packed with explosives), (police arrested suspect), would be given to the model as: packed truck explosives with [tup] arrested police suspect null , where null is the null preposition token and [tup] is a special separator token between events. each document is first partitioned into segments of four sentences each. all events extracted from each segment are concatenated (in discourse order) to form an event sequence. this is a simple heuristic to avoid considering event sequences that can drift or connect otherwise unrelated events. tuples with common verbs (is, are, be, ...) and repeating predicates are also ignored. the training, development, and test splits have 7.1m, 19k, and 29k event sequences respectively. during training, depending on the perturbation strategy used, a number of sequences are added to the initial sets. the numbers are hyperparameters, selected differently for each model. details are given in the following sections. autoregressive models our baseline autoregressive event lm is a pretrained gpt-2 model (radford et al., 2019) fine-tuned on the event sequences. once the perturbations are applied to the original sequence, the modified sequence is used as both the input and the output of the model. we trained variants of gpt-2 with different sequence perturbations as shown in figure 2 in our experiments. for the dropout and masked versions, we created n/3 new sequences with n being the number of events in the sequence. each sequence has n/3 of its events either dropped or masked. autoencoding models we use hierarchical quantized autoencoder (haqae) @xcite as a strong autoencoding model. haqae is an lstm-based autoencoder, which uses a hierarchical latent space to model event sequences. haqae uses categorical global latent variables to represent a tree-structured hierarchy which allow it to model different types of schemas and their possible tracks. different levels of this hierarchical structure capture different levels of features of the schemas. for training the haqae model, instead of reconstructing a perturbed sequence, we explore a denoising style training objective, where we only perturb the input part of the sequence keeping the output the same as the original. our hypothesis is that these models learn a perturbation-invariant latent space representation in both cases, which will help break the dependence on discourse order. we use the denoising variant in our experiments as it worked better than the standard reconstruction objective in our initial experiments. for each sequence in the permutation model, we generated permuted sequences for 10% of the original sequences. as for the dropout and masked models, we created n/4 new sequences with n being the number of events in the sequence. each sequence has n/3 of its events either dropped or masked. preliminary experiments showed little difference between using all the data vs a subset. ## evaluation we ran different experiments to answer the following questions: how do sequence perturbation techniques improve event language modeling? we evaluate perplexity as is standard in the inc evaluation starts with a gold sequence of events from a real document, and then includes 5 other event sequences pulled from confounding documents. you insert the first gold event artificially at the start of each of these. the gold event sequence should have high probability compared to the confounding event sequences. figure 3 shows a gold sequence and one confounding sequence generated for it. the six sequences are ranked based on the probabilities assigned by the model, and then the accuracy is the number of predictions where the gold sequence is ranked first. a random model will uniformly choose one among the six sequences and thus will have an accuracy of 16.6%. the perplexity foot_1 and the inc accuracy of different variants of both autoregressive and autoencoding models are shown in using sequence perturbations improves the inc accuracy on both test and validation sets for both categories of models. further, the sequence perturbations gain in terms of inc accuracy is much higher with haqae. how do models trained with perturbation techniques perform on out-of-domain data? the nyt corpus used for training the models in this study is newswire. the journalistic writing style does not always follow the temporal ordering of events, but represents the events in various orders going backwards or forward in time. one might argue that the reason the sequence perturbations work better in terms of inc accuracy is that the events extracted from news do not necessarily follow the temporal order and therefore the perturbations will not create an issue. to show the effectiveness of our approach, we evaluated the performance of our models on the event sequences extracted from narratives coming from different domains: novels, blogs and news @xcite . we used the openie extraction system in a similar fashion to extract the event tuples from the narrative sequences. we used our best-performing model from the previous section and with no finetuning applied the models to see how our sequence perturbations performed in terms of inc accuracy on these narrative texts. the results of this analysis are presented in how effective are the sequence perturbation techniques with respect to the number of training instances? our sequence perturbations can be seen as data augmentation strategies which will help models learn new aspects of data that can not be learned from the original sequences. as the number of training samples increases, the model has more opportunities to learn these aspects. therefore, the sequence perturbations will be more useful for domains with fewer training samples. seed generated events ppl(g-s) ppl(g-ps) we plotted the perplexity with respect to the number of training sequences for the gpt-2 baseline system as well as permuted and dropout models. as can be seen in figure 4 , the gap between the perplexity scores are higher when the number of sequences are lower. this observation suggests that our approach will result in better language models for domains with limited data. how do schemas generated by different models differ from each other? we generated schemas for 46 two-event seeds using the haqae baseline and permuted models. we wanted to see how the generated schemas differ in two different aspects: first, for each seed, we permuted the events and generated schemas for both models. we expect the permuted model to have less variation in generating events for original and permuted seeds. we calculated the perplexity of the generated events for both the original order of events as well as the permuted order. second, we want to see how dependent the generation is upon the most recent event in the sequence. we generated schemas for two-event seeds in which the last event is the same while the first event indicates a different path. ## conclusion we proposed a set of simple sequence perturbations to relax the model's reliance on the discourse order of event mentions for event language modeling. by predicting the next event based on perturbed sequences, the model is encouraged to treat the input as a set of events. our experiments show that these perturbations can improve identifying event schemas measured by inc accuracy both on in-domain and out-of-domain data.