Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020)

Abstract: In this paper, we propose a neural architecture and a set of training methods for ordering events by predicting temporal relations. Our proposed models receive a pair of events within a span of text as input and they identify temporal relations (Before, After, Equal, Vague) between them. Given that a key challenge with this task is the scarcity of annotated data, our models rely on either pretrained representations (i.e. RoBERTa, BERT or ELMo), transfer and multi-task learning (by leveraging complementary datasets), and self-training techniques. Experiments on the MATRES dataset of English documents establish a new state-of-the-art on this task.

## introduction the task of temporal ordering of events involves predicting the temporal relation between a pair of input events in a span of text (figure 1 ). this task is challenging as it requires deep understanding of temporal aspects of language and the amount of annotated data is scarce. albright (e1, came) to the state department to (e2, offer) condolences. the matres dataset @xcite has become a de facto standard for temporal ordering of events. 1 it contains 13,577 pairs of events annotated with a temporal relation (before, after, equal, vague) within 256 english documents (and 20 more for evaluation) from timebank foot_0 @xcite , aquaint 3 @xcite and @xcite . in this paper, we present a set of neural architectures for temporal ordering of events. our main model (section 2) is similar to the temporal ordering models designed by @xcite , @xcite and @xcite . our main contributions are: (1) a neural architecture that can flexibly adapt different encoders and pretrained word embedders to form a contextual pairwise argument representation. given the scarcity of training data, (2) we explore the application of an existing framework for scheduled multitask-learning (henceforth smtl) (kiperwasser and ballesteros, 2018) by leveraging complementary (temporal and non temporal) information to our models; this imitates pretraining and finetuning. this consumes timex information in a different way than @xcite . (3) a self-training method that incorporates the predictions of our model and learns from them; we test it jointly with the smtl method. our baseline model that uses roberta @xcite already surpasses the state-of-the-art by 2 f1 points. applying smtl techniques affords further improvements with at least one of our auxiliary tasks. finally, our self-training experiments, explored via smtl as well, establishes yet another state-of-the-art yielding a total improvement of almost 4 f1 points over results from past work. ## our baseline model our pairwise temporal ordering model receives as input a sequence x [0,n) of n tokens (or subword units for bert-like models) i.e. {x 0 , x 1 , ..., x n-1 }, representing the input text. a subsequence span i is defined by start i , end @xmath0, n). subsequences span 1 and span 2 represent the input pair of argument events e1 and e2 respectively. the goal of the model is to predict the temporal relation between e1 and e2. first, the model embeds the input sequence into a vector representation using either static wang2vec representations @xcite , or contextualized representations from elmo @xcite , bert @xcite , or roberta @xcite . these embedded sequences are then optionally encoded with either lstms or transformers. when bert or roberta is used to embed the input, we do not use any sequence encoders. the final sequence representation h [0,n) comprises of individual token representations i.e. {h 0 , h 1 , ..., h n-1 }. while the goal is to predict the temporal relation between span 1 and span 2 , the context around these two spans also has linguistic signals that connect the two arguments. to use this contextual information, we extract five constituent subsequences from the sequence representation h [0,n) : (1) s 1 , the subsequence before span 1 i.e., h [0,start 1 ) , (2) s 2 , the subsequence corresponding to span 1 i.e., h [start 1 ,end 1 ) , (3) s 3 , the subsequence between span 1 and span 2 i.e, h [end 1 ,start 2 ) , (4) s 4 , the subsequence corresponding to span 2 i.e., h [start 2 ,end 2 ) and (5) s 5 , the subsequence after span 2 , i.e. h [end 2 ,n) . each of these subsequences s i has a variable number of tokens which are pooled to yield a fixed size representation s i : the final contextual pair representation c is formed by concatenating 4 the five span representations s i with a sequence representation r. for models with bert and roberta, r is the cls @xmath1,n) ). this final contextual pair representation c is then projected with a fully connected layer followed by 4 is used to denote concatenation a softmax function to get a distribution over the output classes. the entire model is trained end-toend using the cross entropy loss. ## multi-task learning while the model described in the previous section can be directly trained using labeled training data, the amount of annotated training data for this task (in the matres dataset) is limited. we enrich our model with useful information from other complementary tasks via smtl. ## experiments and results the matres dataset is our primary dataset for training and validation. as in previous work, we use timebank and aquaint (256 articles) for training, 25 articles of which are selected at random for validation and platinum (20 articles) as a held-out test set @xcite @xcite . articles from timebank and aquaint at full length are about 400 tokens long on average. we believe that the document in its entirety is not required to infer the temporality between a given pair of events. moreover, bert style models are also often pre-trained for shorter inputs than this. for these reasons, we truncate our input text to a window of sentences 8 starting with one sentence before the first event argument up to and including one sentence after the second event argument. we use one set of hyperparameters for all lstm models and another set for all the transformer models (both with and without elmo embedder). 9 bert and/or roberta are loaded as a replacement of the transformer parameters and they are therefore used both as embedders and encoders. we run our smtl and self training experiments with our best baseline model on the development data: the roberta model. for the smtl experiments, we explore the α hyperparameter, and we pick the one that produces the highest scores in our development data. finally, we picked our best smtl model on the development data (see table , this is the constant scheduler with silver data) parameters and continue training on the gold data only; we reduce the learning rate to 10 -6 . this is because the model trained in the first step is already in a good state and we want to avoid distorting it with aggressive updates. we compare our results ( importantly, however, our base model that uses pretrained roberta surpasses the previous stateof-the-art @xcite which uses bert. our bert models yield very similar results to them. the main differences are that they do not finetune bert along with the updates to the model, while we do and also, we model the context around the argument spans explicitly as part of s 1 , s 3 and s 5 . the reason why roberta is better than bert in this case is likely due to the fact that it has been trained longer, over more data, and over longer sequences. this matters because our temporal ordering model usually takes into account a long span in which both events occur. the smtl experiments show that the auxiliary task with timex annotations provides nonnegligible improvements of almost 1 f1 point on top of our roberta model. learning from the timex annotations makes our model more aware of time relations and thus, better at ordering events in time. the sigmoid and exponent schedulers perform better than the constant scheduler, suggesting that the model needs to first learn about temporality, and then learn to be more specialized on predicting temporal ordering relations later. we believe this timex multi-tasking setup to be an implicit yet effective way to teach our model about timexes in general without timex embeddings used in @xcite . when we use the ace relation extraction dataset as an auxiliary task, none of the schedulers produce improvements while the sigmoid and exponent scheduler fare significantly worse. this result suggests that if the tasks differ too much, smtl might not be a helpful strategy. the self-training experiments (including smtl with silver data) show that the silver data helps to reach better performance with constant being the best scheduler. furthermore, fine-tuning of the best model (according to development set score, which in this case it is the same as test set score) on the gold data gives us another boost in performance establishing a new state of the art in the task that is 2.7 f1 points better than our roberta baseline, and almost 4 points better than the previous published results. ## conclusions and future work this paper presents neural architectures for ordering events in time. it establishes a new state-ofthe-art on the task through pretraining, leveraging complementary tasks through smtl and selftraining techniques. for the future, instead of using the roberta baseline model for the self-training experiments, we could run several iterations by retraining on the data produced by our best self-trained model(s); this could be a good avenue for further improvements. in addition we plan to extend our work by moving to other languages beyond english (we currently have not tried this due to lack of data) using cross-lingual models, @xcite , applying other architectures like cnns (nguyen and grishman, 2015), incorporating tree structure in our models @xcite and/or by handling jointly performing event recognition and temporal ordering @xcite .