Effective Distant Supervision for Temporal Relation Extraction (2021)

Abstract: A principal barrier to training temporal relation extraction models in new domains is the lack of varied, high quality examples and the challenge of collecting more. We present a method of automatically collecting distantly-supervised examples of temporal relations. We scrape and automatically label event pairs where the temporal relations are made explicit in text, then mask out those explicit cues, forcing a model trained on this data to learn other signals. We demonstrate that a pre-trained Transformer model is able to transfer from the weakly labeled examples to human-annotated benchmarks in both zero-shot and few-shot settings, and that the masking scheme is important in improving generalization.

## introduction temporal relation extraction has largely focused on identifying pairwise relationships between events in text. past work on annotating temporal relations has struggled to devise annotations schemes which are both comprehensive and easy to judge @xcite . however, even simplified annotation schemes designed for crowdsourcing @xcite can struggle to acquire high-accuracy judgments about nebulous phenomena, leading to a scarcity of high-quality labeled data. compared to tasks like syntactic parsing @xcite or natural language inference @xcite , there are thus fewer resources for temporal relation extraction in other domains. in this work, we present a method of automatically gathering distantly-labeled temporal relation examples. unlike traditional distant supervision methods @xcite , we do not rely on a knowledge base, but instead on heuristic cues that we will then mask out, forcing the model to make inferences from the remaining context. we explore two types of cues, but focus primarily on events that are anchored to orderable timexes @xcite . these examples can be collected and labeled using an automatic system @xcite . by then masking the explicit temporal indicators, a model trained on these examples can no longer learn trivial timex-based rules, but must instead attend to more general temporal context cues. we show that a pre-trained model fine-tuned on this data learns general, implicit cues that transfer more broadly to human-annotated benchmarks. this observation follows a trend of recent work showing pre-trained models' ability to generalize from synthetic data to natural data @xcite . we implement our approach with pre-trained transformer models @xcite @xcite similar to a state-of-theart temporal relation extraction model from the literature @xcite . our model is able to effectively transfer from a distantly-labeled dataset to the matres benchmark @xcite when used to supplement a small number of indomain or out-of-domain samples. ## classification model our base classification model consists of a pretrained transformer @xcite model with an appended linear classification layer, represented in figure 1 . for the majority of our experiments, we use roberta @xcite , which we found to work better than bert @xcite and electra @xcite ) for domain transfer. we chose a single set of hyperparameters by tuning to match the performance of @xcite ; for details see appendix a. a single example consists of an event pair in context, which may be a single sentence or two sentences, and a label from {after, be-fore, equals, vague}, following the annotation scheme in matres. each example is tokenized to yield input tokens @xmath0, t 2 , ..., t n ], with event tokens t i , t @xmath1. for events consisting of multiple sub-word tokens, we track only the first token position. we use the convention of passing events in text order, where the language model then produces output embeddings [e 1 , e 2 , ..., e n ]. for classification, we select the embeddings e i , e j corresponding to the event token positions, and combine them into a classification vector, @xmath2; e j ; e i e j ; e i -e j ] where represents elementwise multiplication. finally, a linear classification layer produces a distribution over the four relation labels. training is done by maximizing likelihood of labeled samples. we implement this model using pytorch @xcite and pre-trained models from huggingface's transformers library @xcite . we benchmark our classification model by training and evaluating on matres. we achieve an f1 of 79.8 with roberta @xcite and an f1 of 80.3 with electra @xcite , demonstrating that our model replicates state-of-the-art performance achieved by local models (only considering arcs in isolation), currently 80.3 f1 @xcite . ## learning from distant data we aim to create a method of automatically gathering high-quality data that can be applied to unlabeled text. to this end, we focus on two techniques identifying explicit temporal indicators. first, we identify single-sentence examples where event pairs are automatically labeled via explicit discourse connectives. second, we scrape occurrences of event pairs that are anchorable to timexes which determine their relation. we will see that this second technique is substantially better, and analyze some factors contributing to the performance delta. although neither technique captures the gamut of phenomena found in human-labeled data, pre-trained models' generalization capabilities and a masking technique tailored for this setting are two tools that enable effective transfer. for both techniques, we scrape distant examples from english gigaword fifth edition @xcite . we extract samples from a balance of the different news sources present in the dataset. in both cases, we use the stanford corenlp lexicalized parser @xcite to generate parse trees for the source text, which can be timeconsuming at scale. however, we can pre-filter sentences based on the presence of timexes or target discourse connectives, and so in practice we only rarely need to invoke the parser. ## results we evaluate our distant dataset on several axes. (1) in zero-shot, few-shot, or transfer settings, to what extent can this help existing models? (2) how important is each facet of our distant extraction setup? (3) what can we say about the distribution of our data from these techniques? we focus our evaluation on the english matres dataset @xcite , a four-class temporal relation dataset of chiefly newswire data drawn from a number of different sources. 2 we found significant variance in model performance in our transfer for small data settings, so most results use average best performance or majority-vote ensembled performance of three randomly seeded models trained in the same setting. ## related work there is little direct prior work on using this kind of distant supervision for temporal relation extraction. past work has studied automatic extraction of typical inter-event orderings @xcite @xcite to aid downstream temporal tasks, but these approaches represent events as single words (predicates) taken out of context, so the knowledge they can capture is limited. the commonsense acquisition method of zhou et al. (2020) learns more sophisticated information, but more about unary properties of events (typical time, duration) rather than relational knowledge. lin et al. (2020) achieve a somewhat similar goal, but make a strong assumption about narrative-structured corpora and do not evaluate on in-context temporal relation extraction. our technique does not use a knowledge base like classic distant supervision methods @xcite . however, because we eventually mask out the explicit temporal indicators, we are still using temporal information "external" to the final example to derive the label, hence why we invoke this term. a related concept is the idea of labeling functions @xcite , which are used to automatically construct training data for new domains. however, to our knowledge, these techniques have not been applied to temporal relation extraction, nor used in conjunction with masking as we do. ## discussion we use explicit temporal cues to automatically identify examples of temporal relations between events in text. by masking these trivial features, a pre-trained transformer model can learn from the remaining context and generalize to humanannotated benchmarks. comparing performance for two distant labeling methods-using discourse connectives and linking events to time expressionsindicates that richer temporal cues exist in the second case. the scope of identified time expressions encompasses both explicit datetimes and natural language indicators ("now", "recently", etc.). while datetimes may be more common in news and historical articles, relative time expressions are present in diverse domains such as literature and colloquial texts. where such indicators exist, our approach may be used to automatically collect distantly labeled temporal relations. more broadly, we believe that this label-and-mask paradigm could be used to collect targeted training data for a variety of nlp tasks.