Probing Language Models for Understanding of Temporal Expressions (2021)

Abstract: We present three Natural Language Inference (NLI) challenge sets that can evaluate NLI models on their understanding of temporal expressions. More specifically, we probe these models for three temporal properties: (a) the order between points in time, (b) the duration between two points in time, (c) the relation between the magnitude of times specified in different units. We find that although large language models fine-tuned on MNLI have some basic perception of the order between points in time, at large, these models do not have a thorough understanding of the relation between temporal expressions.

## introduction while contextualized embeddings obtained from recent transformer-based models such as bert @xcite have proven to contain a lot of semantic and syntactic information about the tokens they encode, recent studies have shown that there are still gaps in their understanding @xcite . on the semantic side, for instance, bert struggles with representations of numbers @xcite and cannot reason based on its world knowledge @xcite . work in nli has also developed challenge sets showing that the reported performance of these language models on various tasks can be exaggerated @xcite , and they rely on lexical cues in the dataset instead of actual language comprehension. our work explores the grasp of such models on the relation between temporal expressions. temporal expressions, or time expressions, in text are a sequence of tokens that denote time, such as a point in time (6 may 1980, monday, 12 pm) or duration (7 minutes, 5 years, 2 months). more specifically, we try to determine whether these models capture the ordering and duration relationships between different points in time. we also analyze if these models can reason about durations specified in different units. recognition of temporal expressions has had applications in timeline construction @xcite and clinical analysis @xcite previously, and can be beneficial for dialogue assistants in scheduling reminders and meetings, which shapes our motivation behind conducting such an analysis. we evaluate these models on the above temporal properties by presenting three nli challenge sets. our experiments demonstrate that language models such as roberta @xcite and de-berta @xcite fine-tuned on existing large nli datasets are unable to completely reason about the ordering and duration between temporal expressions. we further analyze the examples and find that while these models recognize whether a point in time lies within an interval, they cannot capture other relations between time instances and durations foot_0 . ## related work much work has been done on the extraction of events, temporal expressions @xcite , and the temporal relations between the two. timebank @xcite was one of the first annotated corpora for this task. it utilized the timeml @xcite standard for annotation. tempeval-1 @xcite , tempeval-2 @xcite , and tempeval-3 (uzzaman et al., 2013) are shared tasks created for evaluating models on various temporal properties, and most methods used were traditional rule-based @xcite or grammar-based @xcite solutions. various corpora have been developed that test for different temporal properties. @xcite map events to their fine-grained duration, and event pairs to their relative timelines. @xcite create additional annotations in the existing timebank-dense corpus @xcite for discourse-level temporal ordering. @xcite create a reading comprehension dataset that tests for temporal ordering. @xcite test for various temporal commonsense properties using a multiple-choice question-answering dataset. @xcite recast existing temporal datasets into nli format to test for temporal ordering and duration. our goal is to create similar datasets to probe for a semantic understanding of temporal expressions in pre-trained language models. we create these datasets in an nli format and use them to evaluate nli models trained on mnli @xcite , which is a generic nli dataset. we choose mnli because it is large and diverse. the dataset contains time terms in 36% of the development instances, including examples containing temporal expressions like months and days of the week. we investigate whether these examples are sufficient for a general perception of temporal expressions. to our knowledge, there has been little work investigating the implicit understanding of time expressions in pre-trained large language models. the most similar work to ours is @xcite . they produce five nli datasets recast from existing temporal reasoning corpora and test nli models for event duration (how long an event lasts) and event ordering (how events are temporally arranged). however, there are some key differences: we construct three nli datasets that aim to test different relations between temporal expressions. the datasets use templates from a manually curated list of 71 events, labeled with their temporal occurrence (when the event is likely to occur) and temporal duration (how long the event is expected to last) values. for instance: template: i went to paris occurrence: day, month, year • i went to paris on monday. each temporal unit corresponds to some list(s) spanning different magnitudes of time ( ## experimental setup we evaluate three different nli models on each of our challenge sets. the first model is a pretrained roberta-large model fine-tuned on the mnli corpus, which reports 90.8% accuracy on the mnli-matched task. the second model is microsoft's deberta-large model fine-tuned on the mnli corpus, which reports 91.9% accuracy on the mnli-matched task. both these models are trained on all three labels: entailment, contradiction, and neutral. the third model comes from @xcite , which is a roberta-large model finetuned on their temporal nli datasets. for temp-order set, we evaluate their model trained on the uds-nli (order) corpus as it explored ordering relations between events, and we wanted to analyze if any of that knowledge transfers over for determining the ordering between temporal expressions. similarly, for temp-duration and cross-unit duration sets, we evaluate their model trained on the uds-nli (duration) corpus, which explored more coarse-grained duration of events. in contrast to the models fine-tuned on mnli, these models are only trained on binary classification -producing 'entailed' for the entailment label and 'not-entailed' for the contradiction and neutral labels. we have a separate majority baseline corresponding to these models. model method accuracy f1 score majority ternary classification 40.29 23.14 binary classification 65.19 51.46 roberta (mnli) direct evaluation 52.75 45.36 hypothesis only training 40.29 ± 0 23.14 ± 0 train and evaluate 99.81 ± 0.03 99.81 ± 0.03 deberta (mnli) direct evaluation 51.57 44.29 hypothesis only training 40.29 ± 0 23.14 ± 0 train and evaluate 99.76 ± 0.04 99.76 ± 0.04 uds-nli (order) direct evaluation 56.36 57.20 we do not train the models fine-tuned on uds-nli corpora, and only report their performance under the direct evaluation setting, as the architecture of those models is similar to that of the pretrained roberta-mnli model and we hypothesize that this may lead to similar results on training. more details on the training process are mentioned in appendix a. ## results & discussions we present the results of all three challenge sets separately. ## conclusion we create three challenge sets that test different kinds of relationships between temporal expressions. we evaluate these challenge sets on popular nli models like roberta and deberta trained on mnli, and find that while they can reason about simple cases of ordering between time instances, they fail when presented with more complicated cases or when temporal reasoning requires determining fine-grained duration. since our challenge sets were synthetically created, training on them helped the nli models to figure out the label generation process, and they produced near-perfect accuracy for the temp-order and the cross-unit duration sets. a direction for future research could be evaluating and comparing models, trained on other nli datasets containing temporal expressions, on our challenge sets. another direction could be to collect naturally occurring sentences that contain temporal expressions from large corpora and recast them into nli format for similar testing of understanding of temporal expressions.