How about Time? Probing a Multilingual Language Model for Temporal Relations (2022)

Abstract: This paper presents a comprehensive set of probing experiments using a multilingual language model, XLM-R, for temporal relation classification between events in four languages. Results show an advantage of contextualized embeddings over static ones and a detrimen- tal role of sentence level embeddings. While obtaining competitive results against state-of-the-art systems, our probes indicate a lack of suitable encoded information to properly address this task.

## introduction time is a pervasive element of human life with no counterpart in any other cognitive domains @xcite . such pervasiveness is mirrored in natural languages through sets of devices that allows speakers to refer to time, to reason about time and things that unfold in time. reasoning about time is one of the central components of common sense knowledge @xcite @xcite and its modeling has been at the core of many early approaches in computational linguistics and artificial intelligence @xcite @xcite @xcite . more recently, specific natural language understanding (nlu) tasks related to time have been developed, ranging from the identification of temporal expressions @xcite , to measuring the duration of events @xcite , and the ability to order them chronologically @xcite @xcite . more complex tasks have challenged models to extract storylines @xcite @xcite , understand narratives @xcite @xcite , and answer temporally related questions @xcite . recent work has focused on recasting temporal relation classification as a natural language inference (nli) task where fine-tuned pre-trained language models (ptlms) have achieved good results @xcite . embedding representations, both static and contextual, have shown to play a key role to improve systems' results on different time-related benchmarks, especially for the classification of temporal relations between pairs of events @xcite . when it comes to contextualized embeddings the probing of such models for temporal knowledge has not been properly investigated yet. if we embrace the vision of ptlms as large repositories of linguistic knowledge @xcite @xcite , it is a natural question to probe these models for their knowledge about events and time. we present an extensive study on temporal relation probing of ptlms using five temporally annotated corpora in four languages (i.e., english, french, spanish, and italian). although the selected languages all belong to indo-european family, they present differences for the tense-moodaspect (tma) system while showing similarities at the lexico-pragmatic level. our probing tasks focus on temporal ordering of pairs of events (e-e), either in the same sentence or in different ones. our contributions our work has three contributions: (i) it is the first work to probe a multilingual ptlm, xlm-r base, for temporal knowledge between event pairs; (ii) we study the impact of multilingual contextualized representations against monolingual counterparts based on static word embeddings; (iii) we compare zero-shot ptlm against fine-tuned models for temporal reasoning to investigate whether the models have acquired real temporal knowledge. code and data are available. foot_0 ## data overview we have selected five corpora annotated with language specific adaptations of @xcite . iso-timeml is, at the same time, an annotation meta-model for marking events, temporal expressions, and relations between them, and a full-fledged annotation language. iso-timeml has 13 values used to classify temporal relations, based on allen's interval temporal logic @xcite where each value expresses how an event chronologically relates to another event or a temporal expression. in the following paragraphs we present a short overview of the five corpora we have used. for our experiments, we have extracted all temporal relations between event pairs, either occurring in the same sentence or in difference sentences. en-timebank the english timebank @xcite is a corpus of 183 documents manually annotated following the timeml annotation guidelines @xcite . the whole corpus has gone through a curation phase for the semeval 2013 tempeval-3 task @xcite , where an extra test set of 20 documents has been annotated with the same guidelines. in our experiments, we follow the tempeval-3 split for training and test distributions, excluding the automatically annotated data (i.e., silver data distribution). en-timebank uses the full 13 temporal values from iso-timeml to classify event-event relations. it-timebank the italian timebank @xcite has 254 documents, comparable in size and annotation to the en-timebank. we have followed the official split into train and test from the evalita 2014 eventi task @xcite . similarly to en-timebank, the 13 finegrained temporal values have been used to classify temporal relations. fr-timebank french timebank @xcite is a corpus of 107 documents in french annotated following an adaptation to french of timeml. the corpus does not present an official split into train and test. to obviate to this, we have first extracted all temporally annotated event pairs and then created a train and test distribution following a 75-25 split. fr-timebank also uses the full 13 temporal values for classifying temporal relations en-tb-dense the timebank-dense corpus @xcite ) contains only 36 documents from the training portion of the en-timebank. en-tb-dense approximates a complete graph of all possible temporal relations over events and temporal expressions by labeling all pairs locally, i.e., same sentence and adjacent sentence pairs. en-tb-dense simplifies the set of possible temporal relation values by reducing it to five and introducing a new value, vague, for all relations that do not carry a clear semantics. es-timebank the spanish timebank @xcite contains 210 documents in spanish. we have followed the official release into train and test splits. similarly to the en-timebank-dense, the authors have simplified the set of possible temporal relations to five plus vague. however, the overlap is limited only to before after, with the other three being new. ## temporal probing our probing task investigates the capabilities of ptlms to encode information about events and their temporal ordering. to probe such information across multiple languages, we use xlm-r base @xcite ), foot_1 a large multilingual model that has achieved state-of-the-art results on many nlu tasks. following previous work @xcite @xcite , inter alia), we extract embedding representations from each layer and use them to train a linear svm whose objective is to predict the value of a temporal relation between a given pair of events. by default, we feed the svm with four concatenated embeddings: the embeddings of the sentence containing each event in the pair and those of each event. in case the event pair occurs in the same sentence, we duplicate the sentence representation. sentences are represented by averaging the embeddings of the tokens excluding xlm-r base's special tokens. we compare the default settings with three variations: (i) we use only the embeddings of the events in the pair; (ii) we use the embeddings from two xlm-r base models previously finetuned with the en-timebank and en-tb-dense ## results figure 1 summarizes the results for all corpora and settings. details per corpus are in appendix b. although all probing models outperform their respective baselines, our results further confirm that temporal relation classification is a challenging task. xlm-r embeddings consistently obtain the best results across all languages and granularities of the temporal relations, improving the performance of static embeddings. with the exclusion of en-tb-dense, the presence of sentence embeddings is detrimental, confirming previous findings @xcite . although temporal relations are a discourse phenomenon at the interface of the semantics and pragmatics dimensions, it appears that the event only embeddings from xlm-r already store sufficient semantic information to perform this task. regardless of the granularity of the temporal relations, it clearly emerges from all the plots that the best results for xlm-r are obtained between layers 6 and 8. performances are consistently sub-optimal for early layers, especially 1-4. for higher layers, i.e., 10-12, results are disappointing, with the exception for english whose best probe is at layer 11.given the task and previous findings on the encoding of linguistic knowledge in ptlms @xcite , this is not fully expected. ideally, if ptlms tend to encoded more semantic features in the top layers, performances for this task should not degrade on the top layers, as we see for spanish, italian, and french, or, at least, they should remain on a plateau. a further finding concerns the role of finetuned models for temporal reasoning, namely xlm-r_tbd and xlm-r_tb. the models have been fined-tuned using the english corpora en-tb-dense and en-timebank recasted for temporal reasoning. we expected the embeddings from these models to be more competitive than basic xlm-r, but this is not the case. in general, we observe a better performance for xlm-r_tbd than xlm-r_tb, in-line with the results reported by @xcite . the better results of xlm-r_tbd hold also in cross-lingual settings, regardless of the granularity of the temporal values used in the specific corpus. when comparing results across corpora, two dimensions are at play: the first is the granularity of the temporal values; the second is the number of training examples. a general pattern we observe is the following: the less temporal values are to be learned, the better the results of a trained model, provided that the annotated data are consis- comparison with state of the art is limited to en-tb-dense and it-timebank. no previous work for this task is available fr-timebank and es-timebank, and for en-timebank we only have access to systems which classified temporal relations from raw text. concerning the en-tb-dense, the best system, sect @xcite , adopts a multi-task learning approach using a gru architecture. on the e-e classification it achieves an f1-score 0.650, gaining 0.098 points with respect to our best training layer. a more similar architecture, catena @xcite a linear svm combining pre-trained word embeddings and additional features, obtains an f1-score of 0.519, only 0.012 points above us. as for italian, the best system, fbk-htl-time @xcite , a feature-based linear svm, achieves an f1-score of 0.688, beating our approach of 0.062 points. ## conclusion this paper investigates the knowledge encoded in a large multilingual ptlm, xlm-r base, for temporal relation classification between pairs of events in four languages and five corpora with varying granularities of temporal values. our results point out that temporal relation classification between events is very challenging and the linguistic knowledge in xlm-r is limited to properly address it. while contextual embeddings are more "powerful" than static ones, current fine-tuned models for temporal reasoning @xcite are not helpful as one would expect. our probes indicate that adding more information, i.e., sentence representations, to lexical entities is detrimental, meaning that "global" semantic information is already encoded at the lexical level. finally, our models are competitive with state-of-the-art systems, indicating that improvements are due either to specific architectures or extra features capturing additional knowledge not available in the contextual embeddings.