Multilingual Normalization of Temporal Expressions with Masked Language Models (2023)

Abstract: The detection and normalization of temporal expressions is an important task and preprocessing step for many applications. However, prior work on normalization is rule-based, which severely limits the applicability in real-world multilingual settings, due to the costly creation of new rules. We propose a novel neural method for normalizing temporal expressions based on masked language modeling. Our multilingual method outperforms prior rule-based systems in many languages, and in particular, for low-resource languages with performance improvements of up to 33 F1 on average compared to the state of the art.

## introduction temporal tagging consists of the extraction of temporal expressions (te) from texts and their normalization to a standard format (e.g., may '22: 2022-05). while there are deep-learning approaches for the extraction, temporal tagging as a whole is usually solved with highly specific rulebased systems, such as sutime @xcite or heideltime @xcite . however, transferring rule-based methods to new languages or text domains requires a large manual effort to create rules specific to the target language. although work on the automatic generation of rules for many languages @xcite exists, the rule quality typically does not match the high accuracy of hand-crafted rules. in contrast to rule-based systems, neural networks are known for their ability to generalize to new targets, in particular, for cross-and multilingual applications @xcite . in the context of temporal tagging, recent works have shown promising results of neural networks for te extraction in monolingual @xcite and multilingual settings @xcite where a single neural model is trained on many languages at once. however, te ... appear @xmath0. ## background on temporal tagging temporal tagging addresses the detection, classification and normalization of temporal expressions in unstructured texts -often following the timeml specifications @xcite . timeml's most important attributes are type (the class of an expression, e.g., date, time, du-ration or set), and value (the normalized meaning of an expression, e.g., yyyy-mm-dd for specific days, such as 2022-05-01 for may 1, 2022). while some tes contain all necessary information for the normalization, e.g., "may 1, 2022", many expressions are incomplete w.r.t. the temporal information required for a normalization. an example is a relative expression like "yesterday" which needs an anchor point. given the anchor point may 1, 2022, for example from the document creation time, "yesterday" should be annotated with @xmath1. determining the anchor point can be challenging as it requires additional context information that could be given anywhere in the document. therefore, systems for te normalization, such as heideltime @xcite , create an intermediate context-independent representation (cir) of the value. in the syntax of heideltime, the expression yesterday would result in a cir of undef-last-day. similarly, an underspecified expression, such as "may" would be represented with a cir of undef-year-05. note that such a syntax for cirs is language-independent. see appendix a for more details. to determine the final value, the cir needs to be anchored given, e.g., a reference date and further cues (such as tense information). ## approach we propose to approach multilingual temporal tagging in three steps as shown in figure 1: (1) extraction of temporal expressions and their types using a multilingual sequence tagger; (2) normalization of tes to cirs with our novel mlm-based normalization model; (3) anchoring of cirs given a reference time, e.g., using heideltime rules. our main contribution is a neural model for the second subtask, the normalization to a cir. to the best of our knowledge, this has not been addressed with neural networks before. in this section, we detail all components of our approach. information on the models that we apply for the first and third subtasks as well as an ablation study of directly predicting the normalized anchored value (without cirs) are given in section 5. ## experiments this section describes our experiments and discusses the results. we compare our model to hei-deltime @xcite , the current state of the art for multilingual temporal tagging. for evaluation, we use the tempeval3 evaluation script @xcite and report strict, relaxed and type f 1 for the extraction and value f 1 for our normalization experiments, respectively. evaluation data. our models are evaluated on gold-standard corpora in 17 languages. details on the corpora are given in appendix b.2. we divide the languages into high-and low-resource depending on whether manually created heideltime rules are available for the respective language. training data. for training the normalization model, we create a large-scale weakly-supervised dataset covering 87 languages. 3 reasons are that (i) existing gold training data is too small to cover the wide range of different values and (ii) cirs are not part of existing annotations. for all languages, we take the data from globalvoices foot_3 (news-style documents) and wikipedia foot_4 (narrative-style documents), use spacy for tokenization and heideltime for the annotation with temporal expressions. the number and quality of annotations is highly dependent on the amount of available data for that language and the quality of heideltime's rules. details on the weakly-supervised data are given in appendix b.1. ## conclusion in this paper, we introduced a new method for normalizing temporal expressions based on masked language modeling and a new slot-based prediction scheme of context-independent representations. with this approach, we were able to train a single multilingual model for the task. we evaluated our method in 17 languages and set the new state of the art in low-resource languages with massive improvements of 35 f 1 points on average. the success of our method demonstrates the potential of neural networks for temporal normalization and we are convinced that it will enable future research on this topic. an interesting research direction is the joint modeling of extraction and normalization.