XLT ime: A Cross-Lingual Knowledge Transfer Framework for Temporal Expression Extraction (2022)

Abstract: Temporal Expression Extraction (TEE) is essential for understanding time in natural language. It has applications in Natural Language Processing (NLP) tasks such as question answering, information retrieval, and causal inference. To date, work in this area has mostly focused on English as there is a scarcity of labeled data for other languages. We propose XLTime, a novel framework for multilingual TEE. XLTime works on top of pre-trained language models and leverages multi-task learning to prompt cross-language knowledge transfer both from English and within the non-English languages. XLTime alleviates problems caused by a shortage of data in the target language. We apply XLTime with different language models and show that it outperforms the previous automatic SOTA methods on French, Spanish, Portuguese, and Basque, by large margins. XLTime also closes the gap considerably on the handcrafted HeidelTime method.

## introduction temporal expression extraction (tee) refers to the detection of temporal expressions (such as dates, durations, etc., as shown in moreover, methods that work on other languages are rare, because of the scarcity of annotated data. we find that that there is considerable room for improving tee, especially for low-resource languages. for example, the previous sota performance on the english te3 dataset (uzzaman in the last three months duration , net revenue rose 4.3% to @xmath0504.2 million last year date . the official news agency, which gives the daily set tally of inspections, updated on friday evening time . developing an approach that can learn using the existing limited amount of training data is crucial for this field because of the effort required to develop high-quality rules for each language. thus we propose a cross-lingual knowledge transfer framework for multilingual tee, namely, xl-time. we base our framework on pre-trained multilingual models @xcite . we then use multi-task learning (mtl) @xcite to prompt knowledge transfer both from english and within the low-resource languages. for this, we design primary and secondary tasks. the primary task leverages the existing, annotated tee data of the other languages. it transfers explicit knowledge that tells the forms of the temporal expressions in a source language. the secondary task maps the annotated source language tee data samples to the target language using machine-translation tools, such as google translate, and acquires sentence-level labels (of the presence of one or more time expressions) from the original token-level labels. it constructs training data in a weakly-supervised manner. the secondary task transfers implicit knowledge by teaching the model to detect the presence of temporal expressions in text from the target language. contributions. 1) we propose xltime, which prompts cross-lingual knowledge transfer using mtl to address multilingual tee. 2) we show that xltime outperforms the previous automatic sota methods by large margins on four languages including french (fr), spanish (es), portuguese (pt), and basque (eu), which are "low-resource" for the tee task. 3) we show that xltime also approaches the performance of the heavily handcrafted heideltime @xcite , and xltime even outperforms it on two languages (portuguese and basque). we make our code and data publicly available. foot_0 ## related work while tee is an important problem in nlp, there is relatively little work in the area, and most of this work focuses on english. prior art can be divided into two classes: rule/pattern-based and deep learning approaches. in the first class, heidel-time @xcite is the top performing approach to date, and covers over a dozen languages. it is driven by a collection of finely-tuned rules. the approach was later extended to more languages with heideltime-auto @xcite , which leverages language-independent processing and rules. other approaches include syn-time @xcite , which is based on heuristic rules, and sutime @xcite and ptime @xcite , which leverages pattern learning. for the second class, @xcite proposes a model based on rnns. @xcite uses bert with a linear classifier. @xcite inputs mbert embeddings to a bilstm with a crf layer and outperforms heideltime-auto on four languages. however, the reported performances of the deep learning-based methods are inferior to the rule-based ones, which is, in part, due to the complexity of the problem and training data paucity. in our work, we propose a new model which outperforms prior deep learning methods but also closes the gap considerably on heideltime, despite the data issues. in addition, we are aware that applying label projection methods @xcite can be a straightforward way to address the data scarcity in non-english tee. tmp @xcite , originally proposed for cross-lingual named entity recogni-tion (ner) @xcite , projects english data in iob (inside outside beginning) tagging format @xcite to that of the other languages using machine translation, orthographic, and phonetic similarity packages. we show that the proposed xltime, specifically designed to transfer temporal knowledge between languages, outperforms tmp by large margins. ## proposed method we formalize tee as a sequence labeling task, similar to ner @xcite . the architecture is shown in figure 1 . ## experiments this section evaluates the proposed xltime framework. section 4.1 introduces the datasets, models evaluated, metrics, and experimental settings. section 4.2 quantitatively shows how xltime alleviates data scarcity and prompts tee performances. section 4.3 studies the effect of transferring knowledge from other languages in addition to english. we also qualitatively show how xltime transfers knowledge to the target languages in an error analysis in appendix e. ## conclusion we propose xltime for multilingual language tee in low-resource scenarios. it is based on language models and leverages mtl to prompt crosslanguage knowledge transfer. it greatly alleviates the problems caused by the shortage in training data and shows results superior to the previous automatic sota methods on four languages. it also approaches the performance of a highly engineered rule-based system.