Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text (2020)

Abstract: Although temporal tagging is still dominated by rule-based systems, there have been recent attempts at neural temporal taggers. However, all of them focus on monolingual settings. In this paper, we explore multilingual methods for the extraction of temporal expressions from text and investigate adversarial training for aligning embedding spaces to one common space. With this, we create a single multilingual model that can also be transferred to unseen languages and set the new state of the art in those cross-lingual transfer experiments.

## introduction the extraction of temporal expressions from text is an important processing step for many applications, such as topic detection and questions answering @xcite . however, there is a lack of multilingual models for this task. while recent temporal taggers, such as the work by @xcite focus on english, only little work was dedicated to multilingual temporal tagging so far. @xcite proposed to automatically generate language resources for the rulebased temporal tagger heideltime, but all of these models are language specific and can only process texts from a fixed language. in this paper, we propose to overcome this limitation by training a single model on multiple languages to extract temporal expressions from text. we experiment with recurrent neural networks using fasttext embeddings @xcite and the multilingual version of bert @xcite . in order to process multilingual texts, we investigate an unsupervised alignment technique based on adversarial training, making it applicable to zero-or low-resource scenarios and compare it to standard dictionary-based alternatives @xcite . we demonstrate that it is possible to achieve competitive performance with a single multilingual model trained jointly on english, spanish and portuguese. further, we demonstrate that this multilingual model can be transferred to new languages, for which the model has not seen any labeled sentences during training by applying it to unseen french, catalan, basque, and german data. our model shows superior performance compared to heidel-time @xcite and sets new stateof-the-art results in the cross-lingual extraction of temporal expressions. ## related work temporal tagging. the current state of the art for temporal tagging are rule-based systems, such as heideltime @xcite or su-time @xcite . in particular, heideltime uses a different set of rules depending on the language and domain. @xcite automatically generated heideltime rules for more than 200 languages in order to support many languages. however, the quality of these rules does not match the high quality of manually created rules and the models are still language specific. aside from rule-based systems, @xcite proposed to learn context-dependent semantic parsers for extracting temporal expressions from text. @xcite made a first step towards neural models by using recurrent neural networks. however, they only performed experiments on english corpora using monolingual models. in contrast, we propose a truly multilingual model. ## methods we model the task of extracting temporal expressions as a sequence tagging problem and explore the performance of state-of-the-art recurrent neural networks with fasttext and bert embeddings, respectively. in particular, we train multilingual models that process all languages in the same model. to create and improve the multilingual embedding spaces, we propose an unsupervised alignment approach based on adversarial training and compare it to two baseline approaches. figure 1 provides an overview of the system. the different components are described in detail in the following. ## conclusion in this paper, we investigated how a multilingual neural model with fasttext or bert embeddings can be used to extract temporal expressions from text. we investigated adversarial training for creating multilingual embedding spaces. the model can effectively be transferred to unseen languages in a cross-lingual setting and outperforms a stateof-the-art model by a large margin.