Cross-Lingual Event Detection via Optimized Adversarial Training (2022)

Abstract: In this work, we focus on Cross-Lingual Event Detection where a model is trained on data from a \textit{source} language but its performance is evaluated on data from a second, \textit{target} , language. Most recent works in this area have harnessed the language-invariant qualities displayed by pre-trained Multi-lingual Language Models. Their performance, however, reveals there is room for improvement as the cross-lingual setting entails particular challenges. We employ Adversarial Language Adaptation to train a Language Discriminator to discern between the source and target languages using unlabeled data. The discriminator is trained in an adversarial manner so that the encoder learns to produce refined, language-invariant representations that lead to improved performance. More importantly, we optimize the adversarial training process by only presenting the discriminator with the most informative samples. We base our intuition about what makes a sample informative on two disparate metrics: sample similarity and event presence. Thus, we propose leveraging Optimal Transport as a solution to naturally combine these two distinct information sources into the selection process. Extensive experiments on 8 different language pairs, using 4 languages from unrelated families, show the flexibility and effectiveness of our model that achieves state-of-the-art results.

## introduction event detection (ed) is an important sub-task within the broader information extraction (ie) task. event detection consists of being able to identify the words, commonly referred to as triggers, that denote the occurrence of events in a sentence, and classify them into a discrete set of event types. for example, in the sentence "jamie bought a car yesterday.", bought is considered the trigger of a transaction:transfer-ownership 1 1 event type taken from the ace05 dataset. event type. it is a very well studied task in which there have been lots of previous research efforts that have recently been primarily deep learningbased @xcite @xcite @xcite @xcite @xcite . nonetheless, ed remains quite a challenging task as the context in which a trigger occurs can change its corresponding type completely. furthermore, the same event might also be expressed by entirely different words/phrases. additionally, the vast majority of the aforementioned efforts are limited to a monolingual setting -performing ed on text belonging to a single language. alternatively, cross-lingual ed (cled) proposes the scenario of creating models that effectively perform ed on data belonging to more than one language, which brings about additional challenges. for instance, trigger words present in one language might not exist in another one. an frequent example of this phenomenon are verb conjugations where some tenses only exist in some languages. accurate verb handling is of particular importance for the ed task as event triggers are usually related to the verbs in a sentence. some recent work @xcite has attempted to address this issue by injecting external verb knowledge into the training process. another similar problematic issue for cled are triggers with different meanings that are each distinct words in different languages. for instance, the word "juicio" in spanish can either mean "judgement" or "trial" in english, depending on the context. a compelling approach to creating a crosslingual model is to use transfer learning which carries the performance of a model trained on a source language over onto a second target language. the general idea is leveraging the existing high-quality annotated data available for a high-resource language to train a model in a way that allows it to learn the language-invariant characteristics of the task at hand, ed in this case, so that it also performs effectively on text from a second language. prior works on transfer learning for cled have relied on pre-trained multilingual language models (mlms), such as multilingual bert (mbert) @xcite , to take advantage of their innate language-invariant qualities. yet, their performance still shows room for improvement as they sometimes struggle to handle the difficult instances, unique to cross-lingual settings, mentioned earlier. we identify a significant shortcoming of previous cled efforts in that they do not exploit the abundant supply of unlabeled data: even though mlms are trained on immense amounts of it, unlabeled data is not used when fine-tuning for the ed task. it is our intuition that by integrating unlabeled target-language data into the training process, the model is exposed to more language context which should help deal with issues such as verb variation and multiple connotations. as such, we propose making use of adversarial language adaptation (ala) @xcite to train a cled model. the key idea is to generate language-invariant representations that are not indicative of language but remain informative for the ed task. unlabeled data from both the source and target languages is used to train a language discriminator (ld) network that learns to discern between the two. the adversarial part comes from the fact that the encoder and discriminator are trained with opposing objectives: as the ld becomes better at distinguishing between languages, the encoder learns to generate more language-invariant representations in an attempt to fool the ld. to the best of our knowledge, our work is the first one proposing the use of ala for the cled task. nonetheless, contrary to past uses of ala where the same importance is given to all unlabeled samples, we recognize that such course of action is suboptimal as certain samples are bound to be more informative for the discriminator than others. for example, we would like to present the ld with the samples that allow it to learn the fine-grained distinctions between the source and target languages, instead of relying on syntactic differences. moreover, in the context of ed, we suggest it would be beneficial for the ld to be trained with examples containing events, instead of non-event samples, as the presence of an event can then be incorporated into the generated representations. hence, we propose refining the adversarial training process by only keeping the most informative examples while disregarding less useful ones. our intuition as to what makes samples more informative for cled is two-fold: first, we presume that presenting the ld with examples that are too different makes the discrimination task too simple. as mentioned previously, we would like the ld to learn a fine-grained distinction between the source and target languages which, in turn, improves the language-invariance of the encoder's representations. thus, we suggest presenting the ld with examples that have similar contextual semantics, i.e., similar contextualized representations. second, we consider that sentences containing events should provide an ed system with additional task-relevant information when compared against non-event samples. accordingly, we argue that event-containing sentences should have a larger probability of being selected for ala training. with these intuitions in mind, we propose optimal transport (ot) @xcite as a natural solution to simultaneously incorporate both the similarity between sample representations and the likelihood of the samples containing an event into a single framework. therefore, we cast sample selection as an ot problem in which we attempt to find the best alignment between the samples from the source and target languages. for our experiments, we focus on the widely used ace05 @xcite and ere @xcite datasets which, in conjuction, contain event-annotations in 4 different languages: english, spanish, chinese, and arabic. we work on 8 different language pairs by selecting different languages as the source and target. our proposed model obtains new state-of-the-art results with considerable performance improvements (+ 2-3% in f1 scores) over competitive baselines and previously published results @xcite . we believe these results demonstrate our model's efficacy and applicability at creating cled systems. the rest of this paper is organized as follows: section 2 provides an thorough description of our proposed model, section 3 presents and analyses the results from our experiments, section 4 provides a brief review of related work, and section 5 includes our conclusions. ## related work research efforts on monolingual ed are extensive and varied. hand-crafted, feature-based, languagespecific methods were the basis of early ed approaches @xcite @xcite @xcite @xcite @xcite . more recent efforts have primarily made use of deep learning techniques such as convolutional neural networks @xcite @xcite , recurrent neural networks @xcite @xcite , graph convolutional networks @xcite @xcite , adversarial networks @xcite , and pre-trained language models @xcite @xcite @xcite . works on cross-lingual ed are not as prevalent and generally make use of cross-lingual resources employed to address the differences between languages such as bilingual dictionaries or parallel corpora @xcite and, more recently, pre-trained multilingual language models @xcite @xcite . unlike these previous efforts, our method leverages unlabeled data to further refine the language-invariant qualities of the language models. adversarial language adaptation, inspired by models in domain adaptation research @xcite @xcite , has been successfuly applied at generating language-invariant models @xcite @xcite . our method improves upon these approaches optimizing the adversarial training process by selecting the most informative examples from the unlabeled data. additional examples of downstream applications of cross-lingual learning are document classification @xcite , named entity recognition @xcite and part-of-speech tagging @xcite . for a thorough review on cross-lingual learning, we refer the reader to @xcite . ## conclusion we present oacled, a new model for crosslingual event detection that learns fine-grained language-invariant representations by optimizing the standard ala training through optimaltransport-based sample selection. our model achieves new state-of-the-art performance in our experiments on 8 different language pairs which demonstrate its robustness and effectiveness at generating refined language-invariant representations that allow for better event detection results. our analysis of its intermediate outputs and predictions confirm that oacled's representations are indeed closer to each other and that this proximity translates into better handling of difficult cross-lingual instances. we also note that, while this work focuses on the event detection task, our proposed optimization of the adversarial training process is task independent and can be generalized to other related ie tasks when leveraging ala is deemed beneficial.