Bridging the Gap between Native Text and Translated Text through Adversarial Learning: A Case Study on Cross-Lingual Event Extraction (2023)

Abstract: Recent research in cross-lingual learning has found that combining large-scale pretrained multilingual language models with machine translation can yield good performance. We explore this idea for cross-lingual event extraction with a new model architecture that jointly encodes a source language input sentence with its translation to the target language during training, and takes a target language sentence with its translation back to the source language as input during evaluation. However, we observe significant representational gap between the native source language texts during training and the texts translated into source language during evaluation, as well as the texts translated into target language during training and the native target language texts during evaluation. This representational gap undermines the effectiveness of cross-lingual transfer learning for event extraction with machine-translated data. In order to mitigate this problem, we propose an adversarial training framework that encourages the language model to produce more similar representations for the translated text and the native text. To be specific, we train the language model such that its hidden representations are able to fool a jointly trained discriminator that distinguishes translated texts’ representations from native texts’ representations. We conduct experiments on cross-lingual for event extraction across three languages. Results demonstrate that our proposed adversarial training can effectively incorporate machine translation to improve event extraction, while simply adding machine-translated data yields unstable performance due to the representational gap.

## introduction there are over 6,000 living languages in the world, and for many of them, too little appropriate data exists to build natural language processing (nlp) models. cross-lingual learning has been proposed to leverage resources in data-rich languages to train nlp models for data-scarce languages @xcite . there are two main strategies for building cross-lingual models: (1) train models with multilingual language models and languageuniversal features that are transferable to the target language @xcite @xcite @xcite @xcite @xcite ; (2) use machine translation models in a pipeline, either by transforming annotated training data into the desired target language to build target-language models, or by translating data at inference time into the source language and applying source-language models @xcite @xcite . the first approach relies on the quality of the constructed multilingual semantic space; the discrepancy between source-language training data and target-language evaluation data may cause overfitting. the second approach does not require a perfect multilingual semantic space since models can be trained in a monolingual fashion, but it depends on the quality of machine translation. a combination of both approaches showed good performance on a variety of tasks such as natural language inference and question answering @xcite , but is underexplored for event extraction. compared with previous research in cross-lingual event extraction mainly adopting the first approach @xcite @xcite , we explore the idea of combining both machine translations and language-universal representations for cross-lingual event extraction in this work. we perform translation by extending the previous effort on cross-lingual reading comprehension @xcite and question answering @xcite by adding special tags around the trigger and entity spans to translate the annotations. we use a multilingual language model to simultaneously encode a sentence and its corresponding multilingual language model source language input target language input source language representation target language representation task loss task loss labeled source language corpus unlabeled target language corpus machine translation multilingual language model source language input target language input source language representation target language representation adv loss adv loss machine translation figure 1 : overall cross-lingual information extraction framework translation as shown on the left side of figure 1 . for example, in an english-to-chinese cross-lingual learning setting, we would train a model with english sentences with their chinese translations as training data, and evaluate our model with chinese sentences and their english translations as inputs. since our work includes both cross-lingual learning and machine translation, to avoid ambiguity, we will use "source" language as the one we perform cross-lingual learning from, and "target" language as the one we perform cross-lingual learning to. we will call texts before translation "native" text and text after translation "translated" text for the machine-translation-related descriptions. we found that one challenge in cross-lingual event learning with machine translations is that the machine-translated text m @xmath0. this difference is also introduced and studied as the problem of "translationese" (translated text as a different language) in previous machine translation research @xcite . in cross-lingual event extraction, we observe from a simple preliminary experiment that there indeed exists a distinguishable gap between representations of native texts h(n l ) and translated text h(m @xmath1. the pretrained language models appear to be "unaccustomed" to the translated text. the representational gap will negatively impact the cross-lingual learning with machine-translated data. since we, as introduced above, simultaneously encode a native source language sentence n s and its translation into the target m @xmath2, and a native target language sentence n t and its translation back to the source language m @xmath3, the problem of representational gap between n s and m @xmath4, as well as n t and m @xmath5. here s and t refer to the source and the target language respectively. in order to mitigate the representational gap problem between machine-translated text m and native text n in both source and target languages, we propose to take advantage of an unlabeled corpus in the target language and use adversarial training to make the encoder produce more similar representations for n s and m @xmath6, as well as n t and m @xmath7. the adversarial framework trains the language model h such that its hidden representations can fool a jointly trained discriminator that distinguishes translated texts' representations h(m) from native texts' representations h(n ). our complete cross-lingual ie framework is shown in figure 1 , which combines translationbased methods with transfer-based methods, and uses an unlabeled target language corpus to improve the representations in multilingual language models. our method shows superior performance on event trigger labeling and argument role labeling, and through quantitative studies, we observe that adversarial training indeed makes the multilingual language model generate closer representations for the translated text and the native text. we believe our proposed adversarial training can also be helpful in other nlp tasks where machine translation can boost performance. to summarize, our contributions are two-fold: ## approach in this section, we will start with a simple preliminary experiment to validate the problem of the representational gap, and then introduce our approaches to cross-lingual event trigger and argument role labeling. for both tasks, we first design specific methods to use machine translation models to translate source language annotations into the target language. we then use xlm-roberta @xcite to encode pairs of parallel sentences simultaneously into hidden representations. task-specific losses are used on top of the hidden representations. in order to make the multilingual language model produce more similar representations for translated sentences and native sentences, we further use an unlabeled target language corpus for adversarial training. ## conclusions and future work in this paper, we proposed a new cross-lingual event extraction framework and evaluated the framework on the ace 2005 dataset. our framework combines the multilingual language models with a machine-translation-based method. meanwhile, we observe the representational gap between the translated text and the native text in multilingual language models that may affect the performance and propose an adversarial training approach to make the language model produce more similar representations for these two types of text. one potential reason for remaining errors in cross-lingual transfer learning could be that the source and the target languages may differ in the common expressions of an event type. it will be helpful to detect such differences from pretrained multilingual language models and incorporate them for training. although we focus on cross-lingual event extraction in this work, our adversarial training approach could be extended to other crosslingual language understanding tasks.