Retrieving Relevant Context to Align Representations for Cross-lingual Event Detection (2023)

Abstract: We study the problem of cross-lingual transfer learning for event detection (ED) where models trained on a source language are expected to perform well on data for a new target language. Among a few recent works for this problem, the main approaches involve representation matching (e.g., adversarial training) that aims to eliminate language-specific features from the representations to achieve the language-invariant representations. However, due to the mix of language-specific features with event-discriminative context, representation matching methods might also remove important features for event prediction, thus hindering the performance for ED. To address this issue, we introduce a novel approach for cross-lingual ED where representations are augmented with additional context (i.e., not eliminating) to bridge the gap between languages while enriching the contextual information to facilitate ED. At the core of our method involves a retrieval model that retrieves relevant sentences in the target language for an input sentence to compute augmentation representations. Experiments on three languages demonstrate the state-of-the-art performance of our model for cross-lingual ED.

## introduction as one of the core tasks in information extraction (ie), the goal of event detection (ed) is to identify and classify the word(s) that most clearly evoke events in text (called event triggers). for instance, in the sentence "he was fired from the corporation yesterday.", an ed system needs to predict "fired" as an event trigger of the type attack. due to its applications, ed has been well studied over the last decade, featuring deep learning as the most recent approach with state-of-the-art performance @xcite @xcite @xcite . however, despite intensive studies, most prior work has focused on monolingual learning settings for ed where models are trained and evaluated on labeled data of the same languages @xcite @xcite @xcite @xcite ). as such, to extend current ed models to another language, monolingual learning will require new annotated data to train the models, which can be very expensive to obtain for different languages. to this end, there has been a growing interest in cross-lingual transfer learning for ed where models trained on a source language are directly applied to data of a new target language @xcite . in this way, labeled data from high-resource languages (e.g., english) can be leveraged to develop effective ed models for other languages (e.g., low-resource ones). in this work, we focus on zero-shot cross-lingual transfer learning for ed to avoid the need for labeled data in the target languages, thus enabling fast adaptation of ed models to multiple languages. a key strategy for cross-lingual transfer learning is to align input text representations for the source and target languages to facilitate cross-lingual extraction of events. as such, prior work on crosslingual ed has explored multilingual word embeddings (e.g., muse) @xcite or recent multilingual pre-trained language models (e.g., mbert) @xcite to represent source-and target-language texts in the same space. recently, state-of-the-art crosslingual ed methods have leveraged unlabeled data in the target language with representation matching frameworks to further align text representations for the source and target languages @xcite . given two sentences in the source and target languages, these methods aim to encode the two sentences to obtain representation vectors for language-universal objects, e.g., sentences, event types, universal dependency relations or parts of speech @xcite . afterward, the representations of the same language-universal objects (computed with the source or language data) are regulated to be similar to each other to improve the alignment between the source and target languages for cross-lingual ed (e.g., with adversarial training to fool the language discriminators). as such, to achieve representation similarity between languages, previous representation matching methods for ed will need to filter information/features that are specific to each language in the representations @xcite . however, as the representations for each language are computed from input sentences, the languagespecific information might involve/mix with important contextual structures, which are necessary to reveal event types in the representations. consequently, removing language-specific information might also eliminate important discriminative context features, thus limiting the performance for ed models. to address this issue, our work explores a novel approach for alignment of language representations for cross-lingual ed that avoids direct similarity regularization and removal of important context information from the representation vectors. instead, our approach seeks to add relevant context information into original input representations for the source and target language texts to make them closer for effective cross-lingual ed. in particular, starting with the representation vectors s and t to perform ed in the source and target languages (respectively), we aim to induce additional context representations a(s) and a(t ) that will be added into s and t (i.e., leading to the augmented representations s + a(s) and t + a(t )) to achieve greater similarity between representations for the source and target languages. one the one hand, the additional representations a(s) and a(t ) will be obtained over sentences in the target language to bias the prediction representations toward the target space and enhance representation alignment for cross-lingual ed. on the other hand, we will leverage external sentences with relevant/related contexts to the original representations s and t to compute the augmented representations a(s) and a(t ). as such, with enriched context information, we expect that the augmented representations s + a(s) and t + a(t ) will facilitate the prediction of event types to boost performance for cross-lingual ed. note that this representation augmentation with relevant context is not possible in previous cross-lingual transfer learning methods for ed, thus highlighting the advantage of our proposed approach for cross-lingual ed in this work. to implement the representation augmentation idea for cross-lingual ed, we introduce a "retrievethen-classify" framework with two major steps. in the first step of retrieval, given an input sentence, our model first retrieves relevant/related sentences from an unlabeled dataset (e.g., focusing on sentences with similar event types). next, the retrieved sentences will be encoded and their representations will be injected into the representation for the input sentence to perform ed. in our method, the unlabeled dataset will be taken from the target language (for input sentences from both the source and target languages) to shift the augmented representations to the target language space, thus implicitly bridging the gap between representations for different languages for ed. in addition, to better customize the context retrieval for our cross-lingual ed task, the retrieval model will be jointly trained with the ed model in an end-to-end fashion to encourage their interactions/feedback for better overall performance. our framework also introduces a novel regularization mechanism to promote the shared awareness of relevant context sentences for an input sentence between retrieval and ed models to further improve the induced representations for cross-lingual ed. finally, we conduct extensive experiments on the multilingual ace 2005 dataset for ed (with three languages: english, chinese, and arabic), demonstrating the state-of-the-art performance of the proposed method over different language pairs for cross-lingual transfer learning of ed. to our knowledge, this is the first work on retrieval-based models for cross-lingual ed. ## model we follow prior work @xcite to formalize the cross-lingual transfer learning (cltl) task for ed as a sequence labeling problem. given an input sentence @xmath0, w 2 , . . . , w n with n words, we need to assign a label y i for each word w @xmath1. in cltl, the input sentence w belongs to the source language in the training time while sentences in the new target language are leveraged for evaluation in test time. similar to recent methods on cltl for ed @xcite , our model assumes an unlabeled dataset @xmath2, u 2 , . . . , u m } that contains m sentences in the target language (u t is the t-th sentence in u ). given the input sentence w , the first step in our model involves retrieving a set of relevant sentences r in the unlabeled dataset u (i.e., r ⊂ u ) to provide augmented context information for w for cross-lingual ed. note that the unlabeled set u will be used to retrieve sentences for input texts in both training and testing phases. the representations for the retrieved sentences in r will later be integrated into the representation vectors for the words in w to perform sequence labeling for ed. the benefit of this representation augmentation approach is twofold. first, as the representations for the retrieved sentences r are computed over the target language sentences, during the training time with the source-language input sentence w , the representation augmentation will shift the representations for the words w @xmath3. this helps to bridge the gap between the source-and target-language representation spaces that enables the training of ed models over source-language data to better generalize to data in the target language (i.e., cross-lingual generalization). second, during the test time with the target language, incorporating context information from the retrieved relevant sentences r will enrich/strengthen the representations for the words in the original input sentence w , thus facilitating the predictions of labels to boost performance for cross-lingual ed. our following sections will describe the retrieval and ed models in our method. ## related work ed has been studied mostly for monolingual settings, involving feature-based models @xcite @xcite and recent deep learning models @xcite @xcite @xcite @xcite . cross-lingual transfer learning for ed has gained more interests recently where different resources are leveraged to project the representations for different languages into the same space, including bilingual dictionaries/parallel corpora @xcite and multilingual language models (m @xcite @xcite . to further bridge the gap between the representations for cross-lingual ed, @xcite explores adversarial training with language discriminators @xcite @xcite and representation matching of similar objects to remove language-specific features. we also note that these methods are motivated from domain adaptation methods that aim to avoid domain-specific features @xcite @xcite @xcite . in contrast, our model introduces additional augmentation representations from retrieval to achieve language-universal representations. ## conclusion we present a novel method for cross-lingual transfer learning for ed. instead of removing languagespecific features, our model augments the representations for the input sentences with those from relevant sentences in the target language to align the representations for the source and target languages. our method involves a retrieval component to obtain relevant sentences that is jointly trained with the ed model. our proposed method demonstrates the state-of-the-art cross-lingual performance over six different language pairs.