Hybrid Knowledge Transfer for Improved Cross-Lingual Event Detection via Hierarchical Sample Selection (2023)

Abstract: In this paper, we address the Event Detection task under a zero-shot cross-lingual setting where a model is trained on a source language but evaluated on a distinct target language for which there is no labeled data available. Most recent efforts in this field follow a direct transfer approach in which the model is trained using language-invariant features and then directly applied to the target language. However, we argue that these methods fail to take advantage of the benefits of the data transfer approach where a cross-lingual model is trained on target-language data and is able to learn task-specific information from syntactical features or word-label relations in the target language. As such, we propose a hybrid knowledge-transfer approach that leverages a teacher-student framework where the teacher and student networks are trained following the direct and data transfer approaches, respectively. Our method is complemented by a hierarchical training-sample selection scheme designed to address the issue of noisy labels being generated by the teacher model. Our model achieves state-of-the-art results on 9 morphologically-diverse target languages across 3 distinct datasets, highlighting the importance of exploiting the benefits of hybrid transfer.

## introduction event detection (ed) is a sub-task of the encompassing information extraction (ie) natural language processing (nlp) task. the main objective of ed is to detect and categorize the event triggers in a sentence, i.e., the words that most clearly indicate the occurrence of an event. event triggers are known to be frequently related to the verb in a sentence @xcite . however, they can also be other parts of speech such as nouns or adjectives. for instance, in the sentence "the ceremony was chaired by the former secretary of state", an ed system should recognize former as the trigger of a personnel:end-position event 1 . generating labeled data for ie tasks such as ed can be a long and expensive endeavor. as such, most labeled ed datasets pertain to a small set of popular languages (e.g., english, chinese, spanish). in turn, labeled data is scarce or non-existent for a vast majority of languages. this imbalance in annotated data availability has prompted many research efforts into zero-shot cross-lingual transfer learning which attempts to transfer knowledge obtained from annotated data in a high-resource source language to a low-resource target language for which no labeled data is available. there are two predominant knowledge-transfer paradigms employed by such cross-lingual methods: data transfer and direct transfer. approaches that adhere to the data transfer paradigm generate pseudo-labeled data in the target language and then train a model on such data. this pseudo-training data can be constructed by mapping the gold source labels into parallel, or translated, versions of the source data, or by leveraging source-trained models to annotate unlabeled target data. since models in this category are trained on the target language, they can directly exploit wordlabel relations and other target-language-specific information such as word order and lexical features @xcite . however, annotated parallel corpora are extremely scarce, and misaligned or incorrect translations introduce noise that affects the model performance. in contrast, direct-transfer-based approaches aim at creating cross-lingual models by training them with delexicalized, language-independent features obtained from the labeled, source-language data. the resulting language-agnostic models can then be applied directly to unlabeled data in the target language. in recent years, direct transfer has become the favored transfer paradigm as such models have less need for cross-lingual resources and can be applied to a broader range of languages. as such, previous research efforts on cross-lingual event detection (cled) have mostly focused on the direct transfer approach @xcite and, in consequence, have failed to exploit the aforementioned advantages of training with target-language data. more recent approaches have attempted to address this issue by incorporating unlabeled targetlanguage data into the training process. for example, @xcite propose a classaware, cross-lingual alignment mechanism where they align examples from the source and target languages based on class information. @xcite instead propose to improve standard adversarial language adaptation (ala) @xcite by only presenting the language discriminator with informative samples. despite their improved results, these models only learn task-related information from the source language and fail to make use of the potentially useful information contained in wordlabel relations in the target language. furthermore, previous studies on similar tasks have shown that, even for direct transfer methods, lexical features are useful if the source and target languages are close to each other @xcite . given that the data transfer and direct transfer paradigms are orthogonal, in this paper we present a hybrid transfer approach for cross-lingual event detection that (1) exploits the desirable features of both and (2) minimizes their respective shortcomings. for this purpose, we propose a knowledge distillation framework which has already been proven effective on similar cross-lingual tasks @xcite @xcite . in our proposed framework, a teacher model is trained using a direct transfer approach (i.e., with languageinvariant features obtained from annotated source data) and applied to unlabeled target-language data. then, this pseudo-labeled data is utilized to train a student model so that it benefits from the advantages of the data transfer paradigm. nonetheless, we recognize that the pseudo-labels obtained from the teacher model are prone to containing noisy predictions which can be hurtful for student training. to address this issue, we argue that the teacher model should produce more dependable predictions on target-language examples that share some similarities with their source-language counterparts. as such, we propose to improve the teacher-student learning process by restricting student training to samples with such desirable characteristics. we perform our trainingsample selection in a hierarchical manner: first, we leverage optimal transport (ot, @xcite to compute similarity scores between batch samples in the source and target languages. only samples with similarity scores above a certain threshold are selected in this first step. ot has already been shown to be effective at estimating cross-lingual similarities for sample selection @xcite . then, in the second step, we make use of cross-domain similarity local scaling (csls, @xcite to refine our sample selection. csls provides an enhanced measure to obtain reliable matches between samples in the source and target languages by addressing the hubness phenomenon that plagues nearestneighbor-based pair-matching methods. the student model is then trained on the hierarchicallyselected target-language samples exclusively. in order to validate our approach, we compare our model's performance against current state-ofthe-art models for cled. for this purpose, we report our results on the most commonly used cled benchmarking datasets: ace05 @xcite and ace05-ere @xcite . these datasets, in conjunction, contain ed annotations for 3 distinct target languages. our experimental results show that our approach consistently outperforms such state-of-the-art cled models. additionally, we further evaluate the flexibility and applicability of our model by leveraging the recently released minion dataset @xcite which contains ed annotations for 8 typologically different languages. the remainder of this document is organized as follows: section 2 presents the definition of the ed task and an in-depth description of our model and approach, section 3 includes the main results from our experiments and related analysis, section 4 provides a review of previous relevant work, and finally, section 5 presents our conclusions. ## related work event detection (ed) is an active research area in nlp @xcite @xcite , featuring cross-lingual ed as a recent direction with growing interests. the work by @xcite presents a data transfer method that learns a mapping between monolingual word embeddings, translates the source training data on a word-by-word basis and uses a graph convolutional network to generate orderindependent representations. m' hamdi et al. (2019) leverage mbert as an encoder to perform zeroshot transfer learning and a crf layer to account for label dependency. @xcite present a cross-lingual structure transfer approach that represents sentences as language-universal structures (trees, graphs). in their work, @xcite argue that event triggers are usually related to the verb in a sentence and propose to incorporate external verb knowledge by pre-training their encoder to classify whether two verbs belong to the same class according to two distinct ontologies verbnet, @xcite and framenet, @xcite . model prim-ing @xcite is a simple, yet effective method that consists in augmenting the encoder inputs by concatenating a candidate trigger to the input sentence so that the encoder learns to generate task-specific representations. nguyen et al. (2021) leverage class information and word categories as language-independent sources of information and condition their encoder to generate representations that are consistent in both the source and target languages. finally, @xcite propose to optimize standard adversarial language adaptation by restricting the language discriminator training to informative examples. our approach is also closely related to knowledge distillation models for cross-lingual named entity recognition (ner). @xcite were the first to train a ner student model on the label distributions obtained from a teacher model. @xcite improved upon this initial approach with a multi-step training method that involved fine-tuning the teacher model with pseudolabeled data and generating hard labels that were later used for student training. more recent proposals improve the knowledge distillation with either reinforcement learning @xcite or adversarial training @xcite . nonetheless, our approach is the first to leverage a knowledge distillation framework for cled, and our novel hierarchical training-sample selection scheme further differentiates our work from previous efforts. ## conclusion in this work, we present the first effort to leverage a hybrid knowledge-transfer approach for the cross-lingual event detection task. we propose a teacher-student framework complemented by a hierarchical training-sample selection scheme that effectively constrains the student-training process to pseudo-labeled target-language samples that are similar to their source-language counterparts. our hkt-cled model sets a new state-of-the-art performance on the most popular benchmarking datasets ace05 and ace05-ere, and obtains substantial performance improvements on the recentlyreleased, and more diverse, minion dataset with an average improvement of +7.74 f1 points across 7 distinct target languages. we believe these results demonstrate our model's robustness and applicability and validate our claim that combining the benefits of the direct transfer and data transfer approaches is beneficial for cross-lingual learning.