Transitioning Representations between Languages for Cross-lingual Event Detection via Langevin Dynamics (2023)

Abstract: Cross-lingual transfer learning (CLTL) for event detection (ED) aims to develop models in high-resource source languages that can be directly applied to produce effective performance for lower-resource target languages. Previous research in this area has focused on representation matching methods to develop a language-universal representation space into which source- and target-language example representations can be mapped to achieve cross-lingual transfer. However, as this approach modifies the representations for the source-language examples, the models might lose discriminative features for ED that are learned over training data of the source language to prevent effective predictions. To this end, our work introduces a novel approach for cross-lingual ED where we only aim to transition the representations for the target-language examples into the source-language space, thus preserving the representations in the source language and their discriminative information. Our method introduces Langevin Dynamics to perform representation transition and a semantic preservation framework to retain event type features during the transition process. Extensive experiments over three languages demonstrate the state-of-the-art performance for ED in CLTL.

## introduction event detection (ed) is one of the fundamental problems in information extraction (ie). this task aims to identify and classify the words or phrases that most clearly evoke events in text (called event triggers). for instance, given a sentence "he was fired from the corporation yesterday.", an ed system needs to recognize the word "fired" as an event trigger of the type attack. with the advances in deep learning, recent research has produced impressive performance for ed @xcite @xcite @xcite . however, despite such progress, existing datasets for ed are limited to a small set of popular languages (e.g., english and chinese) due to the high cost of data annotation for this task. the lack of annotated data has thus hindered the development of ed systems for a larger set of languages and the benefits of the technology for broader society. to enable ed for more languages while avoiding expensive annotation efforts, recent research has focused on zero-shot cross-lingual transfer learning (cltl) for ed where models are trained on annotated data of a high-resource source language and directly applied to extract events for text in another low-resource target language @xcite @xcite . in this setting, labeled data is not available for the target language; however, unlabeled data can be used to facilitate knowledge transfer between languages. the key challenging in cltl for ed is to bridge the gap between representation spaces for the source and target languages to enable knowledge transfer. as such, previous work in this area has focused on the representation matching approaches, aiming to transform the representations for the source and target languages into a common space to train language-general ed models over source language data, e.g., via similarity regularization @xcite or adversarial learning @xcite . however, to achieve a common space, the source-language representations might need to sacrifice languagespecific information, which can involve discriminative features for event prediction due to the mix of information in the representations for input text. consequently, losing discriminative features might lead to impaired performance for ed. to this end, our work explores a new direction for cltl for ed using representation transition. instead of transforming the source-language representations, which might eliminate discriminative features for ed, we only seek to adapt the targetlanguage representations into the source space whose results can be directly fed into the sourcelanguage models for event prediction. in this way, we can preserve the source-language representations learned over training data for ed to maximize prediction ability. to achieve this idea, we propose to employ an energy function to model the representations for the source-language training examples where low-energy regions correspond to high likelihood of belonging to the source language @xcite . afterward, target-language representations can be adapted to the source-language space by transitioning them into low-energy regions (i.e., minimizing the energy function). as such, our method leverages langevin dynamics @xcite to perform efficient representation transition for the target language for cross-lingual ed where gradient descent is utilized to iteratively update the initial target representations into the source language with low-energy regions. to our knowledge, this is the first work to explore energy functions and langevin dynamics for cltl in ie and ed. finally, we introduce a mechanism to regulate the adapted representations for the target language examples from langevin dynamics so they can maintain semantic similarity of event types with the original representations. this regularization is necessary to ensure that the predictions of the source language model over the adapted representations are correct for the target examples. extensive evaluations over three languages demonstrate our state-of-the-art performance for cross-lingual ed. ## model base model: following prior work @xcite , we formulate ed as a sequence labeling problem to facilitate cross-lingual transfer learning (cltl). given an input sentence of n tokens @xmath0, w 2 , . . . , w n }, our ed model needs to predict its corresponding sequence of labels @xmath1, y 2 , . . . , y n }. here, y i is the label for the token w i using the bio annotation schema to capture event mentions and types in w . in our model, we first use the multilingual pre-trained language model xlm-r @xcite as the feature extractor for the input sentence w , leading to a representation vector h i for each word here, θ x captures the parameters of xlm-r. afterward, the representation h i is fed into a feed-forward network (ffn) with parameters θ c to produce a label distribution for w i for event prediction. for convenience, in the following, we will use the subscripts src and tgt to specify words and representations from the source and target languages (respectively). energy model and langevin dynamics: as discussed above, to perform cltl for ed, we propose to learn an energy function @xcite to model the representation distribution of the source-language examples. the representations of the target-language examples can then be transitioned toward this source distribution to perform ed. formally, in our method, an energy function e θ e (h) : r @xmath2. this function can then be used to define a distribution over the representation space: in the second step, we freeze the parameters θ x for xlm-r and train the energy function e θ e by optimizing the negative log-likelihood for the induced representations h i for w @xmath3: l @xmath4, where l involves indexes for the words in the event trigger spans in w . here, although computing the normalizing term z θ e for e is intractable, the gradient of log p θ e (h) can still be estimated to perform training in our second step @xcite : as such, the data for the first term is drawn from the representations for the source language examples p data (h) while those for the second term can be sampled from the model representation distribution p θ e (h) using our langevin dynamics process. at inference time for cltl, given an input sentence w tgt in the target language, the representation h tgt i for each word w tgt @xmath5, resulting in the adapted representation h tgt i . afterward, due to their compatibility, we can apply the source-trained ed model θ c over h tgt i to predict the event type for w tgt i . semantic preservation: although langevin dynamics can adapt the representation h tgt i to achieve features in the source language, it cannot guarantee that the adapted representation h tgt i can preserve semantic information of event types from h tgt i to produce correct prediction based on h tgt i . to facilitate semantic maintenance in the adaption, we propose to decompose a representation h into languageinvariant and language-specific components z and l (respectively). as such, our intuition for semantic maintenance is that the target-to-source adaptation process should retain language-invariant information while eliminating target-specific features and absorbing source-language features along the way. we employ both labeled data in the source language and unlabeled data in the target language to achieve semantic preservation in this step. representation decomposition: to disentangle the representations, we employ the variationl auto-encoder (vae) framework @xcite to introduce two encoders q ϕz (z|h) and q ϕ l (l|h) to transform a representation h into stochastic dimensional spaces for language-invariant and language-specific representations z and l respectively. in addition, a decoder p ϕ h (h|z, l) is introduced to infer the representation h from z and l. here, the distributions q ϕz (z|h), q ϕ l (l|h), and p ϕ h (h|z, l) are assumed to be gaussian and feed-forward neural networks with parameters ϕ z , ϕ l , and ϕ h are leveraged to compute the means and variances for the distributions from corresponding representations. to this end, vaes learn the parameters by minimizing the negation of the variational lower bound: for the language-invariant representation, we expect z to encode information on event types to allow transferability across languages. as such, we utilize z to predict the event type y using training examples in the source language. in particular, a feed-forward network is introduced to compute the distribution p (y|z) from z and the log-likelihood e q ϕz (z|h) [log p (y|z)] is used to train the network. for the representation l, its language-specific information is achieved by computing the distribution p (d|l) with a feed-forward network f f l to predict the language identity d for h (i.e., @xmath6, target}). f f l can then be trained with the objective e q ϕ l (l|h) [log p (d|l)] using unlabeled data in both source and target languages. consequently, to promote the expected features for z and l, we minimize the combined function: each column corresponds to one language pair where source languages are shown above the target languages. the proposed model is significantly better than other models with @xmath7.01. ## related work a large body of previous ed research is dedicated to monolingual learning, i.e., training and testing over the same languages @xcite @xcite @xcite . the models might consider different domains @xcite . to address the language barriers, recent work has studied cltl for ed, focusing on mapping representations for different languages into the same space for transfer learning, e.g., using bilingual resources @xcite , multilingual language models (m'hamdi et al., 2019), and adversarial training @xcite . our work is also related to previous work on data projection for ed and ie where texts in the target language are translated and aligned into the source language for prediction at test time @xcite @xcite . however, data projection requires high-quality machine translation and alignment systems, which might not be available for different languages. our method is different as we project representations (not actual texts) in the target to the source language space to avoid translation systems. ## conclusion we introduce a novel method for cltl for ed where representations of examples in the target language are transferred into the source language space at test time for prediction with the source language model. our extensive experiments confirm the benefits of the proposed method. in the future, we plan to extend our method to other tasks in ie.