Learning Cross-lingual Representations for Event Coreference Resolution with Multi-view Alignment and Optimal Transport (2021)

Abstract: We study a new problem of cross-lingual transfer learning for event coreference resolution (ECR) where models trained on data from a source language are adapted for evaluations in different target languages. We introduce the first baseline model for this task based on XLM-RoBERTa, a state-of-the-art multilingual pre-trained language model. We also explore language adversarial neural networks (LANN) that present language discriminators to distinguish texts from the source and target languages to improve the language generalization for ECR. In addition, we introduce two novel mechanisms to further enhance the general representation learning of LANN, featuring: (i) multi-view alignment to penalize cross coreference-label alignment of examples in the source and target languages, and (ii) optimal transport to select close examples in the source and target languages to provide better training signals for the language discriminators. Finally, we perform extensive experiments for cross-lingual ECR from English to Spanish and Chinese to demonstrate the effectiveness of the proposed methods.

## introduction event coreference resolution (ecr) aims to link event-trigger expressions (event mentions) in a document that refer to the same event in real world. technically, the core problem in ecr involves predicting if two event mentions in a document corefer to each other or not (i.e., a binary classification problem). for instance, consider the following text: with national outrage boiling over, bangladeshi paramilitary officers tracked down and arrested sohel rana. when loudspeakers at the rescue site announced his capture, local news reports said, the crowd broke out in cheers. an ecr system in information extraction (ie) should be able to recognize the coreference of the two event mentions associated with the trigger words "arrested" and "capture" in this text. prior work on ecr assumes the monolingual setting where training and test data are presented in the same languages. current state-of-the-art ecr systems thus rely on large monolingual datasets to train advanced models @xcite @xcite @xcite that are only annotated for popular languages (e.g., english). as document annotation for ecr is an expensive process, porting ecr models for english to other languages is crucial and appealing to enhance the accessibility of ecr systems. to this end, this paper explores cross-lingual transfer learning for ecr where models are trained on annotated documents in english (source language) and tested on documents from other languages (target languages). to be clear, our work considers zero-resource cross-lingual learning that requires no labeled data for ecr in the target languages as well as human or machine generated parallel text. the systems in this work only have access to unlabeled text in the target languages to aid the cross-lingual learning for ecr. to our knowledge, this is the first work on cross-lingual transfer learning for event coreference resolution in the literature. recent advances in contextualized word embeddings have featured multilingual pre-trained language models, e.g., multilingual bert @xcite , @xcite , that overcome the vocabulary difference of languages and produce language-universal representations for cross-lingual transfer learning in different nlp tasks @xcite . in fact, such pre-trained language models have set a new standard for multilingual learning in nlp @xcite , serving as the baseline models for our cross-lingual transfer learning problem for ecr in this work. how can we improve the cross-lingual performance of ecr models over multilingual language model baselines? treating the source and target languages as the source and target domains in domain adaptation @xcite @xcite , one can borrow the popular technique of domain adversarial neural networks (dann) @xcite to induce better language-general representations for ecr, called language adversarial neural networks (lann) to make it consistent with our language generalization problem. as such, in addition to traditional learning objectives (e.g., cross-entropy), the key idea of lann is to introduce a language discriminator that seeks to differentiate representation vectors for text inputs from the source and target languages. to enhance the language generalization, models will attempt to generate representation vectors so the language discriminator is fooled, i.e., its performance is minimized to align the source and target languages @xcite . however, there are two major limitations with lann that will be addressed to improve the cross-lingual performance for ecr models in this work. first, taking the binary classification setting for ecr, inputs to the language discriminator in lann involve two pairs of event mentions in the source and target languages. as coreference labels for pairs of event mentions in target languages are not available, the language discriminator will thus only aim to align marginal distributions of event mention pairs (called examples) in the source and target languages (without considering the coreference labels for the pairs). this is less optimal as the lack of coreference labels in the alignment might unexpectedly cause coreferring examples in the source language to be mapped or aligned with non-coreferring examples in the target languages and vice versa, thus impairing the discriminative nature of representation vectors for ecr. to overcome this issue, we propose to use two network architectures to obtain two complementary representation vectors for each example in both source and target languages. representation vectors from each network will be first aligned between source and target languages using the usual lann technique. in addition, representation vectors from the two networks will be regularized to agree with each other over same examples in target languages. as demonstrated later, this regularization helps to penalize the alignment between coreferring examples in the source language and non-coreferring exam-ples in the target languages (and vice versa) in lann, thus improving the representation quality. second, as lann attempts to discriminate all examples in the source language from all examples in the target languages, it also employs training signals from examples whose representations are far away from each other in the source and target languages. however, it is intuitive that the most useful information for model training comes from close examples in the source and target language spaces. including long-distance examples might even introduce noise and hurt the models' performance. consequently, instead of using all examples for lann, we propose to only leverage examples with close representation vectors for the language discriminator in ecr models. as such, our approach involves measuring distances between representation vectors of examples in the source and target languages to determine which examples are used for the language discriminator. to access the distance between two examples in the source and target languages, instead of only relying on the similarity of learned representations, we propose to additionally consider coreference likelihoods of examples that assign higher similarity if two examples have similar coreference likelihoods (i.e., examples with the same coreference labels are more similar to each other than others in ecr). accordingly, our model employs optimal transport, a method to determine the cheapest transformation between two data distributions, as a natural solution to simultaneously incorporate both representation vectors and coreference likelihoods of examples into the distance estimation for example selection in the language discriminator of lann. we conduct cross-lingual ecr evaluation for english, spanish and chinese that demonstrates the benefits of the proposed methods by significantly outperforming the baseline models. we will release experiment setups and code to push forward research on cross-lingual ecr in the future. ## related work regarding coreference resolution, our work is related to studies in entity coreference resolution that aim to resolve nouns phrases/mentions for entities @xcite @xcite @xcite . this work focuses on event coreference resolution that is often considered as a more challenging task than entity resolution due to the more complex structures of event mentions @xcite . for event coreference resolution, although there have been works on cross-document resolution @xcite @xcite , this work is more related to prior work on within-document ecr @xcite . in particular, previous within-document ecr methods have applied feature-based models for pairwise classifiers @xcite @xcite , spectral graph clustering @xcite , information propagation @xcite , markov logic networks @xcite , end-to-end modeling with event detection @xcite @xcite , and recent deep learning models @xcite @xcite @xcite . our work is different from such prior work as we investigate a novel setting of cross-lingual transfer learning for ecr. cross-lingual transfer learning has been studied for other nlp and ie tasks, including sentiment analysis @xcite , relation extraction @xcite @xcite , event extraction @xcite @xcite , and entity coreference resolution @xcite @xcite @xcite . compared to such prior work, this paper presents two novel approaches to improve the language generalization of representation vectors based on multi-view alignment and ot. finally, our work involves lann that bears some similarity with dann models in domain adaptation research of machine learning @xcite @xcite @xcite . compared to such work, our work explores a new dimension of adversarial networks for language-invariant representation learning for texts in ecr. ## model we formalize our ecr problem using a pairwise approach @xcite @xcite . let @xmath0, w 2 , . . . , w n be a document (with n words) that contains two input event mentions with event trig-gers located at w e 1 and w e 2 in w (1 ≤ e 1 < e 2 ≤ n). as such, the core problem in ecr is to perform a binary prediction to determine whether two event mentions w e 1 and w e 2 refer to the same event or not. an example in our ecr task thus involves an input tuple @xmath1, e 1 , e 2 ) and a binary output variable y to indicate the coreference of w e 1 and w e 2 . this work focuses on crosslingual transfer learning for ecr where training data involve input documents w in english (the source language) while sentences in test data are presented in another language (the target language). to enable the zero-resource cross-lingual setting for ecr, our model takes two following inputs: ## conclusion this paper presents the first study on cross-lingual transfer learning for event coreference resolution. we introduce the first baseline models for this problem, leveraging a state-of-the-art pre-trained language models for multilingual nlp (i.e., xlm-roberta) and lann for language-invariant representation learning. we propose two novel techniques for cross-lingual transfer learning based multi-view alignment to avoid cross-label alignment in the source and target languages and optimal transport for example selection in lann. our experiments provide baselines for future research and demonstrate the benefits of the proposed methods for cross-lingual transfer learning for ecr.