Crosslingual Transfer Learning for Relation and Event Extraction via Word Category and Class Alignments (2021)

Abstract: Previous work on crosslingual Relation and Event Extraction (REE) suffers from the monolingual bias issue due to the training of models on only the source language data. An approach to overcome this issue is to use unlabeled data in the target language to aid the alignment of crosslingual representations, i.e., via fooling a language discriminator. However, as this approach does not condition on class information, a target language example of a class could be incorrectly aligned to a source language example of a different class. To address this issue, we propose a novel crosslingual alignment method that leverages class information of REE tasks for representation learning. In particular, we propose to learn two versions of representation vectors for each class in an REE task based on either source or target language examples. Representation vectors for corresponding classes will then be aligned to achieve class-aware alignment for crosslingual representations. In addition, we propose to further align representation vectors for language-universal word categories (i.e., parts of speech and dependency relations). As such, a novel filtering mechanism is presented to facilitate the learning of word category representations from contextualized representations on input texts based on adversarial learning. We conduct extensive crosslingual experiments with English, Chinese, and Arabic over REE tasks. The results demonstrate the benefits of the proposed method that significantly advances the state-of-the-art performance in these settings.

## introduction relation and event extraction (ree) are important tasks of information extraction (ie), whose goal is to extract structured information from unstructured text @xcite . due to their complexity, annotations for ree tasks are costly and only available in a few languages. thus, there have been growing interests on crosslingual learning for ree in which a model is trained on a language, i.e., source language, and applied to another language, i.e., target language, where the annotations are not available. recent approaches for crosslingual ree have mainly employed multilingual word embeddings, e.g., muse, @xcite @xcite or multilingual pre-trained language models, e.g., multilingual bert, @xcite @xcite to learn crosslingual representation vectors for ree. however, previous work on crosslingual ree suffers from the monolingual bias issue due to the monolingual training of models on only the source language data, leading to non-optimal crosslingual performance. a solution for this issue can resort to language adversarial training @xcite @xcite @xcite where unlabeled data in the target language is used to aid crosslingual representations via fooling a language discriminator. the underlying principle for this approach is to encourage the closeness of representation vectors for sentences in the source and target languages (i.e., aligning representation vectors). however, a critical drawback of language adversarial training is the failure to condition on classes/types of examples in the alignment process. as such, a target language example of a class could be incorrectly aligned to a source language example of a different class in ree, causing confusion and hindering the performance of the models. the middle sub-figure in figure 2 demonstrates the class misalignment of representation vectors in crosslingual ree. to this end, we propose a crosslingual alignment method that explicitly conditions on class information of ree tasks to enhance representation alignment and learning. our major intuition is that the semantics of the classes in ree tasks (e.g., the event type of attack in event extraction) are generally invariant across languages that can be leveraged as anchors to bridge representation vectors for examples in different languages. as such, we can obtain two semantic representation vectors for each class in an ree task based on representation vectors of examples in either source or target language. afterward, the representation vectors of the same class can be regulated to match each other, serving as a mechanism for class-aware crosslingual alignment of representation vectors for source and target examples. to implement this idea, we use multilingual bert (mbert) to obtain same-space representations for examples in both source and target languages to facilitate the alignment process. afterward, the source-language representation vector for a class is computed via representation vectors of source-language examples that belong to the corresponding class. for the target language, as class information is not provided, we seek to compute target-language representation vector for a class by aggregating representation vectors for unlabeled examples, weighted on an estimation of the probabilities for the examples to exhibit the class. in addition to class semantics, we propose to further exploit universal parts of speech and dependency relations in parsing trees (i.e., word categories) to improve the cross-lingual alignment for representation vectors in ree. as such universal word categories have been consistently annotated for more than 100 languages @xcite and can be generated with high accuracy via existing toolkits, e.g., the transformer-based toolkit trankit for multilingual nlp @xcite @xcite , we expect this information to provide helpful anchor knowledge for cross-lingual representation learning. to this end, similar to the class-aware alignment, we propose to align representation vectors of the same universal word categories that are computed using contextualized representations of examples in the source and target languages to further improve the language-independence of representation vectors for ree. a potential issue with the computation of word category representations via contextualized representations of examples is the preservation of context word information in representations for word categories that might introduce noise and hinder the representation alignment. to address this issue, we propose an adversarial training model that seeks to explicitly filter context information from word category representations. this is achieved by using gradient reversal layer @xcite to prevent word category representations from being able to recognize the context words in the original examples. we expect that this filtering mechanism can improve the word category pureness of the representations, thus providing appropriate inputs for the alignment process for improved representation learning. we conduct extensive experiments with different crosslingual settings on english, chinese, and arabic for three ree tasks, i.e., relation extraction, event detection, and event argument extraction. the results demonstrate the benefits of the proposed method that significantly advances the state-of-the-art performance in these settings. ## problem statement we study cross-lingual transfer learning for three ree tasks as defined in the ace 2005 dataset @xcite , i.e., relation extraction (re), event detection (ed), and event argument extraction (eae). given two entity mentions in an input sentence, the goal of re is to determine the semantic relationship between the mentions according to predefined relation types/classes (e.g., employment). for ed, its purpose is to identify event triggers, which can be verbs/normalization with one or multiple words, that express occurrences of events of predefined types (e.g., attack). finally, given an event trigger and an entity mention, eae aims to predict the role (e.g., victim) that the entity mention plays in the corresponding event. note that, we have a special type none to indicate nonrelation, non-trigger, or non-argument for re, ed, and eae respectively. for further discussion, let d @xmath0, y src )} (|d src | = n src ) be the labeled training set in the source language. as such, for ed, x src is an input sentence and y src serves as the golden sequence tag (using bio) for the words in x src . for re and eae, x src involves an input sentence along with indexes of the given trigger word and entity mentions while y src represents the golden relation type or argument role for the input. we also assume access to an unlabeled dataset d @xmath1. ## baseline methods to prepare for our cross-lingual representation alignment techniques for ree, we first describe the baseline models explored in this work. ## related work ree has been extensively studied for english, featuring traditional machine learning methods @xcite @xcite and advanced deep learning models @xcite @xcite @xcite @xcite @xcite . recently, several works have considered cross-lingual transfer learning for three ree tasks @xcite @xcite where multilingual pre-trained language models (e.g., mbert) have been proved as an important encoding component @xcite . however, a fundamental limitation of existing crosslingual models for ree is the monolingual bias due to the sole reliance on source language data for training. in other nlp tasks, ladv has been explored to address this issue by leveraging 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% percentage of source language data used 35 40 45 50 55 60 65 70 f1 score (%) re:gate+cccar re:gate ed:bertcrf+cccar ed:bertcrf eae:gate+cccar eae:gate figure 3: performance on test data of the models in the english-to-chinese setting. dash lines represent the performance of the source-only baselines using 100% of the source-language training data. ## conclusions we present a novel method for crosslingual transfer learning for ree that leverages unlabeled data in the target language to support language-universal representation learning. our method exploits class semantics in ree tasks and universal word categories (i.e., upos and dependency relations) as bridges to align representation vectors across languages. in our method, representation vectors for classes and word categories are computed via contextualized representations of examples to implement representation matching for crosslingual alignment. extensive experiments show that the proposed method achieves sota performance for three ree tasks in different crosslingual settings. in the future, we plan to extend our methods to related problems in ie (e.g., coreference resolution). thien huu nguyen, kyunghyun cho, and ralph grishman. 2016a. joint event extraction via recurrent neural networks. in proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies (naacl-hlt). thien huu nguyen and ralph grishman. 2015. event detection and domain adaptation with convolutional neural networks. in the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (acl-ijcnlp). thien huu nguyen and ralph grishman. 2018. graph convolutional networks with argument-aware pooling for event detection. in proceedings of the association for the advancement of artificial intelligence (aaai). jian ni and radu florian. 2019. neural cross-lingual relation extraction based on bilingual word embedding mapping. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp). siddharth patwardhan and ellen riloff. 2009. a unified model of phrasal and sentential evidence for information extraction. in proceedings of the annual meeting of the association for computational linguistics (acl).