Constructing Code-mixed U niversal D ependency Forest for Unbiased Cross-lingual Relation Extraction (2023)

Abstract: Latest efforts on cross-lingual relation extraction (XRE) aggressively leverage the language-consistent structural features from the universal dependency (UD) resource, while they may largely suffer from biased transfer (e.g., either target-biased or source-biased) due to the inevitable linguistic disparity between languages. In this work, we investigate an unbiased UD- based XRE transfer by constructing a type of code-mixed UD forest. We first translate the sentence of the source language to the parallel target-side language, for both of which we parse the UD tree respectively. Then, we merge the source-/target-side UD structures as a unified code-mixed UD forest. With such forest features, the gaps of UD-based XRE between the training and predicting phases can be effectively closed. We conduct experiments on the ACE XRE benchmark datasets, where the results demonstrate that the proposed code-mixed UD forests help unbiased UD-based XRE transfer, with which we achieve significant XRE performance gains.

## introduction relation extraction (re) aims at extracting from the plain texts the meaningful entity mentions paired with semantic relations. one widelyacknowledged key bottleneck of re is called the long-range dependence (lrd) issue, i.e., the decay of dependence clues of two mention entities with increasing distance in between @xcite @xcite . fortunately, prior work extensively reveals that the syntactic dependency trees help resolve lrd issue effectively, by taking advantage of the close relevance between the dependency structure and the relational re pair @xcite . in cross-lingual re, likewise, the universal dependency trees @xcite are leveraged as effective language-persistent features in the latest work for better transfer from source (src) language to target (tgt) language @xcite @xcite . current state-of-the-art (sota) xre work leverages the ud trees based on the model transfer paradigm, i.e., training with src-side ud features while predicting with tgt-side ud features @xcite . model transfer method transfers the shareable parts of features from src to tgt, while unfortunately it could fail to model the tgt-side language-specific features, and thus results in a clear tgt-side bias. in fact, the tgt-side bias can be exacerbated in ud-based model transfer, cf. fig. 1 @xcite . given that ud has a universal annotation standard, inevitably, there is still a syntax discrepancy between the two languages due to their intrinsic linguistic nature. we show (cf. §3 for more discussion) that between the parallel sentences in english and arabic, around 30% words are misaligned and over 35% ud word-pairs have no correspondence. such structural discrepancies consequently undermine the model transfer efficacy. one alternative solution is using annotation projection @xcite @xcite . the main idea is directly synthesizing the pseudo tgtside training data, so that the tgt-side linguistic features (i.e., ud trees) are well preserved. however, it could be a double side of the sword in the annotation projection paradigm. it manages to learn the language-specific features, while at the cost of losing some high-efficient structural knowledge from src-side ud, thus leading to the src-biased ud feature transfer. as illustrated in fig. 1(b) , the dependence paths in the src ud tree that effectively solves the lrd issues for the task are sacrificed when transforming the src tree into the tgt tree. this motivates us to pursue an unbiased and holistic ud-based xre transfer by considering both the src and tgt ud syntax features. to reach the goal, in this work, we propose combining the view of model transfer and annotation projection paradigm, and constructing a type of codemixed ud forests. technically, we first project the src training instances and tgt predicting instances into the opposite languages, respectively. then, we parse the parallel ud trees of both sides respectively via existing ud parsers. next, merge each pair of src and tgt ud trees together into the code-mixed ud forest, in which the wellaligned word pairs are merged to the tgt ones in the forest, and the unaligned words will all be kept in the forest. with these code-mixed syntactic features, the gap between training and predicting phases can be closed, as depicted in fig. 1(c ). we encode the ud forest with the graph attention model (gat; @xcite for feature encoding. we perform experiments on the representative xre benchmark, ace05 (hristopher @xcite , where the transfer results from english to chinese and arabic show that the proposed code-mixed forests bring significant improvement over the current best-performing ud-based system, obtaining the new sota results. further analyses verify that 1) the code-mixed ud forests help maintain the debiased cross-lingual transfer of re task, and 2) the larger the difference between src and tgt languages, the bigger the boosts offered by code-mixed forests. to our knowledge, we are the first taking the complementary advantages of annotation projection and model transfer paradigm for unbiased xre transfer. we verify that the gap between training and predicting of ud-based xre can be bridged by synthesizing a type of code-mixed ud forests. the resource can be found at https://github.com/ scofield7419/xlsie/ . ## related work different from the sequential type of information extraction (ie), e.g., named entity recognition (ner) @xcite , re not only detects the mentions but also recognizes the semantic relations between mentions. re has long received extensive research attention within the last decades @xcite . within the community, research has revealed that the syntactic dependency trees share close correlations with re or broad-covering information extraction tasks in structure @xcite @xcite , and thus the former is frequently leveraged as supporting features for enhancing re. in xre, the key relational features between words need to be transferred between languages, which motivates the incorporation of ud tree features that have consistent annotations and principles across various languages. thus, ud-based systems extensively achieve the current sota xre @xcite @xcite . this work inherits the prior wisdom, and leverages the ud features. model transfer @xcite @xcite and annotation projection @xcite @xcite @xcite are two mainstream avenues in structural cross-lingual transfer track. the former trains a model on src annotations and them make predictions with tgt instances, i.e., transferring the shared language-invariant features. the latter directly synthesizes the pseudo training instances in tgt language based on some parallel sentences, in which the tgt-specific features are retained to the largest extent. as we indicated earlier, in both two paradigms the ud tree features can be unfortunately biased during the transfer, thus leading to the underutilization of ud resource. this work considers a holistic viewpoint, integrating both the two cross-lingual transfer schemes and combining both the src and tgt syntax trees by code mixing. several prior studies have shown that combining the raw src and pseudo tgt (from projection) data for training helps better transfer. it is shown that although the two data are semantically identical, src data still can offer some complementary language-biased features @xcite ). yet we emphasize that different from regular cross-lingual text classification or sequential prediction, xre relies particularly on the syntactic structure features, e.g., ud, and thus needs a more fine-grained approach for src-tgt data ensembling, instead of simply instance stack-ing. thus, we propose merging the src and tgt syntax trees into the code-mixed forests. code mixing has been explored in several different nlp applications @xcite @xcite , where the core idea is creating data piece containing words from different languages simultaneously. for example, @xcite introduce a novel data augmentation method for enhancing the recognition of code-switched sentiment analysis, where they replace the constituent phrases with code-mixed alternatives. @xcite propose generating code-switching data to augment the existing multilingual language models for better zero-shot cross-lingual tasks. while we notice that most of the works focus on the development of code-mixed sequential texts, this work considers the one for structural syntax trees. our work is partially similar to @xcite on the code-mixed ud tree construction. but ours differentiate theirs in that @xcite target better ud parsing itself, while we aim to improve downstream tasks. ## code-mixed ud forest construction to eliminate such discrepancies for unbiased udfeature transfer, we build the code-mixed ud forests, via the following six steps. ## xre with code-mixed ud forest along with the ud forest f src , we also assemble the code-mixed sequential text x src from the src and translated pseudo-tgt sentences (i.e., x src and x tgt ), and the same for the tgt sentences x tgt . an xre system, being trained with src-side annotated data (<x src , f src >, y src ), needs to determine the label y tgt of relational pair e r ↷ s e o given a tgt sentence and ud forest (<x tgt , f @xmath0. the xre system takes as input @xmath1. we use the multilingual language model (mlm) for representing the input code-mixed sentence x: xre aims to determine the semantic relation labels between two given mention entities. for example, given a sentence 'john smith works at google', re should identify that there is a relationship of "works at" between the entities "john smith" and "google". our xre model needs to predict the relation label y. we adopt the biaffine decoder @xcite to make prediction: here both h s and h o are given. ## conclusion and future work universal dependencies (ud) have been served as effective language-consistent syntactic features for cross-lingual relation extraction (xre). in this work, we reveal the intrinsic language discrepancies with respect to the ud structural annotations, which limit the utility of the ud features. we enhance the efficacy of ud features for an unbiased ud-based transfer, by constructing code-mixed ud forests from both the source and target ud trees. experimental results demonstrate that the ud forests effectively debias the syntactic disparity in the ud-based xre transfer, especially for those language pairs with larger gaps. leveraging the syntactic dependency features is a long-standing practice for strengthening the performance of re tasks. in this work, we propose a novel type of syntactic feature, code-mixed ud forests, for cross-lingual relation extraction. we note that this feature can be applied broadly to other cross-lingual structured information extraction tasks that share the same task definition besides re, such as event detection (ed) @xcite and semantic role labeling (srl) @xcite . besides, how to fur-ther increase the utility of the ud forests with a better modeling method is a promising research direction, i.e., filtering the noisy structures in the ud forests.