Representation Alignment and Adversarial Networks for Cross-lingual Dependency Parsing (2024)

Abstract: With the strong representational capabilities of pre-trained language models, dependency parsing in resource-rich languages has seen significant advancements. However, the parsing accuracy drops sharply when the model is transferred to low-resource language due to distribution shifts. To alleviate this issue, we propose a representation alignment and adversarial model to filter out useful knowledge from rich-resource language and ignore useless ones. Our proposed model consists of two components, i.e., an alignment network in the input layer for selecting useful language-specific features and an adversarial network in the encoder layer for augmenting the language-invariant contextualized features. Experiments on the benchmark datasets show that our proposed model outperforms RoBERTa-enhanced strong baseline models by 1.37 LAS and 1.34 UAS. Detailed analysis shows that both alignment and adversarial networks are equally important in alleviating the distribution shifts problem and can complement each other. In addition, the comparative experiments demonstrate that both the alignment and adversarial networks can substantially facilitate extracting and utilizing relevant target language features, thereby increasing the adaptation capability of our proposed model.

## introduction dependency parsing is a fundamental task in natural language processing that aims to identify the grammatical and syntactic relationships between words in a sentence by constructing a dependency tree. as shown in figure 1 , the dependency tree includes a dependency arc (illustrated with red arrows) from the headword "voi (elephant)" to the modifier "thông minh (intelligent)" with the relation label "amod". this indicates that "thông minh † corresponding author. in the past decades, pre-trained language model enhanced dependency parsers have achieved outstanding performances in rich-resource languages @xcite @xcite @xcite . most significantly, @xcite propose a biaffine parser that leverages multi-layer bilstms to encode input sentences and a biaffine operation to compute scores, thus achieving better performance on various languages. then, @xcite develop a self-attentive biaffine parser and further improve the model performance with elmo and bert representations. however, these model performances drop sharply in low-resource languages due to the lack of annotated data @xcite @xcite . as shown in figure 1 , both sentences from vietnamese and chinese have a similar core grammatical structure "subject-predicate-object", but they also have differences in the attributive positions where vietnamese adopts "post-modifier" while chinese is the opposite. hence, how to construct the discrepancy and similarity between different languages becomes the key challenge for crosslingual dependency parsing @xcite @xcite @xcite . a series of previous works have explored feature transfer to improve low-resource parsing. most recently, al @xcite propose a hierarchical transfer learning (htl) approach to exploit a source and an intermediate language to improve the parsing accuracy in low-resource languages. similarly, choudhary and o'riordan (2023) incorporate linguistic typology knowledge as an auxiliary task, further improving the low-resource dependency parsing performances. although transfer learning from rich-resource to low-resource language has shown its promising advantages, how to further emphasize the helpful knowledge and filter out the harmful ones automatically is still an important problem. to address this issue, we propose a novel representation alignment and adversarial networks for cross-lingual dependency parsing. on the one hand, we propose an alignment network on the input layer to select useful language-specific word information. on the other hand, a language-aware adversarial network is applied on the encoder layer to excavate potential language-invariant knowledge. experiments on the benchmark dataset show that our proposed model achieves notable performance improvements, leading to new state-of-the-art results. detailed analysis shows that alignment and adversarial networks are complementary and can complement each other. in-depth comparative experiments demonstrate that both alignment and adversarial networks are equally important for filtering out effective knowledge from the source language. in addition, our codes are released at https://github.com/noteljj/align to facilitate future research. adversarial learning. adversarial learning has become increasingly central in nlp, notably for its role in fortifying model robustness and counteracting data biases @xcite , @xcite and @xcite have demonstrated the efficacy of adversarial examples in bolstering the resilience of nlp models to linguistic variations and malicious attacks. extending this, @xcite and @xcite have successfully integrated adversarial learning into domain adaptation, effectively reducing domainspecific biases. a recent novel approach by @xcite and @xcite involves using adversarial training to mitigate biases in training. additionally, the advent of adversarial data aug-mentation, as investigated by @xcite , has shown promise in diversifying training datasets, further enhancing model robustness. despite these advancements, adversarial learning still confronts challenges in balancing model stability and performance, particularly when dealing with highly complex and nuanced linguistic data, underscoring the need for ongoing research and development in this dynamic area of nlp. feature alignment and transfer. in the field of feature alignment and transfer, existing research can be categorized into deep learning-based methods, instance-based methods, and model-based methods. deep learning-based methods automatically learn feature mapping relationships between source and target domains through neural networks, such as aligning feature distributions in the space through adversarial training @xcite @xcite . instancebased methods select and weight examples from the source domain to have a greater impact in the target domain, like instance selection based on conditional adversarial learning @xcite . model-based methods focus on how to use the source domain's model to assist learning in the target domain, such as progressive neural networks that learn to transfer knowledge across domains @xcite . these methods have their own advantages and can effectively improve the performance of cross-domain learning in different scenarios. ## our approach considering not all rich-source language information is equally important for cross-lingual dependency parsing, we propose the alignment and adversarial networks for effective representation selection. concretely, we first leverage the multi-lingual pre-trained language model xlm-roberta to improve the word representation capability of both source and target languages. then, a representation alignment network is applied on the input layer to emphasize useful language-specific information and ignore the harmful one. next, we exploit an adversarial network on the encoder layer to enhance language-invariant representations. finally, all selected representations are utilized to search for the best dependency tree. figure 2 illustrates the framework of our proposed model, which is organized into three components, i.e., input layer based on the alignment network, encoder layer enhanced with an adversarial network, mlp and biaffine layers. ## conclusion we propose a feature selection approach to emphasize useful representative features and ignore the useless ones, thus improving the performance of cross-lingual dependency parsing. our model not only exploits a representation alignment network that selectively filters advantageous source language representations at the input layer but also utilizes an adversarial network to strengthen contextinvariant features within the encoding layer. experiments on a benchmark dataset illustrate that our proposed model significantly outperforms several strong baseline models. detailed comparative experiments show that both the alignment and adversarial networks can substantially facilitate extracting and utilizing relevant target language features, thereby increasing the adaptation capability of our model. furthermore, in-depth analysis reveals that our model achieves notable improvements in parsing long-distance dependencies and exhibits robustness capabilities, confirming its comprehensive applicative value in cross-lingual settings.