APGN : Adversarial and Parameter Generation Networks for Multi-Source Cross-Domain Dependency Parsing (2021)

Abstract: Thanks to the strong representation learning capability of deep learning, especially pre-training techniques with language model loss, dependency parsing has achieved great performance boost in the in-domain scenario with abundant labeled training data for target domains. However, the parsing community has to face the more realistic setting where the parsing performance drops drastically when labeled data only exists for several fixed out-domains. In this work, we propose a novel model for multi-source cross-domain dependency parsing. The model consists of two components, i.e., a parameter generation network for distinguishing domain-specific features, and an adversarial network for learning domain-invariant representations. Experiments on a recently released NLPCC-2019 dataset for multi-domain dependency parsing show that our model can consistently improve cross-domain parsing performance by about 2 points in averaged labeled attachment accuracy (LAS) over strong BERT-enhanced baselines. Detailed analysis is conducted to gain more insights on contributions of the two components.

## introduction dependency parsing aims to derive syntactic and semantic tree structures over input words @xcite . given an input sentence @xmath0. . . w n , a dependency tree, as depicted in figure 1 , is defined as @xmath1, m, l), 0 ≤ @xmath2, 1 ≤ @xmath3, @xmath4, where (h, m, l) is a dependency from the head word w h to the child word w m with the relation label @xmath5. recently, supervised neural dependency parsing models have achieved great success, leading to impressive performance @xcite @xcite . remarkably, the biaffine parsing model can obtain a uas of 96.67 and a las of 95.03 on standard penn treebank benchmark for the english language. in order to obtain competitive performance, supervised dependency parsing models rely on a sufficient amount of training data, which is inevitably dominated to several fixed domains. when the test data is sourced from similar domains, good performance could be achieved. however, the performance could be decreased significantly when the test data is from a different domain which has a large gap between the training domains. thus domain adaptation for dependency parsing has been concerned by a number of studies @xcite @xcite @xcite . these works mostly focus on single-source cross-domain dependency parsing, assuming the training data is from a single source domain @xcite . in fact, multi-source cross-domain dependency parsing is a more practical setting, considering that several dependency parsing corpora from different domains have been developed @xcite . intuitively, an effective exploration of all these corpora can give better performance for the target domain compared with the single-source domain adaptation. separating domain-invariant and domainspecific features is one popular way for domain adaptation to distinguish the similarity and discrepancy of different domains @xcite @xcite . domain-invariant features indicate the shared feature space across domains, which have been widely-adopted as knowledge transferring. domain-specific features imply the differences between domains, which could be helpful if the domain gaps could be accurately measured and effectively modeled. the learning of domain invariant and specific features are actually complementary because of mutual exclusivity, especially for single-source domain adaptation. although single-source and multi-source settings are easily separated domain-invariant and domain-specific features via independent bilstms, the in-depth relevance of domain-specific representations becomes more complicated with the increasing of source domains. hence, how to construct the relationship between different domain-specific features after a simple feature separation becomes more challenge for multi-source dependency parsing. in this work, we for the first time apply adversarial and parameter generation networks (apgn) to multi-source cross-domain dependency parsing for extracting domain-invariant and domain-specific features. experiments on a benchmark dataset show that our proposed model can boost the parsing performance significantly, leading to averaged las improvements by 2 points over strong bertenhanced baselines. first, explorations on different unlabeled data sizes reveal that unlabeled data is an useful resource and proper utilization of unlabeled data further improves our model performance by a large margin. then, we conduct in-depth analysis to gain crucial insights on the effect of adversarial and parameter generation networks, finding the two components are complementary and both have the capability of modeling short-or long-range dependencies. finally, detailed comparative experiments on alternative domain representation strategies show that our designed distributed domain representation can accurately measure domain gaps and extract more reliable domain knowledge that benefits the dependency parsing task. we will release our code at https://github.com/ suda-yingli/emnlp2021-apgn for facilitating future researches. ## baseline model in this work, we adopt the state-of-the-art deep biaffine parser @xcite as our baseline model. figure 2 shows the framework of the parser, which mainly contains four components, i.e., input layer, encoder layer, mlp layer, and biaffine layer. input layer. the input layer maps each word w i into a dense vector representation x i . first, we apply a bilstm to encode the constituent characters of each word w i into its character representation rep char i . then, we concatenate rep char i with emb word i as the input vector x i . encoder layer. following dozat and manning (2017), we employ a three-layer bilstm to sequentially encode the inputs x 0 . . . x n and generate context-aware word representations h 0 . . . h n . we omit the detailed computation of the bilstm due to space limitation. mlp layer. the mlp layer uses two independent mlps to get lower-dimensional vectors of each position 0 ≤ @xmath6. parsing loss. the parsing loss for each position i is computed as: ## proposed apgn approach the goal of multi-source cross-domain dependency parsing is to train a parser that generates well to the target domain with labeled training data from multiple source domains and unlabeled data from the target domain. the most straightforward approach is training a parser with the concatenation of all source-domain training data. this can extract common features across different domains but fail to capture domain-specific knowledge. to address this issue, we propose an apgn approach for modeling the discrepancy and commonality between different domains simultaneously. as shown in figure 3 , our apgn model mainly contains two components, i.e., a pgn for distinguishing domainspecific features, and an adversarial network for learning domain-invariant representations. in this section, we first give a detailed illustration of the parameter generation network which takes distributed domain embedding as input to alleviate potential domain conflicts caused by the fixed one. then, we introduce the adversarial network which encourages the bilstm to extract more pure shared information by fooling the domain classifier. finally, we propose a new strategy for our model training to make full use of all labeled and unlabeled data. @xcite first propose pgn to generate bil-stm parameters based on fixed task and domain embeddings for ner domain adaptation, finding that the pgn can effectively extract domain differences. however, the vanilla pgn requires crossdomain language model task as a bridge to help fixed domain embeddings training. considering the development of pre-training techniques with language model loss and computational complexity, we first remove the language model from the pgn component and use pre-trained bert to enhance model performance in the final experiments. intuitively, each input word has its unique domain distributions, initializing these words with the same fixed domain embedding may lead to potential domain conflicts. we then improve the pgn via replacing the fixed domain embedding with distributed one to more accurately integrate multidomain information. as shown in the right part of figure 3 , our pgn takes distributed representations as inputs and dynamically generates the domain-related pgn-bilstm parameters. ## conclusion this work for the first time apply the apgn approach to multi-source cross-domain dependency parsing, obtaining better performance than multiple baselines, even when all models are enhanced with bert representations. the ablation study reveals that both adversarial and parameter generation networks are equally important and complementary in capturing domain-related features, which motivates us to make a deep analysis to gain crucial insights on the effectiveness of the two components. based on the in-depth error analysis, we find that in spite of local divergences, domain-invariant and domain-specific features generated by adversarial and parameter generation networks actually both have the power of modeling short-or long-range dependencies and can certainly benefit from each other. furthermore, detailed comparative experiments demonstrate that the distributed domain representation is extremely useful to reduce domain conflicts and accurately measure the domain similarity, thus extracting more reliable domain-specific features to boost the parsing performance.