Semi-supervised Domain Adaptation for Dependency Parsing with Dynamic Matching Network (2022)

Abstract: Supervised parsing models have achieved impressive results on in-domain texts. However, their performances drop drastically on out-of-domain texts due to the data distribution shift. The shared-private model has shown its promising advantages for alleviating this problem via feature separation, whereas prior works pay more attention to enhance shared features but neglect the in-depth relevance of specific ones. To address this issue, we for the first time apply a dynamic matching network on the shared-private model for semi-supervised cross-domain dependency parsing. Meanwhile, considering the scarcity of target-domain labeled data, we leverage unlabeled data from two aspects, i.e., designing a new training strategy to improve the capability of the dynamic matching network and fine-tuning BERT to obtain domain-related contextualized representations. Experiments on benchmark datasets show that our proposed model consistently outperforms various baselines, leading to new state-of-the-art results on all domains. Detailed analysis on different matching strategies demonstrates that it is essential to learn suitable matching weights to emphasize useful features and ignore useless or even harmful ones. Besides, our proposed model can be directly extended to multi-source domain adaptation and achieves best performances among various baselines, further verifying the effectiveness and robustness.

## introduction dependency parsing aims to capture syntactic and semantic information over input words via a dependency tree. as depicted in figure 1 , given an input sentence @xmath1. . . w n , a dependency tree is defined as @xmath2, m, l), 0 ≤ @xmath3, 1 ≤ @xmath4, @xmath5, where (h, m, l) is a dependency from the head word w h to the modifier word w m with the relation label @xmath6. recently, supervised neural models have achieved significant improvements in dependency parsing @xcite manning, @xmath0 very cute , very lovely ! root adv punc sasubj adv punc figure 1: an example of a dependency tree which is from the target-domain product comment (pc) data. in order to obtain better performance, supervised parsing models rely on sufficient in-domain training data. however, the parsing accuracy degrades significantly when the training data is from out-ofdomain that has a large gap between the in-domain data.the main reason can be attributed to different feature distributions between source and target domains. thus modeling the relevance of these distributions becomes the key challenge for crossdomain dependency parsing. in the past few years, semi-supervised dependency parsing has attracted more attention with the surge of labeled web data that are user-generated non-canonical texts @xcite @xcite . as shown in figure 2 , these approaches for modeling the similarity and discrepancy among different domains can be classified into three categories. the fully-shared model treats source and target domains equally and shares all model parameters, which may extract domain-invariant features but fail to capture domain-specific ones. in contrast, the fully-private model exploits completely independent encoders for each domain, which can better capture domain-specific features but ignore domain-invariant ones. to combine the advantages of fully-shared and fully-private models, the sharedprivate model naturally separates domain-invariant and domain-specific features via shared and private encoders @xcite . however, this model still has two issues, i.e., neglecting the in-depth relevance of specific ones and failing to utilize unlabeled data effectively. for the first issue, we investigate feature transfer approaches that encourage the target feature space to learn useful knowledge from source domain @xcite @xcite . particularly, @xcite successfully use meta-learning to learn transfer weights between heterogeneous architectures and tasks. motivated by this work, we propose a dynamic matching network based on the shared-private model for semisupervised dependency parsing. concretely, our model automatically generates matching weights to emphasize useful information and filter useless or even harmful features, thus further improving the power of target feature space. for the second issue, considering that manually annotating samples for a new domain is timeconsuming and expensive, we endeavour to effectively utilize target-domain unlabeled data. we design a new training strategy to use unlabeled data to enhance the power of matching network, thus modeling more effective specific features for the target domain. meanwhile, we fine-tune bert model with language model loss to obtain more reliable domain-related contextualized representations. experiments on benchmark datasets show that our proposed model outperforms the top submitted system in the nlpcc-2019 shared task @xcite , leading to new state-of-the-art results on all domains. in addition, detailed analysis on differ-ent matching settings reveals insights on the effect of intermediate source features. the extension on multi-source domain adaptation further verifies the effectiveness and robustness of our model. the code is released at https://github.com/ suda-yingli/acl2022-match to facilitate future research. ## biaffine parser in this work, we use the simple yet effective bi-affine parser @xcite as our basic model, which consists of four components, i.e., input layer, bilstm encoder, mlps (multilayer perceptron), and biaffines. ## our approach semi-supervised dependency parsing aims at learning a parser that generalizes well to the target domain. although supervised parser has achieved good results on in-domain data, the parsing performance drops dramatically when the training data is mainly from the out-of-domain. the shared-private model has been proven effective for alleviating this problem. however, the model ignores the in-depth relevance of specific ones and fails to directly use unlabeled data for model training. to address these problems, we for the first time apply a dynamic matching network on the shared-private model to learn appropriate matching weights automatically via mimicking well-trained source features. as shown in figure 3 , our model mainly contains two components, i.e., a shared-private schema for feature separation and a dynamic matching network for capturing the relevance of domain-specific features. in addition, we propose a new strategy for our model training to make full use of all labeled and unlabeled data. ## related work domain adaptation generally falls into two categories: semi-supervised where large-scale labeled data for the source and small-scale labeled data for the target are available and unsupervised where only the labeled data for the source domain is given. ## conclusion this work proposes a feature matching sharedprivate model for semi-supervised dependency parsing. meanwhile, we utilize unlabeled data to enhance the power of feature matching network and the bert representations. our proposed approach achieves consistent improvements among various baseline models, leading to new state-ofthe-art results on all domains. the detailed analysis shows that compared with manual matching setting, the automatically learned matching weights by our designed dynamic matching network can improve the parsing accuracy. furthermore, our proposed model can be directly extended to multi-source domain adaptation and achieves the best performance among various baselines, further demonstrating the effectiveness and robustness of our proposed method.