Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations (2020)

Abstract: In recent years, parsing performance is dramatically improved on in-domain texts thanks to the rapid progress of deep neural network models. The major challenge for current parsing research is to improve parsing performance on out-of-domain texts that are very different from the in-domain training data when there is only a small-scale out-domain labeled data. To deal with this problem, we propose to improve the contextualized word representations via adversarial learning and fine-tuning BERT processes. Concretely, we apply adversarial learning to three representative semi-supervised domain adaption methods, i.e., direct concatenation (CON), feature augmentation (FA), and domain embedding (DE) with two useful strategies, i.e., fused target-domain word representations and orthogonality constraints, thus enabling to model more pure yet effective domain-specific and domain-invariant representations. Simultaneously, we utilize a large-scale target-domain unlabeled data to fine-tune BERT with only the language model loss, thus obtaining reliable contextualized word representations that benefit for the cross-domain dependency parsing. Experiments on a benchmark dataset show that our proposed adversarial approaches achieve consistent improvement, and fine-tuning BERT further boosts parsing accuracy by a large margin. Our single model achieves the same state-of-the-art performance as the top submitted system in the NLPCC-2019 shared task, which uses ensemble models and BERT.

## introduction dependency parsing aims to capture syntax with a dependency tree and is proven to be helpful for various natural language processing (nlp) tasks, such as semantic role labeling @xcite , natural language generation @xcite , and machine translation @xcite . given an input sentence @xmath0. . . w n , a dependency tree, as depicted in figure 1 , is defined as @xmath1, m, l), 0 ≤ @xmath2, 1 ≤ @xmath3, @xmath4, where (h, m, l) is a dependency from the head word w h to the child word w m with the relation label @xmath5, and w 0 is a pseudo word that points to the root word of the sentence. in recent years, neural network based approaches have achieved remarkable improvement and outperformed the traditional discrete-feature based approaches by a large margin in dependency parsing @xcite @xcite . most remarkably, @xcite propose a simple yet effective deep biaffine parser and achieve the state-of-the-art accuracy on a variety of datasets and languages. however, the domain adaptation problem, i.e., how to improve parsing performance on texts that are very different from the training data, remains a key challenge for the parsing community, especially when trying to apply the parsing technique to real-life web data. taking the examples in figure 1 , we can see that as user-generated texts, the left sentence from the product comment (pc) domain is quite non-canonical and contains a lot of ellipsis phenomena. in contrast, the right one from the balanced corpus (bc) domain is a typical sentence from newswire texts and is much more formal. hence, domain differences can be represented with both sentence and parse tree distribution changes due to new words and phrases, new expression structures, etc. the key for domain adaptation is how to model differences and commonalities between different domains. most previous works focus on unsupervised cross-domain parsing, assuming there is no target-domain labeled data. typical methods include self-training @xcite and co-training @xcite . however, due to the intrinsic difficulty of domain adaptation, progress in this direction is very slow. in the past few years, semi-supervised cross-domain parsing attracts more attention due to the emergence of more labeled data. particularly, @xcite release large-scale labeled and unlabeled datasets, and they find that their proposed domain embedding (de) approach is more effective than the direct concatenation (con) method. the feature augmentation (fa) method, as another typical technique for semi-supervised domain adaptation, is first proposed by @xcite . @xcite successfully apply it to a neural model which leverages multiple bilstms to extract shared and private domain features. to learn the differences and commonalities between source and target domains, the de method uses explicit domain indicators as extra inputs, whereas the fa method employs a shared and two private bilstm encoders for the feature separation. this work proposes to improve the contextualized word representation by adversarial learning and fine-tuning bert, thus further modeling more pure yet effective domain-specific and domain-invariant representations. to alleviate the domain-invariant representations from being contaminated by domainspecific ones, we apply adversarial learning to enhance three typical semi-supervised approaches, i.e., con, fa, and de with two useful strategies, i.e., fused target-domain word representations and orthogonality constraints. at the same time, we utilize a large-scale target-domain unlabeled data to fine-tune bert and obtain more reliable contextualized word representations, leading to a large improvement over using off-the-shelf bert representations. our final single model achieves nearly the same state-ofthe-art performance as the ensemble models with bert of @xcite , which won the first place in the cross-domain parsing shared task recently organized at the international conference on natural language processing and chinese computing (nlpcc-2019). although we focus on semi-supervised domain adaptation for dependency parsing, the techniques and findings may be applicable to domain adaptation for other nlp tasks. all codes are released publicly available for the research purpose foot_0 . ## base model in this work, we select the state-of-the-art biaffine parser as our strong baseline model. as shown in the left part of figure 2 , the parser mainly contains four components: input layer, bilstm encoder, mlp layer, and biaffine layer. ## approaches in this work, we propose to improve contextualized word representations by adversarial learning and fine-tuning bert processes to boost the performance of cross-domain dependency parsing. concretely, we apply adversarial learning to three typical semi-supervised approaches with two useful strategies, thus obtaining more pure word representations. simultaneously, we propose to fine-tune bert with all target-domain unlabeled data to obtain more reliable word representations. ## experiments datasets. we use the chinese multi-domain dependency parsing datasets released at the nlpcc-2019 shared task foot_2 , containing four domains: one source domain which is a balanced corpus (bc) from newswire, three target domains which are the product comments (pc) data from taobao, the product blog (pb) data from taobao headline, and a web fiction data named "zhuxian" (zx). the detailed data statistics are shown in evaluation. we use unlabeled attachment score (uas) and labeled attachment score (las) to evaluate the dependency parsing accuracy. each parser is trained for at most 1000 iterations, and the performance is evaluated on the dev data after each iteration for model selection. we stop the training if the peak performance does not increase in 100 consecutive iterations. hyper-parameters. we follow the hyper-parameter settings of @xcite , such as learning rate and dropout ratios. the loss weights α and β are set to 0.001. the grl hyper-parameter λ is 10 -5 . for pre-trained word embeddings, we train word2vec embeddings on chinese gigaword third edition @xcite , consisting of about 1.2 million sentences. ## conclusions this work successfully exploits adversarial learning and fine-tuning bert to model pure yet effective word representations that benefit for the cross-domain dependency parsing. we have demonstrated the effectiveness of adversarial learning and fine-tuning bert by applying them to three representative semisupervised approaches. experimental results show that our proposed adversarial approaches achieve consistent improvement, and fine-tuning bert further boosts parsing accuracy by a large margin. the detailed comparison experiments demonstrate that both the fused target-domain word representation and orthogonality loss are useful for adversarial models to alleviate the domain-invariant representations from being contaminated by domain-specific ones. the analysis on the utilization of bert indicates that the fine-tuning bert with the target-domain unlabeled data encourages bert to learn more reliable contextualized word representations, leading to a large improvement over using off-the-shelf bert on both non-adversarial and adversarial models.