Unsupervised Domain Adaptation Method with Semantic-Structural Alignment for Dependency Parsing (2021)

Abstract: Unsupervised cross-domain dependency parsing is to accomplish domain adaptation for dependency parsing without using labeled data in target domain. Existing methods are often of the pseudo-annotation type, which generates data through self-annotation of the base model and performing iterative training. However, these methods fail to consider the change of model structure for domain adaptation. In addition, the structural information contained in the text cannot be fully exploited. To remedy these drawbacks, we propose a Semantics-Structure Adaptative Dependency Parser (SSADP), which accomplishes unsupervised cross-domain dependency parsing without relying on pseudo-annotation or data selection. In particular, we design two feature extractors to extract semantic and structural features respectively. For each type of features, a corresponding feature adaptation method is utilized to achieve domain adaptation to align the domain distribution, which effectively enhances the unsupervised cross-domain transfer capability of the model. We validate the effectiveness of our model by conducting experiments on the CODT1 and CTB9 respectively, and the results demonstrate that our model can achieve consistent performance improvement. Besides, we verify the structure transfer ability of the proposed model by introducing Weisfeiler-Lehman Test.

## introduction dependency parsing is to extract the dependency structure of a sentence that shows its grammatical structure and the relationships between "head" words and associated "dependent" words. dependency parsing can provide the syntactic structure information for sentence, which can be used to enhance other natural language processing task such as named entity recognition @xcite and sentence semantic similarity @xcite ., ⇤ corresponding authors ssadp out train train proposed method data-selection method selector parser out parser out train predict train filter pseudo-annotation method target domain source domain x x x y y y x x x x x x 𝒚 y y x x y y x x x y y y x x x x x y y x x x y y y x x x y y y x x x x x x y y y unlabeled set unlabeled set unlabeled set train set train set train set pseudo train set predict output predict output predict output sub train set ' ' ' ' ' ' ' ' ' ' ' figure 1: difference between previous methods and proposed method. the proposed ssadp method is more advantageous in that knowledge transfer and parsing are conducted in a joint manner. there are mainly two ways to achieve unsupervised cdp: (1) pseudo-annotation iterative methods, such as self-training @xcite , cotraining @xcite and tri-training @xcite , which mainly rely on a model trained on the source domain to generate credible pseudolabeled data in the target domain; (2) data selection methods, which mainly focus on selecting a subdataset in the source domain that is similar to the target domain for training @xcite . these approaches focus on designing training or data filtering strategies. however, feature-based transfer is not considered being taken into the previous two ways, which means the transfer is separated with the parsing and the hidden feature produced by parser is not used for domain adaptation. besides, previous methods only use the unlabeled data for filtering, and more information (such as feature statistics) contained in such data can not be fully exploited. to remedy this drawback, we propose a semantic-structural adaptative dependency parser (ssadp) to accomplish transfer and parsing simultaneously for unsupervised cdp without using pseudo-annotation and data-selection strategy. the proposed method is graph-based, which means that the words are treated as nodes and sentence as a graph. the cross-domain transfer is mainly achieved by: 1) using biaffine @xcite to extract semantic feature and query-key cnn (qkcnn) to extract structural features from the domain data respectively to enhance the ability in describing the domain of model; 2) integrating metric-learning method local maximum measure discrepancy (lmmd) @xcite and graph attention network (gat) @xcite to align the domains according to characteristics of the extracted semantic and structural features. effectiveness of the proposed ssadp is demonstrated by the extensive experiments on the quite new chinese open dependency treebank 1.0 (codt1) @xcite and chinese tree bank 9.0 (ctb9) @xcite datasets, where we propose a data division standard for ctb9. we introduce the weisfeiler-lehman test to verify that the proposed ssadp has the ability of transfer structural information. our main contributions of this paper are summarized as follows: ## methods the proposed model is mainly composed of four parts: semantic feature extractor, structural feature extractor, semantic feature transfer module and structural feature transfer module. semantic feature extractor and structural feature extractor are designed to extract domain features. semantic feature transfer module and structural transfer module are used to align domain feature. the architecture of the proposed ssadp is shown in the figure 2 , and each of the four modules is depicted as follows. ## conclusion we propose a novel parsing model termed ssadp that can achieve unsupervised cross-domain dependency parsing by extracting semantic and structural features and performing domain alignment without using pseudo annotation and data selection. the experimental results on the codt1 and ctb9 datasets demonstrate the effectiveness of our model. moreover, we adopted weisferiler-lehman test to verify the structural transfer ability of the proposed ssadp and other baselines. finally, by extending ssadp with pseudo-annotation method, we show that proposed ssadp can be combined with the