A Fine-Grained Domain Adaption Model for Joint Word Segmentation and POS Tagging (2021)

Abstract: Domain adaption for word segmentation and POS tagging is a challenging problem for Chinese lexical processing. Self-training is one promising solution for it, which struggles to construct a set of high-quality pseudo training instances for the target domain. Previous work usually assumes a universal source-to-target adaption to collect such pseudo corpus, ignoring the different gaps from the target sentences to the source domain. In this work, we start from joint word segmentation and POS tagging, presenting a fine-grained domain adaption method to model the gaps accurately. We measure the gaps by one simple and intuitive metric, and adopt it to develop a pseudo target domain corpus based on fine-grained subdomains incrementally. A novel domain-mixed representation learning model is proposed accordingly to encode the multiple subdomains effectively. The whole process is performed progressively for both corpus construction and model training. Experimental results on a benchmark dataset show that our method can gain significant improvements over a vary of baselines. Extensive analyses are performed to show the advantages of our final domain adaption model as well.

## introduction chinese word segmentation (cws) and part-of-speech (pos) tagging are two fundamental tasks for natural language processing (nlp) in chinese @xcite , serving as backbones for a number of downstream nlp tasks. the joint models of the two tasks can lead to better performance because they are closely-related and the pipeline models suffer from the error propagation problem @xcite @xcite @xcite , which can be alleviated in the joint architecture. currently, joint cws and pos tagging has gained great achievements with bert inputs @xcite . our preliminary results show that the f1-score of joint pos tagging can be close to 95% when the training and test corpus both belong to a standard newswire domain. unfortunately, it is not always the case in real applications. the performance might be degraded dramatically when the source and target domains are highly different. taken the zhuxian (a novel from internet) as an example @xcite , the same model can only obtain an f1-score of 89% for pos tagging according to our results. it is a typical domain adaption problem targeted to joint cws and pos tagging. self-training could be one promising solution @xcite @xcite which can accomplish the goal in a fully-automatic manner without any human intervention @xcite . by using a source model to automatically label a largescale raw corpus of the target domain, and then selecting a set of high-confidence pseudo-labeled instances as additional training data, we can obtain boosted performance on the target domain. the quality of pseudo corpus is the key to success. for the target sentences which are far from the source domain, the generated corpus based on them might be of extremely-low quality @xcite . thus, these sentences should be either filtered, resulting in a biased corpus to the target domain, or be kept with great noises to degrade the overall target performance. in this work, we suggest a fine-grained domain adaption method to alleviate the above problem of self-training. we define a simple and intuitive metric to measure the distance (gap) of a target sentence to the source domain. based on the metric, we create a set of high-quality training corpora incrementally according to the distances of the target sentences to the source domain. figure 1 shows the main idea. the process is conducted by several iterations in a progressive manner, where at each new iteration, we add a small set of high-quality instances which are not as distant from the previous iteration. finally, we arrive at a training corpus covering the target domain of various distances fully. at each iteration, we go only a little further by the distance, thus the quality of the pseudo corpus can be greatly ensured by the previous model. by the fine-grained domain adaption, we can obtain a training corpus of multiple types from different iterations, where each type differs from the other in both quality and input distribution. during the early iterations, the produced instances are possible with higher quality and close to the source domain, while for the later iterations, the quality might be lower and the distance to the source domain is larger. to make full use of the corpus together with the source training set, we present a domain-mixed model for sophisticated representation learning to capture domain-aware and domaininvariant features @xcite @xcite , which is also strengthened progressively by the incremental style of the fine-grained domain adaption. we conduct experiments on a benchmark zhuxian dataset @xcite to show the effectiveness of our method. in detail, the penn chinese treebank version @xcite 6.0 (ctb6) is used as the source corpus, belonging to the newswire domain, while the target zhuxian corpus is from an internet novel. experimental results show that our fine-grained domain adaption is significantly better than previous self-training studies. moreover, we find that our domain-mixed representation learning model suits the fine-grained framework perfectly. we also conduct extensive analyses to understand our model comprehensively. we will release our codes at github.com/jzx555/fgda under apache license 2.0 to help the reproduction. ## joint cws and pos tagging this section describes the basic model of our joint cws and pos tagging. concretely, we regard our joint task as a character-level sequence labeling problem following @xcite . given an input character sequence @xmath0, ..., x n ], the output labels @xmath1, ..., y n ] are concatenations of word boundaries (i.e., bmes) and pos tags for all sentential characters. we exploit an adbert-bilstm-crf model as our basic model, which is very strong in performance and highly parameter efficient. the model includes two parts sequentially: (1) adbert for character representation, (2) bilstm-crf for feature extraction, label inference and training. below, we introduce the ad-bert directly and the bilstm-crf is exactly the same as @xcite which can be referred to in their work for the details. adapter • bert we exploit bert @xcite to derive character representations for a given sentence @xmath2, ..., x n ], as it brings state-of-the-art performances for a range of chinese language processing tasks. in particular, we patch bert with adapters @xcite inside all the included transformer units. by this way, fine-tuning bert parameters is no longer necessary across different tasks, and we only need to tune the adapter parameters. more particularly, we let all adapters across different transformer units use a shared set of parameters to reduce the scale of tunable model parameters of our joint task. here we refer to this method as adbert: ## our method the above joint cws and pos tagging model can perform well on the standard setting when the test domain is similar to the training domain @xcite . however, the performance might be degraded dramatically when the test (i.e., target) domain differs from the training (i.e., source) domain significantly. there have two studies for crossdomain of joint cws and pos tagging @xcite , both of which have exploited self-training due to its effectiveness as well as simplicity for domain adaption. the selftraining aims to produce a set of high-confidence training instances of the target domain which are used to train a target model. here we follow this line of work, presenting a novel fine-grained domain adaption strategy. the fine-grained domain adaption is an extension of the standard self-training, aiming to produce a helpful pseudo training corpus of the target domain. the line of work is essentially orthogonal to the representation learning methods which aim to learn sophisticated (e.g., domain-aware and domain-invariant) features for domain adaption. thus, we also present a novel domain-mixed model based on the basic adbert-bilstm-crf for effective exploration of our fine-grained domain adaption. in the following, we first describe the fine-grained domain adaption method in detail, and then introduce our representation learning model. ## related work cws and pos tagging are closely-related tasks for chinese processing, which could be handled either jointly or in a pipeline way @xcite @xcite @xcite @xcite . the joint models are able to obtain better performances, as they can alleviate the error propagation problem between two tasks @xcite @xcite . recently, neural models lead to state-of-the-arts for joint cws and pos tagging @xcite @xcite . in particular, the bert representations @xcite and the bilstm neural network @xcite have shown impressive results for the joint task @xcite @xcite . in this work, we adopt both bert and bilstm to reach a strong baseline for cross-domain adaption. domain adaptation has been extensively studied in both the machine learning and nlp communities @xcite @xcite @xcite . the typical methods of domain adaption can be divided into two categories mainly. the first category aims to create a set of pseudo training corpora for the target domain, while the second category attempts to learn transferable features from the source domain to the target. self-training is one most representative methods of the first category @xcite @xcite . for the second category, the representation learning of domain-specific and domain-invariant features has received the most attention recently @xcite @xcite @xcite . for the joint cws and pos tagging task, @xcite and @xcite investigate the task under the cross-domain adaption setting, both of which exploit self-training. in particular, @xcite suggest a lexicon-based type-supervised model for further enhancement, and meanwhile publish a benchmark dataset which is publicly available for cross-domain adaption of joint cws and pos tagging. unfortunately, there is no future work for the joint task since then, while the majority of studies focus on the cross-domain of the two individual tasks @xcite @xcite @xcite @xcite . we propose a novel fine-grained domain adaption method with a domain-mixed representation learning model for the joint task. ## conclusion we suggested a novel fine-grained domain adaption method for joint word segmentation and pos tagging. we started from self-training strategy, which exploits various transfers to generate pseudo training instances for the target domain, and argued that the strategy might lead to low-quality of the autolabeled instances when the target sentences are distant from the source domain. to address the problem, we proposed fine-grained domain adaption, regarding the oov number to the source training corpus as the main advancing indicator to construct a higher quality corpus progressively. in addition, we combined our method with another line of representation learning of domain adaption, presenting a domain-mixed model for full exploration of the produced training instances. we evaluated our method on the benchmark zhuxian dataset by using ctb6 as the source domain. the results showed that our method is highly effective, and our final model can achieve significant improvements on the joint task.