A Joint Multiple Criteria Model in Transfer Learning for Cross-domain C hinese Word Segmentation (2020)

Abstract: Word-level information is important in natural language processing (NLP), especially for the Chinese language due to its high linguistic complexity. Chinese word segmentation (CWS) is an essential task for Chinese downstream NLP tasks. Existing methods have already achieved a competitive performance for CWS on large-scale annotated corpora. However, the accuracy of the method will drop dramatically when it handles an unsegmented text with lots of out-of-vocabulary (OOV) words. In addition, there are many different segmentation criteria for addressing different requirements of downstream NLP tasks. Excessive amounts of models with saving different criteria will generate the explosive growth of the total parameters. To this end, we propose a joint multiple criteria model that shares all parameters to integrate different segmentation criteria into one model. Besides, we utilize a transfer learning method to improve the performance of OOV words. Our proposed method is evaluated by designing comprehensive experiments on multiple benchmark datasets (e.g., Bakeoff 2005, Bakeoff 2008 and SIGHAN 2010). Our method achieves the state-of-the-art performances on all datasets. Importantly, our method also shows a competitive practicability and generalization ability for the CWS task.

## introduction in the extensive researches on natural language processing (nlp), most of the tasks are based on word-level methods because word is the smallest linguistic unit in natural languages. it has rich feature information. however, the situation is totally different when dealing with the chinese language. there is not clearly delimiter between chinese words, instead the blank space is regarded as a delimiter in most western languages. different segmentation results may lead to different feature information. thus, chinese word segmentation (cws) is an essential task, which will significantly affect the effectiveness of downstream chinese nlp tasks. recently, the approaches for cws have already achieved a good performance in largescale annotated corpora, as reported by related researches @xcite . most of the effective approaches fall into two major research fields: the statistical machine learning method and the neural network method. the former is mainly based on conditional random fields (crf), which is considered as the most effective statistical machine learning method for cws @xcite . however, the statistical machine learning method always heavily relies on hand-craft features. to minimize the efforts in feature engineering, more and more researches are focus on neural network method @xcite . furthermore, following the rapid development of neural network models, variations on neural network methods for cws have already gained comparable results as the state-of-the-art statistical machine learning techniques @xcite @xcite . nevertheless, there are still two important issues on the cws task. one important issue is that almost all effective methods are limited by large-scale annotated corpora, these methods will lead to a weak generalization ability. the results may decline rapidly when the methods deal with a cross-domain situation. since there are many out-of-vocabulary (oov) words in cross-domain scenarios, and the character feature information is different in another unrelated domain. for example, the chinese character "莫(mo/not to do)" is always trained as a surname of chinese people in most domains, especially when a slice "莫言(mo yan/not to say)" appears in a sentence, it probably should be a chinese person who won the nobel prize for literature. however, the situation is totally different in the chinese novel. in chinese novel domain, "莫(mo/not to do)" always means "not to do something". when the slice "莫言(mo yan/not to say)" appears in a famous chinese novel, the meaning of "莫言(mo yan/not to say)" is "do/does not to say" definitely. the current methods hardly segment it correctly because of the low generalization ability and robustness. the other important issue is that chinese word segmentation criterion is multiple, and most novel methods depend on large-scale corpora. if the large-scale corpora have different criteria, which are shown in in this paper, we propose a joint multiple criteria method for both standard and cross-domain simplified chinese word segmentation. the method utilizes a novel pre-trained (roberta-wwm) model @xcite , which adequately trained a rich chinese character vector embedding. with the richness of the pre-trained model, our method for cws can obtain a robust generalization ability to deal with the cross-domain situation. in order to further improve the performance of the model, we consider improving the amount of training data through the process of transfer learning. we adopt a strategy that integrates several different segmentation criteria into a single straightforward model. the benefit is that we do not need many models to fit multiple segmentation criteria, and the method improves the amount of training data in disguised simultaneously. to sum up, the contributions of this paper are as follows. ## related work since @xcite first formalizes cws task as a sequence labeling problem, many researches depending on supervised machine learning methods have already achieved good performance for cws. @xcite utilized the crf methods for cws, since then crf became the most popular machine learning method for cws task. variations of crf based model achieved good performances for cws @xcite @xcite @xcite . with the development of neural network, more and more researchers made gradual progress with a wide range of neural methods @xcite @xcite @xcite , and the performances on neural methods have already approximated stateof-the-art performances on statistical methods. neural network methods can incorporate the information in the model easily and effectively by means of automatic feature extraction. thus, it is possible to train multiple segmentation criteria into a single model well with neural network methods while it is a challenge on previous statistical methods. @xcite was first to propose a multi-criteria learning method for cws, in which the method adopted shared layers and private layers. however, there is still a gap with independent segmentation criterion method. @xcite improved the performance on the same base bi-lstm (bidirectional long short-term memory network) @xcite model, it adopted a simple and effective method to integrate different segmentation criteria by adding two tags in a sentence @xcite . @xcite changed the base model to transformer @xcite utilized bert @xcite that is based on transformer to extract feature information. both of these two methods achieved the state-of-the-art performances on benchmark datasets. the approaches mentioned above mainly focus on in-domain benchmarks, and there is still much room for improvement using the neural network method @xcite @xcite . most of these methods leverage external resources to alleviate the oov issue. our proposed method is inspired by this thought, and uses rich pre-trained embeddings to relieve the weakness in cross-domain. the method not only solves the multiple segmentation criteria issue with a straightforward architecture, but also solves the cross-domain cws problem. with the help of the pre-trained embedding, the transfer learning method does not need to learn from scratch, and has a robust generalization ability. ## model architecture figure 1 shows our proposed model architecture which is quite brief. we do not pay attention to complicating the neural network. meanwhile, the strategy of integrating criteria is first proposed on @xcite for machine translation transfers multiple segmentation criteria into one model with minimal effort. ## conclusion in this paper, we construct a transfer layer structure that leverages the pre-trained feature information for cws and exploit a transfer tag to boost joint multiple criteria learning. the model could relieve the oov problem for chinese word segmentation and achieves the best performance in comparison with state-of-the-art techniques for both in-domain and out-of-domain chinese word segmentation. extensive experiments on seven in-domain and four cross-domain datasets for chinese word segmentation confirm the superiority of our model over all other advanced methods. in summary, the advantages of our model are twofold. first, the model has a stronger robustness with a straightforward transfer learning method. the performance of our model is better, especially when dealing with high oov rate data. second, our model effectively solves the parameters exploding due to different segmentation criteria. we do not need to design any redundant structures. nevertheless, there is still a gap in a real-word situation. in the future, we will continue studying the efficiency of the neural architecture, and pay attention to improving the speed of both training and testing steps on an ever-increasing dataset. in particular, we will enhance the practicability of chinese word segmentation to improve the effectiveness of other downstream chinese nlp tasks.