Segment, Mask, and Predict: Augmenting C hinese Word Segmentation with Self-Supervision (2021)

Abstract: Recent state-of-the-art (SOTA) effective neural network methods and fine-tuning methods based on pre-trained models (PTM) have been used in Chinese word segmentation (CWS), and they achieve great results. However, previous works focus on training the models with the fixed corpus at every iteration. The intermediate generated information is also valuable. Besides, the robustness of the previous neural methods is limited by the large-scale annotated data. There are a few noises in the annotated corpus. Limited efforts have been made by previous studies to deal with such problems. In this work, we propose a self-supervised CWS approach with a straightforward and effective architecture. First, we train a word segmentation model and use it to generate the segmentation results. Then, we use a revised masked language model (MLM) to evaluate the quality of the segmentation results based on the predictions of the MLM. Finally, we leverage the evaluations to aid the training of the segmenter by improved minimum risk training. Experimental results show that our approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness.

## introduction in extensive natural language processing (nlp) scenarios, most of the tasks are based on word-level methods. when we deal with the chinese language, there is no specific boundary between two chinese words. the situation is different in western languages. for instance, there is a space between two words. thus, chinese word segmentation (cws) is considered an essential task, which will accurately represent semantic information of chinese nlp tasks. besides, the length of the sentence is shortened by word segmentation. the shorter length of a sentence is effective for the deep learning method in some cases. recently, good performance for cws has already been achieved in large-scale annotated corpora as reported by related research @xcite . most methods start with data-driven to improve the performance for cws. for instance, some neural methods try to incorporate external resources to achieve good performance for in-domain and cross-domain cws @xcite . the previous methods fall into two categories: (1) the statistical machine learning methods and (2) neural network methods. in statistical machine learning meth-ods, conditional random fields (crf) is the most effective model for the sequence labeling problem @xcite . however, the performance of the crf model depends on the quality of the hand-crafted features. to minimize the effects of different hand-crafted features, neural network methods @xcite @xcite have been widely used. on the other hand, these supervised learning methods are usually limited by the training data. recent sota approaches utilize the pre-trained models (ptm) to improve the quality of cws @xcite . however, the cws methods based on the ptm only utilize the large-scale annotated data to finetune the parameters. it omits much-generated information of the training step. besides, the annotated data has some incorrect labels due to lexical diversity in chinese, therefore the robustness of methods is quite important for the cws. in this work, we propose a self-supervised cws approach to enhance the performance of cws model. in addition, we also investigate on the crossdomain and low-quality datasets to analyze the robustness of cws models. as depicted in figure 1 , our model consists of two parts: segmenter and predictor. we leverage the transformer encoder as a word segmenter. we exploit the revised masked language model (mlm) as a predictor to improve the segmentation model. we generate masked sequences with respect to the segmentation results. then we exploit mlm to predict the masked part and evaluate the quality of the segmentation based on the quality of the predictions. we leverage an improved version of minimum risk training (mrt) @xcite to enhance the segmentation. our contributions are as follows: ## related work chinese word segmentation (cws) has been studied for several years as an essential chinese nlp task. cws methods are divided into two streams of approaches: word-based methods and characterbased methods. since @xcite first formalizes the cws task as a sequence labeling problem, almost all methods transfer the cws results into the sequence labels. as a sequence labeling task, the crf-based model can achieve a competitive performance with multiple features @xcite @xcite . however, the effect of each method is determined by the quality of manual features. to reduce the influence of feature engineering, neural cws methods have been studied and further progress has been made @xcite @xcite @xcite . neural methods gradually replaces traditional machine learning methods. @xcite propose the basic lstm model that is the same with @xcite . but the former study could achieve sota performance through tuning the hyper-parameters. some studies leverage the rich pre-trained embeddings to improve the performance for neural cws methods @xcite @xcite . to alleviate the issue of oov words for cws, some researches have been studied for cross-domain cws. @xcite incorporate the domain dictionary into the neural network, and @xcite utilize the unlabeled data to enhance the ability to recognize oov words. with the development of pre-trained language models (plm) @xcite , cws methods also make further progress. previous sota methods effectively achieve good performance for cws @xcite @xcite , and they take the advantages of plms rather than the pure models themselves. the redundant components get slight improvements that are not as much as the plms learning paradigm. ## method the overall process of our method is shown in algorithm 1: first, we train a word segmentation model and use it to generate segmentation results. then, according to the segmentation results, the masked sentence is generated based on certain strategies, and an mlm is trained with the masked sentence. afterward, we mask the sentences in the training algorithm 1 self-supervised word segmentation o) ) to segment d and achieve segmented sequence d (t) . ## conclusion and future work in this work, we propose a self-supervised method for cws. we first generate masked sequences based on the segmentation results and then use revised mlm to evaluate the quality of segmentation and enhance the segmentation by improved mrt. experimental results show that our approach outperforms previous methods on both popular and cross-domain cws datasets, and has better robustness on noised-labeled data. in the future, we can also extend our work to tasks of morphological word segmentation (e.g., morphological analysis).