Improved Unsupervised C hinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer (2023)

Abstract: Unsupervised Chinese word segmentation (UCWS) has made progress by incorporating linguistic knowledge from pre-trained language models using parameter-free probing techniques. However, such approaches suffer from increased training time due to the need for multiple inferences using a pre-trained language model to perform word segmentation. This work introduces a novel way to enhance UCWS performance while maintaining training efficiency. Our proposed method integrates the segmentation signal from the unsupervised segmental language model to the pre-trained BERT classifier under a pseudo-labeling framework. Experimental results demonstrate that our approach achieves state-of-the-art performance on the eight UCWS tasks while considerably reducing the training time compared to previous approaches.

## introduction word segmentation is critical in natural language processing (nlp) tasks. different from english, chinese text does not show explicit delimiters between words (e.g., whitespaces), making chinese word segmentation (cws) a challenging task. traditional unsupervised chinese word segmentation (ucws) approaches include rule-based and statistical methods @xcite @xcite . in recent years, neural approaches based on word embeddings and recurrent neural networks have been studied for ucws. @xcite proposed segmental language model (slm) with the pretrained word embeddings @xcite and the lstm-based language model @xcite . by calculating the probabilities between words in a fixed-length segmentation size, slm performs better than the traditional approaches on the ucws tasks. * the first two authors contributed equally. recently, as large-scale pre-trained language models have become mainstream solutions for several nlp tasks, @xcite successfully applied bert @xcite to ucws and demonstrated state-of-the-art (sota) performance by incorporating perturb masking @xcite and the self-training loops into the bert classifier. however, perturb masking is computationally inefficient due to the requirement of performing at least twice bert forward passes for each token in left-to-right order @xcite . in other words, for a sequence with a length of n , the additional training time complexity using perturb masking @xcite will become 2n , which significantly increases the training time of fine-tuning bert for ucws. this paper introduces a simple unsupervised training framework to leverage the pre-trained bert @xcite for ucws efficiently. following @xcite , we view the cws task as a sequence tagging problem. to make the bert model learn how to segment words with its implicit pre-trained knowledge, we propose a pseudo-labeling approach to fine-tune bert with the pseudo-labels generated by an unsupervised segment model. our experiments demonstrate that the proposed method brings substantial performance gains compared to the previous studies @xcite @xcite . in addition, the proposed method provides an 80% decrease in training time than the existing sota method @xcite , which also utilized the pre-trained language model for ucws. ## method there are two modules in the proposed framework: the segment model and the classifier, which we show in figure 1 . the segment model produces the text segmentation results, which serve as pseudolabels. the classifier, which is a bert-based classification model, then uses these labels to learn how to separate the words in a sequence. in other words, the segment model acts as a teacher for the classifier. the following sections first introduce the details of the two modules and then describe the training process in our proposed framework. ## analysis ## conclusion this work presents an improved training framework for unsupervised chinese word segmentation (ucws). our framework leverages the pseudolabeling approach to bridge the two-stage training of the lstm-based segment model and the pretrained bert-chinese classifier. the experiments show that the proposed framework outperforms the previous ucws methods. in addition, without using perturb masking @xcite and self-training @xcite , our framework significantly reduces training time compared with the sota approach @xcite . our code is available at https://github.com/ikmlab/ improveducws-knowledgetransfer .