Unsupervised C hinese Word Segmentation with BERT Oriented Probing and Transformation (2022)

Abstract: Word Segmentation is a fundamental step for understanding Chinese language. Previous neural approaches for unsupervised Chinese Word Segmentation (CWS) only exploits shallow semantic information, which can miss important context. Large scale Pre-trained language models (PLM) have achieved great success in many areas because of its ability to capture the deep contextual semantic relation. In this paper, we propose to take advantage of the deep semantic information embedded in PLM (e.g., BERT) with a self-training manner, which iteratively probes and transforms the semantic information in PLM into explicit word segmentation ability. Extensive experiment results show that our proposed approach achieves state-of-the-art F1 score on two CWS benchmark datasets.

## introduction there exist many low resource fields and languages where labeled word segmentation is inaccessible, which makes unsupervised word segmentation desirable. previous unsupervised word segmentation methods mainly apply statistical models to either evaluate the quality of possible segmented sequence with discriminative models (e.g., mutual information @xcite ) or estimate the generative probabilities with generative models (e.g., hidden markov model @xcite ). however, these statistical methods can only make use of the limited contextual information, thus yielding less competitive performance. with the thrive of neural networks, researchers have applied neural models for unsupervised word segmentation. @xcite propose a segmental language model (slm) to estimate the generative probability with recurrent networks. although slm can exploit more contextual information compared with statistical models, it is still weak in modeling deep semantic information, limited by its model capacity and training data scale. pre-trained language models trained on large scale data have shown superior ability to model contextual information, and achieve great success in various tasks @xcite @xcite . inspired by the attempt for interpreting @xcite , we propose to take advantage of the semantic representation ability of bert to evaluate the closeness between characters in a probing manner. to be more specific, we assume that the difference between masking one character and masking several adjacent characters as a whole reveals the closeness between that character and the adjacent ones. although this probing-based method can take advantage of the large amount of knowledge embedded in bert, it only implicitly exploits the representation ability of bert. to transfer the implicit knowledge into explicit segmentation boundary, we propose to apply a self-training method that transforms the segmentation decision from generative methods with high confidence into traditional "bi" sequence labeling system, which is then treated as the supervision signals for a discriminative model. to combine the advantage of both generative and discriminative models, we propose to iteratively train the discriminative model and generative model under the supervision signal from their counterparts. to select the model with the best performance in the unsupervised setting, we propose an evaluation module that evaluates the quality of the word boundaries with masked prediction accuracy based on the assumption that the closer two characters are, the bigger loss masking one adjacent character would bring. we conduct experiments on two chinese word segmentation benchmark datasets in an unsuper-vised manner. experiment results show that our method can outperform the strong baseline models and achieve state-of-the-art results in unsupervised cws. extensive analysis shows the effectiveness of the proposed modules. we conclude our contributions as follows: ## related work previous unsupervised word segmentation methods can be roughly classified as generative and discriminative two ways. generative models focus on finding the segmented sequence with the highest posterior probability. hierarchical dirichlet process (hdp) model @xcite , nested pit-manyor process (npy) @xcite , hidden markov model (hmm) @xcite and slm (sun and deng, 2018) are all different ways to estimate the generative probabilities for segmented sequences. on the other hand, discriminative models focus on designing a measure to evaluate the segmented sequences. mutual information (mi) @xcite , normalized variation of branching entropy (nvbe) @xcite and esa @xcite apply co-occurrence based measurement to evaluate the segmented sequences. ## approach in this section, we describe our bert oriented probing and transformation based unsupervised word segmentation approach. our model mainly consists of three parts, a generative module that suggests the plausible word boundaries by probing bert, a discriminative module that transforms the implicit boundary information into explicit sequence labels, and an evaluation module that estimates the performance of the model in an unsupervised manner. algorithm 1 unsupervised word segmentation procedure require: generative module g, discriminative module d, evaluation module e, sequences to be segmented x. because our method works in an unsupervised manner, we propose to get the original word boundary information by probing bert, which reveals the word boundaries by measuring the distance between masking a span and masking a token using the generative module. this distance reflects the closeness between the masked token and the masked span (separately). then the discriminative module transforms the word boundaries suggested by the generative module into explicit segmentation labels to enable the self-training process. to combine the advantages of both generative and discriminative modules, two modules are iteratively trained with the word boundaries suggested by the updated counterpart with high confidence. to decide when to stop this iterative self-training procedure, an evaluation module is proposed to evaluate the segmented sequence, which early stops the iterative process with the model parameters that yields the best performance. ## experiment in this section, we show the results and analysis on two cws benchmark datasets, pku and msr for a fair comparison, which are provided by the second segmentation bake-off (sighan 2005) @xcite . there are 104k and 107k words in the test set of pku and msr datasets respectively. ## conclusion in this paper, we propose a bert oriented probing and transformation method for unsupervised word segmentation. our proposed model reveals the semantic information encoded in plm into word boundary information by probing and transforming the token representations into explicit sequence labels. experiment results on two benchmark cws datasets show that our method achieves state-ofthe-art f1 score. the proposed method works in an unsupervised manner, which can help understand low resource and endangered languages and thus protecting language diversity.