u C hecker: Masked Pretrained Language Models as Unsupervised C hinese Spelling Checkers (2022)

Abstract: The task of Chinese Spelling Check (CSC) is aiming to detect and correct spelling errors that can be found in the text. While manually annotating a high-quality dataset is expensive and time-consuming, thus the scale of the training dataset is usually very small (e.g., SIGHAN15 only contains 2339 samples for training), therefore supervised-learning based models usually suffer the data sparsity limitation and over-fitting issue, especially in the era of big language models. In this paper, we are dedicated to investigating the unsupervised paradigm to address the CSC problem and we propose a framework named uChecker to conduct unsupervised spelling error detection and correction. Masked pretrained language models such as BERT are introduced as the backbone model considering their powerful language diagnosis capability. Benefiting from the various and flexible MASKing operations, we propose a Confusionset-guided masking strategy to fine-train the masked language model to further improve the performance of unsupervised detection and correction. Experimental results on standard datasets demonstrate the effectiveness of our proposed model uChecker in terms of character-level and sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of spelling error detection and correction respectively.

## introduction chinese spelling check (csc) is a crucial and essential task in the area of natural language processing. it aims to detect and correct spelling errors in the chinese text @xcite . generally, sequence translation @xcite @xcite @xcite and sequence tagging @xcite @xcite are the two most typical technical paradigms to tackle the problem. benefiting from the development of pretraining techniques, many researchers fine-tune the pretrained language models such as bert @xcite on the task of csc and obtain encouraging performance @xcite @xcite @xcite @xcite @xcite . meanwhile, it should be emphasized that almost all of the above mentioned models are trained via the supervised learning paradigm. however, during the investigating stage about those newly typical state-of-the-art models, we observe some spiny and serious phenomenons: (1) occasionally those models may generate some special over-correcting results. as shown in figure 1 , operation path i is the regular spelling error detection and correction path, while operation path ii is also observable in the inference stage where the models can detect the errors correctly but rectify them using some other error tokens in the correction stage. (2) the spelling error detection and correction performance will drop dramatically when those models did not see the spelling error cases in the training dataset or the text are from different genres and domains. this issue tells us that the generalization capability of those models are limited and need to be enhanced. then what are the causes of these phenomenons? since some of the models are already strong enough (which are constructed based on big pretrained models), then we shift our eyes to the data perspective. in real practical scenarios, natural human-labeled spelling error corpus are difficult and expensive to obtain. although some works such as @xcite employ ocr and asr based techniques to automatically synthetic the paired samples by replacing the correct tokens using visually or phonologically similar characters, obviously, the constructed data is unrealistic and far from the real and objective scenarios. therefore, actually, the scale of the typical corpus for the task of chinese spelling check is very small. considering that almost all the research works have used sighan series datasets @xcite to train and evaluate their algorithms, we conduct counting on those three corpora, and the statistics results are shown in thus, sticking to train the supervised learning models based on those scale-limited resources might not be a wise direction. therefore, in this paper, we are dedicated to exploring unsupervised frameworks to conduct chinese spelling error detection and correction. fortunately, masked pretrained language models such as bert @xcite , roberta @xcite , elec-tra @xcite , etc. can satisfy the needs of detecting and correcting spelling errors in an unsupervised manner. first, the masked training strategy is naturally a convenient and perfect shortcut for us to conduct token-grained detection and correction. for example, we can mask any token and predict it based on the bi-directional context to see if the current token is appropriate or not. second, the pretrained language models are usually trained using large-scale corpora, thus the language diagnosis capability is very strong. intuitively, these models can also guarantee the generalization capability considering the corpora may contain text from a wide range of domains and genres. therefore, based on the masked pretrained lan-guage models, we propose a framework named uchecker to conduct unsupervised spelling error detection and correction respectively. uchecker is a two-stage framework and it will detect the text token-by-token first and then correct the abnormal tokens. models such as bert are introduced as the backbone model. inspired by the previous works @xcite , benefiting from the various and flexible masking operations, we also introduce a confusionsetguided masking strategy to fine-train the masked language model to further improve the performance of unsupervised detection and correction. though uchecker is a two-stage framework, we design an elegant method to let the information pass bert only once to guarantee the time efficiency. moreover, interestingly, in unsupervised settings, we experimentally find the performance of error detection is crucial to the global and general performance. it means that the correction capability of the pretrained language models are strong enough, then the key-point is how to improve the performance of detection. therefore, in uchecker, we also design several algorithms to improve the performance of seplling error detection. @xcite employ gpt2-like models to conduct unsupervised english grammatical error correction which also verifies the feasible of our direction. in summary, our contributions are as follows: bert is constructed based on the model of transformer @xcite . after preparing the input samples, an embedding layer and a stack of transformer layers are followed to conduct the bi-directional semantic modeling. specifically, for the input, we first obtain the representations by summing the word embeddings with the positional embeddings: the masking strategies used in bert are shown in figure 2 . there are 20% tokens will be masked, and among them there are 80% tokens are replaced with a special symbol such @xmath0, and 10% are replaced with a random token, and the left 10% keep unchanged. what should be emphasized here is the random replacing operation which plays an crucial role in the following model designs about the unsupervised detection and correction as well as the confusionset-guided fine-training. finally, a linear function g with softmax activation is used to predict the masked token x t via: ## conclusion in this paper, we propose a framework named uchecker to conduct unsupervised spelling error detection and correction. masked pretrained language models such as bert are introduced as the backbone model. we also propose a confusionsetguided masking strategy to fine-train the model to further improve the performance. experimental results on standard datasets demonstrate the effectiveness of our proposed model uchecker.