An Error-Guided Correction Model for C hinese Spelling Error Correction (2022)

Abstract: Although existing neural network approaches have achieved great progress on Chinese spelling correction, there is still room to improve. The model is required to avoid over-correction and to distinguish a correct token from its phonological and visual similar ones. In this paper, we propose an error-guided correction model to address these issues. By borrowing the powerful ability of the pre-trained BERT model, we propose a novel zero-shot error detection method to do a preliminary detection, which guides our model to attend more on the probably wrong tokens in encoding and to avoid modifying the correct tokens in generating. Furthermore, we introduce a new loss function to integrate the error confusion set, which enables our model to distinguish similar tokens. Moreover, our model supports highly parallel decoding to meet real applications. Experiments are conducted on widely used benchmarks. Our model achieves superior performance against state-of-the-art approaches by a remarkable margin, on both the quality and computation speed.

## introduction chinese spelling correction (csc) attracts wide attention in recent years, which is significant for many real applications, such as search engine @xcite , optical character recognition(ocr) @xcite and automatic speech recognition(asr) @xcite . given an input sentence with spelling errors, the model is trained to detect and correct these errors and output a correct sentence. according to @xcite , phonologically and visually similar characters are major contributing factors for errors in chinese text. as shown in figure 1 , in the first example, the error is caused by the misuse of "派"(send) and "拍"(take) which have similar chinese pronunciation. in the second example, the error is caused by the misuse of "门"(door) and "们"(they) which have similar shapes. recently, the advanced neural network models and pre-trained models have achieved great success in csc, such as plome @xcite , realise @xcite , phmospell @xcite , spellbert @xcite , gad @xcite , mlm-phonetics @xcite , roberta-dcn @xcite and ecspell @xcite . although much progress has been made, there are still limitations in previous methods. first, given an input sequence, only a small fragment might be misspelled. however, for most of the previous models, they are totally blind to the errors at start, and so they attend on all tokens equally in encoding and generate every token from left to right for inference. as a result, previous models are inefficient and might create over-correction. as these models obtain a stronger ability to correct the errors, they also tend to modify the correct tokens by mistake. second, the confusion set, where a set of phonological and visual similar tokens are defined for each chinese token, provides valuable knowledge for spelling correction, as shown in figure 2 . but the methodology to use it should be further improved. for example, @xcite propose a confusion set based masking strategy, in which they remove a token and replace the token with a random character in the confusion set. as the model randomly chooses a token from the confusion set each time, some tokens might be ignored. besides, this method can't pay more attention to the token that is more easily to be misused. @xcite propose to generate a character from the confusion set rather than the entire vocabulary. in this hard restriction, the model cannot generate tokens that are not in the confusion set. third, when a csc model is deployed in real applications, the time cost of inference is a critical problem to be considered. however, most previous models try to improve the generation quality but ignore the computation speed. to address these issues mentioned above, we propose an error-guided correction model (egcm) for csc. firstly, taking advantage of the strong ability of bert @xcite , we propose a novel zero-shot error detection method to do a preliminary detection, which provides precise guidance signals to the correction model. following the guidance, our model attends more on the probably wrong tokens in encoding, and fixes the probably correct tokens during generation to avoid overcorrection. furthermore, we introduce a new loss function that effectively integrates the confusion set. by applying this loss function, every similar token in the confusion set is learned to be distinguished from the target token, and the most similar token with a high possibility of being misused is given more attention. to speed up the inference, we apply a mask-predict strategy @xcite to support parallel decoding, where the tokens with low generation probability are masked and predicted iteratively. we conduct extensive experiments on the widely used benchmark dataset sighan @xcite @xcite . experimental results show that our model significantly outperforms all previous approaches, achieving a new state-of-the-art performance for chinese spelling correction. moreover, our model has a distinct speed advantage over other models, which is 6.3 times faster than the standard transformer and 1.5 times faster than the recent non-autoregressive model ttt @xcite . we summarize our contributions as follows: recent work tends to utilize character similarity as an external knowledge. the confusion set where similar characters are stored is widely used @xcite @xcite @xcite . there are several ways of using the confusion set. the first is to augment the training data by replacing the original token with it's similar tokens @xcite . @xcite proposes to generate a character from the confusion set rather than the entire vocabulary. @xcite proposes to produce candidates by retrieving the confusion set and then filter them via language models. cheng et al. (2020) uses similarity graphs derived from the confusion set and use graph convolution operation to absorb the informa-tion from neighboring characters in the graph. ## methodology the proposed error-guided correction model (egcm) is illustrated in figure 3 . we apply the conditional masked language model (cmlm) @xcite as a backbone, which is an encoder-decoder architecture trained with a masked language model objective @xcite . in the cmlm architecture, the source wrong sentence with n tokens is denoted as @xmath0, x 2 , x 3 , ..., x n ), the target sentence is denoted as @xmath1, y 2 , y 3 , ..., y n ). several tokens in y are replaced with @xcite . these masked tokens construct the set y mask . and the rest of the tokens in y that are unmasked construct the set of y obs . for chinese spelling correction, given a source sentence x and the set of unmasked target tokens y obs , the objec is to predict the probability p (y|x, y obs ) and generate token y for each @xmath2. we first propose a zero-shot spelling error detection method to provide two guidance signals to the correction model, as shown in figure 4 . the first guidance signal is the guidance attention mask that is used in the error-focused encoder, in which the probably correct tokens are masked to push our model to attend more on the wrong tokens. the second guidance signal is the guidance for inference that serves as the start of decoding to avoid modifying correct tokens by mistake. moreover, we introduce a new loss function to take advantage of the confusion set. during inference, we apply an error-guided mask-predict strategy in which the correct tokens are fixed and the probably wrong tokens are masked and repredicted iteratively. ## conclusion we propose an error-guided correction model for the csc task. a zero-shot error detection method is proposed to provide guidance signals for training and inference. we apply a conditional masked language model as a backbone, where we improve the encoder-decoder architecture by adding an errorfocused encoder which pushes our model to focus on the wrong tokens. during training, we introduce a new confusion loss to help the model distinguish similar tokens. during inference, the error-guided mask-predict decoding strategy is adopted to mask and repredict the tokens that are probably wrong. experimental results show that our model not only achieves superior performance against state-of-theart approaches but also is cost-saving and green.