CRAS pell: A Contextual Typo Robust Approach to Improve C hinese Spelling Correction (2022)

Abstract: Recently, Bert-based models have dominated the research of Chinese spelling correction (CSC). These methods have two limitations: (1) they have poor performance on multi-typo texts. In such texts, the context of each typo contains at least one misspelled character, which brings noise information. Such noisy context leads to the declining performance on multi-typo texts. (2) they tend to overcorrect valid expressions to more frequent expressions due to the masked token recovering task of Bert. We attempt to address these limitations in this paper. To make our model robust to contextual noise brought by typos, our approach first constructs a noisy context for each training sample. Then the correction model is forced to yield similar outputs based on the noisy and original contexts. Moreover, to address the overcorrection problem, copy mechanism is incorporated to encourage our model to prefer to choose the input character when the miscorrected and input character are both valid according to the given context. Experiments are conducted on widely used benchmarks. Our model achieves superior performance against state-of-the-art methods by a remarkable gain.

## introduction chinese spelling correction (csc) is a task to detect and correct spelling errors in texts @xcite . it is a challenging yet important task, which plays an important role in various nlp applications such as optical character recognition @xcite , automatic speech recognition @xcite and search engine @xcite in chinese, spelling error is mainly caused by the misuse of phonologically and visually similar characters @xcite . according to @xcite , about 83% of errors are related to phonological similarity and 48% are related to visual similarity. table1 illustrates two examples. in recent years, bert @xcite based models have dominated the research of chinese spelling correction @xcite @xcite @xcite , which follow the paradigm of non-autoregressive generation. typically, these models generate corrected characters for all input characters in parallel, where the generated characters can be the same as the input characters. thanks to the success of bert, these models significantly improved the results of csc benchmarks. however, they have two limitations. first, multi-typo texts are very common in csc datasets, which is defined as texts containing more than one typos. the multi-typo statistics of sighan13 @xcite , sighan14 @xcite and @xcite . however, we find the performance of existing csc models declines sharply on multi-typo texts. second, bert is a masked language model @xcite . it learned how to recover a masked token based on its context from large corpus. when there are multiple valid characters for a masked position, the model prefers to recover it with the most frequent one in the training corpus. as a consequence, bert-based models tend to overcorrect an infrequent but valid expression to a more frequent expression. for instance, both bert @xcite and plome @xcite wrongly correct "这并非 非 非是说..." to "这并 不 不 不是说...". we call this problem overcorrection. in this paper, we attempt to address the aforementioned problems by proposing a new model called craspell, which is short for contextual typo robust approach for chinese spelling correction. figure 1 illustrates the framework. the contextual typo disturbance problem is caused by the noise of contextual typos, therefore we try to make our model robust to such noise. to this end, our approach first generates a noisy context for each training instance, and then forces the correction model to yield similar outputs based on the original and noisy context. moreover, to address the overcorrection problem, we incorporate the copy mechanism @xcite in our model. finally, the output for each position in a given text is the sum of generative distribution and copy distribution. thus, our model has more chances to keep the input character unchanged when the miscorrected and input characters are both valid according to the given context. we conduct experiments on the widely used benchmark dataset sighan @xcite @xcite . the experimental results show that our model outperforms all the compared approaches. furthermore, we extract samples containing multiple typos from sighan to construct a multi-typo evaluation set. experimental results show that our approach achieves 2.6% absolute improvement in detection and 2.7% absolute improvement in correction on the multitypo set, which demonstrates the effectiveness of our approach for multi-typo texts. we summarize our contributions as follows. ## related work chinese spelling correction is a challenging task in natural language processing, which plays important roles in many applications, such as search engine @xcite , automatic essay scoring @xcite , and optical character recognition @xcite . it has been an active topic, and various approaches have been proposed in recent years @xcite @xcite @xcite @xcite . early work on csc followed the pipeline of error identification, candidate generation and selection. some researchers focused on unsupervised approaches, which typically adopted a confusion set to find correct candidates and employed language model to select the correct one @xcite @xcite @xcite . however, these methods failed to condition the correction on the input sentence. in order to model the input context, discriminative sequence tagging methods @xcite and sequence-to-sequence generative models @xcite @xcite were employed. bert @xcite is a bidirectional language model based on transformer encoder @xcite . it has been demonstrated effective in a wide range of applications, such as question answering @xcite , information extraction @xcite , and semantic matching @xcite . recently, it has dominated the researches on csc @xcite @xcite @xcite . @xcite adopted the dae-decoder paradigm with bert as encoder. zhang et al. (2020) introduced a detection network to generate the masking vector for the bertbased correction network. @xcite employed graph convolution network (gcn) @xcite combined with bert to model character inter-dependence. in our previous work @xcite , we proposed a task-specific pretrained model for chinese spelling correction, which masked tokens by similar characters according to confusion set. @xcite proposed to decode with the crf module. @xcite proposed to incorporate phonological and visual features via a multi-modal module. although these models achieved some success on benchmarks, all of them suffer from the typo disturbance problem and overcorrection problem. ## approach in this section, we describe the proposed approach and its detailed implementation. figure 1 illustrates the framework of our model, which is composed of correction module and noise modeling module. ## conclusions in this work, we first point out two limitations of previous csc models, which are called contextual typo disturbance problem and overcorrection problem. to solve the first problem, we propose the noise modeling module to generate noisy context in training process. experimental results show that this module is effective on multi-typo texts. to solve the overcorrection problem, we incorporate a copy block in the correction model, which encourages our model to prefer to keep the input character when the miscorrected and input characters are both valid in the given context. experimental results demonstrate its effectiveness on both bert and cbert. moreover, the proposed model craspell outperforms all compared models and achieve new state-of-the-art performances on sighan dataset.