C hinese Spelling Corrector Is Just a Language Learner (2024)

Abstract: This paper emphasizes Chinese spelling correction by means of self-supervised learning, which means there are no annotated errors within the training data. Our intuition is that humans are naturally good correctors with exposure to error-free sentences, which contrasts with current unsupervised methods that strongly rely on the usage of confusion sets to produce parallel sentences. In this paper, we demonstrate that learning a spelling correction model is identical to learning a language model from error-free data alone, with decoding it in a greater search space. We propose Denoising Decoding Correction (D2C) , which selectively imposes noise upon the source sentence to determine the underlying correct characters. Our method is largely inspired by the ability of language models to perform correction, including both BERT-based models and large language models (LLMs). We show that the self-supervised learning manner generally outperforms the confusion set in specific domains because it bypasses the need to introduce error characters to the training data which can impair the error patterns not included in the introduced error characters.

## introduction chinese spelling correction (csc) is a fundamental natural language processing task for a series of ai applications @xcite @xcite @xcite . recent studies @xcite show that simply using the supervised signals within parallel sentences to fine-tune pretrained language models (plms) achieves notable results across a series of benchmarks. however, the high cost of annotation is blamed for the low accessibility of parallel sentences. therefore, these models remain mediocre in handling massive domains in real applications, which makes the application of powerful self-supervised learning to csc a pivotal issue that has received broad attention in the community. this paper emphasizes the value of self-supervised learning, where only error-free data is used to adapt models to specific target domains, which has still achieved marginal progress in recent years. previous unsupervised methods @xcite @xcite focus on synthesizing pseudo parallel sentences, while the supervised signals do not derive from the real distribution but from the confusion set (an empirically constructed word set of common misspelled cases). by replacing certain characters in the original sentences with those in the confusion set, parallel sentences are obtained for fine-tuning the models. however, the gap between the confusion set and the real error patterns in the target domain can induce a high false positive rate @xcite . this paper raises a bold idea: can machine spelling correction learn from error-free data alone? intriguingly, humans naturally learn to rectify mistakes in a sentence with minimal exposure to parallel data. we illustrate in figure 1 that humans only learn to use correct sentences (error-free data) in daily life. when encountering a sentence with an error character "模" (mold), they can correct it to "膜" (cornea) with ease based on their knowledge. in contrast, machine spelling correction models cannot do this if they are not exposed to annotated edit pairs like "模" → "膜" in the training process. in this paper, we demonstrate that a machine spelling corrector can also be learned from solely error-free data as illustrated at the bottom of figure 1 . the key is to have the model learn semantics rather than character-to-character editing, where the source sentence will first be encoded into the semantic space, and then rephrased to the correct sentence, demonstrate this ability. we call this manner self-supervised spelling correction. however, the resultant models still exhibit a low recall. to address this problem, we propose a novel decoding algorithm denoising decoding correction (d 2 c), which selectively imposes noise upon the source sentence to solve the underlying correct characters. we apply d 2 c to two architectures: bidirectional models (represented by relm @xcite , the state-of-the-art model in chinese spelling correction) and auto-regressive models (represented by a series of llms @xcite @xcite ). d 2 c achieves a significant performance boost over raw language models trained with errorfree data. to evaluate our method across different domains, we created a synthesized training set for lemon @xcite using gpt-3.5 as a sentence generator, which contains only error-free sentences. this dataset permits the fine-tuning and evaluation of self-supervised models in various domains. we summarize the contributions of this paper. ## related works correcting spelling errors poses a challenging yet crucial task in natural language processing. early endeavors primarily relied on unsupervised techniques, assessing sentence perplexity as a key metric @xcite @xcite . recent methods model spelling correction as a sequence tagging problem that maps each character in a given sentence to its accurate counterpart @xcite . on top of pre-trained language models (plms), some bert-based models with the sequence tagging training objective are proposed. @xcite identify the potential error characters by a detection network and then leverage the soft masking strategy to enhance the eventual correction decision. @xcite use a multi-task network to minimize the misleading impact of the misspelled characters @xcite . there is also a line of work that incorporates phonological and morphological knowledge through data augmentation and enhances the bertbased encoder to assist mapping the error to the correct one @xcite @xcite @xcite . recent studies @xcite focus on the rephrasing training objective, which achieves notable results. while in the unsupervised spelling correction domain, previous works focus on generating pseudo annotated data or detecting error characters with confusion dataset @xcite @xcite . while these methods are based on heuristics, our method is based on self-supervised learning @xcite @xcite which seeks to perturb the language representation of plms. ## from language modeling to spelling correction this section provides the motivation for our work. the basic goal is to learn spelling correction from error-free data, which we term self-supervised spelling correction. first, we discuss the transferability between language modeling and spelling correction. second, we highlight that rephrasing is the primary training objective for self-supervised spelling correction. we discuss the transferability from two perspectives: (1) the coherence of training objectives between rephrasing spelling correction and language modeling, and (2) the inclusion of knowledge about spelling correction into the pre-training process. ## synthetic lemon training set to evaluate self-supervised models' performance across multiple domains, we release a gpt-3.5generated synthetic lemon training set. lemon @xcite ) is a multi-domain benchmark that allows us to evaluate the multidomain generalization of csc models. however, it only includes a test set without a training set. the synthetic data is generated in two steps: (1) extract the words in each domain. (2) randomly select words and request gpt-3.5 to generate error-free sentences mimicking the style of specific domains. see our prompts in appendix a. statistical information about the size of the different training sets is provided in ## method in this section, we first introduce two rephrasing architectures. then, we propose an enhanced decoding method to unleash the potential of pre-trained language models. additionally, we suggest using a confusion dataset to improve the recall score. ## experiments in this section, we report the empirical results of a series of spelling correction benchmarks. we concentrate on two benchmarks: ## case study we further showcase some examples to illustrate how d 2 c improves the decoding process. ## conclusion this paper studies self-supervised spelling correction based on rephrasing-based models. we demonstrate that machine spelling correction does not necessitate parallel data and can be learned from error-free data alone. we propose a novel decoding algorithm named d 2 c to effectively enhance the recall ability of the self-supervised model. we also compare the d 2 c method with the confusion set method. results on chinese spelling correction showcase the significant improvement brought by our method. we hope this paper can bring new in-