Rethinking Masked Language Modeling for C hinese Spelling Correction (2023)

Abstract: In this paper, we study Chinese Spelling Correction (CSC) as a joint decision made by two separate models: a language model and an error model. Through empirical analysis, we find that fine-tuning BERT tends to over-fit the error model while under-fit the language model, resulting in poor generalization to out-of-distribution error patterns. Given that BERT is the backbone of most CSC models, this phenomenon has a significant negative impact. To address this issue, we are releasing a multi-domain benchmark LEMON, with higher quality and diversity than existing benchmarks, to allow a comprehensive assessment of the open domain generalization of CSC models. Then, we demonstrate that a very simple strategy – randomly masking 20% non-error tokens from the input sequence during fine-tuning – is sufficient for learning a much better language model without sacrificing the error model. This technique can be applied to any model architecture and achieves new state-of-the-art results on SIGHAN, ECSpell, and LEMON.

## introduction chinese spelling correction (csc) is a crucial task in natural language processing (nlp) behind many downstream applications, e.g, web search @xcite , named entity recognition, optical character recognition @xcite . it aims to detect and correct the potential spelling errors in a sentence. bert @xcite and its enhanced variants have achieved state-of-the-art results in the current csc community (name a few) @xcite @xcite . from a high-level perspective, csc requires a language model and an error model working collaboratively to make a decision @xcite . suppose that the input sentence contains n characters @xmath0, ..., x n ). the model predicts the corrected character at each position @xmath1, ..., y n ). at each position i, let x -i indicate the characters at all other positions, then by bayes rule @xcite , we have: in this paper, we make a key observation that bert-based csc models typically over-fit the error model, yet under-fit the language model, be-cause the error model is much easier to memorize compared to the language model. as a result, the model generalizes very poor to unseen edit pairs (x i , y i ) and fails to exploit the context x -i . we illustrate this fact in figure 1 . here, the model has been exposed to edit pairs "生硬→声音" (correct stiff to sound) and "生音→声音" (correct raw to sound) during training. during testing, the model fails to detect an unseen edit pair "声影→声音" (correct shadow to sound) and meanwhile overcorrects "生 硬→声 音" (correct stiff to sound). this is due to the fact that the model naively memorizes the training edit pairs, failing to identify if they fit the broader context. we will present qualitative analysis of this phenomenon in later sections. the consequence of a sub-optimal or under-fit language model is that the model struggles to generalize to new contexts and new domains. sighan is the current most widely-used benchmark in csc, but it is limited in two ways: (1) a narrow sentence corpus sourced exclusively from the chinese essays by foreign speakers @xcite ; (2) a low diversity of edit pairs (i.e. 370 edit pairs in its test set). as a result, it does not pose enough challenge to the model's generalization ability. to this end, we present lemon, a new benchmark that is a large-scale multi-domain dataset with natural spelling errors, which spans 7 domains and contains over 22,000 examples with 7,627 distinct edit pairs collected from real human daily writing. it provides a comprehensive evaluation of csc models in real-world scenarios. based on lemon and other public benchmarks, we demonstrate that a very simple method can effectively enhance language modeling without causing adverse effect to error modeling, thus significantly improves csc model performances. the method is to randomly mask 20% of the non-error tokens from the input sentence during fine-tuning (this is different from masking 15% tokens during pre-training in bert). if x i is masked, it forces the model to predict y i given x -i without any clue about x i , equivalent to training p (y i |x -i ). this masked-fine-tuning (masked-ft) technique is unlike other data augmentation methods based on homophone substitution, random substitution or confusion sets @xcite , in that it does not impose any assumption about human errors. as a result, it enables learning a completely unbiased error model from real human data. this property let masked-ft achieve new state-of-the-art across csc benchmarks. we also show that masked-ft is effective in domain transfer. suppose that there is an annotated parallel corpus for a certain domain, and we want to transfer the model of such a domain to a new domain where only monolingual (i.e. unannotated) corpus is available. we propose to train the model with the parallel data along with a masked language modeling (mlm) loss from the monolingual corpus. the idea behind is to transfer the language model to the new domain while preserving the error model that is learned through the parallel data. empirical results demonstrate that this way of using monolingual data produces a better model than data synthesis methods based on confusion sets. our contributions are summarized as follows. (1) we perform empirical analysis showing that bert-based csc models learn a sub-optimal language model, resulting in a bad performance on out-of-distribution edit pairs. (2) we release a largescale and multi-domain benchmark for csc, which is more challenging than existing ones. (3) we demonstrate that a simple masked-fine-tuning strategy significantly enhance language modeling without hurting error modeling, leading to new state-ofthe-art results across benchmarks. ## analysis of bert fine-tuning in this section, we report empirical analysis on bert-based models. we study their top-k performance, generalization to unseen edit pairs, and gradient scales during training. the observation is that the bert-based models, with regular fine-tuning, easily over-fits the edit pairs in the training set and learns a degenerated language model. for some analyses, we also include the result of masked-ft (randomly mask 20% input tokens) for comparative study. ## lemon benchmark sighan @xcite is the current most widely-used benchmark in csc, but as described in the introduction, it doesn't pose enough challenge to test the generalization ability of csc models. sighan is exclusively collected from the chinese essays written by foreign speakers @xcite . that includes 1,100 test examples with a narrow content coverage. besides, there are 370 distinct edit pairs in the test set, with nearly 70% overlap with the training set. as a result, a model can achieve a decent score by memorizing the error patterns. in this paper, we present lemon, a large-scale multi-domain dataset with natural spelling errors, which spans 7 domains, including game (gam), encyclopedia (enc), contract (cot), medical care (mec), car (car), novel (nov), and news (new). as opposed to ecspell @xcite , where the typos are deliberately created by human on correct sentences, lemon consists of over 22,000 examples with natural spelling errors identified from daily human writing, annotated by well-educated native chinese speakers. the idea is to be as close to the real-life language distribution as possible. lemon contains 7,627 edit pairs from all domains, which is much more diversified than sighan. figure 3 shows some concrete pieces of examples in lemon. in mec, for example, we see tyrosinase is misspelled, which is a professional word in medicine. the model thus requires certain expertise to correct it. additionally, the language style of context varies greatly from one domain to another. for example, the expressions in gam are idiomatic while those in cot are relatively regularized and formal. the bottom part of each block shows the histogram of all characters in this domain, indicating its lexical distribution. we can see that the lexicon of each domain varies greatly, suggesting different domain-specific language styles. due to space limitation, further analysis for lemon is reported in appendix b. ## masked fine-tuning the intuition behind masked fine-tuning (masked-ft) is simple: we want to enhance the learning of language model without perturbing the error model. by equation ( 1 ), the language model predicts a token given all other tokens. thus, we propose to randomly mask a fraction of tokens and train the model to restore them. for training with parallel data, this is equivalent to randomly substituting a fraction of input tokens by a special mask token. the mask token can be any token, as long as it never occurs in an ordinary input. it can be understood as a special "typo" that human never makes, thus introducing zero bias to the error model. this technique can be applied to any model architecture. empirically, we find that masking 20% of non-error tokens by [mask] is the most effective. other variants, such as using a different masking rate, selecting from both error and non-error tokens, and substituting by @xcite , also works, but they achieve slightly worse results. the ablation study is presented in section 6. for training with both parallel (annotated) data and monolingual (unannotated) data, we propose to randomly mask 20% tokens from the monolingual data, then construct mlm loss @xcite and add it to the training objective. this is different from generating parallel data by corrupting 20% tokens. any corruption rule (e.g. confusion sets) would make assumptions on human errors, thus introduce a bias to the error model. the mlm loss does not introduce any error model bias, and as section 5 shows, it achieves better results in domain transfer. and evaluate them on sighan-15, ecspell, and lemon. our implementation is based on transformers @xcite . ## further analysis mask rate we investigate the impact from the mask rate p. a large p can hurt the training as it wipes out too much contextual information. from ## related work for chinese spelling correction, bert @xcite @xcite is the straightforward backbone model. there is a line of work on improving the model architecture on top of bert, such as imposing masking signals to those potential error tokens to improve error detection @xcite , incorporating multi-modal knowledge (e.g. pronunciation, glyph) @xcite @xcite @xcite , using multi-task network to explicitly let the model detect @xcite or predict the pronunciation @xcite . another major category is data augmentation, with the goal of synthesizing efficient training data. existing data augmentation techniques are based on homophone substitution, random substitution or confusion sets @xcite @xcite . the decomposition of csc into a language model and an error model is inspired by the classi-cal noisy channel theory @xcite . the masked-ft method proposed in this paper is similar to the "dynamic masking" method proposed by @xcite . however, there are a few differences between the two studies. first, @xcite describes dynamic masking as a data augmentation method, and proposes to mix it with other data augmentation techniques such as confusion set substitution; in contrast, we describe masked-ft as a mean to enhance language modeling without perturbing error modeling, demonstrating both theoretically and empirically that it should be carried out alone without mixing with data augmentation. second, we study domain transfer with monolingual data, showing that mlm training performs better than training with synthesized data. again, it verifies our language/error decomposition theory and to the best of our knowledge, was not discussed in previous work. ## conclusion this paper presents qualitative analysis and shows that existing csc models lean to over-fit the error model and under-fit the language model. a simple yet effective method is thus presented to encourage a better language model learning. empirical results demonstrate that the simple method achieves new state-of-the-art results on public benchmarks, including on lenon, a new large-scale challenging benchmark released with this paper.