Two Issues with C hinese Spelling Correction and A Refinement Solution (2024)

Abstract: The Chinese Spelling Correction (CSC) task aims to detect and correct misspelled characters in Chinese text, and has received lots of attention in the past few years. Most recent studies adopt a Transformer-based model and leverage different features of characters such as pronunciation, glyph and contextual information to enhance the model’s ability to complete the task. Despite their state-of-the-art performance, we observe two issues that should be addressed to further advance the CSC task. First, the widely-used benchmark datasets SIGHAN13, SIGHAN14 and SIGHAN15, contain many mistakes. Hence the performance of existing models is not accurate and should be re-evaluated. Second, existing models seem to have reached a performance bottleneck, where the improvements on the SIGHAN’s testing sets are increasingly smaller and unstable. To deal with the two issues, we make two contributions: (1) we manually fix the SIGHAN datasets and re-evaluate four representative CSC models using the fixed datasets; (2) we analyze the new results to identify the spelling errors that none of the four models successfully corrects, based on which we propose a simple yet effective refinement solution. Experimental results show that our solution improves the four models in all metrics by notable margins.

## introduction chinese spelling correction (csc) aims to detect and correct misspelled characters in chinese text. the task is challenging yet important, being used in various nlp applications such as search engines @xcite , optical character recognition @xcite and international chinese education @xcite . to solve the task, recent studies have employed transformer @xcite or @xcite as the base model and incorporated rich semantic features of characters to promote performance @xcite @xcite @xcite @xcite . despite the promising results, we observe two issues with the current research for csc. first, the widely-used benchmark datasets, sighan13 @xcite , sighan14 @xcite and @xcite , contain many mistakes, most of which are the meaningless sentences and the spelling errors in the target sentences. the former are the common mistakes made by chinese beginners, as the sighan datasets are collected from the chinese essay section of test for foreigners. these mistakes make the meaning of the sentences unclear and may affect the correction of spelling errors. the latter are the spelling errors that were not identified by the chinese teachers in the test. specifically, it is known that sighan13 contains many misuses of "的", "地" and "得" in the target sentences. these mistakes definitely affect the accuracy of the evaluation results. surprisingly, previous studies have never attempted to fix the mistakes to better evaluate their models. second, recent models seem to have reached a performance bottleneck on the sighan's testing sets, as evidenced by the increasingly smaller and unstable improvements (i.e., a newly proposed model does not perform better in all metrics) in the evaluation metrics. for instance, scope @xcite performs worse than mlm-phonetics @xcite in detection recall and correction recall on sighan14 and performs worse than realise @xcite in detection precision and correction precision on sighan15. furthermore, scope combined with dr-csc @xcite improves scope by only around 1 point in all metrics and also performs worse than comparative models in several metrics on sighan13 and sighan14. while these models are focused on different aspects of spelling errors, we speculate the reason is that there exist certain errors which none of them can stably correct. to tackle the two issues, we make two contributions in this paper. first, we examine the sighan datasets sentence by sentence and fix all possible mistakes. then, we retrain four representative csc models using the fixed datasets and re-evaluate their performance. second, we analyze the evaluation results and identify the spelling errors that none of the models successfully corrects, based on which we propose a simple solution to refine the output of the models without training. experimental results show that our simple solution improves the four models in all metrics by notable margins. type 1: meaningless sentences original 连忙我都没有时间跟父母见面！ quickly i don't even have time to meet my parents! fixed 忙得我都没有时间跟父母见面！ i'm so busy that i don't even have time to meet my parents! type 2: spelling errors in target sentences original 很多伪心的路，在我们面前挥手。 many roads of false hearts wave in front of us. fixed 很多违心的路，在我们面前挥手。 many roads against our will wave in front of us. type 3: unconverted traditional chinese characters original 一张又一张地念著， read one page after another, fixed 一张又一张地念着， read one page after another, two authors of the paper independently examine the sighan datasets and identify the sentences with mistakes. then they review each identified sentence and discuss whether it should be fixed and how to fix it. to ensure the accuracy of fixing, we fix the datasets in two rounds and both rounds take the same steps. first, we examine the fluency of the sentences and identify those that are meaningless. in this case, both a source sentence and the corresponding target sentence need to be fixed, and the spelling errors remain unchanged. second, we identify the spelling errors in the target sentences. third, we identify the traditional chinese characters that are not converted into simplified ones by opencc foot_0 in both source and target sentences. comparing the results of "original" and "retrained", we observe that the results are largely changed. on sighan13, all results decrease drastically. this is mainly because the "original" results are calculated after excluding "的", "地" and "得", since the targets are almost not correct, whereas the "retrained" results are calculated on all spelling errors. this indicates the models can still not correct "的", "地" and "得" well, especially for scope which has the largest performance drop. on sighan14 and sighan15, the results generally increase after the datasets are fixed. based on the results, we suggest to use the fixed datasets for more accurate evaluation in the future. an interesting observation is that the "retrained" results generally show the models ranked by performance from high to low are scope foot_2 , lead, realise and plome, which coincides with the "original" results. this indicates that we have correctly retrained the models and the fixed sighan can reflect their performance discrepancies. ## a refinement solution using chinesebert we extract the sentences from the testing sets that none of the four models successfully reproduces the target sentence, and analyze the reasons of failures. we observe three main types of failures. first, the models often fail to correct the particles "的", "地" and "得" and the pronouns such as "他(们)", "她(们)", "它(们)", "那" and "哪". second, the models often fail to correct the spelling errors in special terms, including idioms, proverbs, proper nouns and other commonly-used expressions. third, the models often make overcorrections. most of the above failures can be solved by inferring the correct character using the contextual information of the corresponding sentence. based on the idea, we propose a simple refinement solution with chinesebert @xcite on top of the output of the four models. specifically, given a sentence output by any model, we mask the character pertaining to the above failure cases, and let chinesebert infer the new character without training. then we measure the phonological distance between the masked character and the inferred character, where the distance is calculated as the edit distance between the pinyins (with tone) of the two characters. if the distance is below a threshold foot_3 , we keep the inferred character; otherwise, we keep the masked character. the intuition is that about 83% spelling errors have similar pronunciation with the correct character @xcite , so if the inferred character has a very different pinyin than the masked character, it is unlikely to be the correct character. if there are multiple characters to mask in a sentence, we mask them one at a time and infer using chinesebert, from beginning to end. once there is no character to mask, we stop the process and use the last output of chinesebert as the refined sentence. note that if a sentence output by the above four models contains no character to mask, the sentence is the final output and the refinement process does not run. the problem at hand is how to identify the characters to be masked. we design three strategies for the three failure types, respectively. first, we directly mask the particles "的", "地" and "得" and the pronouns "他", "她", "它", "那" and "哪". second, for a special term with spelling errors, we notice that the jieba foot_4 tokenizer produces different tokens with and without the hidden markov model (hmm). the former tends to regard it as a new word and the latter tends to tokenize it into single characters. hence, for a sentence output by the above models, we use the two methods to tokenize it and regard the parts with different tokenization results as the special terms to mask. note that this approach may mask phrases other than special terms if there exist spelling errors. third, to identify over-corrections, we calculate the edit distance between the pinyins (with tone) of the changed character and the original character in the source sentence. if the distance is above 3 as discussed in the last paragraph, we regard it as a potential over-correction and mask the character. the results are presented in ## related work recent studies mainly adopt transformer or bert/chinesebert as the base model to solve the csc task, and incorporate rich semantic features of the chinese language to enhance the ability of the base model. for instance, cheng et al. (2020) other studies learn contextual information in sentences to detect and correct spelling errors @xcite @xcite . ## conclusion in this work, we discuss two issues with the chinese spelling correction task: the existence of mistakes in the sighan datasets and the smaller and unstable improvements of new models. we manually fix the mistakes and re-evaluate four representative csc models on the fixed datasets. we analyze the common types of failures of the models and propose a simple yet effective refinement solution. experimental results show our solution can stably improve the base models in all metrics. while the current refinement solution is purely rule based, in the future we will develop data-driven methods to further improve the performance.