Refining Corpora from a Model Calibration Perspective for C hinese Spelling Correction (2024)

Abstract: Chinese Spelling Correction (CSC) commonly lacks large-scale high-quality corpora, due to the labor-intensive labeling of spelling errors in real-life human writing or typing scenarios. Two data augmentation methods are widely adopted: (1) *Random Replacement* with the guidance of confusion sets and (2) *OCR/ASR-based Generation* that simulates character misusing. However, both methods inevitably introduce noisy data (e.g., false spelling errors), potentially leading to over-correction. By carefully analyzing the two types of corpora, we find that though the latter achieves more robust generalization performance, the former yields better-calibrated CSC models. We then provide a theoretical analysis of this empirical observation, based on which a corpus refining strategy is proposed. Specifically, OCR/ASR-based data samples are fed into a well-calibrated CSC model trained on random replacement-based corpora and then filtered based on prediction confidence. By learning a simple BERT-based model on the refined OCR/ASR-based corpus, we set up impressive state-of-the-art performance on three widely-used benchmarks, while significantly alleviating over-correction (e.g., lowering false positive predictions).

## introduction chinese spelling correction (csc) aims to detect and correct misspellings in the text while maintaining the sentence length @xcite . it can not only directly facilitate human writing and typing but also serve as a critical pre-processing step for many downstream chinese nlp tasks such as search engine @xcite and optical character recognition @xcite . one common challenge of applying csc is the lack of large-scale high-quality corpora in practice since labeling spelling errors in real-life writing or typing scenarios is labor-extensive @xcite . therefore, two data augmentation methods are widely adopted for this task. the first one is figure 1 : calibration curves and performance of bertbased csc models trained on random replacement and ocr/asr-based data. ece means the metric of expected calibration error @xcite , and fpr means the sentence-level false positive rate that measures over-corrections. combing subplots (a), (b), and (c), ocr/asr-based data produce better performances on standard metrics (e.g., p, r, and f1), while random replacement yields better calibration and fpr. these observations inspire us to denoise ocr/asr-based data with well-calibrated csc models trained on random replacement data, to improve performance and mitigate over-corrections. compared to random replacement, ocr/asrbased generation better mimics human misspelling scenarios, becoming the mainstream strategy used by many recent csc efforts @xcite . unfortunately, both data augmentation methods inevitably introduce noises. for example, we randomly sample 300 sentences in the ocr/asr-based corpus @xcite and check the annotated misused characters manually, finding that 11.3% of them are false spelling errors. training on these noisy samples can produce unintended over-correction (e.g., a high false positive rate). previous works mainly alleviate the problem through sophisticated model designs, e.g., integrating phonological and morphological information using multi-modal approaches @xcite . unlike these efforts, in this paper, we propose to improve csc by directly purifying noisy samples in csc corpora. considering model confidence is commonly exploited to denoise data @xcite , we first analyze the two types of csc corpora by checking the calibration characteristics and performance of models trained on them (see section 2 for experiment details). the experimental results on the sighan 13 @xcite benchmark are shown in figure 1 1 . comparing subplots (a) and (b), we find that although the csc model trained on ocr/asr-based data performs better (e.g., with a better f1 score), it is worse calibrated than its counterpart of random replacement. its calibration curve continuously lies below the dotted line (representing perfectly calibrated), indicating that the model tends to make over-confident predictions. this observation is consistent with its higher false positive rate (despite overall better performance) in subplot (c). to explain the empirical observation, we then perform a theoretical analysis of model confidence based on bayesian inference (section 3). we reveal why the calibration curve differs between the two categories of training data and identify which data samples negatively affect model confidence. guided by the empirical observations and theoretical findings, we propose to refine the ocr/asrbased corpus with a csc model trained on random replacement data. thanks to this csc model's more trustful confidence, we can use it to filter noisy ocr/asr-based samples according to their prediction scores. we achieve competitive performance on three open csc benchmarks by training a simple bert-based model on the refined corpus. notably, the model also produces a much lower false positive rate and demonstrates better 1 appendix b shows the results on sighan 14/15 calibration, which is essential in real-world csc applications. in summary, our contributions are as follows: ## a pilot study of data characteristics figure 1 illustrates the properties of ocr/asrbased and random replacement data through the calibration curves and performance of their respective models. the expected calibration error (ece) metric is explained in detail in appendix a. in this section, we provide a comprehensive description of the experimental methodology and procedures. ## conclusion we propose a simple, efficient, and interpretable data filtering method to purify chinese spelling correction (csc) corpora. we empirically reveal and theoretically prove the promising calibration characteristic of csc models trained on random replacement datasets. using a well-calibrated csc model to filter the ocr/asr-based corpora, we learn a final csc model that integrates the strong generalization performance from osc/asrbased data and the trustful model confidence from random replacement data. our method impres-sively achieves state-of-the-art performance on sighan 13/14/15 and significantly alleviates overcorrections.