PTCS pell: Pre-trained Corrector Based on Character Shape and P inyin for C hinese Spelling Correction (2023)

Abstract: Chinese spelling correction (CSC) is a challenging task with the goal of correcting each wrong character in Chinese texts. Incorrect characters in a Chinese text are mainly due to the similar shape and similar pronunciation of Chinese characters. Recently, the paradigm of pre-training and fine-tuning has achieved remarkable success in natural language processing. However, the pre-training objectives in existing methods are not tailored for the CSC task since they neglect the visual and phonetic properties of characters, resulting in suboptimal spelling correction. In this work, we propose to pre-train a new corrector named PTCSpell for the CSC task under the detector-corrector architecture. The corrector we propose has the following two improvements. First, we design two novel pre-training objectives to capture pronunciation and shape information in Chinese characters. Second, we propose a new strategy to tackle the issue that the detector’s prediction results mislead the corrector by balancing the loss of wrong characters and correct characters. Experiments on three benchmarks (i.e., SIGHAN 2013, 2014, and 2015) show that our model achieves an average of 5.8% F1 improvements at the correction level over state-of-the-art methods, verifying its effectiveness.

## introduction chinese spelling correction (csc) is a task which detects incorrect characters in chinese text and corrects them. csc is often used as post-processing to ensure the quality of a search engine query text @xcite and academic papers @xcite . csc plays an important role in correcting recognition errors due to similar pronunciation and character shape @xcite , which is a common issue with automatic speech recognition (asr) and optical character recognition (ocr) systems. compared with other languages, chinese has distinct characteristics, such as its unique pronunciation system (usually represented as pinyin) and writing norms, which often lead to the two problems, namely the same pronunciation may correspond to multiple characters and different characters have similar shapes. according to @xcite , around 83% and 48% of errors in chinese texts can be attributed to phonological and visual similarity respectively. the first sentence in chinese spelling correction is challenging because different people have different writing habits, resulting in a variety of mistakes for each character. as such, it is difficult for previous rule-based methods to address these issues effectively. most of the recent works based on large pre-trained language models @xcite perform well in the csc task. for example, @xcite ; @xcite used artificially constructed confusion sets to pre-train language models for the csc task. however, pre-training objectives tailored for csc have not yet been explored. the visual and phonetic properties of characters are not fully considered in the pre-training process. moreover, most works are based on the architecture of the detector-corrector @xcite @xcite . however, an inherent problem occurs if the detector predicts a correct character as being wrong, in which case the corrector may change the correct character to another one which is also a reasonable response. for example, in to solve these problems, we propose a pretrained corrector based on the visual and pronunciation features of characters for the csc task. by doing so, the corrector can capture the phonetic similarity and visual similarity between characters, and such a pre-training strategy is more tailored to the csc task. to address the problem of the detector's prediction results misleading the corrector, our basic idea is to strengthen the ability of the corrector to recognize correct characters to ease the errors caused by detector. in this work, we propose a two-stage pre-trained corrector. in the first stage of pre-training, the visual and phonological features of characters are taken into account, and the pre-training strategies matching csc are designed, respectively from similar vision to character and from similar pinyin to character. the pre-trained models are denoted as similar character shape bert (scsbert) and similar pinyin bert (spbert). in the second stage of pre-training, three different pre-trained models, scsbert, bert, and spbert are fully fused. in addition, we propose a novel loss function to fine-tune the corrector. we calculate the loss of a small number of correct characters and the loss of all the wrong characters in the original text, so that it not only corrects the wrong characters, it also prevents the correct characters from being corrected by mistake. to verify the effectiveness of our method, we conduct experiments with our model on the sighan 2013, 2014 and 2015 test set using the official sighan testing tool, which achieve an average improvement in f1 of 5.2% and 5.8% at the detection-level and correction-level compared to the latest works, mdcspell @xcite and realise @xcite . our main contributions are summarized as follows: (i) we propose a pre-trained corrector which is tailored for the csc task; (ii) we design two pre-training objectives based on vision and pronunciation to enable the corrector to capture the shape and pinyin similarity between characters; (iii) we propose a new strategy to solve the problem where the prediction results of the detector mislead the corrector by balancing the loss of incorrect characters and correct characters. large pre-trained language models (ptms) based on transformer @xcite , such as bert @xcite , xlnet @xcite and spanbert @xcite have been proposed and are being increasingly used for csc tasks. in soft-masked bert @xcite , bert is used as a corrector to correct wrong positions predicted by the gate recurrent unit (gru). @xcite set different masking schemes to fine-tune bert, and selected the optimal candidate results as the error correction results. in addition to using bert as a corrector, electra's discriminator @xcite is also used to detect errors @xcite . although bert can be used in csc tasks and achieves good results, the similarity between the target character and original character is not easy to learn. therefore, a confusion set @xcite is used to pre-train the bert-like model to solve the above problem. @xcite used artificially constructed confusion sets to pre-train bert, which makes bert more capable of correcting phonologically and visually similar characters. @xcite set different proportions to select phonologically and visually similar characters in the process of using a confusion set for training. chinese spelling errors are mainly caused by characters which are similar in shape and pronunciation. recent works focus on how to make full use of visual and phonetic features to improve the correction of these two types of errors. to correct errors due to characters having a similar shape, the chinese characters are encoded into images or split into strokes. to correct errors due to characters having a similar pronunciation, the original text is converted into pinyin or speech features @xcite @xcite . although prior research has achieved good results on the csc task, the following two problems remain unresolved. first, the visual and phonetic features of similar chinese characters cannot be directly fused into bert for pre-training. second, in relation to the detector-corrector architecture, the corrector cannot effectively alleviate the error caused by detector. to solve these problems, we propose a pre-trained language model based on visual features and pinyin features and a special loss function for the corrector. ## methodology in this section, we briefly introduce the preliminaries of the our method, then we detail the proposed ptcspell. ## conclusion we propose a pre-trained corrector, which is a part of the detector-corrector architecture, for modeling similar shape and pronunciation errors in the csc task. our main contribution is to enhance the ability of the corrector. the architecture of ptcspell is based on bert, but is pre-trained from scratch based on similar character shapes and pinyin re-spectively. in addition, we propose a special loss function that enhances the ability of the corrector to retain the correct characters in the original text. finally, the experimental results show that our model is effective. in the future, we will design a better detector to further improve the recall score for the whole detector-corrector model.