PLOME : Pre-training with Misspelled Knowledge for C hinese Spelling Correction (2021)

Abstract: Chinese spelling correction (CSC) is a task to detect and correct spelling errors in texts. CSC is essentially a linguistic problem, thus the ability of language understanding is crucial to this task. In this paper, we propose a Pre-trained masked Language model with Misspelled knowledgE (PLOME) for CSC, which jointly learns how to understand language and correct spelling errors. To this end, PLOME masks the chosen tokens with similar characters according to a confusion set rather than the fixed token “[MASK]” as in BERT. Besides character prediction, PLOME also introduces pronunciation prediction to learn the misspelled knowledge on phonic level. Moreover, phonological and visual similarity knowledge is important to this task. PLOME utilizes GRU networks to model such knowledge based on characters’ phonics and strokes. Experiments are conducted on widely used benchmarks. Our method achieves superior performance against state-of-the-art approaches by a remarkable margin. We release the source code and pre-trained model for further use by the community ( https://github.com/liushulinle/PLOME ).

## introduction chinese spelling correction (csc) aims to detect and correct spelling errors in texts @xcite . it is a challenging yet important task in natural language processing, which plays an important role in various nlp applications such as search engine @xcite and optical character recognition @xcite . in chinese, spelling errors can be mainly divided into two types: phonological errors and visual errors, which are separately caused by the misuse of phonologically similar characters and visually similar characters. according to @xcite , about 83% of errors are phonological and 48% are visual. figure 1 illustrates examples of such errors. the first case is caused by the misuse of "没(gone)" and "美(beautiful)" with the same phonics, and the second case is caused by the misuse of "人(human)" and "入(enter)" with very similar shape. chinese spelling correction is a challenging task because it requires human-level language understanding ability to completely solve this problem @xcite . therefore, language model plays an important role in csc. in fact, one of the mainstream solutions to this task is based on language models @xcite @xcite . currently, the latest approaches @xcite are based on bert @xcite , which is a masked language model. in these approaches, (masked) language models are independently pretrained from the csc task. as a consequence, they did not learn any task-specific knowledge during pre-training. therefore, language models in these approaches are sub-optimal for csc. chinese spelling errors are mainly caused by the misuse of phonologically or visually similar characters. thus, knowledge of the similarity between characters is crucial to this task. some work leveraged the confusion set, i.e. a set of similar characters, to fuse such information @xcite @xcite . however, confu-sion set is usually generated by heuristic rules or manual annotations, thus its coverage is limited. to circumvent this problem, @xcite computed the similarity based on character's strokes and phonics. the similarity was measured via rules rather than learned by the model, therefore such knowledge was not fully utilized. in this paper, we propose plome, a pre-trained masked language model with misspelled knowl-edge, for chinese spelling correction. the following characteristics make plome more effective than vanilla bert for csc. first, we propose the confusion set based masking strategy, where each chosen token is randomly replaced by a similar character according to a confusion set rather than the fixed token "[mask]" as in bert. thus, plome jointly learns the semantics and misspelled knowledge during pre-training. second, the proposed model takes each character's strokes and phonics as input, which enables plome to model the similarity between arbitrary characters. third, plome learns the misspelled knowledge on both character and phonic level by jointly recovering the true character and phonics for masked tokens. we conduct experiments on the widely used benchmark dataset sighan @xcite @xcite . experimental results show that plome significantly outperforms all the compared approaches, including the latest soft-masked bert @xcite and spell-gcn @xcite . we summarize our contributions as follows: (1) plome is the first task-specific language model designed for chinese spelling correction. the proposed confusion set based masking strategy enables our model to jointly learn the semantics and misspelled knowledge during pre-training. (2) plome incorporates phonics and strokes, which enables it to model the similarity between arbitrary characters. (3) plome is the first to model this task on both character and phonic level. ## related work chinese spelling correction is a challenging task in natural language processing, which plays important roles in many applications, such as search engine @xcite , automatic essay scoring @xcite , and optical character recognition @xcite . it has been an active topic, and vari-ous approaches have been proposed in recent years @xcite @xcite @xcite . early work on csc followed the pipeline of error identification, candidate generation and selection. some researchers focused on unsupervised approaches, which typically adopted a confusion set to find correct candidates and employed language model to select the correct one @xcite @xcite @xcite . however, these methods failed to condition the correction on the input sentence. in order to model the input context, discriminative sequence tagging methods @xcite and sequence-to-sequence generative models @xcite @xcite were employed. bert @xcite is a bidirectional language model based on transformer encoder @xcite . it has been demonstrated effective in a wide range of applications, such as question answering @xcite , information extraction @xcite , and semantic matching @xcite . recently, it has dominated the researches on csc @xcite @xcite . @xcite adopted the dae-decoder paradigm with bert as encoder. @xcite introduced a detection network to generate the masking vector for the bert-based correction network. @xcite employed the graph convolution network (gcn) @xcite combined with bert to model character interdependence. however, bert is designed and pretrained independently from the csc task, thus it is sub-optimal. to improve the performance, we propose a task-specific language model for csc. ## approach we introduce plome and its detailed implementation in this section. figure 2 illustrates the framework of plome. similar to bert @xcite , the proposed model also follows the pre-training&fine-tuning paradigm. in the following subsections, we first introduce the confusion set based masking strategy, then present the architecture of plome and the learning objectives, finally show the details of fine-tuning. ## experiments in this section, we present the details for pretraining plome and the fine-tuning results on the most widely used benchmark dataset. ## conclusions we propose plome, a pre-trained masked language model with misspelled knowledge for csc. to the best of our knowledge, plome is the first task-specific language model for csc, which jointly learns semantics and misspelled knowledge thanks to the confusion set based masking strategy. previous work demonstrated that phonological and visual similarity between characters is essential to this task. we introduce phonic and shape gru networks to model such features. moreover, plome is also the first model that makes decision via jointly considering the target pronunciation and character distributions. experimental results showed that plome outperforms all the compared models with remarkable gains.