S pell BERT : A Lightweight Pretrained Model for C hinese Spelling Check (2021)

Abstract: Chinese Spelling Check (CSC) is to detect and correct Chinese spelling errors. Many models utilize a predefined confusion set to learn a mapping between correct characters and its visually similar or phonetically similar misuses but the mapping may be out-of-domain. To that end, we propose SpellBERT, a pretrained model with graph-based extra features and independent on confusion set. To explicitly capture the two erroneous patterns, we employ a graph neural network to introduce radical and pinyin information as visual and phonetic features. For better fusing these features with character representations, we devise masked language model alike pre-training tasks. With this feature-rich pre-training, SpellBERT with only half size of BERT can show competitive performance and make a state-of-the-art result on the OCR dataset where most of the errors are not covered by the existing confusion set.

## introduction spelling check is to detect and correct chinese spelling errors in sentences. however, it is a nontrivial task for chinese spelling check because of the nature of ideographic language. chinese has a large vocabulary including at least 3,500 common characters which leads to huge search space and an unbalanced distribution of errors. though hard to cover most of the misuses, their patterns could be roughly reduced to visual or phonetic errors @xcite as shown in figure 1 . the former type of errors have similar shapes as correct ones and they are often caused by optical character recognition (ocr) or morphology-based input method. the other type of errors have similar pronunciation as original ones and they are usually caused by automatic speech recognition (asr) or phonetic-based input method. previous work @xcite @xcite to employ a predefined confusion set to find and filter correction candidates. confusion set is constructed by incorrect stats @xcite and it has a mapping between visually similar pairs and phonetically similar pairs in accord with erroneous patterns. however, these models only learn a shallow mapping from confusion set and their performance is heavily dependent on the quality of confusion set. but it is hard to find an up-to-date and in-domain confusion set. in this paper, we devise two pre-training tasks to model the two aforementioned erroneous patterns explicitly. to model visual errors, we introduce radical features. chinese characters can be decomposed into various components namely radical. as for phonetic errors, we employ pinyin as features which are descriptions of pronunciation. we fuse these visual and phonetic features with character representations by relational graph convolutional network @xcite . likewise masked language model in bert @xcite , we randomly replace some characters and then predict the original visual and phonetic features with false input. our model, spellbert, can intrinsically learn to correct errors based on visual or phonetic patterns rather than simple mapping. on the ocr dataset, where only a few errors are covered by confusion set, we make a state-ofthe-art result and this indicates that spellbert can generalize well without depending on confusion set. on resource-constrained scenarios for deployment, making a model lightweight is necessary. spellbert only has half size of bert and is more efficient for these scenarios. in summary, spellbert is independent on confusion set in training and inference phase. with only half size of bert, spellbert can show competitive performance and generalize well. ## related work current methods consider csc as sequence generation problem or sequence labeling problem. @xcite introduce copy mechanism to generate corrected sequence. @xcite unify single-character and multi-character correction by a chunk-based generative model. pretrained models (ptms) have made a success on sequence labeling tasks @xcite . masked language model (mlm) is introduced as pre-training task to predict masked or replaced words conditioned on context. the mode of mlm is intuitively appropriate to be transformed to predict spelling errors and correct them. significant progress has been made by power of ptm @xcite . based on mlm, confusion set is applied to narrow search space for predicting correct characters. cheng et al. (2020) constructed a graph by confusion set to help final prediction. nguyen et al. (2020) raised an adaptable confusion set but its training process is not end-to-end. ideally, csc corpus can be infinitely constructed by replacing words based on confusion set. wang et al. (2018) generated 270k data by ocr-and asr-based approaches. zhang et al. (2020) created 5 million augmented data and li et al. (2021) created 9 million augmented data by substitutionbased method. zhang et al. (2021) corrupted input sentence by randomly replacing characters with noisy-pinyin and the new pre-training task fitted well for csc. more recently, some methods also utilized phonetic and visual features in csc. @xcite employed a gru @xcite to encode pinyin sequence and chinese strokes sequence as extra features. @xcite had similar design but they encoded pictures of characters to get visual features. @xcite enriched character representations by knowledge of audio and visual modalities. our method is different from all of these work. for phonetic features, we regard pinyin as a whole but not a sequence. for visual features, we used radicals which are higher-level features than strokes. and we incorporate these extra features by graph neural network. ## approach we treat csc as a sequence labeling problem. an input sequence with n characters is represented as @xmath0, x 2 , • • • , x n }. our goal is to transform it into a target sequence @xmath1, y 2 , • • • , y n }. during which, incorrect characters will be detected and corrected. obviously, the input and output share the same vocabulary and most of the output characters can be directly copied from input. the framework of our model is shown in figure 2 . it contains three parts, i.e., a bert-based encoder, a feature-fusing module and a component for pretraining. we will progressively elaborate our design in detail. ## conclusion in this work, we propose a lightweight pretrained model, spellbert, for chinese spelling check. we incorporate pinyin and radicals as phonetic and visual features and design two pre-training tasks to encourage the pre-trained model to explicitly capture erroneous patterns. experiments show that spellbert has competitive performance compared to the large pretrained models. besides, spellbert can be directly used without confusion set in the fine-tuning and inference phase, which is more convenient to use and easier to deal with the errors uncovered by the existing confusion sets.