Spelling Error Correction with Soft-Masked BERT (2020)

Abstract: Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using ‘Soft-Masked BERT’ is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets, including one large dataset which we create and plan to release, demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.

## introduction spelling error correction is an important task which aims to correct spelling errors in a text either at word-level or at character-level @xcite @xcite @xcite . it is crucial for many natural language applications such as search @xcite , optical character recognition (ocr) @xcite , and essay scoring @xcite . in this pa- wrong: 埃及有金子塔。egypt has golden towers. correct: 埃及有金字塔。egypt has pyramids. wrong: 他的求胜欲很强，为了越狱在挖洞。 he has a strong desire to win and is digging for prison breaks correct: 他的求生欲很强，为了越狱在挖洞。 he has a strong desire to survive and is digging for prison breaks. spelling error correction is also a very challenging task, because to completely solve the problem the system needs to have human-level language understanding ability. there are at least two challenges here, as shown in many methods have been proposed for csc or more generally spelling error correction. previous approaches can be mainly divided into two categories. one employs traditional machine learning and the other deep learning @xcite @xcite . @xcite , for example, proposed a unified framework for csc consisting of a pipeline of error de-tection, candidate generation, and final candidate selection using traditional machine learning. @xcite proposed a seq2seq model with copy mechanism which transforms an input sentence into a new sentence with spelling errors corrected. more recently, bert @xcite , the language representation model, is successfully applied to many language understanding tasks including csc (cf., @xcite ). in the state-ofthe-art method using bert, a character-level bert is first pre-trained using a large unlabelled dataset and then fine-tuned using a labeled dataset. the labeled data can be obtained via data augmentation in which examples of spelling errors are generated using a large confusion table. finally the model is utilized to predict the most likely character from a list of candidates at each position of the given sentence. the method is powerful because bert has certain ability to acquire knowledge for language understanding. our experimental results show that the accuracy of the method can be further improved, however. one observation is that the error detection capability of the model is not sufficiently high, and once an error is detected, the model has a better chance to make a right correction. we hypothesize that this might be due to the way of pre-training bert with mask language modeling in which only about 15% of the characters in the text are masked, and thus it only learns the distribution of masked tokens and tends to choose not to make any correction. this phenomenon is prevalent and represents a fundamental challenge for using bert in certain tasks like spelling error correction. to address the above issue, we propose a novel neural architecture in this work, referred to as soft-masked bert. soft-masked bert contains two networks, a detection network and a correction network based on bert. the correction network is similar to that in the method of solely using bert. the detection network is a bi-gru network that predicts the probability that the character is an error at each position. the probability is then utilized to conduct soft-masking of embedding of character at the position. soft masking is an extension of conventional 'hard masking' in the sense that the former degenerates to the latter, when the probability of error equals one. the soft-masked embedding at each position is then inputted into the correction network. the correction network conducts error correction using bert. this approach can force the model to learn the right context for error correc-tion under the help of the detection network, during end-to-end joint training. we conducted experiments to compare soft-masked bert and several baselines including the method of using bert alone. as datasets we utilized the benchmark dataset of sighan. we also created a large and high-quality dataset for evaluation named news title. the dataset, which contains titles of news articles, is ten times larger than the previous datasets. experimental results show that soft-masked bert significantly outperforms the baselines on the two datasets in terms of accuracy measures. the contributions of this work include (1) proposal of the novel neural architecture soft-masked bert for the csc problem, (2) empirical verification of the effectiveness of soft-masked bert. ## related work various studies have been conducted on spelling error correction so far, which plays an important role in many applications, including search @xcite , optical character recognition (ocr) @xcite , and essay scoring @xcite . chinese spelling error correction (csc) is a special case, but is more challenging due to its conflation with chinese word segmentation, which received a considerable number of investigations @xcite @xcite . early work in csc followed the pipeline of error detection, candidate generation, and final candidate selection. some researchers employed unsupervised methods using language models and rules @xcite and the others viewed it as a sequential labeling problem and employed conditional random fields or hidden markov models @xcite . recently, deep learning was applied to spelling error correction @xcite , and for example, a seq2seq model with bert as encoder was employed @xcite , which transforms the input sentence into a new sentence with spelling errors corrected. bert @xcite is a language representation model with transformer encoder as its architecture. bert is first pre-trained using a very large corpus in a self-supervised fashion (mask language modeling and next sentence prediction). then, it is fine-tuned using a small amount of labeled data in a down-stream task. since its inception bert has demonstrated superior performances in almost all the language understanding tasks, such as those in the glue challenge @xcite . bert has shown strong ability to acquire and utilize knowledge for language understanding. recently, other language representation models have also been proposed, such as xlnet @xcite , roberta @xcite , and albert @xcite . in this work, we extend bert to soft masked bert for spelling error correction and as far as we know no similar architecture was proposed before. ## conclusion in this paper, we have proposed a novel neural network architecture for spelling error correction, more specifically chinese spelling error correction (csc). our model called soft-masked bert is composed of a detection network and a correction network based on bert. the detection network identifies likely incorrect characters in the given sentence and soft-masks the characters. the correction network takes the soft-masked characters as input and makes correction on the characters. the technique of soft-masking is general and potentially useful in other detection-correction tasks. experimental results on two datasets show that soft-masked bert significantly outperforms the state-of-art method of solely utilizing bert. as future work, we plan to extend soft-masked bert to other problems like grammatical error correction and explore other possibilities of implementing the detection network.