WS peller: Robust Word Segmentation for Enhancing C hinese Spelling Check (2022)

Abstract: Chinese spelling check (CSC) detects and corrects spelling errors in Chinese texts. Previous approaches have combined character-level phonetic and graphic information, ignoring the importance of segment-level information. According to our pilot study, spelling errors are always associated with incorrect word segmentation. When appropriate word boundaries are provided, CSC performance is greatly enhanced. Based on these findings, we present WSpeller, a CSC model that takes into account word segmentation. A fundamental component of WSpeller is a W-MLM, which is trained by predicting visually and phonetically similar words. Through modification of the embedding layer’s input, word segmentation information can be incorporated. Additionally, a robust module is trained to assist the W-MLM-based correction module by predicting the correct word segmentations from sentences containing spelling errors. We evaluate WSpeller on the widely used benchmark datasets SIGHAN13, SIGHAN14, and SIGHAN15. Our model is superior to state-of-the-art baselines on SIGHAN13 and SIGHAN15 and maintains equal performance on SIGHAN14.

## introduction chinese spelling check (csc) aims to detect and correct spelling errors in chinese text, and can be used in many other natural language processing (nlp) applications, including search optimization @xcite , location extraction @xcite , optical character recognition (ocr) @xcite , automatic speech recognition (asr) @xcite ) and text classification @xcite . typos in the text are mostly phonetic or graphic of the correct character @xcite work focus on how to use the features of pronunciation and shape @xcite . however, little attention has been devoted to the incorrect word segmentation induced by typos. as shown in to cope with the problem, we propose wspeller (word speller), which consists of two submodules, namely the word segmentation module (ws-module) and the correction module (c-module). the ws-module predicts the right word boundaries based on the original text (may have typos). following that, the segment-level information is provided to c-module for improving correction. the two submodules adopt different text encoding modules, and the hidden state of the ws-module is passed to the c-module, where the two submodules interact and are trained in a multi-task manner for mutual benefit. w-mlm (word-masked language model) is proposed as the text encoding module of c-module because it adapts bert @xcite with different embedding information and special replacement strategies and pre-trained on a large chinese corpus. wspeller is evaluated against three widely used datasets: sighan13 @xcite , sighan14 @xcite , @xcite . the experimental results show that wspeller improves the performance of csc. wspeller increased the f1 scores by 3% and 2.1% in sighan13 and sighan15 and achieved almost the same performance as the state-of-the-art on sighan14. ablation studies reveal that both the word segmentation and the w-mlm contribute considerably to the performance. the contribution of this paper is summarized as follows: ## related work in recent years, csc has received widespread attention. early works focus on rules and confusion sets to deal with csc, follows pipelines, including detection, candidate generation and selection @xcite @xcite . these methods are limited by manually set rules and fixed confusion sets. with the development of deep learning, most methods focus on how to integrate the phonetic and graphic features of characters into the model. hong et al. @xcite proposed faspell, which exploited the phonetic and graphic information to exclude candidates with low similarity. cheng et al. @xcite proposed spell-gcn, which employed gcn to combine the phonetic and graphic information as the output layer. ji et al. @xcite proposed spellbert, which used gcn to model the pinyin and radical of characters as the embedding layer. wang et al. @xcite proposed dcn, which added phonetic embedding when generating candidates, used attention mechanism to calculate the fraction between adjacent characters, and obtained the best path. bao et al. @xcite proposed to use semantic token embedding token embedding segment embedding segment embedding position embedding position embedding embedding of original bert token embedding segment embedding position embedding embedding of original bert token embedding token embedding word embedding word embedding position embedding position embedding embedding of preliminary experiment token embedding word embedding position embedding embedding of preliminary experiment candidates to expand the confusion set and use a block-based structure to correct errors. some methods focus on training skills and the positive impact of detection on correction. zhang et al. @xcite adopted gru as detection network and bert as correction network, and propose a soft masking strategy. gan et al. @xcite propose to judge the difficulty of learning samples through loss and apply self-supervised curriculum learning to csc. in recent years, the methods focus on integrating the similarity between characters into the model. however, they ignore the importance of word segmentation to csc. in this paper, the word segmentation information of text is integrated into the model through pre-training and fine-tuning. only a simple model structure can achieve a good correction effect. ## preliminary experiment our primary premise is that the presence of typos hurts word segmentation and the ability correct would improve if better word boundaries were available. to validate this, we conduct a preliminary experiment in which we inject the precise word segmentation into bert. as shown in figure 1 , the boundaries of words are represented by 0s and 1s, where 1 indicates segmentation is required before the current character. the word segmentation is obtained by lac foot_0 . the result is shown in we, therefore, propose wspeller, which includes word segmentation information for csc. however, obtaining correct word boundaries for sentences containing typos is a challenge. wspeller has been further enhanced with a robust ws-module so that it can predict the word boundaries correctly. ## problem definition csc aims to detect and correct typos in a given text @xmath0, x 2 , • • • , x n ), and finally get the detec- ## the proposed model in this section, we present the wspeller, whose overall architecture is illustrated in figure 2 . wspeller consists of a ws-module (5.1) and a c-module (5.3). while the ws-module aims to predict the correct word boundaries in spite of spelling errors, the c-module aims to correct the typos in conjunction with the ws-module. to fully exploit the word segmentation information, the c-module uses a pre-trained language model w-mlm (5.2). ## limitations in our study, we discovered that correct word segmentation would improve the ability of correction, but achieving accurate word segmentation from source texts that contain typos remains a challenge. however, although wspeller is capable of segmenting words with high accuracy, there is still a lot of room for improvement. by having more accurate word boundaries available, the model's correction capabilities can be further enhanced, as the upper bound in the preliminary experimental results in section 3. as wspeller encodes text twice, its inference speed is limited. a future attempt will be made to improve ws-module to reduce its complexity.