Improving C hinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity (2022)

Abstract: Chinese spelling check (CSC) is a fundamental NLP task that detects and corrects spelling errors in Chinese texts. As most of these spelling errors are caused by phonetic similarity, effectively modeling the pronunciation of Chinese characters is a key factor for CSC. In this paper, we consider introducing an auxiliary task of Chinese pronunciation prediction (CPP) to improve CSC, and, for the first time, systematically discuss the adaptivity and granularity of this auxiliary task. We propose SCOPE which builds upon a shared encoder two parallel decoders, one for the primary CSC task and the other for a fine-grained auxiliary CPP task, with a novel adaptive weighting scheme to balance the two tasks. In addition, we design a delicate iterative correction strategy for further improvements during inference. Empirical evaluation shows that SCOPE achieves new state-of-the-art on three CSC benchmarks, demonstrating the effectiveness and superiority of the auxiliary CPP task. Comprehensive ablation studies further verify the positive effects of adaptivity and granularity of the task.

## introduction chinese spelling check (csc), which aims to detect and correct spelling errors in chinese texts, is a fundamental task in chinese natural language processing. spelling errors mainly originate from human writing errors and machine recognition errors, e.g., errors caused by automatic speech recognition (asr) and optical character recognition (ocr) systems @xcite . with the latest development of deep neural networks, neural csc methods, instance similarity coarse fine w: 我觉得你们会好好的完(wan2/w,an,2)。 1 1 i think you will finish well. r: 我觉得你们会好好的玩(wan2/w,an,2)。 i think you will play well. w:我以前想要高(gao1/g,ao,1)诉你。 0 2 /3 i tried to high you before. r: 我以前想要告(gao4/g,ao,4)诉你。 i tried to tell you before. w:他收(shou1/sh,ou,1)到山上的时候。 0 1 /3 when he received the mountain. r: 他走(zou3/z,ou,3)到山上的时候。 when he walked up the mountain. w:行为都被蓝(lan2/l,an,2)控设备录影。 0 0 actions are recorded by blue control devices. r: 行为都被监(jian1/j,ian,1)控设备录影。 actions are recorded by surveillance devices. in particular those based on encoder-decoder architectures, have become the mainstream of research in recent years @xcite . encoder-decoder models regard csc as a special sequence-to-sequence (seq2seq) problem, where a sentence with spelling errors is given as the input and a corrected sentence of the same length will be generated as the output. previous research has shown that about 76% of chinese spelling errors are induced by phonological similarity @xcite . hence, it is a crucial factor to effectively model the pronunciation of chinese characters for the csc task. in fact, almost all current advanced csc approaches have actually exploited, either explicitly or implicitly, character pronunciation. the implicit use takes into account phonological similarities between pairs of characters, e.g., by increasing the decoding probability of characters with similar pronunciation @xcite or integrating such similarities into the encoding process via graph convolutional networks (gcns) @xcite . the explicit use considers directly the pronunciation, or more specifically, pinyin foot_0 , of individual characters, encoding the pinyin of input characters to produce extra phonetic features @xcite or decoding the pinyin of target correct characters to serve as an auxiliary prediction task @xcite . this paper also considers improving csc with auxiliary character pronunciation prediction (cpp), but focuses specifically on the adaptivity and granularity of the auxiliary task, which have never been systematically studied before. first, all the prior attempts in similar spirit simply assigned a universal trade-off between the primary and auxiliary tasks for all instances during training, while ignoring the fact that the auxiliary task might provide different levels of benefits given different instances. take for example the instances shown in based on the above intuitions we devise scope (i.e., spelling check by pronunciation prediction), which introduces a fine-grained cpp task with an adaptive task weighting scheme to improve csc. figure 1 provides an overview of scope. given a sentence with spelling errors as input, we encode it using chinesebert @xcite to produce semantic and phonetic features. then we build on top of the encoder two parallel decoders, one to generate target correct characters, i.e., the primary csc task, and the other to predict the initial, final and tone of the pinyin of each target character, i.e., the auxiliary fine-grained cpp task. the trade-off between the two tasks can be further adjusted adaptively for each instance, according to the phonological similarity between input and target characters therein. in addition, we design an iterative correction strategy during inference to address the over-correction issue and tackle difficult instances with consecutive errors. we empirically evaluate scope on three shared benchmarks, and achieve substantial and consistent improvements over previous state-of-the-art on all three benchmarks, demonstrating the effectiveness and superiority of our auxiliary cpp task. comprehensive ablation studies further verify the positive effects of adaptivity and granularity of the task. the main contributions of this paper are summarized as follows: (1) we investigate the possibility of introducing an auxiliary cpp task to improve csc and, for the first time, systematically discuss the adaptivity and granularity of this auxiliary task. ## related work csc is a fundamental nlp task that has received wide attention over the past decades. early work on this topic was mainly based on manually designed rules @xcite . after that, statistical language models became the mainstream for csc @xcite @xcite . methods of this kind in general followed a pipeline of error detection, candidate generation, and candidate selection. given a sentence, the error positions are first detected by the perplexity of a language model. the candidates for corrections can then be generated according to similarity between characters, typically by using figure 1 : overview of scope. top: the one-encoder-two-decoder structure for csc and cpp. the input sentence x is fed into the encoder and then, after character-/pronunciation-specific feature projection, two parallel decoders, one to predict the characters, the other to predict the initial, final, and tone of each character in the target sentence. bottom: adaptive task weighting between csc and cpp (detached in the backward pass). the target sentence y is fed into the encoder and the pronunciation-specific feature projection layer. then the similarities between input and target sentences on character level are calculated and the adaptive weights are accordingly defined. note: only the csc decoder branch (along with the encoder) will be used at inference time. a confusion set. and the final corrections can be determined by scoring the sentence replaced by the candidates with the language model @xcite . in the era of deep learning, especially after transformer @xcite and pre-trained language models like bert @xcite were proposed, a large number of neural csc methods have emerged. @xcite used transformer as an encoder to produce candidates and designed a confidence-similarity decoder to filter these candidates. @xcite designed a detection network based on bi-gru to predict the error probability of each character and passed the probabilities to a bert-based correction network via a soft masking mechanism. @xcite employed gcns combined with bert to further model interdependences between characters. recent work of @xcite @xcite proposed to encode phonetic and glyph information in addition to semantic information, and then combine phonetic, glyph and semantic features to make final predictions. as we could see, modeling pronunciation information is prevailing in csc research @xcite , typically via an encoding process to extract phonetic features. @xcite proposed the first work that considered predicting the pronunciation of target characters as an auxiliary task. their work, however, employed pronunciation prediction in a coarse-grained, non-adaptive manner, which is quite different to ours. ## our approach this section presents our approach scope for the csc task. below, we first define the problem formulation and then describe our approach in detail. ## experiments and results in this section, we introduce our experiments and results on sighan benchmarks @xcite @xcite . we then verify the effectiveness of our model design, in particular the adaptivity and granularity of the auxiliary cpp task, via extensive ablation studies and analyses. ## conclusions this paper proposes scope, which employs a finegrained chinese pronunciation prediction (cpp) task with adaptive task weighting to improve the performance of chinese spelling check (csc). our method builds upon a shared encoder two parallel decoders, one to predict target characters i.e., csc, and the other to predict initials, finals, and tones of target characters, i.e., fine-grained cpp. the two decoders are then balanced adaptively according to the phonetic similarity between input and target characters. an iterative correction strategy is further designed during inference. scope establishes new state-of-the-art on three sighan benchmarks, verifying the effectiveness and superiority of introducing an auxiliary cpp task to improve csc. extensive ablation studies further verify the positive effects of dynamic adaptivity and fine granularity of this auxiliary task.