Disentangled Phonetic Representation for C hinese Spelling Correction (2023)

Abstract: Chinese Spelling Correction (CSC) aims to detect and correct erroneous characters in Chinese texts. Although efforts have been made to introduce phonetic information (Hanyu Pinyin) in this task, they typically merge phonetic representations with character representations, which tends to weaken the representation effect of normal texts. In this work, we propose to disentangle the two types of features to allow for direct interaction between textual and phonetic information. To learn useful phonetic representations, we introduce a pinyin-to-character objective to ask the model to predict the correct characters based solely on phonetic information, where a separation mask is imposed to disable attention from phonetic input to text. To avoid overfitting the phonetics, we further design a self-distillation module to ensure that semantic information plays a major role in the prediction. Extensive experiments on three CSC benchmarks demonstrate the superiority of our method in using phonetic information.

## introduction chinese spelling correction (csc) is a task to detect and correct erroneous characters in chinese sentences, which plays an indispensable role in many natural language processing (nlp) applications @xcite . previous research @xcite shows that the misuse of homophonic characters accounts for roughly 83% of the spelling errors. we present two such cases in source 可是我忘了，我真户秃(hu tu)。 but i forgot, i am so household bald. target 可是我忘了，我真糊涂(hu tu)。 but i forgot, i am so silly. bert 可是我忘了，我真护突。 pinyinbert 可是我忘了，我真糊涂。 realise 可是我忘了，我真户涂。 our dorm 可是我忘了，我真糊涂。 source 可是现在我什么事都不济的(ji de)。 but i can't do anything right now. target 可是现在我什么事都不记得(ji de)。 but i don't remember anything now. bert 可是现在我什么事都不记得。 pinyinbert 可是现在我什么事都不记的。 realise 可是现在我什么事都不记的。 our dorm 可是现在我什么事都不记得。 as the pronunciation similarity of chinese characters is primarily determined by their initial or final sounds rather than their tones, we focus solely on the initials and finals as the phonetic features of chinese characters. as pre-trained language models like bert @xcite have dominated various nlp tasks, researchers explore incorporating pinyin features into pre-trained language models for the csc task. there are mainly two approaches. first, the pinyin of a chinese character is encoded and fused into the character representation with a gate mechanism @xcite @xcite . second, a pronunciation prediction objective is introduced to model the relationship among phonologically similar characters @xcite @xcite . despite considerable performance gain, these methods suffer from two potential issues. first, pinyin information may be neglected or dominated by textual information during training because of the entanglement between pinyin and textual representations. as the first case shows in based on the above observations, we propose disentangled phonetic representation model (dorm) for csc. our motivation is to decouple text and pinyin representations to allow for direct interaction between them to make better use of phonetic information. specifically, we first construct a phonetics-aware input sequence by appending the pinyin sequence to the original textual input, where a common set of position embeddings is used to relate the two sub-sequences. in doing so, textual features are allowed to capture phonetic information as needed from the pinyin part during training and inference. then, to learn useful pinyin representations, we introduce a pinyin-to-character prediction objective, where a separation mask is imposed to disallow attention from pinyin to text to ask the model to recover the correct characters only from pinyin information. the pinyin-to-character task is auxiliary during training and its prediction will be discarded at inference time. intuitively, pinyin should serve to complement but not replace textual information in csc for two reasons. first, there is a one-to-many relation be-tween pinyin and chinese characters, and it is more difficult to recover the correct characters solely from pinyin than from chinese characters. second, pinyin representations are not pre-trained as textual representations in existing language models. therefore, the model should avoid overly relying on pinyin which may cause overfitting. inspired by deep mutual learning @xcite and self-distillation @xcite , we propose a self-distillation module to force the prediction of our model to be consistent with that when a rawtext input is supplied. to this end, kl-divergence is applied to the two sets of soft labels. experiments are conducted on three sighan benchmarks and the results show that our model achieves substantial performance improvement over state-of-the-art models. further analysis demonstrates that phonetic information is better utilized in our model. the contributions of this work are summarized threefold. first, we disentangle text and pinyin representations to allow for direct interaction between them. second, we introduce a pinyin-to-character task to enhance phonetic representation learning with a separation mask imposed to disable attention from pinyin to text. third, a self-distillation module is proposed to prevent overreliance on phonetic features. through this work, we demonstrate the merit of our approach to modeling pinyin information separately from the text. ## methodology the motivation of our disentangled phonetic representation model (dorm) for chinese spelling correction (csc) is to allow for direct and deep interaction between textual and phonetic fea-tures by decoupling chinese character and pinyin representations. to enable effective pinyin representations, we introduce a pinyin-to-character objective that requires the model to restore the correct characters purely from pinyin information. inspired by deep mutual learning @xcite and self-distillation @xcite , we further introduce a self-distillation module to prevent the model from overfitting pinyin information. in the following, we first formulate the task ( §3.1) and then introduce dorm in detail ( §3.2). finally, we introduce how to pre-train the model for better textual and pinyin representations ( §3.3). ## experiments in this section, we introduce the details of our experiments to evaluate the proposed model. ## analysis and discussion in this section, we further analyze and discuss our model quantitatively and qualitatively. ## conclusion in this paper, we propose dorm in an attempt to improve the effect of using phonetic knowledge in chinese spelling correction (csc). to this end, we propose to disentangle textual and phonetic features and construct a phonetics-aware input to allow for direct interaction between them. we also introduce a pinyin-to-character objective to force the model to recover the correct characters based solely on pinyin information, where a separation mask is applied to prevent exposing textual information to phonetic representations. besides, we propose a novel self-distillation module for dorm to avoid overfitting pinyin information. extensive experiments on three widely-used csc datasets show that this model outperforms existing stateof-the-art baselines. detailed analysis and studies show that direct interaction between characters and pinyin is beneficial to better restore misspelled characters. through this work, we demonstrate the merit of disentangling phonetic features from textual representations when solving csc. ying zhang, tao xiang, timothy m. hospedales, and huchuan lu. 2018. deep mutual learning. in proceedings of the ieee conference on computer vision and pattern recognition (cvpr). chenxi zhu, ziqiang ying, boyu zhang, and feng mao. 2022. mdcspell: a multi-task detector-corrector framework for chinese spelling correction. in findings of the association for computational linguistics: acl 2022, pages 1244-1253, dublin, ireland. association for computational linguistics.