Training a Better C hinese Spelling Correction Model via Prior-knowledge Guided Teacher (2024)

Abstract: Recent advancements in Chinese Spelling Correction (CSC) predominantly leverage pre-trained language models (PLMs). However, a notable challenge with fine-tuned PLM-based CSC models is their tendency to over-correct, leading to poor generalization for error patterns outside the standard distribution. To address this, we developed a teacher network guided by prior knowledge for distillation learning of CSC models. Unlike traditional teacher networks, which depend on task-related pre-training, our method infuses task-related prior information into the teacher network, offering guidance beyond mere labels to the student network. This strategy significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. Importantly, our approach is model-independent and the teacher network does not require task-related pre-training, making it broadly applicable for enhancing various PLM-based CSC models with minimal additional computational resources. Extensive experiments on widely used benchmarks demonstrate that our method achieves new state-of-the-art results. Additionally, we explored the potential of generalizing our method to other non-autoregressive text-generation tasks.

## introduction the primary objective of chinese spelling correction is to detect and correct spelling errors in chinese sentences, a task that holds substantial utility in various applications. csc can play a pivotal role in enhancing tasks such as optical character recognition, text editing, and speech recognition @xcite @xcite . moreover, it acts as an invaluable tool for individuals learning a new language, enabling quicker and more economical learning processes. the importance of csc spans numerous fine-tuned bert 哪里有上大学，不想念书的道理？ is there any reason to go to college and not want to study? ## methodology in this section, we provide a comprehensive description of our proposed prior-knowledge guided teachers network (pgt), as depicted in figure 2 . pgt adopts a teacher network guided by taskrelated prior knowledge instead of task-related pretrained teacher networks. pgt offers several key advantages. firstly, our approach does not necessitate any alterations to the input or structure of the csc model, allowing for the training of an unbiased csc model from real human data. secondly, the model-independent of pgt facilitates superior ## experiments this section will give a comprehensive overview of the data, settings, and outcomes of the experiments in this paper. moreover, we will perform essential analysis and discussion to show the merits of our approach. ## conclusion building on the idea of augmenting the language modeling capabilities of the csc model, this paper proposes to use a prior-knowledge guided teacher (pgt) network to distill the csc model. extensive experiments conducted on widely used benchmarks affirm that our method achieves new state-of-the-art results on the csc task. furthermore, the experiment results validate that pgt indeed enhances the language modeling capabilities of the csc model. in terms of knowledge distillation, pgt outperforms general kd schemes in both performance and training efficiency for csc tasks, while also demonstrating notable model compression efficiency. lastly, we verify the potential of pgt to generalize to other non-autoregressive textgeneration tasks through a simple experiment.