Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for C hinese Spell Checking (2022)

Abstract: Chinese Spell Checking (CSC) aims to detect and correct Chinese spelling errors. Recent researches start from the pretrained knowledge of language models and take multimodal information into CSC models to improve the performance. However, they overlook the rich knowledge in the dictionary, the reference book where one can learn how one character should be pronounced, written, and used. In this paper, we propose the LEAD framework, which renders the CSC model to learn heterogeneous knowledge from the dictionary in terms of phonetics, vision, and meaning. LEAD first constructs positive and negative samples according to the knowledge of character phonetics, glyphs, and definitions in the dictionary. Then a unified contrastive learning-based training scheme is employed to refine the representations of the CSC models. Extensive experiments and detailed analyses on the SIGHAN benchmark datasets demonstrate the effectiveness of our proposed methods.

## introduction as a crucial chinese processing task, chinese spell checking (csc) aims to detect and correct chinese spelling errors @xcite , which are mainly caused by phonetically or visually similar characters @xcite . recent researches propose to introduce phonetics and vision information to help pretrained language models (plms) deal with confusing characters @xcite @xcite . however, csc is challenging because it requires not only phonetics/vision information but also complex definition knowledge to assist in finding the truly correct character. as shown in to help people learn chinese, the meanings of chinese characters and words have been preorganized as the definition sentences in the dictionary. the dictionary contains a wealth of useful knowledge for csc, including character phonetics, glyphs, and definitions. it is also the most impor-tant resource for chinese beginners to learn how to pronounce, write, and use one character. inspired by this, we focus on utilizing the rich knowledge in the dictionary to improve the csc performance. in this paper, we propose lead, a unified finetuning framework to guide the csc models to learn heterogeneous knowledge from the dictionary. in general, lead has one training paradigm but three different training objectives besides the traditional csc objective. this enables models to learn three different kinds of knowledge, namely phonetics, vision, and definition knowledge. specifically, we construct various positive and negative samples according to the respective characteristics of different knowledge, and then utilize these generated sample pairs to train models with our designed unified contrastive learning paradigm. through the optimization of lead, the finetuned model handles various phonetically/visually similar character errors as well as previous multimodal models, and goes a further step to deal with more confusing errors with the help of the definition knowledge contained in the dictionary. additionally, lead is a model-agnostic fine-tuning framework, which has no restrictions on the finetuned models. in practice, we fine-tune bert and a more complex multimodal csc model @xcite with lead, and experimental results on the sighan datasets show consistent improvements. to summarize, the contributions of our work are in three folds: (1) we focus on the importance of the dictionary knowledge for the csc task, which is instructive for future csc research. (2) we propose the lead framework, which fine-tunes the models to learn heterogeneous knowledge beneficial to the csc task in a unified manner. ## methodology in this section, we first introduce the overview of the lead framework, as illustrated in figure 1 , and describe our designed unified contrastive learning mechanism for heterogeneous dictionary knowledge. then, for each knowledge-guided finetuning, we explain its motivation, positive/negative pairs construction, and representation metric which is used in the contrastive learning mechanism. ## experiments in this section, we first introduce the experiment settings and the main performance of lead. then we conduct detailed discussions and analyses to verify the effectiveness of our proposed methods. ## conclusion in this paper, we propose to promote csc by utilizing various knowledge contained in the dictionary. we introduce lead, a unified fine-tuning framework that aims to perform contrastive learning for three kinds of heterogeneous knowledge. extensive experiments and empirical analyses verify the motivation of our study and the effectiveness of our proposed methods. the dictionary knowledge we focus on is not only beneficial for csc, but also crucial for other chinese text processing tasks. therefore, in the future, we will continue to mine the knowledge contained in the dictionary to improve other chinese text processing tasks.