The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for C hinese Spell Checking (2022)

Abstract: Chinese Spell Checking (CSC) aims to detect and correct Chinese spelling errors, which are mainly caused by the phonological or visual similarity. Recently, pre-trained language models (PLMs) promote the progress of CSC task. However, there exists a gap between the learned knowledge of PLMs and the goal of CSC task. PLMs focus on the semantics in text and tend to correct the erroneous characters to semantically proper or commonly used ones, but these aren’t the ground-truth corrections. To address this issue, we propose an Error-driven COntrastive Probability Optimization (ECOPO) framework for CSC task. ECOPO refines the knowledge representations of PLMs, and guides the model to avoid predicting these common characters through an error-driven way. Particularly, ECOPO is model-agnostic and it can be combined with existing CSC methods to achieve better performance. Extensive experiments and detailed analyses on SIGHAN datasets demonstrate that ECOPO is simple yet effective.

## introduction chinese spell checking (csc) aims to detect and correct spelling errors in chinese texts @xcite . it is a crucial research field for various nlp downstream applications, such as optical character recognition @xcite , search query correction @xcite and essay scoring @xcite . however, csc is also very challenging because it mainly suffers from confusing characters, such as phonologically and visually similar characters @xcite . as illustrated in figure 1 , "素(sù, plain)" and "诉(sù, sue)" are confusing characters for each other due to the shared pronunciation "sù". figure 1 : examples of chinese spelling errors. previous research @xcite shows that 83% of errors belong to phonological error and 48% belong to visual error. we give the characters with their pronunciation and translation. we mark the input confusing/golden/common candidate characters in red/blue/orange. the characters in "candidate" sentences are all predicted by fine-tuned bert. recently, pre-trained language models (plms) such as bert @xcite have been utilized in the csc task and became mainstream solutions @xcite . however, there exists a significant gap between the learned knowledge of plms and the goal of csc task. plms provide informative representations from the perspective of semantics, but if only considering the semantics in csc, there are multiple appropriate characters as the correction. without the constraints of phonological and visual similarities, plms easily predict semantically proper or common characters due to the masking strategy in the pre-training procedure. figure 1 presents two predictions of bert to better understand the gap mentioned before. the first example is caused by the misuse of "素(sù, plain)" and "诉(sù, sue)". an ideal csc model should pay attention to the pronunciation information "sù" and output the golden character "诉(sue)" as a correction for the input confusing character. however, as pre-trained on general corpora, bert tend to predict semantically proper characters, such as "争(zhēng, fight)", "谋(móu, plan)" and "获(huò, acquire)". these characters are also from more commonly used phrases. in the second example, bert also overlooks the visual similarity between "解(jiě, understand)" and "懈(xiè, slack)", resulting in wrong correction. to alleviate this gap, we propose to empower the plms to avoid predicting the above-mentioned common characters by optimizing the knowledge representation of plms. intuitively, if we guide the model to not make the same mistakes it would prone to make before, the model performance should be improved. hence, the mistakes that the model has ever made can be utilized as constraints on the knowledge representation of the model. in other words, we exploit the past mistakes that the model may make to further enhance the model itself, this is the meaning of our title, "the past mistake is the future wisdom". ## methodology in this section, we introduce the proposed ecopo in details, as illustrated in figure 2 . ecopo aims to refine the knowledge representation of plms to narrow the gap between it and the essential of csc task. as mentioned in section 1, with the model before our optimization process, we select the mistakes generated by this model itself to be the negative samples. then through the contrastive probability optimization objective, we maximize the prediction probabilities of the model for correct answers and minimize the prediction probabilities of the model for negative samples. in this errordriven way, the original prediction probabilities of the model are refined, improving the performance of the model on the csc task. therefore, the model will grow and progress after making mistakes again and again, just as humans do. note that the proposed ecopo is a model-agnostic framework, we can choose different plms or csc models to be optimized in practice for better performance. ## experiments in this section, we introduce the details of experiments and main results we obtained. then we conduct detailed analyses and discussions to verify the effectiveness of our method. ## conclusion in this paper, we introduce to promote csc by narrowing the gap between the knowledge of plms and the goal of csc. we propose the ecopo, a simple yet effective training framework that aims to perform an error-driven optimization for the plms based on their original probability representation. extensive experiments and empirical results show the competitive performance of our method. in the future, we will study how to automatically measure the quality of negative samples to further enhance our method. additionally, applying our core idea to other tasks will be an interesting direction.