Alirector: Alignment-Enhanced C hinese Grammatical Error Corrector (2024)

Abstract: Chinese grammatical error correction (CGEC) faces serious overcorrection challenges when employing autoregressive generative models such as sequence-to-sequence (Seq2Seq) models and decoder-only large language models (LLMs). While previous methods aim to address overcorrection in Seq2Seq models, they are difficult to adapt to decoder-only LLMs. In this paper, we propose an alignment-enhanced corrector for the overcorrection problem that applies to both Seq2Seq models and decoder-only LLMs. Our method first trains a correction model to generate an initial correction of the source sentence. Then, we combine the source sentence with the initial correction and feed it through an alignment model for another round of correction, aiming to enforce the alignment model to focus on potential overcorrection. Moreover, to enhance the model’s ability to identify nuances, we further explore the reverse alignment of the source sentence and the initial correction. Finally, we transfer the alignment knowledge from two alignment models to the correction model, instructing it on how to avoid overcorrection. Experimental results on three CGEC datasets demonstrate the effectiveness of our approach in alleviating overcorrection and improving overall performance. Our code has been made publicly available.

## introduction chinese grammatical error correction (cgec) @xcite , which aims to identify and correct potential grammatical errors in given chinese sentences while adhering to the principle of minimal editing, has broad applications in scenarios such as writing assistant and search engine @xcite . chinese grammatical errors can be basically categorized into component missing, component redundancy, improper collocation, and improper word order @xcite , which are 我校采取了一系列卓有成效的改进方法。 我校采取了一系列卓有成效的改进措施。 我们采取了一系列卓有成效的解决措施。 source sentence ## methodology as depicted in figure 3 , our alignment-enhanced corrector (alirector) for chinese grammatical error correction (cgec) comprises three main steps to build. first, we train a correction model to generate an initial correction of the source sentence. second, we perform bidirectional alignment by combining the source sentence with the initial correction forward and backward respectively, and passing each combination through an alignment model for another round of correction. third, we employ knowledge distillation to transfer the knowledge from the two alignment models to the correction model. in the following sections, we first formulate the cgec task and introduce the correction model in section 3.1. then, we delve into the details of the alignment models in section 3.2 and specify the knowledge distillation in section 3.3. ## analysis 5.1 overcorrection mitigation to further verify the effectiveness of alirector in mitigating overcorrection, we use baichuan2-7b as the backbone and present fine-grained precision results across the four categories of cgec errors, including missing (m), redundant (r), substitution ## conclusion in this paper, we first investigate a predict-and-align method that effectively leverages alignment information between the source sentence and the initial correction to alleviate the overcorrection issue in cgec. then, we propose transferring knowledge from the alignment process to enhance the correction model, resulting in an improved model termed alirector. experimental results on three cgec datasets showcase the efficacy of our approach in mitigating overcorrection for both seq2seq models and decoder-only llms. detailed analysis further demonstrates the effectiveness of this method across various error types, as well as the pivotal role of alignment in enhancing performance. broadly speaking, the overcorrection challenge falls within the realm of uncontrollability of generative language models. besides straightforward efforts to acquire more high-quality training data or employ specific pre-training strategies such as bart, this study introduces an alignment-based method that has demonstrated effectiveness in addressing this issue. despite the improvement of our approach for decoder-only llms, their performance in cgec still lags behind that of the strongest seq2seq models, even though they are smaller in size, which contradicts their outstanding performance in other nlp tasks. in future research, we will further exploring enhancing the performance of decoder-only llms for cgec.