Detection-Correction Structure via General Language Model for Grammatical Error Correction (2024)

Abstract: Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits, which can be decoupled into two components: detection and correction. However, previous works have predominantly focused on direct correction, with no prior efforts to integrate both into a single model. Moreover, the exploration of the detection-correction paradigm by large language models (LLMs) remains underdeveloped. This paper introduces an integrated detection-correction structure, named DeCoGLM, based on the General Language Model (GLM). The detection phase employs a fault-tolerant detection template, while the correction phase leverages autoregressive mask infilling for localized error correction. Through the strategic organization of input tokens and modification of attention masks, we facilitate multi-task learning within a single model. Our model demonstrates competitive performance against the state-of-the-art models on English and Chinese GEC datasets. Further experiments present the effectiveness of the detection-correction structure in LLMs, suggesting a promising direction for GEC.

## introduction grammatical error correction (gec) is a task focused on automatically rectifying grammatical errors in human-written text @xcite . gec models are applied in language learning @xcite @xcite , enhancing automatic speech recognition @xcite , and aiding in text data labeling @xcite . the two primary approaches in gec are sequence-to-sequence (seq2seq) and sequenceto-edit (seq2edit). without detection, seq2seq treats gec as the direct generation for correct text, providing high flexibility @xcite . on the other hand, seq2edit views gec as a sequence labeling task for edit la-finally, i will have high salary when i am journalist. source finally, i will have @xcite high salary when i am @xcite . ## methods our proposed model leverages the design of the glm. given a sentence with mask tokens, glm utilizes autoregressive blank infilling @xcite to generate a corresponding segment for each mask position. these segments are termed as text pieces. this section describes how glm is utilized to integrate detection and correction into a single model, as depicted in figure 2 . additionally, the design of multi-task training is also outlined here. ## experiments 4.1 datasets and evaluation for the english gec task, we evaluate the performance on the conll-14 test set @xcite using the m 2 scorer @xcite , and on the bea-19 test set @xcite using the errant scorer @xcite . the model is pretrained on synthetic dataset c4-200m @xcite and fine-tuned on the cleaned lang8 dataset (clang8) @xcite . for the large version of coglm model, we utilize smaller datasets including fce @xcite , nucle @xcite , and w&i+locness @xcite for fine-tuning, following @xcite . the bea-19 dev set is used for model selection. for the chinese gec task, we synthesize pretraining data from the people's daily corpus foot_1 using rule-based insertion, replacement, and deletion. the models are fine-tuned on the chinese lang8 dataset @xcite and the hsk dataset, following @xcite , and on the fcgec training set, respectively. the models are evaluated on mucgec and fcgec test sets using cher-rant @xcite . further details are provided in appendix a. ## conclusion we introduce a novel language-agnostic detectioncorrection structure via glm for the gec task. the structure employs a three-label error detection pattern and uses focal loss for aggressive detection. the correction phase leverages the maskinfilling capability of glm to generate correct text pieces. a multi-task learning approach is designed to integrate both functionalities within the same model, optimized using a weighted loss function. experimental results show proposed model decoglm outperforms previous detectioncorrection structures and achieves f 0.5 scores comparable to sota on english and chinese gec benchmarks. the effectiveness of the detectioncorrection structure is further validated by applying it to open-source llms and gpt-4, indicating that incorporating error detection information improves the performance of llms on gec datasets by reducing over-correction. ablation studies confirm the efficacy of our model design and the ability to trade off precision and recall can be realized by detection control. we aim for this work to further guide gec research within the detectioncorrection paradigm. the code and related models are available at https://github.com/gmagoleway/gecframework .