LET : Leveraging Error Type Information for Grammatical Error Correction (2023)

Abstract: Grammatical error correction (GEC) aims to correct errors in given sentences and is significant to many downstream natural language understanding tasks. Recent work introduces the idea of grammatical error detection (GED) to improve the GEC task performance. In contrast, these explicit multi-stage works propagate and amplify the problem of misclassification of the GED module. To introduce more convincing error type information, we propose an end-to-end framework in this paper, which Leverages Error Type (LET) information in the generation process. First, the input text is fed into a classification module to obtain the error type corresponding to each token. Then, we introduce the category information into the decoder’s input and cross-attention module in two ways, respectively. Experiments on various datasets show that our proposed method outperforms existing methods by a clear margin.

## introduction the grammatical error correction (gec) task aims to correct grammatical errors in natural language texts, including spelling, punctuation, grammar, word selection, and more. as shown in figure 1 , a gec model receives text containing errors and produces its corrected version. current gec algorithms are mainly divided into two categories: detection-based models and end-toend generative models. detection-based models treat gec as a token classification problem @xcite . by classifying each token in the sentence, we can make a detailed transformation according to the classification result to obtain the modified sentence. this method has strong interpretability and a transparent error correction process. however, to achieve precise error correction, it is necessary first to identify and classify all possible grammatical * these authors contributed equally to this work. to avoid manually designing wrong categories and labeling data, many works @xcite have built end-toend generative gec systems from the perspective of machine translation (mt), which is also the current mainstream method. in this approach, erroneous sentences correspond to the source language, and error-free sentences correspond to the target language. most recent generative models @xcite are based on the transformer encoder-decoder architecture @xcite . they also achieve competitive results compared to detection-based models. the most significant advantage of the end-toend generative model is that we do not need to design complex error categories manually or perform labor-intensive labeling work on the data. we can only use parallel corpora to train the model. recent works @xcite have shown that if the error type results obtained in the grammatical error detect (ged) task are introduced into the generative model in some form, the error correction ability of the model will be further improved. this is because the entire training and inference process can be viewed as a black-box operation in an end-to-end generative model. furthermore, the model can generate more accurate results if additional information guides this process (e.g., the classification result of some location is "delete"). @xcite extends the transformer encoder-decoder model based on introducing error type information. they classify input tokens into different error types, transform them into representations, and feed them into the decoder's crossattention module. however, this method suffers from two fundamental limitations: therefore, to solve the above problems, we propose a simple yet novel generative model to improve the performance of gec, termed let (leveraging error type information). first, we utilize the intermediate representation of the error type classification module as the error type vector. it would not discard the probabilities of other classes, even if their values are small. this operation ensures more convincing guidance of the type vectors to the generated modules. second, to discard the mismatch in the crossattention module, we transfer the input from the previous sub-layer in the decoder to the classification vector. thus, both parts of the input are in the same semantic space. therefore, the crossattention for them is more reasonable. in summary, our contributions can be summarized in the following points: ## related work much progress in the gec task can be attributed to transforming the problem into a machine translation task @xcite ) from an ungrammatical source sentence to a grammatical target sentence. early gec-mt methods leveraged phrase-based statistical machine translation (pb-smt) @xcite . with the rapid development of related work on machine translation, statistical machine translation (smt) and neural machine translation (nmt) have been successfully applied to various task-specific adaptations of gec @xcite @xcite with the introduction of transformer architectures, this approach rapidly evolved to powerful transformerbased seq2seq models @xcite . transformer-based models autoregressively capture the complete dependency among output tokens @xcite . @xcite leveraged a transformer model pre-trained on synthetic gec data. several improvement strategies of bert were also adopted in the gec model @xcite . with the development of large-scale pre-trained models recently, @xcite built their system on top of t5 @xcite and reached new state-of-the-art results. grammatical error detection is usually formulated as a sequence tagging task, where each erroneous token is assigned with an error type, e.g., selection errors and redundant words. early ged methods mainly used rules to identify specific sentence error types, such as preposition errors @xcite . with the development of neural networks, @xcite presented the first work using a neural approach and framed ged as a binary sequence labeling problem, classifying each token in a sentence as correct or incorrect. sequence labeling methods are widely used for ged, such as feature-based statistical models @xcite and neural models @xcite . due to the effectiveness of bert @xcite in many other nlp applications, recent studies adopt bert as the basic architecture of ged models @xcite . recent work has attempted to explore a different approach to using ged in gec, which aims to use the detection results of ged to guide gec generation. @xcite introduced token-level and sentence-level ged as auxiliary tasks when training for gec. @xcite employed multitask learning to utilize the detection results of ged to guide gec generation. similarly, chen et al. (2020) fine-tuned roberta @xcite for ged and improved the efficiency for gec by dividing the task into two sub-tasks: erroneous span detection and erroneous span correction. @xcite treated ged as a sequence labeling task and gec as a sequence-to-sequence task and additionally investigated ways to use multiclass ged predictions to inform gec. ## method in this section, we first describe the problem definition and the basic model, our baseline. then we describe the let (leveraging error type information) model, which explicitly applies the classification information (error types) of tokens to guide the generative model to generate better-corrected sentences. the whole architecture of let is shown in figure 2 . ## experiments to test the performance of the let system, we conduct evaluation experiments on two mainstream gec benchmarks: bea-test @xcite and @xcite and compare with previous state-of-the-art approaches. ## discussion we discuss many details of the model in more depth in this section. unless stated otherwise, all experiments in this section are tested on the bea-dev dataset under the constrained data setting. ## conclusion grammar error correction is significant for many downstream natural language understanding tasks. in this paper, we propose an end-to-end framework termed let, which effectively leverages the error type information generated by the ged task to guide the gec task. our work solves two critical problems in the previous work. firstly, we have alleviated the problem of error propagation caused by hard-coded error types by introducing soft-encoded error types. secondly, we have introduced the concept of error type alignment, which is more reasonable and adequate. we transfer the original semantic vectors into classification vectors to ensure that the two parts of the input of the proposed cross-attention module are both in the same semantic space. experiments and ablation studies show that alignment leads to better results. overall, let provides a better sample for research in the gec field and addresses some potential issues with previous technical solutions.