R obust GEC : Robust Grammatical Error Correction Against Subtle Context Perturbation (2023)

Abstract: Grammatical Error Correction (GEC) systems play a vital role in assisting people with their daily writing tasks. However, users may sometimes come across a GEC system that initially performs well but fails to correct errors when the inputs are slightly modified. To ensure an ideal user experience, a reliable GEC system should have the ability to provide consistent and accurate suggestions when encountering irrelevant context perturbations, which we refer to as context robustness. In this paper, we introduce RobustGEC, a benchmark designed to evaluate the context robustness of GEC systems. RobustGEC comprises 5,000 GEC cases, each with one original error-correct sentence pair and five variants carefully devised by human annotators. Utilizing RobustGEC, we reveal that state-of-the-art GEC systems still lack sufficient robustness against context perturbations. Moreover, we propose a simple yet effective method for remitting this issue.

## introduction grammatical error correction (gec) is the task of automatically fixing various textual errors in a given sentence. this task has many real-life applications, such as writing assistance and language teaching, thus receiving considerable interest from both academia and industry @xcite @xcite . nowadays, cutting-edge gec systems perform decently in academic benchmarks @xcite @xcite . however, in practical applications, the gec system may offer different error correction suggestions when users perform irrelevant modifications. figure 1 presents a real example of this phenomenon, where a user interacts with a gec system based on t5 @xcite . given the original input "i * work was done during the internship at tencent ai lab. † corresponding author. user gec system i like play basketball. i like playing basketball. user gec system he loves play basketball. he loves play basketball. user gec system i like play hockey. i like playing ice hockey. like play basketball", the gec system effectively corrects the error (play ⇒ playing), because "like" should be followed by a gerund. however, when the user introduces minor modifications unrelated to the error correction, the gec system starts making mistakes, including under-correction (play ⇒ play) and over-correction (hockey ⇒ ice hockey). this observation may lead to confusion and frustration, as users would expect the gec system to have a more stable and reliable performance. a key desirable property of strong gec systems is context robustness, which refers to the ability to maintain consistent and accurate results without being disrupted by irrelevant context perturbations. to test this, we contribute the robustgec benchmark based on traditional gec datasets ( §3). we choose 5,000 error-correct sentence pairs from three sources for annotation, i.e., @xcite , bea-19 @xcite , and our newly released tem-8. the first two are collected from essays and corrected by native englishspeaking teachers. the last is derived from the ungrammatical sentence modification question in tem-8, a high-level exam in china for evaluating english major students' english ability, where errors are designed by experts. we ask annotators proficient in english to carefully craft five variants for each original sample by perturbing the context unrelated to the correction. the annotators are allowed to freely replace, insert, or delete content without introducing new errors or altering the original errors. in this way, robustgec can examine whether gec systems consistently and accurately rectify errors disregarding natural and realistic irrelevant context perturbations. utilizing robustgec, we evaluate the context robustness of five state-of-the-art (sota) gec systems ( §4), i.e., seq2seq-based bart (katsumata and komachi, 2020), seq2seq-based syn-gec with linguistic knowledge @xcite , seq2edit-based gector @xcite , llm-based llama with gec finetuning @xcite , and llm-based chat-gpt with zero-shot prompting @xcite . despite their notable performance on conventional gec benchmarks, all evaluated systems show dramatic performance fluctuations when facing context perturbations, revealing their lack of context robustness. taking bart as an example, only 43.5% of gec instances exhibit consistent corrections during perturbation. one potential explanation is that current gec systems do not truly comprehend grammar, but rather depend on some spurious correlations @xcite to make corrections. in order to gain more in-depth insights, we also conduct detailed analyses to investigate how context perturbations affect gec systems, in terms of perturbing action, distance, word frequency, etc. to improve the context robustness, we further propose a context perturbation robust (cpr) posttraining approach ( §5). the proposed cpr method optimizes the gec system to output the same prediction distribution at the non-perturb positions by minimizing the bi-directional kullback-leibler (kl) divergence, which is lightweight and easy to train. experiments demonstrate that the cpr method can improve context robustness by a large margin and maintain the original gec performance. for example, after post-training with cpr, the gector model performs consistent corrections for 15.1% more gec cases in robustgec. we hope this research could spur future investigations into the robustness of gec systems. to facilitate further studies, we have made all data and code publicly accessible at https://github .com/hillzhang1999/robustgec. in the context of the llm era, we believe that there is still much more to explore for gec, extending beyond the mere pursuit of superior leaderboard scores. ## evaluating robustness of gec systems in this section, we employ the robustgec benchmark to conduct a comprehensive test and analysis on the context robustness of existing gec systems. ## improving robustness of gec systems after witnessing the unsatisfactory context robustness of existing gec systems, we further explore the approach to improving it. in this section, we try to propose a simple yet effective method to enhance the context robustness of gec systems. ## conclusion this paper investigated the context robustness of gec systems. this is the ability to provide consistent correction suggestions when modifications unrelated to errors are introduced into the inputs. to quantitatively analyze this, we developed a diagnostic benchmark named robustgec. using ro-bustgec, we disclosed that popular gec systems have significant room for improvement in context robustness. furthermore, we proposed a simple yet effective method for enhancing context robustness. we hope our work can offer insights for future research aimed at developing robust gec systems.