M ix E dit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction (2023)

Abstract: Data Augmentation through generating pseudo data has been proven effective in mitigating the challenge of data scarcity in the field of Grammatical Error Correction (GEC). Various augmentation strategies have been widely explored, most of which are motivated by two heuristics, i.e., increasing the distribution similarity and diversity of pseudo data. However, the underlying mechanism responsible for the effectiveness of these strategies remains poorly understood. In this paper, we aim to clarify how data augmentation improves GEC models. To this end, we introduce two interpretable and computationally efficient measures: Affinity and Diversity. Our findings indicate that an excellent GEC data augmentation strategy characterized by high Affinity and appropriate Diversity can better improve the performance of GEC models. Based on this observation, we propose MixEdit, a data augmentation approach that strategically and dynamically augments realistic data, without requiring extra monolingual corpora. To verify the correctness of our findings and the effectiveness of the proposed MixEdit, we conduct experiments on mainstream English and Chinese GEC datasets. The results show that MixEdit substantially improves GEC models and is complementary to traditional data augmentation methods. All the source codes of MixEdit are released at https://github.com/THUKElab/MixEdit.

## introduction grammatical error correction (gec) is a task that involves making locally substitutions in text to correct all grammatical errors in a text @xcite @xcite . gec is often considered a monolingual machine translation (mt) task, which is typically tackled using sequence-to-sequence (seq2seq) architecture @xcite . numerous studies tend to improve the performance of seq2seq gec models by increasing the amount of training data @xcite @xcite . however, high-quality parallel data for gec is not as widely available. despite the great success of seq2seq models, they are prone to overfitting and making predictions based on spurious patterns @xcite , owing to the vast gap between the number of model parameters and the limited high-quality data. this data sparsity issue has motivated research into data augmentation in the field of gec, particularly in the context of resource-heavy seq2seq approaches @xcite . thanks for the ease of constructing pseudo grammatical errors, recent studies focus on generating synthetic parallel data from clean monolingual corpora. various augmentation methods have been widely explored, whose major motivation comes from improving 1) distribution similarity between pseudo and realistic data, including error patterns @xcite and back-translation @xcite @xcite ; and 2) diversity of pseudo data @xcite , such as noise injection @xcite and round-trip translation @xcite . however, prior works have primarily focused on showing the effectiveness of their proposed augmentation methods, without considering sample efficiency. training gec models with excessive samples (e.g. over 100m @xcite ) for poorscalable improvement is expensive and often unfeasible for most researchers. additionally, existing studies suffer from a lack of consistent experimental settings, making it intractable to systematically and fairly compare various data augmentation methods. in this complex landscape, claims regarding distribution similarity and diversity remain unverified heuristics. an uncomplicated approach target this will , if not already , cause problems as there is very limited space for us . ## background to avoid confusion, we use the uppercase symbol x to represent a sentence, and the bold lowercase symbol x to indicate a text segment. in this paper, we focus the discussion on constructing grammatical errors for gec data augmentation. generally, seq2seq-based gec models parameterized by θ learn the translation probability p (y | x; θ), where x denotes an ungrammatical source sentence and y represents a grammatical target sentence. given a parallel training dataset d, the standard training objective is to minimize the empirical risk: direct noise (dn). dn injects noise into grammatically correct sentences in a rule-based manner @xcite . the noise can take the form of 1) masking, 2) deletion, and 3) insertion based on pre-defined probabilities. dn is applicable to all languages since its rules are languageindependent. however, the generated errors are often not genuine and may even distort the original semantics of the sentences. pattern noise (pn). pn, on the other hand, involves injecting grammatical errors that are already present in the realistic gec dataset into sentences @xcite . specifically, this process entails first identifying error patterns in the gec dataset using an automated error annotation tool such as errant @xcite , followed by applying a noising function that randomly substitutes text segments with grammatical errors. ## conclusion this paper introduces two interpretable and computationally efficient measures, affinity and diversity, to investigate how data augmentation improves gec performance.