Data Augmentation of Incorporating Real Error Patterns and Linguistic Knowledge for Grammatical Error Correction (2021)

Abstract: Data augmentation aims at expanding training data with clean text using noising schemes to improve the performance of grammatical error correction (GEC). In practice, there are a great number of real error patterns in the manually annotated training data. We argue that these real error patterns can be introduced into clean text to effectively generate more real and high quality synthetic data, which is not fully explored by previous studies. Moreover, we also find that linguistic knowledge can be incorporated into data augmentation for generating more representative and more diverse synthetic data. In this paper, we propose a novel data augmentation method that fully considers the real error patterns and the linguistic knowledge for the GEC task. We conduct extensive experiments on public data sets and the experimental results show that our method outperforms several strong baselines with far less external unlabeled clean text data, highlighting its extraordinary effectiveness in the GEC task that lacks large-scale labeled training data.

## introduction grammatical error correction (gec) is the task of automatically detecting and correcting grammatical errors of texts into natural and correct forms, which is an important topic in the field of education, especially in language learning. in recent years, different methods have been proposed to improve the performance of the gec models. some early studies @xcite @xcite @xcite take gec as a classification task and rely much on hand-crafted rules. more recently, the technique of neural machine translation is applied to gec and has made remarkable performance @xcite @xcite @xcite . due to insufficient parallel training data publicly available, the gec task is often considered as a low-resource sequence generation task @xcite . in order to address this issue, many data augmentation methods have been proposed for generating synthetic pseudo training data to improve the performance of gec. some works focus on the edit-based approach. for example, @xcite use four basic edit operations for error introduction, which are random deletion, random insertion, random substitution and random shuffling. @xcite use more finegrained noising strategies by replacing the words in a sentence with those of the same type randomly chosen from the dictionary. some works focus on the back-translation-based method, which attempt to train a sequence-to-sequence model to translate correct sentences into wrong ones. for example, @xcite generate synthetic parallel data by first training a noising model with the reversed gec data, and then using the trained noising model to translate correct sentences into error-contained ones. some other studies utilize a large number of edit records from wikipedia in different periods of time for data augmentation. for example, @xcite extract source and target sentence pairs from the wikipedia edit histories and treate them as real-world errors. although previous data augmentation studies have achieved good results, there are still two points that can be improved: (1) we find that most of the existing methods introduce errors through random strategies, such as random deletion of words and random replacement of words, similar to the work of @xcite and @xcite . this random noise introduction strategy will generate unreal and low-quality synthetic errors which could be further propagated into the gec model and harm its performance. however, there are a large number of real error patterns in the manually annotated training data. as shown in figure 1 , there are three real error patterns existed in the annotated sentence "this are gramamtical sentence.", which are "are" → "is", "null" → "a" and "gramamtical" → "grammatical". we argue that these real error patterns can be incorporated into data augmentation to effectively generate more real and high-quality synthetic data, which is less explicitly explored by previous studies. (2) few previous studies have fully considered the linguistic knowledge in data augmentation, which can also be incorporated to improve the representativeness and diversity of the generated synthetic data. to this end, we propose a novel data augmentation method for the gec task, which consists of four types of noising schemes, namely the real error pattern based noising scheme, the synonym noising scheme, the inflection noising scheme and the functional word noising scheme. the real error pattern based noising scheme is designed for introducing the real errors made by english learners into clean text to generate high-quality synthetic data. the other three schemes are designed for incorporating linguistic knowledge to generate more representative and more diverse synthetic data. these generated synthetic data can be used as pseudo training data for the gec task. the main contributions of our work are as followed: ## discussion in order to prove the superiority of our proposed method over other noising strategies and explore different contributions of the proposed noising schemes, we conduct several ablation studies. we also give a case study to show the superiority of our method in generating high-quality and diverse pseudo data. ## related work grammatical error correction (gec) is the task of automatically detecting and correcting grammatical errors of texts into natural and correct forms. in recent years, different methods have been proposed to improve the performance of the gec models. early studies @xcite @xcite @xcite take gec as a classification task and rely much on hand-crafted rules. more recently, the techniques of statistical machine translation and neural machine translation are applied to gec and have made remarkable performance @xcite @xcite @xcite @xcite @xcite . due to insufficient training data in the gec task, many data augmentation methods have been proposed for generating synthetic pseudo training data to improve the performance of gec. we review the methods of data augmentation in detail in the next several subsections. edit-based approach. some data augmentation methods focus on using edit operations for noise introduction. @xcite use four basic edit operations for introducing errors. for each correct sentence, they randomly delete, insert or substitute some words with other randomly chosen words in the dictionary, or shuffle adjacent words. @xcite propose more fine-grained edit-based data augmentation method. they replace words with those of the same type, instead of substituting words with random words chosen from the dictionary. ## conclusion in this paper, we propose a data augmentation method that incorporates real error patterns and linguistic knowledge into the gec task, which effectively generates high-quality and diverse synthetic gec data. experimental results on public datasets demonstrate the effectiveness of our proposed method. especially, our method can produce remarkable results with limited data, proving the superiority of our approach and its potential usage in low-resource scenarios. in the future, we plan to fuse more linguistic knowledge into the gec data augmentation to make training data more diverse. in addition, we are also willing to explore hybrid augmentation methods by combining our proposed noising schemes with other data augmentation methods such as back-translation.