A Comparative Study of Synthetic Data Generation Methods for Grammatical Error Correction (2020)

Abstract: Grammatical Error Correction (GEC) is concerned with correcting grammatical errors in written text. Current GEC systems, namely those leveraging statistical and neural machine translation, require large quantities of annotated training data, which can be expensive or impractical to obtain. This research compares techniques for generating synthetic data utilized by the two highest scoring submissions to the restricted and low-resource tracks in the BEA-2019 Shared Task on Grammatical Error Correction.

## introduction grammatical error correction (gec) is the task of automatically correcting grammatical errors in written text. more recently, significant progress has been made, especially in english gec, within the framework of statistical machine translation (smt) and neural machine translation (nmt) approaches @xcite @xcite @xcite @xcite . the success of these approaches can be partially attributed to the availability of several large training sets. in the most recent building educational applications (bea) 2019 shared task @xcite , which continued the tradition of earlier gec competitions @xcite , all of the 24 participating teams applied nmt and/or smt approaches. one of the goals of bea-2019 was to re-evaluate the field after a long hiatus, as recent gec systems have become difficult to evaluate given a lack of standardised experimental settings: although significant progress has been made since the end of the last conll-2014 shared task, recent systems have been trained, tuned and tested on different combinations of metrics and corpora. the bea-2019 shared task also introduced a new dataset that represents a diverse cross-section of english language levels and domains @xcite , and separate evaluation tracks, namely the restricted, unrestricted and low resource tracks. the unrestricted track allowed the use of any resources; the restricted track limited the use of learner corpora to those that are publicly available, while the low resource track significantly limited the use of annotated data, to encourage development of systems that do not rely on large quantities of human-annotated data. the two top scoring systems in the restricted and low resource tracks, uedin-ms @xcite and kakao&brain @xcite , outperformed the other teams by a large margin in both tracks; furthermore, both systems made use of artificial data for training their nmt systems, but they generated artificial data in different ways. interestingly, in the restricted track, both of the systems scored on par, while in the low resource track kakao&brain exhibited a larger gap in performance (a drop of more than 10 points compared to the restricted track) vs. 4 points for uedin-ms. while both teams used the same model architecture, transformerbased neural machine translation (nmt) @xcite , in addition to the differences in the data generation methods, the systems used different training scenarios, hyperpameter values, and training corpora of native data. the goal of this paper is to compare the techniques for generating synthetic data used by the uedin-ms and kakao&brain systems. the method used in the uedin-ms system utilizes confusion sets generated by a spellchecker, while the kakao&brain method relies on learner patterns extracted from a small annotated sample and on pos-based confusions. henceforth, we refer to these as inverted spellchecker method and pat-terns+pos method, respectively. to ensure a fair comparison of the methods, we control for the other variables, such as model choice, hyperparameters, and the choice of native data. we train nmt systems and evaluate our models on two learner corpora, the w&i+locness corpus introduced in bea-2019, and the fce corpus @xcite . using the automatic error type tool errant @xcite , we also show performance evaluation by error type on the two corpora. the paper makes the following contributions: (1) we provide a fair comparison of two methods for generating synthetic parallel data for gec, using two evaluation datasets; (2) we find that the two methods train different complementary systems and target different types of errors: while the inverted spellchecker approach is good at identifying spelling errors, the patterns+pos approach is better at correcting errors relating to grammar, such as noun number, verb agreement, and verb tense; (3) overall, the patterns+pos method exhibits stronger results, compared to the inverted spellchecker method in multiple training scenarios that include only synthetic parallel data, synthetic data augmented with in-domain learner data, and synthetic data augmented with out-ofdomain learner data; (4) adding an off-the-shelf spellchecker is beneficial, and is especially helpful for the patterns+pos approach. in the next section, we discuss related work. section 3 gives an overview of the w&i+locness and fce learner datasets. section 4 describes the error generation methods. experiments are presented in section 5. section 6 analyzes the results. section 7 concludes the paper. ## related work progress in english gec earlier gec approaches focused on english as a second language learners and made use of linear machine learning algorithms and developed classifiers for specific error types, such as articles, prepositions, or noun number @xcite @xcite . the classifiers can be trained on native english data, learner data, or a combination thereof. the conll shared tasks on english grammar correction provided a first large annotated corpus of learner data for training (nucle, @xcite ), as well as two test sets. all the data was produced by learners of english studying at the national university of singapore (majority of which were native speakers of chinese). the statistical machine translation approach was shown to be successful in the conll-2014 competition for the first time (junczys-dowmunt and grundkiewicz, 2014). since then, the state-of-the-art results on the conll datasets were obtained using smt and nmt approaches. the systems are typically trained on a combination of nucle and the english part of the lang-8 corpus @xcite , even though the latter is known to contain noise, as it is only partially corrected. minimally-supervised and data-augmented gec there has been a lot of work on generating synthetic training data for gec. the approaches can be broken down into those that attempt to make use of additional resources (e.g. wikipedia edits) or noisify correct english data via artificial errors. @xcite augmented training data with edits extracted from wikipedia revision history in german. the edits were classified and only those relating to gec were kept. wikipedia edits are extracted from the revision history using wiki edits @xcite . the contribution of the resulting edits is demonstrated using a multilayer convolutional encoderdecoder neural network model that we also use in this work. @xcite extracted a japanese learners corpus from the revision log of lang-8 (about 1 million sentences) and implemented a character-based machine-translation model. the other approach of generating parallel data creates artificial errors in well-formed native data. this approach was shown to be effective within the classification framework @xcite @xcite . ## the learner datasets we make use of two publicly-available datasets of learner texts for evaluation -the w&i+locness and the fce -described below. the comes from a web-based platform that provides feedback for non-native english students around the world to improve their writing. locness was compiled by researchers at the centre for english corpus linguistics at the university of louvain, and consists of essays written by native english students. w&i+locness contains 3,700 texts, consisting of 43,169 sentences or 801,361 tokens. 34,308 sentences were made available for training and 4,384 for development. to provide an additional benchmark for evaluation, we also evaluate on the test dataset from the fce corpus @xcite . the first certificate in english (fce) corpus is a subset of the cambridge learner corpus (clc) that contains 1,244 written answers to the fce exam, which assesses english at an upper-intermediate level. fce was originally annotated according to a different error type framework, but was reannotated automatically using errant for use in the shared task. ## synthetic data generation methods in this section, we describe the two methods to generate synthetic parallel data for training. ## experiments to compare the inverted spellchecker and pat-terns+pos noising methods, we present a series of experiments that should provide evidence for the efficacy of the noising methods separate from the implementation of the systems as a whole. ## discussion and error analysis results in the previous section indicate that the patterns+pos method outperforms the inverted spellchecker method on w&i+locness, both when used on its own, and when additional learner training data is available, with and without a spellchecker. on the fce dataset, the patterns+pos method is superior only when a spellchecker is added. in general, the pat-terns+pos method benefits more from the addition of a spellchecker in all experiments. adding an off-the-shelf spellchecker to a gec system is a common pre-processing step: a spellchecker is developed to specifically target spelling errors, so a gec system, which is typically more complex, can focus on other language misuse. the greater gap in performance between the methods on w&i+locness, compared to fce, can be attributed to the utilization of in-domain data as part of the patterns+pos noising approach. evaluation by error type to examine the discrepancies in performance between the two noising methods across the two evaluation datasets, we present an evaluation of performance by er-rant error type. type-based evaluation results for the top 10 most common error types for each respective evaluation dataset can be seen in tables 9 and 10 (note that these results do not include the off-the-shelf spellchecker). the inverted spellchecker method significantly outperforms the patterns+pos method on spelling errors on both datasets. as spelling errors make up approximately 10% of errors in the fce test set (double the relative frequency of spelling errors in w&i+locness), this may explain the improved performance of the inverted spellchecker method when evaluated on fce, compared to its own performance on w&i+locness. in contrast, the patterns+pos method outperforms the inverted spellchecker method on verb tense errors and noun number errors. this makes sense, since the pos-based confusion sets produce errors that reflect misuse of these grammatical categories. on the most common and notoriously difficult errors -articles and prepositions -the two methods exhibit similar performance. finally, the patterns+pos method outperforms the inverted spellchecker method by 25 points on punctuation errors on the w&i+locness data, but is outperformed by 2 points on fce. this may be due to the fact that the patterns+pos method utilizes indomain data as part of its noising process. ## conclusions and future work in this paper, we conduct a fair comparison of two methods for generating synthetic parallel data for grammatical error correction -using spellcheckerbased confusion sets and using learner patterns and pos-based confusions. our models are evaluated on two benchmark gec english learner datasets. we show that the methods are bettersuited for different types of language misuse. in general, the patterns+pos methods demonstrated stronger performance than the inverted spellchecker methods. for future work, we will investigate how these noising approaches complement each other. this can be done by training models on a mixture of synthetic data generated from both approaches independently, or by utilizing a hybrid noising method that combines the character-level perturbation method of the inverted spellchecker method with the pattern+pos method in order to generate additional artificial spelling errors. we will also perform experiments with larger training sets. it would also be interesting to examine how these noising scenarios perform for languages other than english.