Data Strategies for Low-Resource Grammatical Error Correction (2021)

Abstract: Grammatical Error Correction (GEC) is a task that has been extensively investigated for the English language. However, for low-resource languages the best practices for training GEC systems have not yet been systematically determined. We investigate how best to take advantage of existing data sources for improving GEC systems for languages with limited quantities of high quality training data. We show that methods for generating artificial training data for GEC can benefit from including morphological errors. We also demonstrate that noisy error correction data gathered from Wikipedia revision histories and the language learning website Lang8, are valuable data sources. Finally, we show that GEC systems pre-trained on noisy data sources can be fine-tuned effectively using small amounts of high quality, human-annotated data.

## introduction grammatical error correction (gec) research has thus far been mostly focused on the english language. one reason for this narrow focus is the difficulty of the task -even for english, which has a reasonable amount of high quality data, the task is challenging. another reason for the englishcentric research has been the lack of available gec benchmark datasets in other languages, which has made it harder to develop gec systems on these languages. in the past few years, there are several languages for which gec benchmarks have become available @xcite @xcite . simultaneously, there has been considerable progress in gec for english using cheap data sources such as artificial data and revision logs @xcite @xcite . since these resources are language-agnostic, the time is ripe for investigating these techniques for other languages. pretraining gec systems on artificially generated errors is now common practice for english. @xcite showed good results on english, russian, and german, using a rule-based error generation approach that @xcite extended to czech. this approach employed the aspell dictionary to create confusion sets of phonologically and lexically similar words. in this work, we additionally investigate the usefulness of morphology-based confusion sets. for english, model-based error generation approaches have also been shown to be useful @xcite . state-of-the-art english gec systems also make use of lower quality data sources, such as wikipedia revision histories and crowd-sourced corrections from the language learning website lang8 @xcite . given that it is possible to extract data from both wikipedia and lang8 in multiple languages, it would be interesting to determine if this data will help improve gec for non-english languages. @xcite have already shown promising results for german using wikipedia revisions with a custom language-specific filtering method. contributions in this work we investigate data strategies for grammatical error correction on languages without large quantities of high quality training data. in particular we answer the following questions: i) can artificial error generation methods benefit from including morphological errors?; ii) how can we best make use of noisy gec data when other data is limited?; iii) how much gold training data is necessary? 2 gec data sources ## systems for all experiments we use the transformer sequence-to-sequence model @xcite available in the tensor2tensor library. 3 the model is trained with early stopping, using adafactor as optimizer with inverse square root decay @xcite . a detailed overview of hyperparameters is listed in appendix a. 4 we compare our results to two baseline gec systems, @xcite junczys-dowmunt (2019) (g&j) and náplava and straka (2019) (n&s), which have both been evaluated on russian and german, and for náplava and straka (2019) additionally on czech. both of these systems are pretrained on artificial data and finetuned on gold data. when training the models several strategies were used: source and target word dropouts, edit-weighted maximum likelihood estimation and checkpoint averaging. in this work we do not employ these techniques because our focus is on comparing methods for data collection and generation and less on surpassing the state-of-the-art. ## experiments we evaluate our models using f 0.5 score computed using the maxmatch scorer @xcite . for all experiments, the reported scores are computed for the model trained on the specified data source, further finetuned on the gold training data. ## conclusion in this paper we have investigated how best to make use of available data sources for gec in low resource scenarios. we have shown a set of best practices for using artificial data, wikipedia revision data and lang8 data, that gives good results across four languages. we show that using unimorph for generating artificial data is useful for russian and czech, which are morphologically rich languages. using wikipedia edits is a valuable source of data, despite its noise. lang8 is an even better source of highquality gec data, despite its smaller size and uncertainties associated with crowdsourcing. when using gold data for fine-tuning, even small amounts of data can yield good results. this is especially true when the initial model has been pretrained on wikipedia edits and lang8. we expect this work to provide a good starting point for developing gec systems for a wider range of languages.