Data Weighted Training Strategies for Grammatical Error Correction (2020)

Abstract: Recent progress in the task of Grammatical Error Correction (GEC) has been driven by addressing data sparsity, both through new methods for generating large and noisy pretraining data and through the publication of small and higher-quality finetuning data in the BEA-2019 shared task. Building upon recent work in Neural Machine Translation (NMT), we make use of both kinds of data by deriving example-level scores on our large pretraining data based on a smaller, higher-quality dataset. In this work, we perform an empirical study to discover how to best incorporate delta-log-perplexity, a type of example scoring, into a training schedule for GEC. In doing so, we perform experiments that shed light on the function and applicability of delta-log-perplexity. Models trained on scored data achieve state- of-the-art results on common GEC test sets.

## introduction grammatical error correction (gec), the task of automatically correcting errors in written text, can be framed as a translation task from 'bad grammar' to 'good grammar.' this framing has enabled gec to borrow models and techniques from the vast literature in machine translation (mt). neural approaches have dominated recent stateof-the-art advances in gec, and have been shown to be more effective in direct comparison with non-neural methods @xcite . nevertheless, gec continues to pose a challenge for data-reliant neural models, given that the publicly available training data is relatively limited, with the largest corpus numbering only 2m examples @xcite . therefore, much recent work in gec has focused on diverse methods to address data sparsity by supplementing available annotated corpora with much larger pretraining data @xcite @xcite @xcite . a contrasting approach to addressing data sparsity in gec has been explored in the building educational application (bea) 2019 shared task on grammatical error correction @xcite . the task introduced the write and improve training set, a new high-quality annotated corpus numbering only ∼34k examples (referred to in this work as bea-19 train), and encouraged exploration of low-resource methods by organizing two tracks specifically for data-restricted competition. despite the relatively small size, many approaches using the bea-19 train data achieved better results on common gec test sets than previous approaches that did not have access to this small but high-quality data @xcite . in the context of neural mt (nmt), models have been shown to be sensitive to noise in the training data @xcite . although much effort has been dedicated to methods which either filter or downweight noisy pretraining data in nmt @xcite , less attention has thus far been paid in gec. to the best of our knowledge, previously explored techniques for filtering pretraining data in gec are limited to hand-engineered heuristic cutoffs @xcite and n-gram language model filtering @xcite . recent work in nmt @xcite presents a training technique for scoring the 'noise' of training data by employing a much smaller, higher-quality 'trusted' dataset. the authors describe a curriculum-style training over data scored by this metric, and demonstrate significant improvements over a baseline. we refer to this score as delta-log-perplexity (∆ppl). ## contributions of this work this work presents an empirical study of training strategies for gec in multiple dimensions. using a standard training setup (without scoring), we explore arrangements of gec corpora into pretraining and finetuning data, establishing a strong ## related work in recent gec work, most approaches pretrain on some synthetic data and then finetune on the union of multiple annotated data sources, with some variation in which datasets are included for fine-tuning @xcite . in a thorough study of incorporating generated pseudo-data into gec training, @xcite report that this typical pretrainfinetune setup scales with size of pretraining data better than a setup in which all data is trained on simultaneously. @xcite describe a 'sequential transfer learning' approach in which the pretrained model, finetuned on all available annotated data, is finetuned again separately for each test set. a thorough review of the gec field is made by @xcite . data selection in mt has been performed along two dimensions: domain-relevance and denoising. multiple researchers @xcite @xcite have used the difference in cross-entropy between two language models as a criteria for the selection of in-domain sentences. in contrast, @xcite and junczys-dowmunt (2018) have used data selection for denoising. recently, @xcite demonstrate that a co-curriculum training for dynamic selection of data that is both clean and in-domain, can outperform independent selection along each of the two dimensions. 2018 ) present a metric defined as the difference in log-probability of an individual training example before and after improving a pretrained model by finetuning on a small trusted dataset. wang et al. use this metric to order the pretrain data, and train a new model via a curriculum-style strategy using this ordering. in their setup, this metric is interpreted as measuring 'noise', describing the change in log probability of an example between a noisy pretrained model and its 'denoised' finetuned counterpart. because log perplexity for an example is the negative of the log-probability, we refer to this score as 'delta-log-perplexity'(∆ppl). foot_0 ## future work the huge jump in performance between unscored and scored pretraining data demonstrates the possibility of making much more effective use of large and noisy datasets through the incorporation of example-level quality scores. while ∆ppl is one such score, there is significant room for @xcite 69.5 64.2 61.2 @xcite 70.2 65.0 61.4 @xcite -60.4 63.3 @xcite 66.6 63.2 62.6 @xcite 73.7 66.5 this work -unscored 71.9 65.3 64.7 this work -scored 73.0 66.8 64.9 in our scored training, we have presented hard-coded training strategies selected for their simplicity. these un-tuned strategies are easy to implement, but do not represent optimal uses of an example-level heuristic score. the fact that there is such variability between them in the two experiments of these two complementary directions of future work, the development of new example-level quality heuristics and the techniques to apply them in scored training, present an intriguing path for future exploration.