Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models (2021)

Abstract: Synthetic data generation is widely known to boost the accuracy of neural grammatical error correction (GEC) systems, but existing methods often lack diversity or are too simplistic to generate the broad range of grammatical errors made by human writers. In this work, we use error type tags from automatic annotation tools such as ERRANT to guide synthetic data generation. We compare several models that can produce an ungrammatical sentence given a clean sentence and an error type tag. We use these models to build a new, large synthetic pre-training data set with error tag frequency distributions matching a given development set. Our synthetic data set yields large and consistent gains, improving the state-of-the-art on the BEA-19 and CoNLL-14 test sets. We also show that our approach is particularly effective in adapting a GEC system, trained on mixed native and non-native English, to a native English test set, even surpassing real training data consisting of high-quality sentence pairs.

## introduction grammatical error correction (gec) systems aim to automatically correct grammatical and other types of writing errors in text. it is common to view this problem as a sequence-to-sequence task (i.e. ungrammatical @xmath0. back-translation @xcite is a synthetic data generation technique for nmt that employs a translation system trained in the reverse direction to synthesize source sentences from sentences in the target language, and is still one of the most effective strategies to use monolingual data in nmt training. similarly, synthetic training data generation for gec has also been widely studied in the literature @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . this work is inspired by previous efforts to use ideas from back-translation for gec @xcite @xcite . in contrast to prior work, we use error type tags such as spell (spelling error) or sva (subject-verb agreement error) to control the output of our corruption models and generate more realistic as well as diverse grammatical errors. our tagged corruption models are trained to output the corrupted sentence given a clean sentence and an error tag, e.g.: "noun:infl there were a lot of sheep." → "there were a lot of sheeps." the tags mitigate the tendency of untagged corruption models to produce simplistic corruptions since many error type tags require more complex rewrites. in general, there is a one-to-many mapping from a clean sentence to a noisy sentence. using a regular corruption model, many of these synthetic errors tend to be simplistic, foot_0 but adding tag information allows the model to generate specific patterns of errors that can be found in actual gec corpora. the benefit of covering a wide range of error types when generating pseudo data for gec has also been demonstrated by @xcite ; @xcite . moreover, the tag distribution in the synthetic data can be made to match the distribution of a specific target domain. we use this distribution matching technique to adapt a gec system to better correct errors by native speakers. as an alternative to adding the tag directly to the input sequence, we add inference-time constraints to the recently proposed seq2edits model @xcite to force the generation of a particular tag. we implement such constraints using finite state transducers (fsts). we then use these corruption models to generate synthetic training data that follows a desired tag distribution, for example the tag distribution on the development set. using our new synthetic pretraining sets foot_1 we report state-of-the-art results on two popular gec test sets (bea-test: 74.9 f 0.5 , conll-14: 68.3 f 0.5 ). in our experiments on gec for native english, a model fine-tuned on synthetic data that follows a native english error-tag distribution can even surpass a model fine-tuned on high-quality, real (i.e. non-synthetic) data. ## tagged corruption models at the core of our approach is a model that generates an ungrammatical sentence from a clean sentence given an error tag @xmath1. t is the set of 25 error type tags supported by the automatic annotation toolkit errant @xcite . a straightforward way to train such a tagged corruption model is to annotate a parallel corpus with errant, prepend the errant tag to the clean sentence, and train a model such as a standard transformer @xcite to generate the ungrammatical sentence. 3 this idea is similar to the multi-lingual nmt system of @xcite that adds the target language id tag to the source sentence. alternatively, the recently proposed seq2edits foot_3 @xcite model is able to directly predict error tags along with the edits, and tag log-prob.corruption model output adj -3.06 there were a lot of many sheep. adj:form -2.49 there were a more better of sheep. adv -2.63 there were a lot of sheep there. conj -1.39 and there were a lot of sheep. contr -0.90 there're a lot of sheep. det -1.06 there were lot of sheep. k -1.45 there were a lot of. morph -0.67 there were a lot of sheeps. noun -3.31 there were a lot of seep. noun:infl -0.61 there were a lot of sheeps. noun:num -1.33 there were a lots of sheep. noun:poss -0.92 there were a lot of sheep's. orth -0.79 there were alot of sheep. other -2.60 there were many sheep. part -1.22 there were a lot off sheep. prep -1.08 there were a lot sheep. pron -1.55 it was a lot of sheep. punct -1.00 there were a lot of sheep spell -2.79 there were a lot of sheeps. verb -2.58 there had a lot of sheep. verb:form -2.34 there being a lot of sheep. verb:infl -1.09 there were a lot of sheeps. verb:sva -0.44 there was a lot of sheep. verb:tense -0.57 there are a lot of sheep. wo -1.87 there were a lot sheep of. ## results for comparability to related work, we report spanbased errant f 0.5 -scores on the development and test sets (bea-dev and bea-test) of the bea-2019 shared task @xcite . we use the m2 scorer @xcite to compute f 0.5 -scores on the conll-13 (ng et al., 2013) and conll-14 (ng et al., 2013) sets, and the gleu metric @xcite on jfleg-dev and jfleg-test @xcite . ## related work the body of literature on synthetic data generation for gec is large. various heuristics have been proposed to inject synthetic noise into grammatical sentences such as random word-or character-level insertion, substitution, deletion, or shuffling operations @xcite @xcite , using spell checkers @xcite , or randomly applying word edits extracted from the training data @xcite . @xcite ; @xcite applied back-translation @xcite to gec and reported substantial gains. similar to mt @xcite , back-translation for gec can be further improved by adding noise to the decoding process @xcite or by using sampling instead of beam search @xcite . fluency-boost learning @xcite can also be used to generate additional sentence pairs during training. @xcite proposed to generate noisy counterparts of grammatical english sentences by translating them to another language (e.g. german) and back ("round-trip translation"), a technique we also use in this work. the use of tags for back-translation in mt has been explored by @xcite . our tagged corruption models are inspired by @xcite who generated synthetic sentences from latent representations that are perturbed using explicit error type tags. our approach of adding the tags to the input sequence is simpler as it requires no modifications to the model architecture or training procedure. ## conclusion we have introduced a synthetic data generation method for grammatical error correction that is able to produce a wide range of realistic grammatical errors. our method is based on grammar corruption models that corrupt a clean sentence given an error type tag. conditioning on the error type tag enables us to control synthetic data generation much more precisely than alternative methods such as round-trip translations or tag-independent back-translation. we explored different ways of using these tagged corruption models to generate synthetic data that follows a certain error tag distribution. we found that fine-tuning a model on synthetic data that follows a native english error tag distribution can even outperform fine-tuning on genuine parallel data from a mixture of proficiency levels. along with this paper we publish a new 200m sentence data set for gec -c4 200m . using c4 200m in pre-training of vanilla transformer gec models yields state-of-the-art performance on two standard gec test sets (bea-test and conll-14). we expect this corpus to further stimulate the development of new data-driven approaches in gec.