Synthetic Data Generation for Low-resource Grammatical Error Correction with Tagged Corruption Models (2024)

Abstract: Tagged corruption models provide precise control over the introduction of grammatical errors into clean text. This capability has made them a powerful tool for generating pre-training data for grammatical error correction (GEC) in English. In this work, we demonstrate their application to four languages with substantially fewer GEC resources than English: German, Romanian, Russian, and Spanish. We release a new tagged-corruption dataset consisting of 2.5M examples per language that was generated by a fine-tuned PaLM 2 foundation model. Pre-training on tagged corruptions yields consistent gains across all four languages, especially for small model sizes and languages with limited human-labelled data.

## introduction grammatical error correction (gec) is the task of correcting writing errors in text (see @xcite for an overview). neural sequenceto-sequence models, commonly used for gec, are hard to train due to limited human-labelled data. a common strategy to mitigate data sparsity is to generate synthetic training data, but most existing methods do not generate sufficiently diverse errors. modern gec systems are expected to handle a broad range of errors involving grammar, spelling, word choice, punctuation and orthography. however, many existing data generation methods that employ rules or character-or word-level noising strategies, cover only a small subset of error types @xcite @xcite @xcite . @xcite improved the diversity of model-based data generation @xcite by introducing tagged corruption models. tagged corruption models are trained to generate an ungrammatical version of a clean sentence given a specific error type tag. for example, the incorrect plural "sheeps" of "sheep" (i.e. a noun inflection error -noun:infl) would be represented in a sentence as follows @xcite : "noun:infl there were a lot of sheep." → "there were a lot of sheeps." in this work, we adapt the tagged corruption approach of @xcite to languages with fewer gec resources than english such as german, spanish, romanian, and russian. we faced two major challenges: first, training tagged corruption models is more challenging due to training data scarcity. we mitigated this issue by leveraging the large language model palm 2 @xcite . second, automatic error type annotation tools such as errant @xcite for english are not available for most other languages. therefore, we developed a multilingual annotation tool based on classification rules that apply to multiple languages and writing systems. using our framework, we generated a new synthetic pre-training dataset with 2.5m examples per language. we demonstrate consistent gains from pre-training mt5 @xcite models on our new dataset and then fine-tuning them on gold data. we achieve the largest improvements (up to 30% relative) for smaller models and languages with limited gold data. we have released the dataset and the error annotation tool to the scientific community. ## multilingual rule-based error type annotation errant @xcite is a rule-based system for english that classifies writing errors into 25 different error categories. some errant rules are specific to english and do not apply to other languages. german (boyd, 2018) 0% 5% 10% 15% 20% 25% 30% adj adp adv cconj det morphnounn:num num orth other part pron propnpunct sconj spell verbv:tense frequency german (de) spanish (es) romanian (ro) russian (ru) figure 1: development set tag distributions for german, spanish, romanian, and russian. tag description adj, adp, adv, cconj, det, part, pron, propn, sconj error classified by spacy partof-speech (pos) tag. morph morphology error. noun noun or noun phrase error. n:num noun number error. num number error. orth orthography error. other unclassified error (no rule matched). punct punctuation error. spell spelling error according to gnu aspell 0.60. verb verb or verb phrase error. v:tense verb tense error. wo word order error. and romanian @xcite versions of er-rant have been developed, but they continue to be language-specific. since our goal is to develop a recipe for low-resource gec that is applicable to a large set of languages, we developed an annotation toolkit that implements a small set of general rules relying on multi-lingual nlp toolkits such as spacy's 1 part-of-speech (pos) tagger or gnu aspell 2 for spelling correction. the error tag set of our tool is shown in ## synthetic data generation using a tagged corruption model tagged corruption models are neural models that corrupt a clean sentence according to an error type tag. we adapt @xcite recipe for english data generation as follows: for each language: ## results like prior work we compute f 0.5 -scores on the german, russian, and spanish test sets with the m2 scorer @xcite , and on the romanian test set with @xcite version of errant. foot_4 fig. 3 shows a log-log plot of the relative improvements between the gold baseline and the @xmath0. the improvements range between 0.5% and 30% depending on the language and model size. our pre dataset is particularly useful for small training sets (ru) and small models (left side of the plot). grammatical error correction models deployed in practice are often small because a low latency is less disruptive for writers. to investigate if pre-training can be further improved by adding external data, we performed experiments using the clang8 corpus @xcite . of pre and clang8 outperforms pre-training on only one of them. ## conclusion we have introduced a new large synthetic dataset for gec that was generated by an llm-based tagged corruption model in german, spanish, romanian, and russian. our dataset consists of 2.5m examples per language. pre-training gec models on this dataset yields consistent gains on all four languages, especially for small gold training sets and small model sizes.