A Simple Recipe for Multilingual Grammatical Error Correction (2021)

Abstract: This paper presents a simple recipe to trainstate-of-the-art multilingual Grammatical Error Correction (GEC) models. We achieve this by first proposing a language-agnostic method to generate a large number of synthetic examples. The second ingredient is to use large-scale multilingual language models (up to 11B parameters). Once fine-tuned on language-specific supervised sets we surpass the previous state-of-the-art results on GEC benchmarks in four languages: English, Czech, German and Russian. Having established a new set of baselines for GEC, we make our results easily reproducible and accessible by releasing a CLANG-8 dataset. It is produced by using our best model, which we call gT5, to clean the targets of a widely used yet noisy Lang-8 dataset. cLang-8 greatly simplifies typical GEC training pipelines composed of multiple fine-tuning stages – we demonstrate that performing a single fine-tuning stepon cLang-8 with the off-the-shelf language models yields further accuracy improvements over an already top-performing gT5 model for English.

## introduction grammatical error correction (gec) is the task of correcting grammatical and other related errors in text. it has been the subject of several modeling efforts in recent years due to its ability to improve grammaticality and readability of user generated texts. this is of particular importance to non-native speakers, children, and individuals with language impairments, who may be more prone to producing texts with grammatical errors. modern approaches often view the gec task as monolingual text-to-text rewriting @xcite @xcite and employ encoderdecoder neural architectures @xcite . these methods typically require large training sets to work well @xcite which are scarce especially for languages other than english. one of the largest and most widely used datasets for gec is the lang-8 learner corpus, which covers 80 languages and has been created by language learners correcting each other's texts. 2 however, the distribution of languages is very skewed, with japanese and english being the most prevalent languages with over a million ungrammatical-grammatical sentence pairs each, while only ten languages have more than 10,000 sentence pairs each. additionally, given the uncontrolled nature of the data collection, many of the examples contain unnecessary paraphrasing and erroneous or incomplete corrections. limited amounts of suitable training data has led to multiple approaches that propose to generate synthetic training data for gec @xcite @xcite @xcite . although using synthetic data as the first fine-tuning step has been shown to improve model accuracy, it introduces practical challenges that make the development and fair comparison of gec models challenging: (i) the synthetic methods often require language-specific tuning (e.g. language-specific hyperparameters and spelling dictionaries @xcite ), and; (ii) due to the inability of synthetic data to capture the complete error distribution of the target eval sets, the final model is obtained by following a multi-stage fine-tuning process @xcite @xcite . because of this, carefully picking the learning rates and number of training steps for each of the fine-tuning stages is required, making it difficult to replicate and build on top of previous best reported models. the ideas of leveraging self-supervised pretraining and increasing the model size have yielded significant improvements on numerous seq2seq tasks in recent years @xcite @xcite @xcite , but these approaches have been applied to gec to only a limited extent. in this paper we adopt the mt5 @xcite as our base model which has already been pretrained on a corpus covering 101 languages. to adapt the model to the gec task, we design a fully unsupervised language-agnostic pre-training objective that mimics corrections typically contained in labeled data. we generate synthetic training data by automatically corrupting grammatical sentences, but in contrast to the previous state-of-the-art by @xcite for low-resources languages, we use our synthetic pre-training to train a single model on all 101 languages, employing no language-specific priors to remain fully languageagnostic. after pre-training we further fine-tune our model on supervised gec data for available languages (with data conditions ranging from millions to tens of thousands). additionally, we explore the effect of scaling up the model size from 60m to 11b parameters. we surpass the previous state-of-the-art results on four evaluated languages: english, czech, german and russian. fine-tuning and running inference with our largest and most accurate models require multi-gpu/tpu infrastructure. to make the results of our research widely accessible we release a clang-8 dataset obtained by using our largest gt5 model to clean up the targets of the frequently used yet noisy lang-8 dataset. we show that offthe-shelf variants of t5 @xcite when fine-tuned only on clang-8, outperform those models trained on the original lang-8 data with and w/o additional fine-tuning data, thus simplifying the complex multi-stage process of training gec models. thus clang-8 not only allows others to easily train highly competitive gec models, but it also greatly simplifies gec training pipeline, basically reducing a multi-step fine-tuning processs to a single fine-tuning step. our contributions in this paper are three-fold: ## model our model builds on top of mt5 @xcite a multilingual version of t5 @xcite )a transformer encoder-decoder model which has been shown to achieve state-of-the-art results on a wide range of nlg tasks. mt5 comes in different sizes, however for this work we use base (600m parameters) and xxl (13b parameters). ## gt5: large multilingual gec model fine-tuning datasets. for english, we fine-tune our pre-trained models on the fce @xcite and w&i @xcite corpora. for czech, german, and russian, we use the akces-gec @xcite , falko-merlin @xcite , and rulec-gec (rozovskaya and roth, 2019) datasets, respectively. training regime. we experimented with several training setups. all of them build on the mt5 pre-trained models (section 2.1). we experimented with a) mixing gec pre-training data (section 2.2) with fine-tuning data (section 3), b) mixing pretraining and finetuning examples but annotating them with different prefixes, and c) first using gec pre-training until convergence and then fine-tuning. while c) is the most computationally expensive approach, it also gave us the best results. gec pre-training as well as finetuning uses a constant results. for english, we evaluate on standard benchmarks from conll-14 and the bea test @xcite , while we use conll-13 as the development set ( ## clang-8: cleaned lang-8 corpus to be able to distill the knowledge learned by gt5 xxl into smaller, more practical models, we create and release clang-8, a cleaned version of the popular lang-8 corpus. as discussed earlier, lang-8 is a large corpus of texts written by language learners and user-annotated corrections to these texts. however, corrected texts frequently contain unnecessary paraphrasing and erroneous or incomplete corrections -phenomena that hurt the performance of a gec model trained on this data. for instance, the following source-target pair we experiment with two approaches for cleaning the data. first, to create clang-8, we generate new targets for lang-8, disregarding the original targets. we tried using both the unsupervised model, which was trained using the gec pretraining objective (section 2.2) and the supervised model (gt5 xxl) (section 3), but the former did not yield comparable results, so all reported numbers use the supervised model. second, to create clang-8-s, we used the unsupervised and the supervised models to score the original targets, disregarding the lowest scoring 20%, 50%, 70%, or 90% targets. disregarding 50% was the best performing setup and there was not a significant difference between the supervised and unsupervised model. we therefore report numbers using the unsupervised model disregarding the worst 50% of the targets. experiments. to evaluate the effect cleaning lang-8 has for english, we train two distinct models on this data: t5 @xcite , a monolingual sequence-to-sequence model, and fe-lix @xcite , a non-auto-regressive text-editing model. 5 we also tried fine-tuning these models on bea (i.e. fce and w&i) after finetuning them on clang-8, but this did not further improve the scores but slightly decreased them, e.g. 0.43 absolute decrease for bea test when using t5 base. this can be explained by the fact that 5 the felixinsert variant which we use does not employ re-ordering. the model used to clean the target texts has already been trained on bea. this suggests that the typical gec training pipeline where a model is first fine-tuned on lang-8 and then on bea can be both simplified and made more accurate by only fine-tuning on clang-8. finally, we train mt5 models on the german and russian portions of the clang-8 dataset and evaluate these models on the test sets from ## conclusion in this paper we report new state-of-the-art results on gec benchmarks in four languages we studied. our simple setup relies on a language-agnostic approach to pretrain large multi-lingual language models. to enable the distillation of our largest model into smaller, more efficient models, we released a cleaned version of the lang-8 dataset, enabling easier and even more accurate training of gec models.