Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model (2020)

Abstract: Studies on grammatical error correction (GEC) have reported on the effectiveness of pretraining a Seq2Seq model with a large amount of pseudodata. However, this approach requires time-consuming pretraining of GEC because of the size of the pseudodata. In this study, we explored the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder-decoder model for GEC. With the use of this generic pretrained model for GEC, the time-consuming pretraining can be eliminated. We find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC.

## introduction grammatical error correction (gec) is the automatic correction of grammatical and other language-related errors in text. most works regard this task as a translation task and use encoderdecoder (enc-dec) architectures to convert ungrammatical sentences to grammatical ones. this enc-dec approach often does not require linguistic knowledge of the target language. strong enc-dec models for gec are pretrained with a large amount of artificially generated data, commonly referred to as 'pseudodata', that is created by introducing artificial error to a monolingual corpus. hereafter, pretraining using pseudodata aimed at the gec task is referred to as task-oriented pretraining @xcite @xcite . for example, @xcite generated a pseudo corpus using back-translation and achieved strong results for english gec. @xcite generated a pseudo corpus by introducing artificial errors into monolingual corpora and achieved the best scores for gec in several languages by adopting the methods proposed by @xcite . these task-oriented pretraining approaches require extensive use of a pseudo-parallel corpus. specifically, @xcite used 100m ungrammatical and grammatical sentence pairs, while @xcite and @xcite used 70m sentence pairs, which required time-consuming pretraining of gec models using the pseudo corpus. in this study, we determined the effectiveness of publicly available pretrained enc-dec models for gec. specifically, we investigated pretrained models without the need for pseudodata. we explored a pretrained model proposed by @xcite called bidirectional and auto-regressive transformers (bart). @xcite also proposed multilingual bart. these models were pretrained by predicting the original sequence, given a masked and shuffled sentence. the motivation for using these models for gec was that it achieved strong results for several text generation tasks, such as summarization; we refer to it as a generic pretrained model. we used generic pretrained bart models to compare with gec models using a pseudo-corpus approach @xcite @xcite . we conducted gec experiments for four languages: english, german, czech, and russian. the enc-dec model based on bart achieved results comparable with those of current strong enc-dec models for english gec. the multilingual model also showed high performance in other languages, despite only requiring fine-tuning. these results suggest that bart can be used as a simple baseline for gec. ## previous work the enc-dec approach for gec often uses the task-oriented pretraining strategy. for example, @xcite al. (2019) and grundkiewicz et al. (2019) reported that pretraining of the enc-dec model using a pseudo corpus is effective for the gec task. in particular, they introduced word-and character-level errors into a sentence in monolingual corpora. they developed a confusion set derived from a spellchecker and randomly replaced a word in a sentence. they also randomly deleted a word, inserted a random word, and swapped a word with an adjacent word. they performed these same operations, i.e., replacing, deleting, inserting, and swapping, for characters. the pseudo corpus made by the above methods consisted of 100m training samples. our study aims to investigate whether the generic pretrained models are effective for gec, because pretraining with such a large corpus is time-consuming. náplava and straka (2019) adopted grundkiewicz et al. (2019)'s method for several languages, including german, czech, and russian. they trained a transformer @xcite with pseudo corpora (10m sentence pairs), and achieved current state-of-the-art (sota) results for german, czech, and russian gec. we compared their results with those of the generic pretrained model to confirm whether the model was effective for gec in several languages. @xcite explored the generation of a pseudo corpus by introducing random errors or using back-translation. they reported that a taskoriented pretraining with back-translation data and character errors is better than that with pseudodata based on random errors. @xcite combined @xcite 's pretraining approach with bert @xcite and improved @xcite 's results. specifically, @xcite fine-tuned bert with a grammatical error detection task. the fine-tuned bert outputs for each token were combined with the original tokens as a gec input. their study is similar to our research in that both studies use publicly available generic pretrained models to perform gec. the difference between these studies is that @xcite used the architecture of the pretrained model as an encoder. therefore, their method still requires pretraining with a large amount of pseudodata. the current sota approach for english gec uses the sequence tagging model proposed by @xcite . they designed tokenlevel transformations to map input tokens to target corrections to produce training data. the sequence tagging model then predicts the transformation corresponding to the input token. we do not attempt to make a comparison with this approach, as the purpose of our study is to create a strong gec model without using pseudodata or linguistic knowledge. ## generic pretrained model bart @xcite is pretrained by predicting an original sequence, given a masked and shuffled sequence using a transformer. they introduced masked tokens with various lengths based on the poisson distribution, inspired by span-bert @xcite , at multiple positions. bart is pretrained with large monolingual corpora (160 gb), including news, books, stories, and web-text domains. this model achieved strong results in several generation tasks; thus, it is regarded as a generic model. they released pretrained models using english monolingual corpora for several tasks, including summarization, which we used for english gec. @xcite proposed multilingual bart (mbart) for a machine translation task, which we used for gec of several languages. the latter model was trained using monolingual corpora for 25 languages simultaneously. they used a special token for representing the language of a sentence. for example, they @xmath0. to fine-tune mbart for german, czech, and russian gec, we set the target language for the special token referring to that language. our models were fine-tuned using a single gpu (nvidia titan rtx), and our implementations were based on publicly available code 3 . we used the hyperparameters provided in some previous works @xcite , unless otherwise noted. the scores excluding the ensemble method were averaged in five fine-tuned experiments with random seeds. english. our setting for the english datasets was almost the same as that of @xcite . we extracted the training data from bea-train for english gec. similar to @xcite , we did not use the unchanged sentences in the source and target sides; thus, the training data consisted of 561,525 sentences. we used bea-dev to determine the best model. we trained the bart-based models by using bart.large. this model was proposed for the summarization task, which required some constraints in inference to ensure appropriate outputs; however, we did not impose any constraints because our task was different. we applied byte pair encoding (bpe) @xcite to the training data for the bart-based model by using the bpe model of @xcite . we used the m 2 scorer @xcite and gleu @xcite for conll-14 and jfleg, respectively, and used the errant scorer @xcite for beatest. we compared these scores with strong results @xcite . german, czech, and russian. the dataset settings in this study were almost the same as those 3 bart, mbart: https://github.com/pytorch/fairseq used by @xcite for each language. we used official training data and decided the best model by using the development data. in addition, we trained the mbart-based models for german, czech, and russian gec. we used mbart.cc25 for the mbart-based models. for the mbart-based model, we followed @xcite ; we detokenized foot_1 the gec training data for the mbart-based model and applied sentencepiece @xcite with the sentencepiece model shared by @xcite . using this preprocessing, the input sentence may not represent grammatical information, compared with the sentence tokenized using a morphological analysis tool and subword tokenizer. however, what preprocessing is appropriate for gec is beyond this paper's scope and will be treated as future work. for evaluation, we tokenized the outputs after recovering the subwords. then, we used a spacy-based foot_2 tokenizer for german foot_3 and russian foot_4 , and the morphodita tokenizer foot_5 for czech. moreover, the m 2 scorer was used for each language. we compared these scores with the current sota results @xcite . ## conclusion we introduced a generic pretrained enc-dec model, bart, for gec. the experimental results indicated that bart better initialized the enc-dec model parameters. the fine-tuned bart achieved remarkable results, which were comparable to the current strong results in english gec. indeed, the monolingual bart seems to be more effective for gec than the model with a multilingual setting. however, although it is not as good as sota, fine-tuned mbart exhibited high performance in other languages. this implies that bart is a simple baseline model for pretraining gec methods because it only requires fine-tuning as training.