A Low-Resource Approach to the Grammatical Error Correction of U krainian (2023)

Abstract: We present our system that participated in the shared task on the grammatical error correction of Ukrainian. We have implemented two approaches that make use of large pre-trained language models and synthetic data, that have been used for error correction of English as well as low-resource languages. The first approach is based on fine-tuning a large multilingual language model (mT5) in two stages: first, on synthetic data, and then on gold data. The second approach trains a (smaller) seq2seq Transformer model pre-trained on synthetic data and fine-tuned on gold data. Our mT5-based model scored first in “GEC only” track, and a very close second in the “GEC+Fluency” track. Our two key innovations are (1) finetuning in stages, first on synthetic, and then on gold data; and (2) a high-quality corruption method based on roundtrip machine translation to complement existing noisification approaches.

## introduction this paper describes our submission in the shared task on the grammatical error correction (gec) of ukrainian @xcite that was organized as part of the workshop on ukrainian natural language processing (unlp 2023), in conjunction with eacl 2023. ukrainian is an indo-european language from the east-slavic language family, and is most closely related to russian and belarusian. in the context of gec, ukrainian is a low-resource language and is under-explored. a dataset of ukrainian native and non-native texts annotated for errors was recently released @xcite , however, to the best of our knowledge, no systems have been benchmarked on this dataset. we have developed two approaches. the first approach is based on the method proposed in earlier work @xcite that finetunes a multilingual mt5 model on gold gec data. 2 because mt5 is pre-trained with an objective that is not appropriate for gec, we propose a 2-stage finetuning strategy, where we finetune first on native data with synthetic noise, and then further finetune on the gold gec data. we show that this two-stage approach is beneficial and provides a large boost compared to an mtt5 model finetuned on gold data only. our model scored first in the "gec only" track and a very close second in the "gec+fluency" track (0.08 point difference from the top submission). our second system is a smaller seq2seq transformer model pre-trained on synthetic data and finetuned on gold data. we propose a novel method of generating synthetic errors using backtranslation. unlike previous approaches, we do not use full-sentence translations but only extract backtranslation pairs that are then used for introducing errors in native data. we present related work on gec in section 2. section 3 describes our approach. section 4 briefly describes the ukrainian gec dataset. section 5 presents our experimental results on the validation and test data, as well as additional evaluation by error type. section 6 concludes. ## background most effort in gec research concentrated on correcting errors made by english as second language writers. more recently, there has been interest in developing approaches and resources in gec for other languages, including arabic @xcite , german @xcite , russian @xcite , chinese, and spanish @xcite . earlier approaches to gec include rule-based methods and machine learning classifiers for correcting a specific type of mistake (e.g. article or preposition) @xcite @xcite . for an overview of approaches and methods in gec, we refer the reader to @xcite . current approaches to gec can be broken down into two categories: sequence-tosequence (seq2seq) generation @xcite @xcite , and sequence-toediting (seq2edits) @xcite @xcite . both approaches achieve state-of-the-art performance on english gec. in the seq2edits framework, the task is viewed as a sequence labeling problem @xcite that tags text spans with appropriate error tags, leaving the rest of the text unchanged. because the seq2edits approach requires human input, as it depends on constructing languagespecific edit operations, we adopt the seq2seq framework. seq2seq approaches have demonstrated strong empirical results in gec @xcite @xcite @xcite @xcite @xcite @xcite . due to lack of gold training data, it is common to first pre-train a model on native data where the source side has been corrupted with artificial noise. the pre-trained model is typically further finetuned on the available gold data. pre-trained language models (plms) recently, finetuning plms has become a standard paradigm for many nlp tasks. in gec, plms have been mainly used in english where models have been finetuned on large amounts of hand-labeled data @xcite @xcite . @xcite apply plms in a multilingual setting, by finetuning bart @xcite . however, even when using a large number of hand-labeled examples, they achieve results that are way below state-of-the-art. in this work we adopt the approach of @xcite and make use of mt5 @xcite , a multilingual variant of t5 @xcite , a pre-trained text-to-text transformer. mt5 has been pre-trained on mc4 corpus, a subset of common crawl, covering 101 languages and composed of about 50 billion documents @xcite . @xcite finetune mt5 on gec gold data for russian, german, and czech languages, although sota results are only achieved, when they re-train mt5 with a different objective and use an extremely large model xxl with 13b parameters. we use the original mt5 models of smaller sizes and show that it is possible to achieve competitive results by pre-training first on synthetic data. ## the models we have implemented two approaches that draw on methods that showed competitive performance in multilingual low-resource settings. our first (larger) model makes use of mt5 but is finetuned in two stages -on synthetic data (we refer to this stage as pre-training on synthetic data), and then finetuning on gold data. our second (smaller) model is a seq2seq transformer model pre-trained on synthetic data (from scratch) and finetuned on gold data. as our baseline for the second model, we use a model pre-trained on synthetic data generated using standard spell-based transformations. we show that adding synthetic noise from back-translations results in a 3-point improvement over the baseline. because both approaches make use of synthetic data, we describe the data generation methods below. generating synthetic data standard data corruption methods typically use a variety of heuristics: random character and token transformations @xcite , confusion sets generated from a spellchecker @xcite , or a morphological analyzer @xcite , or round-trip translation @xcite . we have experimented with two baseline data generation techniques for low-resource settings: (1) spell-based transformations and (2) part-of-speech (pos)-based transformations. both of the methods rely on the idea of using confusion sets that specify for each target word occurring in a native corpus a list of highly confusable words. these lists are used to generate synthetic errors. spell-based transformations this approach showed state-of-the-art performance in english @xcite @xcite , and other languages @xcite . spell-based confusions include highly confusable words based on edit distance obtained from a dictionary available in a spellchecker. because aspell is an open-source spellchecker, it is common to use aspell to generate spell-based confusion sets. we use aspell with the ukrainian dictionary in this work to create spell-based confusions. more detail about the method can be found in @xcite . we follow @xcite for the parameter values for token replacement, deletions, and insertions. ## the ukrainian gec data the data used in the shared task comes from ua-gec, a corpus of social media texts written by native speakers and learners of ukranian @xcite . the shared task organizers have provided training and validation data. the training data is annotated with 1 reference, and the validation set is annotated with 2 references for each sentence. the gold corrections are provided in the standard m2 format @xcite , and the edits are labeled with the corresponding error tags. there are 24 error categories, broadly classified with a prefix into grammar (g) and fluency (f) corrections (with the exception of spelling and punctuation errors that do not have a prefix). the shared task includes 2 tracks: "gec+fluency", where the systems are evaluated with respect to all errors, and the "gec only" track where the fluency edits are removed. ## experiments and results below, we present experimental results for the two models that we implemented. our submissions for both tracks are the same, except that the models are finetuned on the gold data for each respective track. we first present a set of experiments on the data in the "gec+fluency" track. we report results of the submitted systems for both tracks in section 5.3. corrupting monolingual ukrainian data both models use synthetic data. we corrupt sentences from the ukrainian partition of cc-100 @xcite , which contains high-quality data from common crawl. we tokenize the data using stanza @xcite , the same tokenizer that is used to tokenize the gold data. we use spellbased transformations (see section 3) to corrupt the monolingual data (but see also 5.2). ## conclusion we have presented our submission that participated in the shared task on the grammatical error correction of ukrainian. our submission includes two systems. the first system pre-trains mt5 on synthetic data and finetunes on the gold gec data. we have shown that introducing this two-stage approach is crucial to achieving strong results when using mt5. we have also proposed a novel synthetic data generation method that extracts confusion pairs from multiple back-translation hypotheses that are aligned with the original sentence.