Comparative Study of Models Trained on Synthetic Data for U krainian Grammatical Error Correction (2023)

Abstract: The task of Grammatical Error Correction (GEC) has been extensively studied for the English language. However, its application to low-resource languages, such as Ukrainian, remains an open challenge. In this paper, we develop sequence tagging and neural machine translation models for the Ukrainian language as well as a set of algorithmic correction rules to augment those systems. We also develop synthetic data generation techniques for the Ukrainian language to create high-quality human-like errors. Finally, we determine the best combination of synthetically generated data to augment the existing UA-GEC corpus and achieve the state-of-the-art results of 0.663 F0.5 score on the newly established UA-GEC benchmark. The code and trained models will be made publicly available on GitHub and HuggingFace.

## introduction grammatical error correction (gec) models have achieved significant results for english @xcite . however, gec for ukrainian is an open challenge for multiple reasons. even though ukrainian is the official language of ukraine with more than 40 million speakers worldwide foot_2 , there are still few nlp corpora, studies, or tools available @xcite . this lack of resources may be explained by the small pool of speakers (less than one percent of the world population), but also the many intrinsic difficulties of ukrainian, including the historical suppression of the language by the ussr, the high prevalence of its mixture with russian (surzhyk), and the formal context in which texts are commonly written @xcite . the biggest difficulty is that ukrainian is a low-resource language, and has only one annotated gec dataset available @xcite and few high-quality pre-trained transformer models compared to english 4 . additionally, ukrainian is a morphologically complex language, which makes its grammar correction more challenging with token-based models. the ukrainian language also does not have rigid word order, which makes syntactic analysis more difficult. finally, ukrainian grammar contains lots of exceptions that are not widely known within the main body of native speakers, which makes gec even more challenging @xcite . in this paper, we outline the thought process behind and the development of seq2tag and nmt models for the ukrainian language. moreover, we outline the creation of algorithmic correction rules in addition to those architectures. we also assemble a clean corpus of 1mln sentences and develop generators of high-quality human-like errors for it. finally, we compare our models and find the best combination of synthetically-made data with the existing ua-gec corpus and achieve the best results on the newly established ua-gec benchmark. ## synthetic data generation to train and test our models, we rely on the generation of synthetic data. in the following subchapters, we explain what techniques we use to imitate natural errors for the large corpus of correct sentences. many of our modules include detailed controls that allow us to modify data to suit our needs. this means we can provide high-quality data, with error patterns representative of those available in humanannotated corpora. ## results we train and evaluate more than 80 models to compare how well different model types, in conjunction with different synthetically generated data, perform for ukrainian grammar correction. a full list of all models can be found in the appendix. we will provide the most important findings in this section. all models are trained using google colab gpus. we evaluate all our models on the ua-gec development set @xcite set using the m 2 scorer provided by the unlp shared task. ## conclusions we have investigated most of the state-of-the-art gec approaches in the english language and tried to appropriate them for the ukrainian language. we found that the most efficient gec system can be developed using the nmt approach, however, seq2tag has a lot of room for research. our best model gets the 0.632 f 0.5 score on the ua-gec dataset, establishing the state-of-the-art benchmark. moreover, the results suggest that adding a mix of punctuation errors, russism errors, and clean data to the ua-gec training data achieves the best results. overall, we found that data quality is much more important than the amount of raw data. therefore, we suggest that human-annotated gec data is the most promising direction for future research.