UA - GEC : Grammatical Error Correction and Fluency Corpus for the U krainian Language (2023)

Abstract: We present a corpus professionally annotated for grammatical error correction (GEC) and fluency edits in the Ukrainian language. We have built two versions of the corpus – GEC+Fluency and GEC-only – to differentiate the corpus application. To the best of our knowledge, this is the first GEC corpus for the Ukrainian language. We collected texts with errors (33,735 sentences) from a diverse pool of contributors, including both native and non-native speakers. The data cover a wide variety of writing domains, from text chats and essays to formal writing. Professional proofreaders corrected and annotated the corpus for errors relating to fluency, grammar, punctuation, and spelling. This corpus can be used for developing and evaluating GEC systems in Ukrainian. More generally, it can be used for researching multilingual and low-resource NLP, morphologically rich languages, document-level GEC, and fluency correction. The corpus is publicly available at https://github.com/grammarly/ua-gec

## introduction grammatical error correction (gec) is a task of automatically detecting and correcting grammatical errors in written text. gec is typically limited to making a minimal set of grammar, spelling, and punctuation edits so that the text becomes free of such errors. fluency correction is an extension of gec that allows for broader sentence rewrites to make a text more fluent-i.e., sounding natural to a native speaker @xcite . over the past decade, nlp researchers have been primarily focused on english gec, where they indeed made substantial progress: f 0.5 score of the best-performing model in the conll-2014 shared task has increased from 37.33 in 2014 to 68.75 in 2022 @xcite . multiple available datasets and shared tasks were a major contributing factor to that success. however, languages other than english still present a set of challenges for current nlp methods. mainstream models developed with english in mind are suboptimal for morphologically rich languages as well as languages with differing grammar @xcite @xcite . the common issue is a scarcity of data-particularly high-quality annotated data that could be used for evaluation and fine-tuning. more recently, the nlp community has started to pay more attention to non-english nlp @xcite . this positive recent trend manifests itself in the creation of new gec corpora for mid-and lowresource languages: german, czech, and spanish, to name a few @xcite @xcite . these datasets are important to expand nlp research to new languages and to explore new ways of training models in a low-resource setting. furthering that trend, we present a corpus annotated for grammatical errors and fluency in the ukrainian language: ua-gec. we first collected texts from a diverse pool of writers, both native and non-native speakers. the corpus covers a wide variety of domains: essays, social media posts, chats, formal writing, and more. we recruited professional proofreaders to correct errors related to grammar, spelling, punctuation, and fluency. our corpus is open source for the community foot_0 under the cc-by 4.0 license. to summarize, our contributions are as follows: ## data collection in this section, we describe the collection of texts with errors in the ukrainian language. section 3 will explain the annotation details. we have collected 1,872 texts (33,735 sentences) written by 492 unique contributors. the average length of a text snippet is 18 sentences. we partition the corpus into training and test sets. each split consists of texts written by a randomly chosen disjoint set of people: all of a particular person's writing goes to exactly one of the splits. to better account for alternative corrections, we annotated the test set two times @xcite . the resulting statistics are shown in in order to collect the data, we created an online form for text submission. all respondents who contributed to the data collection were volunteers. to attract a socially diverse pool of authors, we shared the form on social media. it contained a list of questions related to gender, native tongue, region of birth, and occupation, making it possible to further balance subcorpora and tailor them so they meet the purpose of various nlp tasks. ## data annotation we enrolled two annotators on the project, both native speakers of ukrainian with a degree in ukrainian linguistics. one of them was a freelance editor, and the other was a teacher of ukrainian. in order to diversify the type of tasks one can perform using the corpus, we released two versions of ua-gec: gec+fluency and gec-only. the former surfaces spelling, punctuation, grammar errors as well as errors associated with unnaturalsounding sentence elements. the latter captures only gec errors, which makes it possible to perform tasks that are narrower and more objective in scope. gec+fluency. the annotation process encompassed two sequential subtasks: error correction followed by error labeling. we found that the given annotation design was more efficient than performing error correction and labeling in a combined mode as it would increase the cognitive load of the task. ## model to prove the utility of our dataset, we trained a simple baseline model. we fine-tuned mbart-50large @xcite on the ua-gec train data without any preprocessing or data augmentation, similarly to @xcite . the model was fine-tuned for 3 epochs using adam optimizer with a learning rate of 5e-5 and batch size of 8. we used greedy decoding. the full training cycle takes around 3 hours on a single nvidia p100 gpu. ## conclusion we release the first professionally annotated corpus. we hope it will facilitate further development of grammatical error correction in the ukrainian language. the corpus is made publicly available at https://github.com/grammarly/ua-gec under the cc-by 4.0 license.