The UNLP 2023 Shared Task on Grammatical Error Correction for U krainian (2023)

Abstract: This paper presents the results of the UNLP 2023 shared task, the first Shared Task on Grammatical Error Correction for the Ukrainian language. The task included two tracks: GEC-only and GEC+Fluency. The dataset and evaluation scripts were provided to the participants, and the final results were evaluated on a hidden test set. Six teams submitted their solutions before the deadline, and four teams submitted papers that were accepted to appear in the UNLP workshop proceedings and are referred to in this report. The CodaLab leaderboard is left open for further submissions.

## introduction grammatical error correction (gec) is an important task in natural language processing (nlp) that aims to automatically detect and correct grammatical errors in a given text. with the rapid growth of digital communication, gec has become increasingly important in improving the quality of written communication. however, gec is a complex task, especially for languages with complex grammar rules and rich morphology such as ukrainian. lack of large annotated and unlabeled datasets poses another challenge. shared tasks were a major contributing factor to the gec progress in other languages: @xcite @xcite @xcite @xcite @xcite @xcite . following that trend and to promote the development of gec systems for ukrainian, we organized the unlp 2023 shared task on grammatical error correction for ukrainian. the shared task was organized as part of the second ukrainian nlp workshop (unlp 2023) colocated with eacl'2023. the remainder of the paper is organized as follows. section 2 describes the task. section 3 describes the dataset. section 4 explains how the submissions were evaluated. finally, section 5 presents the results of the participating teams. ## task description the unlp 2023 shared task required the participating systems to correct a text in the ukrainian language to make it grammatical or both grammatical and fluent. consequently, two tracks were suggested: gec-only and gec+fluency. we made this distinction because fluency errors are more subjective and thus harder to correct. in the gec-only track, the participating systems were expected to correct grammar, spelling, and punctuation errors in the test set. the gec+fluency track added fluency errors to that list. fluency errors include word calques, stylistically inappropriate words, repetitions, or any other constructions that sound unnatural to a native speaker. it was not mandatory to participate in both tracks, i.e., participating in either gec-only or gec+fluency was acceptable. error classification was out of the scope of the shared task. we provided the participants with a preprocessed version of the ua-gec corpus @xcite for training and validation (see section 3 for details) but also encouraged them to use any external data of their choice. evaluation scripts were provided together with the data. we set up a codalab environment 1 to manage system submissions and the leaderboard. the participants submitted their system results to codalab, which automatically evaluated their results on a hidden test set and returned the scores. we used f 0.5 computed by errant @xcite as the primary metric. the leaderboard is still open for further submissions. ## data the unlp 2023 shared task for grammatical error correction utilizes the ua-gec dataset @xcite as the primary source for training, evaluation, and test data. we chose this dataset due to its relevance to the task at hand. for hyperparameter tuning and evaluation during development, we created a separate validation set by extracting 87 documents (1,419 sentences) from the ua-gec test set. in order to assess the final performance of the participating models, we formed a test set containing another 79 documents (1,271 sentences) from the remaining samples in the ua-gec test set. each sentence in both test set and validation set was annotated by two independent annotators. this dual annotation approach ensures a more accurate evaluation of model performance, taking into account the discrepancies and variations between human annotators. we provide training and validation data in three formats: test data is provided only as tokenized and nontokenized source text files. the participants had the freedom to choose which version of the data to utilize for training their models. we employed the stanza tokenization tool @xcite to tokenize the data and prepared a tokenization script for the participants. preserving the document structure allowed the participants to make use of document-level context in their models. to achieve this, sentences were kept in the order in which they appeared within their respective documents. document headers were appended before a sequence of a document's sentences to retain this structure. these headers followed a specific format: "# [0-9]{4}", where an example would be "# 1234". this approach facilitated the incorporation of document-level context while maintaining consistency across the datasets. ## evaluation the primary evaluation metric used for the shared task is the f 0.5 score, which combines the precision and recall metrics while weighing precision more than recall. this metric was computed using the errant tool @xcite , a widely-accepted tool for evaluating grammatical error correction. in addition to reporting the f 0.5 scores, the evaluation script also reports other metrics: precision, recall, true positives (tp), false positives (fp), and false negatives (fn). furthermore, the evaluation script reports error detection metrics. however, these are provided merely for reference and are not considered while comparing the participating models. detection metrics can be insightful in understanding how well a system identifies errors in the text, without necessarily focusing on the correction. all evaluation is done on tokenized data. if the participants choose to train a model that produces ## participating systems a total of fifteen teams registered for the unlp 2023 shared task, but only six teams submitted their solutions before the deadline. four teams submitted papers that were accepted to appear in the unlp workshop proceedings and are referred to in this report. two more teams provided their system descriptions by email. three teams submitted their results for both gec-only and gec+fluency tracks, and three more teams submitted their results only for gec+fluency. we briefly review the systems here; for complete descriptions, please see the corresponding papers. pravopysnyk @xcite , the winners of the gec+fluency track, combined a transformer-based model with a rule-based spelling correction system. for the transformer-based model, they fine-tuned mbart @xcite on ua-gec augmented by synthetically generated errors. to generate more data, the team used round-trip translation, a custom punctuation error generation script, and replacing ukrainian words with their russified versions. for spelling correction, the team applied the symspell algorithm @xcite to the ukrainian language. this algorithm uses a word frequency dictionary and a bigram frequency dictionary based on the dataset of 500k sentences collected from ukrainian books. the most frequent word that passes the spelling criteria is then selected. the transformer-based model was responsible for most corrections. the advantages of the system include high performance, low training cost (training takes 10 minutes on google colab a100 gpu), and its end-to-end training setup, which allows combining different sources of synthetic data. however, the system is slower when compared to sequence tagging models. the authors published the system on the huggingface platform: https://huggingface.co/ pravopysnyk/best-unlp . qc-nlp @xcite , the winners of the gec-only track and second place holders of the gec+fluency track, submitted 2 systems: (1) fpg and (2) rozovska. both systems participated in the two tracks of the shared task. system (1) achieved stronger performance in both tracks than system (2), but system (1) requires more computational resources. in system (1), the authors fine-tuned a pretrained mt5-large @xcite to correct ungrammatical sentences to their grammatical counterparts. they first fine-tuned the model with 10m synthetically generated grammati-cal error correction examples for three epochs and then with the shared task dataset for 10 additional epochs. the synthetic examples were generated using the approach based on the aspell confusion sets proposed in @xcite . the method was applied to the native ukrainian data from the wnt news crawl corpus. fine-tuning on synthetic and learner data was done with 8 nvidia 80gb gpus taking approximately 16 hours to train in total. system (2) is a transformer model proposed in @xcite pre-trained on 35m synthetic examples that use aspell confusions and additional noise from round-trip translation and fine-tuned on the gold learner training data. three models were trained with three different seeds, and the final model is an ensemble of the three best checkpoints. pre-training on 1 nvidia 32gb gpu took 7 hours per epoch for about 10 epochs until convergence. fine-tuning took about an hour until convergence. the authors published the systems on github: webspellchecker (didenko and sameliuk, 2023) used a custom transformer-like architecture called redpennet. the architecture leverages a pre-trained mlm encoder along with a shallow decoder to generate both replacement tokens and spans for editing gec cases. during the generation of edit tokens, the encoder-decoder attention weights determine the edit spans (start and end) that point at the position of the edit in the source sentence. edit tokens are predicted in the autoregressive way. sep tokens separate edits in the output sequence. at each step of the feedback loop, the edit bpe token embedding is combined with a decoder-specific trainable positional encoding embedding. the resulting sum is then concatenated with the span embedding. additionally, compact gec task-specific decoder bpe vocabularies are trained to lower the cost of the pre-softmax dot operation, thus improving the efficiency of predicting replacement tokens. the main advantage of redpennet is the ability to implement any source-to-target transformation using a minimal number of autoregressive steps, which makes it possible to effectively solve the gec cases, including interrelated and multi-token edits. however, due to the tailored architecture of redpennet, there are no out-of-the-box solu-tions available for data preprocessing, training, or fine-tuning. thus, convenient tools like the hug-gingface infrastructure cannot be used for rapid model fine-tuning and deployment. the system repository: https://github.com/ webspellchecker/unlp-2023-shared-task . final submission by maksym tarnavskyi uses a sequence tagging gector model @xcite that contains a transformer-based encoder stacked with two output linear layers that are responsible for error detection and error correction. the author trained the model only on ua-gec data without any synthetic data pre-training or hyperparameter optimization. an ukr-roberta-base foot_2 model is used to initialize the encoder. the system repository: https://github.com/ makstarnavskyi/gector-large . model checkpoints: https://drive.google. com/drive/folders/1zwjjwzrtqacs48z_ h4t1mivzf5nu3h_0 . grammarua by anastasiia hudyma uses mbart50 @xcite , a sequence-tosequence model that was fine-tuned on the shared task training and validation data. this model was chosen because of good results for low-resource languages. the author published the system on the huggingface platform: https://huggingface.co/ smartik/mbart-large-50-finetuned-gec ua-gec system is the baseline for ukrainian gec presented in @xcite . the team used mbart50-large @xcite fine-tuned on the unprocessed training data. training takes around 3 hours on a single nvidia p100 gpu. ## conclusion we believe that the unlp 2023 shared task was instrumental in facilitating research on grammatical error correction for the ukrainian language, and we hope the insights from the teams' research will be useful to the nlp community. all the data and evaluation scripts used in the shared task are available on github, and the competing systems were openly published, which contributes to the reproducibility of the shared task results. the codalab environment remains open for further submissions, although any such submissions will be considered outside of the unlp 2023 competition.