Towards standardizing K orean Grammatical Error Correction: Datasets and Annotation (2023)

Abstract: Research on Korean grammatical error correction (GEC) is limited, compared to other major languages such as English. We attribute this problematic circumstance to the lack of a carefully designed evaluation benchmark for Korean GEC. In this work, we collect three datasets from different sources (Kor-Lang8, Kor-Native, and Kor-Learner) that covers a wide range of Korean grammatical errors. Considering the nature of Korean grammar, We then define 14 error types for Korean and provide KAGAS (Korean Automatic Grammatical error Annotation System), which can automatically annotate error types from parallel corpora. We use KAGAS on our datasets to make an evaluation benchmark for Korean, and present baseline models trained from our datasets. We show that the model trained with our datasets significantly outperforms the currently used statistical Korean GEC system (Hanspell) on a wider range of error types, demonstrating the diversity and usefulness of the datasets. The implementations and datasets are open-sourced.

## introduction writing grammatically correct korean sentences is difficult for learners studying korean as a foreign language (kfl) and even for native korean speakers due to its morphological and orthographical complexity such as particles, spelling, and collocation. its word spacing rule is complex since there are many domain-dependent exceptions, of which only around 20% of native speakers understand thoroughly @xcite . since korean is an agglutinative language @xcite , getting used to korean grammar is time-consuming for kfl learners whose mother tongue is nonagglutinative @xcite . however, despite the growing number of kfl learners @xcite , little research has been conducted on korean grammatical error correction (gec) because of the previously described difficulties of the korean language. another major obstacle to developing a korean gec system is the lack of resources to train a machine learning model. in this paper, we propose three datasets that cover various grammatical errors from different types of annotators and learners. the first dataset named kor-native is crowd-sourced from native korean speakers. second, kor-learner are from kfl learners that consists of essays with detailed corrections and annotations by korean tutors. third, kor-lang8 are similar with kor-learner except that they consist of sentences made by kfl learners but corrected by native koreans on social platforms who are not necessarily linguistic experts. we also analyze our datasets in terms of error type distributions. while our proposed parallel corpora can be served as a valuable resource to train a machine learning model, another concern is about the annotation of the datasets. most existing datasets do not have annotation, which makes it hard to use them for evaluation. a major weakness of human annotation is that (1) experts specialized in korean grammar are expensive to hire, (2) making them annotate a large number of parallel corpora is not scalable, and (3) the error types and schema are different by datasets and annotators, which is counterproductive. another way that we can analyze and evaluate on the dataset is by automatic annotation from parallel corpora. while there is already one for english called errant @xcite , there is no automatic error type detection system for korean. we cannot fully demonstrate and classify error types and edits by using errant, because korean has many different characteristics than english (section 4.5). this motivates us to develop an automated error correction system for korean (kagas), along with annotated error types of refined corpora using kagas. lastly, we build a simple yet effective baseline model based on bart @xcite trained from our datasets. we further analyze the generated outputs of bart on how the accuracy of each system differs by error types when compared with a statistical method called hanspell, 2 providing use cases and insights gained from analysis. to summarize, the contributions of this paper are as follows: evaluation m 2 scorer @xcite which measures precision, recall, and f 0.5 scores based on edits, is the standard evaluation metric for english gec models. it requires an m 2 file with annotations of edit paths from an erroneous sentence to a corrected sentence. however, it is expensive to collect the annotations by human workers as they are often required to have expert linguistic knowledge. when these annotations are not available, gleu @xcite , a simple variant of bleu @xcite , is used instead by the simple n-gram matching. another way of generating an m 2 file for english in a 2 https://speller.cs.pusan.ac.kr/ kor-learner kor-native kor-lang8 models early works on korean gec focus on detecting particle errors with statistical methods @xcite @xcite . a copy-augmented transformer @xcite by pre-training to denoise and fine-tuning with paired data demonstrates remarkable performance and is widely used in gec. recent studies @xcite @xcite apply this method for korean gec. on the other hand, @xcite show that bart @xcite , known to be effective on conditioned generation tasks, can be used to build a strong baseline for gec systems. following this work, we load the pre-trained weights from kobart, foot_0 a korean version of bart, and finetune it using our gec datasets. ## data collection we build three corpora for korean gec: kor-learner ( §3.1), kor-native ( §3.2), and kor-lang8 ( §3.3). the statistics of each dataset is described on ## kagas we propose korean automatic grammatical error annotation system (kagas) that automatically aligns edits and annotate error types on parallel corpora that overcomes many disadvantages of handwritten annotations by human (appendix b.2). figure 2 shows the overview of kagas. as the scope of the system is to extract edits and annotate an error type to each edit, our system assumes the given corrected sentence is grammatically correct. then, our system takes a pair of the original sentence and the corrected sentence as input and output aligned edits with error types. we further extend the usage of kagas to analyze the generated text of our baseline models by each error type in ## experiments we conduct experiments to build an effective model to encourage future research on korean gec models. we report test accuracy using the model with the best validation gleu score. detailed experiment settings to reproduce our results appear in appendix d. ## conclusion in this work, we (1) construct three parallel datasets of grammatically incorrect and corrected korean sentence pairs for training korean gec systems: 25 note that we used the hanspell-cli github wrapper and not the browser to compare inference speed, for fair comparison. https://github.com/9beach/hanspell kor-lang8, kor-native, and kor-learner. our datasets are complementary representing grammatical errors that generated by both native korean speakers and kfl learners. (2) to train and evaluate models with these new datasets, we develop kagas, which considers the linguistic characteristic of korean and automatically aligns and annotates edits between sentence pairs. (3) we show our experimental results based on a pre-trained ko-bart model with fine-tuning on our datasets and compare them with a baseline system, hanspell. we expect that our datasets, evaluation toolkit, and models will foster active future research on korean gec as well as a wide range of korean nlp tasks. future work includes on further refining our proposed method, kagas, by extending the coverage and making more accurate error type classification.