Towards L 2-friendly pipelines for learner corpora: A case of written production by L 2- K orean learners (2023)

Abstract: We introduce the Korean-Learner-Morpheme (KLM) corpus, a manually annotated dataset consisting of 129,784 morphemes from second language (L2) learners of Korean, featuring morpheme tokenization and part-of-speech (POS) tagging. We evaluate the performance of four Korean morphological analyzers in tokenization and POS tagging on the L2- Korean corpus. Results highlight the analyzers’ reduced performance on L2 data, indicating the limitation of advanced deep-learning models when dealing with L2-Korean corpora. We further show that fine-tuning one of the models with the KLM corpus improves its accuracy of tokenization and POS tagging on L2-Korean dataset.

## introduction the use of learner corpora has played a crucial role in understanding language learners' developmental aspects (e.g., @xcite @xcite . with the recent advancement of computational methods and techniques, automatic processing of learner corpora (together with sizeable datasets) is gaining momentum for a better understanding of the properties of learner language (e.g., @xcite @xcite . despite the increasing interest in this approach, we identify two major caveats in the current research practice. one is the sampling bias towards dominant/hegemonic viewpoints and discourse, especially centering around a limited range of languages and language-usage contexts (e.g., l2 english) (c.f., @xcite . this poses a threat to linguistic diversity, equity, and inclusion in the field, as well as weakening the generalizability of previous findings to other (and lesser-studied) languages. the other caveat concerns the degree to which first language (l1)-based automatic processing pipelines work for l2 data. indeed, a line of research has questioned the reliability of currently existing parsing/tagging models, which are trained and tested exclusively on the basis of l1 data, when applied to l2 corpora (e.g., @xcite . this is because these l1-oriented models may not fully account for the characteristics of learner language, including spacing/spelling errors and novel combinations of words and phrases. these factors may negatively impact the performance of l1-based tools when analyzing linguistic features of l2 corpora, thus necessitating empirical investigation. in this study, we aim to address these caveats by developing a sizable l2-korean corpus, featuring enhanced morpheme tokenization and pos tagging of the open-access l2-korean corpus dataset, which comprises 129,784 morphemes (7,527 sentences). using this dataset, we evaluate the morpheme tokenization and pos-tagging accuracy of two language-general parsers incorporating cuttingedge algorithms (stanza, trankit) and two koreanspecific parsers commonly used by researchers in korean studies @xcite . this paper is structured as follows: we discuss the significance of morphological analysis in korean studies and review relevant l2-korean applied research. next, we outline the annotation process employed in our study. we then elaborate on our methodology for evaluating the performance of the morpheme analyzers on our dataset, using an l1 corpus as a reference. following this, we present a comprehensive analysis of the overall performance, including detailed comparisons across different proficiency levels and pos tags, as well as a re-evaluation of performance after training the l2 annotated corpus. finally, we summarize our findings and propose future directions. ## dataset the korean-learner-morpheme (klm) corpus, as it currently stands, comprises 129,784 morphemes (67,284 eojeols, which are sequences of korean characters separated by white-spaces) with morpheme tags grounded in the sejong tag set (appendix a). this corpus was sourced from the kyung hee korean learner written corpus collected by @xcite . the corpus encompasses data on classroom proficiency levels (ranging from 1 to 6 as a proxy for learner proficiency), nationality, gender, and writing topics. to create our dataset, we randomly extracted a total of 600 texts from the original corpus, with each proficiency level represented by 100 texts. despite the presence of morpheme tokenization and pos tags in the original corpus, several issues prevented its direct use for evaluation purposes, which ultimately led us to conduct manual annotations. first, without gold annotations for the data, we were not able to determine the accuracy of the automatic pos tagger (i.e., espresso) that @xcite used for morphological analysis. additionally, we were uncertain whether the annotation scheme in the original cor-pus had been thoroughly tested, taking into account the language-specific properties of korean. second, we were unsure how the characteristics of learner language (e.g., spelling/spacing errors), which were not clearly indicated in the original corpus, were documented in the annotations (e.g., whether they were corrected or neglected during the automatic analysis). on top of these issues, the formatting proved difficult to process the data automatically. to create our corpus, we first reformatted the texts into conll-u format, following the universal dependencies (ud) formalism (c.f., @xcite . to ensure the metadata in the original dataset, we associated the respective # text_id attribute with the extracted metadata (e.g., # @xmath0.g., ## conclusion 6.1 summary of findings in this study, we presented a manually annotated l2-korean corpus and evaluated the performance of korean morphological analyzers pretrained on l1 datasets for tokenization and pos tagging on l2-korean data. the klm corpus and related resources are publicly accessible at: https://github.com/nlpxl2korean/ korean_learner_morpheme_corpus . the results revealed that morphological analyzers exhibited somewhat lower performance on l2-korean data in comparison to their performance on l1 datasets. a detailed analysis of pos tags showed that several essential morphological tags, including predicate-and suffix-related tags, displayed relatively low accuracy. however, the study demonstrated that substantial improvements in morpheme tokenization and pos tagging performance for l2-korean data could be attained by incorporating l2 data into the training sets, even with the relatively small dataset. although no study has specifically focused on l2-korean data so far, these findings align with previous studies on l2-english ud treebanks (e.g., @xcite .