Diversifying language models for lesser-studied languages and language-usage contexts: A case of second language K orean (2023)

Abstract: This study investigates the extent to which currently available morpheme parsers/taggers apply to lesser-studied languages and language-usage contexts, with a focus on second language (L2) Korean. We pursue this inquiry by (1) training a neural-network model (pre-trained on first language [L1] Korean data) on varying L2 datasets and (2) measuring its morpheme parsing/POS tagging performance on L2 test sets from both the same and different sources of the L2 train sets. Results show that the L2 trained models generally excel in domain-specific tokenization and POS tagging compared to the L1 pre-trained baseline model. Interestingly, increasing the size of the L2 training data does not lead to improving model performance consistently.

## introduction computational accounts for language science are gaining momentum in bringing together theories and data on second language (l2) research, offering promising directions toward interdisciplinary collaboration (e.g., @xcite @xcite . this often involves leveraging nlp techniques to address learner language characteristics in sizable l2 corpora. specifically, automatic processing of l2 data, including part-of-speech (pos) tagging, has become crucial to better understand the characteristics of learner language (e.g., @xcite @xcite . despite the emerging trend, we identify two major concerns in the field. one lies in the research practice that applies the currently available nlp tools, predominantly trained on first language (l1) and general-purpose data, directly to l2 data. considering the fact that no single corpus perfectly captures all aspects of how language is used (meurers and @xcite , it is reasonable to think that the types and characteristics of language-usage data affect the way in which specific inquiries are investigated properly on the basis of those data @xcite . given this aspect, it may be the case that currently available parsing/tagging models, exclusively trained and evaluated on l1 data, do not work reliably and optimally for l2 data (cf. @xcite @xcite . another concern is the commitment to diversity, equity, and inclusion (dei) within research practice. notably, there is a strong bias to a limited range of languages (e.g., [l2-]english) and language-usage contexts (e.g., general-purpose l1 usage) @xcite . this sampling bias poses a threat to dei in the field, as well as weakening the generalizability of implications from previous findings to wider situations. this calls for researchers' attention to lesser-studied languages and language-usage contexts. with this background, we investigate how well the current morpheme parsers/taggers cater to l2 korean, a lesser-studied language and languageusage context. in particular, we explore a potential boost in performance when an l1 pre-trained model is additionally trained on l2 data with varying sizes, annotation schemes, and data cleaning processes. to preview, the results demonstrate that (1) models fine-tuned with l2 data outperform the l1 pre-trained baseline when evaluated on test sets from the same domain and (2) simply increasing the size of the l2 training data does not invariably lead to improved the fine-tuned models' performance. this paper is structured as follows: we discuss the importance of morphological analysis in korean studies and how morphological analyzers are applied in l2-korean research ( §2). next, we outline experiments including datasets and evaluation metrics ( §3). we delve into two subsequent experiments: the first involves fine-tuning an l1 model using the full scope of the l2 datasets ( §4), while the second focuses on rigorous data cleaning in one of the l2 datasets ( §5). comprehensive analyses of morpheme tokenization and pos tagging accuracy for both experiments are presented. finally, we summarize our findings and propose future directions ( §6). ## experiment 1 the goal of experiment 1 was to leverage the sizable l2-nikl dataset to the fullest extent, despite some data loss during the data pre-processing stage. ## experiment 2 the goal of experiment 2 was to address potential issues inherent in the l2-nikl-o dataset, which may have aggravated the performance of the model. therefore, we undertook further data-cleaning as outlined below. ## conclusion and future works through the two experiments above, we have revealed three key aspects with regard to constructing l2 models. first, models fine-tuned on l2 data can outperform an l1 baseline when evaluated within the same domain. second, model performance can exhibit asymmetry and variability in zero-shot evaluations. lastly, augmenting the size of the training dataset does not always lead to enhancing the performance of a fine-tuned model. the findings of this study thus confirm domain-specificity in light of data processing and highlight the importance of input quality, including rigorous annotations (conducted by human annotators as in the case of the l2-klm) or cleaning (as exemplified in the case of the l2-nikl-c) when fine-tuning l2 models on the basis of existing l1-oriented, pre-trained models. effectively augmenting computational resources for lesser-studied languages and language-usage contexts requires meticulously engineered and validated data-processing pipelines. this is done not only by expanding target l2 corpora with gold annotations but also by refining existing tagging schemes. for example, in our study, the l2-nikl corpus, while robust in quantity, lacks clear guidelines on whether annotations were manually created from scratch or generated with the assistance of automatic taggers 12 . building upon the find-ings of this study, researchers are encouraged to consider improving the quality of gold annotations for more reliable and robust results in automatic handling of l2 data. beyond the primary focus of this study on morpheme tokenization and pos tagging, extending the scope of investigation toward such tasks as dependency parsing would offer a holistic evaluation of automatic analyses on learner corpora. we believe this extension will contribute to not only promoting a sense of dei in research but also facilitating ai literacy-by improving researchers' understanding of how computational algorithms operate and how these can be applied to research purposes appropriately. moreover, pursuing this research direction would advance nlp applications in educational contexts, offering valuable resources on language learning and teaching.