Testing relevant linguistic features in automatic CEFR skill level classification for Icelandic (2025)

Abstract: This paper explores the use of various linguistic features to develop models for automatic classification of language proficiency on the CEFR scale for Icelandic, a low-resourced and morphologically complex language. We train two classifiers to assess skill level of learner texts. One is used as a baseline and takes in the original unaltered text written by a learner and uses predominantly surface features to assess the level. The other uses both surface and other morphological and lexical features, as well as context vectors from transformer (IceBERT). It takes in both the original and corrected versions of the text and takes into account errors/deviation of the original texts compared to the corrected versions. Both classifiers show promising results, with baseline models achieving between 62.2-67.1% accuracy and dual-version between 75-80.3%.

## introduction language skill level assessment is a critical component in language education and testing, and accurate and scalable methods for assessing skill levels can facilitate personalized learning, enhance testing systems, and contribute to linguistic research. however, automating this process presents significant challenges, especially for lowresourced languages such as icelandic. in this paper, we present findings from an ongoing study focused on using linguistic features to train models for automatic skill level assessment in icelandic as a second language (l2) texts on the cefr scale, a widely adopted framework in language education. by focusing on icelandic we aim to contribute to the growing body of work on underrepresented languages in natural language processing (nlp) and highlight the importance of broadening research efforts beyond high-resourced languages. the icelandic l2 error corpus (icel2ec), published in 2022 @xcite , has served as a foundational dataset for analyzing features associated with the cefr skill level. in particular, manually corrected text versions and error annotation have shown a high value in predicting proficiency levels through machine learning approaches @xcite . to explore automatic assessment further, this study builds on icel2ec and includes additional unpublished texts sourced from the university of iceland. using this combined dataset, several models were trained to test the efficacy of various features for the assessment of the cefr skill level. we present the results of baseline models using k-nearest neighbors (knn) algorithm which uses the learners' original texts and basic linguistic features, and "dual-version" models (logistic regression -lr) which integrate the corrected versions of the data, and more complex features. key research questions addressed in this study include: (1) how accurately can linguistic features predict writing skill levels in icelandic l2 texts, and ( 2 ) to what extent do corrected texts and advanced models like icebert @xcite contribute to improved classification performance? we incorporate surface features, morphological and lexical elements, and icebert-derived context vectors to provide a comprehensive approach to automatic skill level assessment. the paper is organized as follows: section 2 provides a background on l2 icelandic, the cefr scale, and automatic skill level detection. sections 3 and 4 detail our models and evaluation metrics, while section 5 presents the experimental results. ## background icelandic stands out among lesser spoken languages for its relatively robust digital resources @xcite . however, the resources available for icelandic as a learner language (l2) are sparse mainly due to l2 icelandic being a relatively recent phenomenon and collecting written data for l2 icelandic is challenging. however, in recent years the number of foreign nationals in iceland has surged. in the mid-1990s, only 2% of iceland's population were first-generation immigrants; by early 2023, this number reached approximately 17.3%. @xcite . this demographic shift has heightened the importance of developing resources for l2 icelandic. an essential aspect of teaching and assessing a second language is measuring learner skill level. the cefr standardizes skill level assessment with a six-level scale (a1 to c2), focusing on communicative competencies rather than specific linguistic structures (council of europe, 2018). icel2ec, developed under a government-sponsored language technology initiative (see @xcite , is the primary resource available for investigating cefrlabeled learner errors and interlanguage features; data for a new learner corpus is currently being collected to build on these foundations. automatic classification of skill level in written texts remains challenging due to the subjective nature of language proficiency scales like the cefr. a critical component in skill assessment involves the selection of linguistic features that effectively capture learners' proficiency. thus, with learner corpora and error tagging, researchers can identify relevant linguistic patterns that correspond to specific cefr levels. in english, for example, accuracy rates for automatic cefr classification range from 62.7% to 83.8% @xcite . using features derived from lexical, morphological, and syntactic patterns, classifiers like logistic regression and more advanced approaches have achieved promising results in multilingual proficiency assessment tasks, as seen in studies with l2 german, swedish, and estonian @xcite . importantly, model evaluation metrics must consider the proximity between cefr levels, recognizing that misclassifications between adjacent levels (e.g., c1 and b2) are less severe than those between distant levels (e.g., c1 and a1). additionally, language proficiency as-sessment carries significant implications, as its results can influence the learner's educational and professional opportunities. in this context, predicting a higher level is generally less harmful to the learner than predicting a lower one. ## model training this study establishes preliminary models for automated skill level classification. baseline models, utilizing only original texts, are compared with dual-version models that use both original and corrected texts. feature-based approaches yield high prediction accuracy, especially for morphologically rich languages (see @xcite , and this study combines surface, morphologic, and lexical features, as suggested in recent research (see @xcite @xcite , as well as combining context vectors from transformers and perplexity score, typically used to evaluate the performance of language models. for icelandic proficiency classification, we adapt representative models for these approaches, whose established performance in other languages provides additional context for the results we observe in icelandic. in this section we introduce the dataset, models and features selected for our task. ## evaluation both baseline and dual-version models were tested on several algorithms, including linear regression, svm, knn, lr, and mlp. after initial testing, knearest neighbors (@xmath0, while logistic regression was selected for the dual-version evaluations. each model and feature set was assessed using an 80/20 train-test split, with stratified sampling. it was repeated 1000 times with different random splits, and the reported metrics represent averages of these runs. accuracy, as the percentage of correct classifications, is sensitive to data distribution and may overlook false positives, disproportionately affecting smaller classes. additionally, accuracy does not account for the "distance of prediction," where predicting c1 instead of b2, for example, is a less severe error than predicting a1 instead of b2. for a more comprehensive evaluation, f1 scores were also calculated to provide a balance between precision and recall. alongside exact accuracy, we assessed adjacent accuracy, i.e. also viewing predictions within one level above or below the true level (e.g., a2 predicted as either a1, a2, or b1) as correct. this metric reflects the cefr scale's flexibility and the frequent disagreements between human evaluators. ## results baseline models showed varied performance, with the highest exact accuracy achieved by the baseline-lemma knn model, which recorded 67.1% exact accuracy and 89.7% with adjacent accuracy. interestingly, the linear regression model, although performing lower on exact matches at 63.5%, had the highest adjacent accuracy among all models, achieving 97.6%. this suggests that while linear regression may struggle with precise classification, it is particularly effective at capturing a close approximation to the true level. all tested models varied in performance between cefr levels, with levels a1 and c showing better performance across the board, as seen in tables 2 and 3 . ## discussion findings from this study show that baseline models can achieve moderate classification accuracy, with the baseline-lemma knn model reaching the highest baseline performance (67.1% exact, 89.7% adjacent). models performed best at a1 and c levels, likely due to both their highest data support as well as distinctiveness, while b1 and b2 had lower f1 scores, reflecting their similarity to adjacent levels and greater classification difficulty. this level of performance aligns with accuracy rates reported for other languages, suggesting that even simple feature sets can yield effective results for icelandic, despite the limited resources available for l2. we also note that these results are robust to variation in the feature set, as e.g. other two-feature models with different lexical diversity measures in place of hd-d perform about as well as baseline-minimal, without clear preference among a few reasonable alternatives at least within present statistical power. enhanced models using dual versions and more sophisticated linguistic features outper-formed baseline models, with the exception of the linear regression baseline model's strong adjacent accuracy (97.6%). the dual-ling model demonstrated the highest exact accuracy at 80.3% and 96.4% for adjacent. this supports the hypothesis that including corrected versions can improve classifier accuracy by providing insights into corrective changes that reveal interlanguage patterns. additionally, incorporating lexical features, specifically tf-idf weights for both lexical terms and pos tags, proved influential in boosting accuracy, underscoring the importance of lexical diversity and usage patterns in prediction. the absence of icebert embeddings in the topperforming dual-ling model suggests that raw contextual embeddings may not be essential for achieving strong performance in this task. however, it remains a question for future research whether using icebert for classification or extracting embeddings comparatively could improve results in a more balanced dataset or with a larger corpus. ## conclusion we have demonstrated that integrating surface and deeper linguistic features is notably effective in skill level classification, showing that a blend of lexical, morphological, and contextual data can meaningfully reflect learner proficiency. we found that baseline models performed moderately well, with the baseline-lemma knn model achieving the highest exact accuracy (67.1%) and 89.7% when adjacent accuracy was considered. the dual-ling model, relying on both original and corrected text features, achieved the highest overall performance with 80.3% exact and 96.4% adjacent accuracy. these findings have significant implications for future automated tools assessing icelandic learners' skill levels. however, a challenge with the cefr lies in its broad descriptors, which lack specific grammatical and lexical competencies for each level, making it difficult to map concrete linguistic features directly to skill levels. the forthcoming icelandic learner corpus, specifically designed for skill level analysis with balanced data, marks an important step forward. it promises to provide an empirically grounded dataset for further development of automated tools, enabling more accurate skill level assessments.