Cross-Genre Native Language Identification with Open-Source Large Language Models (2025)

Abstract: Native Language Identification (NLI) is a crucial area within computational linguistics, aimed at determining an author’s first language (L1) based on their proficiency in a second language (L2). Recent studies have shown remarkable improvements in NLI accuracy due to advancements in large language models (LLMs). This paper investigates the performance of open-source LLMs on short-form comments from the Reddit-L2 corpus compared to their performance on the TOEFL11 corpus of non-native English essays. Our experiments revealed that fine-tuning on TOEFL11 significantly improved accuracy on Reddit-L2, demonstrating the transferability of linguistic features across different text genres. Conversely, models fine-tuned on Reddit-L2 also generalised well to TOEFL11, achieving over 90% accuracy and F1 scores for the native languages that appear in both corpora. This shows the strong transfer performance from long-form to short-form text and vice versa. Additionally, we explored the task of classifying authors as native or non-native English speakers, where fine-tuned models achieve near-perfect accu- racy on the Reddit-L2 dataset. Our findings emphasize the impact of document length on model performance, with optimal results observed up to approximately 1200 tokens. This study highlights the effectiveness of open-source LLMs in NLI tasks across diverse linguistic contexts, suggesting their potential for broader applications in real-world scenarios.

## introduction native language identification (nli) represents a critical area of study within computational linguistics, focusing on the determination of an author's first language (l1) through their written proficiency in a second language (l2). the relevance of nli extends across various domains, notably in forensic linguistics for authorship profiling and in educational linguistics, where it aids in the customisation of teaching materials tailored to the linguistic background of l2 learners. the significance of nli as a computational challenge was markedly enhanced following the release of the toefl11 corpus @xcite , a comprehensive dataset of non-native english writing, which has since served as a benchmark for advancing research in this domain. historically, nli research has predominantly relied on traditional supervised learning methodologies @xcite . however, recent advancements have undergone a paradigm shift towards employing large language models (llms), particularly leveraging zero-shot @xcite learn-ing and fine-tuning strategies @xcite , with an emphasis on long-form essay-based datasets. while preliminary findings suggest the promise of llm-based approaches in enhancing nli performance, further empirical exploration across diverse real-world datasets remains imperative for elucidating the practical applicability of these methods. in this paper we explore the effectiveness of open-source llms on short-form comments from reddit. in section 2, we discuss the related works for native language identification with large language models. in section 3, we discuss the datasets, models, and prompting techniques we used for nli, and evaluate performance on the longform toefl dataset. more specifically, we fine-tune 3-billion and 8-billion parameter llama-3 models @xcite on the toefl11 @xcite training set, following a similar regime to ng and markov @xcite . in section 4, we seek to measure the transfer performance of these models from long-form to short-form text. more specifically, we explore the performance of those models on a subset of the reddit-l2 corpus @xcite , acting as a validation set. next, we perform the inverse of this by fine-tuning the same foundation models on reddit-l2 and validating on toefl11. this approach seeks to answer whether the models are learning general linguistic characteristics, or simply over-fitting to the training set. additionally, we introduce the sub-task of classifying an author of the reddit-l2 dataset as being a native or non-native english author. we conclude in section 5 and discuss limitations and next steps in section 6. ## related work the progression of nli research has been notably documented through workshops such as nli-2013 @xcite and nli-2017 @xcite , which predominantly utilised the toefl11 corpus. these collaborative efforts underscored the effectiveness of ensemble methods, incorporating various traditional machine learning classifiers. these classifiers, trained on a diverse array of features including lexical, stylistic, and syntactic elements, demonstrate superior performance. among the participants, the italianlp lab @xcite achieved remarkable accuracy, reaching a rate of 88.18% on the toefl11 test set, establishing a benchmark for subsequent research endeavours. the first survey paper on nli came out in 2024 @xcite . recent studies have ventured into the exploration of generative llms within the context of nli, showcasing substantial advancements. an accuracy of 89.0% accuracy on toefl11 was achieved by fine-tuning gpt-2 @xcite . next, the application of gpt-3.5 and gpt-4 for zero-shot learning experiments on the toefl11 corpus set a new precedent in accuracy, achieving 91.7% with gpt-4 @xcite . this exploration into the capabilities of llms revealed the potential for significant improvements in nli accuracy. to our knowledge, llms have not been used yet on the reddit-l2 dataset for nli classification. further extending the boundaries of current methodologies, ng and markov @xcite embarked on the approach of fine-tuning various open-source llms utilising 4-bit quantization-aware training for low rank adaptation (qlora) @xcite . their findings suggest a narrowing accuracy gap between fine-tuned llms and the zero-shot capabilities of gpt-4, with 8 billion parameter models nearly matching the performance of gpt-4 on the toefl11 dataset and surpassing it on the icle-nli dataset. such advancements underscore the rapidly evolving landscape of nli research and its increasing reliance on the sophisticated capabilities of large language models. ## data and models the foundation of our study rests on two primary datasets: the ets corpus of non-native written english (toefl11) and the reddit-l2 corpus. to ensure a comprehensive evaluation, we utilise two llms of varying sizes. these were selected based on their strong performance on the toefl11 test set. this approach allows for an exploration of how model size impacts performance on non-native english text, providing insights into the scalability and efficiency of llms in handling linguistic diversity. ## conclusion in this study, we explored the effectiveness of open-source llms in identifying native languages from short-form comments on reddit, using both the toefl11 and reddit-l2 corpora. our findings highlight several key insights that contribute to the ongoing research in nli. firstly, our experiments demonstrate that fine-tuning smaller llama models (3b and 8b) on toefl11 can yield significant improvements in accuracy when applied to reddit-l2 data. this suggests that the linguistic features captured from structured, academic english texts can generalise well to the more informal and varied language use on social media platforms like reddit. the fine-tuned models significantly outperform baseline models, with accuracy improvements of 20% to 30%, indicating the transferability of learned linguistic characteristics across different text genres. the inverse experiment of fine-tuning on reddit-l2 and validating on toefl11 shows that models trained on informal text can also generalise effectively to more structured academic writing. the finetuned 8b model achieves accuracy and f1 scores exceeding 90% on both datasets, with higher performance on toefl11. this outcome underscores the robustness of the model in handling diverse linguistic contexts and suggests that models trained on a wide range of informal texts can successfully adapt to more formal writing styles. additionally, our study on classifying authors as native or nonnative english speakers reveals that fine-tuned models could achieve near-perfect accuracy on the reddit-l2 dataset. this classification task is crucial for various applications, including educational strategies and security measures. the models retain reasonable performance on the wi-locness dataset, further validating their generalisation capability. one notable observation from our experiments is the influence of document length on model performance. we find that longer documents tend to provide more linguistic features that aid in accurate l1 classification, up to a threshold of approximately 1200 tokens. beyond this point, the benefits of increased document length diminish. this finding is critical for future nli research and practical applications, as it emphasises the need to balance document length with computational efficiency. our research contributes to the field of nli by demonstrating the potential of open-source llms in handling diverse and informal text genres while maintaining high accuracy. the ability of these models to generalise across different datasets and writing styles highlights their versatility and applicability in real-world scenarios.