Leveraging Open-Source Large Language Models for Native Language Identification (2025)

Abstract: Native Language Identification (NLI) – the task of identifying the native language (L1) of a person based on their writing in the second language (L2) – has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.

## introduction native language identification (nli) is the task of automatically identifying an author's native language (l1) based on texts written in their second language (l2). the task is based on the language transfer hypothesis, the phenomenon in which characteristics of l1 influence the production of texts in l2 to the degree that l1 is identifiable @xcite . nli is useful for educational purposes, forensic applications in the context of author profiling, and to inform second language acquisition research @xcite . from a machine learning (ml) perspective, nli is commonly framed as a supervised multiclass classification task, where nli systems are trained to assign an author's l1. while the task has been proven difficult to perform by humans @xcite , automated methods have shown remarkable results using conventional ml approaches based on extensive feature engineering, e.g., @xcite . such methods rely on features that capture l1-indicative linguistic patterns in l2 writing, e.g., spelling errors @xcite @xcite , word choice @xcite , and syntactic patterns @xcite . transformer-based encoder models, like bert @xcite , on the other hand, have yielded poorer performance than conventional ml approaches for the nli task @xcite @xcite . previous research suggests that this is likely because nli concerns very specific linguistic features that models trained on general corpora cannot capture @xcite . recent research has shown that generative large language models (llms) demonstrate promising results for nli. @xcite presented the first study addressing nli using fine-tuned gpt-2 models, which outperformed previous traditional ml approaches and achieved state-of-the-art results on the nli benchmark toefl11 and icle datasets. zhang and salle (2023) explored the ability of gpt-3.5 @xcite and @xcite to perform nli. their results indicate that out-of-the-box gpt models demonstrate outstanding performance, with gpt-4 setting a new performance record of 91.7% accuracy on the toefl11 benchmark dataset, and achieve promising results for open-set classification (without a predefined set of l1s), a useful setting for real-world nli applications. while zhang and salle's results indicate that llms achieve state-of-the-art performance on nli, they only evaluate the performance of gpt-3.5 and gpt-4. the closed-source nature of these models presents a multitude of limitations to research. providers of closed-source models often disclose minimal information regarding the training data or procedure, hindering the evaluation of results achieved with these models and obscuring biases in training data and models @xcite . the undisclosed nature of the training data has also raised concerns among researchers about data contamination risks, as it is challenging to determine whether a model's high performance on a task can be attributed to the model's effective generalization or potential data leakage @xcite . in addition, closed-source models are typically only accessible via an api, causing lack of control over model updates, which are often communicated poorly to users @xcite . in turn, the reproducibility of experiments cannot be guaranteed. the usage of closed-source llms is also highly costly, which negatively impacts the accessibility of llms @xcite . providers of open-source llms, on the other hand, often release more information regarding training data and procedures. as model weights are released openly, open-source llms can be fine-tuned for a down-stream task, which is often highly costly or not supported for closed-source models. despite these advantages, employing opensource llms for nli remains unexplored, and it is therefore important to investigate the difference in performance between open-source and proprietary llms on this task. hence, the research question addressed in this study is: can open-source llms be used for effective native language identification? the contributions of this work are the following: (i) we are the first to explore the performance of open-source llms on nli and quantify the difference in performance with closed-source models, and (ii) we investigate the impact of fine-tuning open-source llms on nli performance. ## data and models to comprehensively evaluate the ability of current llms to perform nli, we compare the performance of two closed-source commercial llms (i.e., gpt-3.5 and gpt-4) with five open-source llms ( §2.2), used out-of-the-box and after finetuning, on two nli benchmark datasets. ## conclusion we explored the performance of a variety of opensource llms for the nli task. our results indicate that open-source llms achieve lower performance than closed-source llms for this task when used out-of-the-box, while domain-specific finetuning of open-source llms allows these models to achieve comparable results to the proprietary llms, such as gpt-4, on the benchmark toefl11 and icle-nli datasets. we believe that our work opens up avenues for future research on llm-based native language identification. future research could explore few-shot prompting and different prompt variations as a way to potentially boost the performance of open-source llms.