Multilingual Native Language Identification with Large Language Models (2025)

Abstract: Native Language Identification (NLI) is the task of automatically identifying the native language (L1) of individuals based on their second language (L2) production. The introduction of Large Language Models (LLMs) with billions of parameters has renewed interest in text-based NLI, with new studies exploring LLM-based approaches to NLI on English L2. The capabilities of state-of-the-art LLMs on non-English NLI corpora, however, have not yet been fully evaluated. To fill this important gap, we present the first evaluation of LLMs for multilingual NLI. We evaluated the performance of several LLMs compared to traditional statistical machine learning models and language-specific BERT-based models on NLI corpora in English, Italian, Norwegian, and Portuguese. Our results show that fine-tuned GPT-4 models achieve state-of-the-art NLI performance.

## introduction individuals proficient in a language have the ability to identify accent patterns in non-native speech @xcite . automatically identifying a speaker's native language (l1) when speaking a second language (l2) on the basis of pronunciation, stress, and prosodic patterns has been substantially explored in speech-based nli @xcite . similarly, in text-based nli, linguistic patterns common to an individual's l1 such as word choices, syntax, and spelling, can be recognized in texts written in a given l2. computational models can be then trained on texts authored by non-native speakers to learn distinctive properties of their l1, aiming to identify the writer's mother tongue @xcite . the underlying assumption in nli is that the native language influences second language acquisition (sla) and production, a phenomenon known as cross-linguistic influence or language transfer @xcite . language transfer results in l1 features manifesting in l2 production, allowing computational models to recognize patterns shared by speakers of the same l1 when communicating in a given l2. text-based nli has numerous important applications such as serving as a corpus-driven approach for sla @xcite and enabling the development of effective l2 teaching materials and computer-aided language learning (call) software. additionally, nli has been shown to improve nlp systems dealing with texts from non-native speakers, contributing to tasks like author profiling, forensics, spam and phishing detection @xcite . as evidenced by a recent survey @xcite , traditional statistical models such as support vector machines (svms) trained on n-grams as features have historically delivered the best performance for text-based nli. a few recent studies @xcite , however, have shown that fine-tuned llms such as gpt-4 deliver state-of-the-art performance for english nli. a key limitation of these studies, as discussed by @xcite is the lack of evaluation of llms for languages other than english. to address this important gap in the literature, we propose the first multilingual evaluation of llms in nli. we evaluate various llms, in a zeroshot and fine-tuned setting, on corpora containing english, italian, norwegian, and portuguese l2 production. we investigate two research questions (rqs): the aforementioned survey by @xcite presents a comprehensive account of textbased nli, covering more than 100 papers on the topic. it describes studies that use a variety of features such as word n-grams @xcite , part-of-speech tags @xcite , and syntactic features @xcite . the survey also covers computational models widely employed in text-based nli from statistical classifiers like svms @xcite and logistic regression @xcite @xcite to deep learning architectures (ajees and idicula, 2018; lotfi et al., 2020; uluslu and schneider, 2022) and llms @xcite . in addition, it reviews shared tasks organized on the topic that provided important benchmark textbased datasets @xcite @xcite . the findings described in goswami et al. (2024) reveal that until recently, approaches that combined statistical classifiers with feature engineering achieved state-of-the-art performance on text-based nli while deep learning architectures achieved limited success. recent studies, however, have showed that the latest generation of llms, most notably gpt-4, are able to outperform statistical and previous neural models (zhang and salle, 2023) particularly when such models are fined-tuned for textbased nli @xcite . the majority of studies referenced here, including recent studies on llm architectures @xcite , only address english nli. this is due to the wider availability of english l2 corpora compared to other languages including widely-used learner corpora such as icle @xcite , toefl11 @xcite , and icnale @xcite . multiple multilingual studies have been conducted that describe data and approaches to text-based nli in other l2s. this includes studies on arabic @xcite @xcite , chinese @xcite , czech (tydlitátová, 2016), finish @xcite , norwegian @xcite , portuguese @xcite , and turkish (uluslu and schneider, 2023). to the best of our knowledge, all text-based nli studies on l2 other than english employed traditional machine learning models combined with fea-ture engineering or early deep learning approaches. the use of llms for l2s other than english remains unexplored. our work fills this gap by presenting the first multilingual evaluation of llms in text-based nli on four languages and five datasets. ## data in this study we use five nli corpora in english, italian, norwegian, and portuguese. nli corpora, and learner corpora in general, are only available for english and a few other high-resource languages @xcite which limits the choice of languages we can study. with the goal of carrying out a multilingual evaluation, we choose italian, norwegian, and portuguese due to the availability of suitable corpora. data splits for toefl11 and nli-pt we follow pre-defined training, development, and testing split from prior work @xcite . for all other corpora, we use a random label wise 80%-10%-10% split for training, development, and testing. to ensure comparability of results, we use the same splits across the different experiments presented in the paper. brief descriptions of the five corpora are presented next. ## models statistical machine learning ensemble we trained a logistic regression (lr) and an svm classifier on pos n-grams of @xmath0. the data was normalized and its dimensionality was reduced using truncatedsvd and pca. we then combine the lr and svm models in a majority voting ensemble @xcite . we refer to this model as ml ensemble. transformers we fine-tune two multilingual models for the four languages, namely mbert @xcite and xlm-r @xcite . we also fine-tune several language specific models. for english we use bert @xcite and roberta @xcite , for italian we use italianbert @xcite , for norwegian we use norbert @xcite , and for portuguese we use bertimbau @xcite . we use learning rate 1e-5 for all models. the hyperparameters for the transformer models for all corpora are presented in ## results we present the results for all languages in terms of accuracy and macro f1, which is the standard in prior work @xcite . the results are presented along with a random and a majority class baseline for comparison. finally, to ensure a fair and comparable analysis across all experiments, we evaluate all models on the same test sets for each particular corpus. ## conclusion and future work this paper presented the first evaluation of llms on multilingual text-based nli, experimenting with four languages and five corpora. our results indicate that larger task fine-tuned llms, such as gpt-4, deliver state-of-the-art performance for textbased nli in the four languages studied. this finding is in line with prior results obtained for english nli @xcite . we further observed that for non-english languages, zero-shot llm prompting approaches are generally outperformed by bert-based and statistical ml approaches. this is likely a limitation of the llms we leveraged here, as they are mostly focused on english. we conclude the paper by revisiting the two rqs below and presenting avenues for future work. rq1 results: our evaluation of llm zero-shot prompting indicates that llms have very little knowledge of nli for the four languages and five corpora explored. a notable exception is toefl11, the most popular nli corpus available, for which the results obtained using gpt-4 were very high using zero-shot prompting. this seems to indicate potential data contamination. when fine-tuned, we observed that llm results have significantly increased (see rq2 results). finally, for all languages, statistical ml classifiers obtained performance superior to several transformers and llm prompting.