Native Language Identification in Texts: A Survey (2024)

Abstract: We present the first comprehensive survey of Native Language Identification (NLI) applied to texts. NLI is the task of automatically identifying an author’s native language (L1) based on their second language (L2) production. NLI is an important task with practical applications in second language teaching and NLP. The task has been widely studied for both text and speech, particularly for L2 English due to the availability of suitable corpora. Speech-based NLI relies heavily on accent modeled by pronunciation patterns and prosodic cues while text-based NLI relies primarily on modeling spelling errors and grammatical patterns that reveal properties of an individuals’ L1 influencing L2 production. We survey over one hundred papers on the topic including the papers associated with the NLI and INLI shared tasks. We describe several text representations and computational techniques used in text-based NLI. Finally, we present a comprehensive account of publicly available datasets used for the task thus far.

## introduction native and non-native speakers of any language are well-equipped to recognize foreign accents in non-native speech @xcite . pronunciation, stress, and prosodic patterns that diverge from those used by native speakers (l1) are widely used to automatically identify a second language (l2) speaker's mother tongue, a task known as speechbased nli @xcite . the same way we hear foreign accents, it is also possible to identify non-native linguistic patterns, mostly related to word choices, syntax, and spelling, in texts written by non-native authors. computational models can therefore be trained on texts belonging to non-native writers to learn common properties of a given l1 with the goal of determining the writer's mother tongue, a task known as text-based nli @xcite . the fundamental assumption behind nli is that the mother tongue influences second language acquisition (sla) and production; the latter is known as cross-linguistic influence @xcite @xcite . language transfer may occur in sla @xcite and it can have negative or positive influence in acquisition. negative transfer occurs when differences between the two languages structures lead to systematic errors in the learning of the l2. this leads to a process called fossilization when the use of particular incorrect patterns becomes a habit thus posing communication problems to the l2 speaker @xcite . positive transfer occurs when areas of similarity (e.g., similar words) between the two languages facilitate learning. due to such positive and negative transfer from the l1 to the l2, it becomes possible to train computational models to recognize patterns shared by speakers of the same l1 when speaking or writing in an l2. there are several reasons to use models to study non-native texts and perform text-based nli. firstly, computational models can be used to investigate the influence of native language in sla @xcite . this can be used to better understand language transfer @xcite and the findings can be applied to the development of l2 teaching materials and computer-aided language learning (call) software. secondly, nli can improve nlp systems for processing texts written by non-native speakers @xcite in applications such as machine translation @xcite . nli is also relevant to forensic linguistics which often entails authorship attribution of disputed documents @xcite or detecting plagiarism @xcite . despite the importance of this task, to the best of our knowledge, no comprehensive survey of text-based nli exists. this paper provides the first comprehensive survey on the topic. we survey over 100 papers on the topic published primarily at the acl anthology, but also at other repositories such as the acm digital library. as evidenced in this survey, most work in nli has relied on feature engineering and traditional machine learning (ml) classifiers (e.g., svms) @xcite @xcite . recent advances in deep learning, and the introduction of large language models (llms) have brought renewed interest in nlp, opening exciting new avenues for new and existing tasks. we believe that by reviewing prior work, this survey can identify knowledge gaps and open research questions that can help shape future research. the remainder of this paper is organized as follows: section 2 presents an overview of text-based nli. section 3 provides a brief description of available nli datasets. section 4 discusses the two most well-known nli shared tasks. section 5 describes features used in nli while section 6 discusses the different models that have been used to approach nli. lastly, section 7 presents concluding remarks and discusses avenues for future work. ## nli task overview text-based nli is typically modeled as a supervised multi-class classification task evaluated using widely-used metrics in text classification such as accuracy, precision, recall, and f-score @xcite . given a collection of texts in a particular l2 written by a set of speakers from n different l1s, nli systems are trained to assign a class label (l1) to each document in the text collection. figure 1 illustrates the nli task with an english l2 and four different l1s. figure 1 : the general concept of an nli system as depicted by @xcite . as discussed in this survey, nli systems rely on a variety of features such as words, characters, morphosyntactic information, dependency relations, and spelling errors. while word-based features (e.g., word n-grams) have shown to deliver high performance in most text classification tasks, topic bias is a significant confounder. topic bias occurs when thematic references in a text reveal the true l1 label, for example, english texts written by l1 speakers of japanese are more likely to talk about tokyo or kyoto than texts from chinese or italian l1 speakers. therefore, to decrease topic influence, several nli studies have incorporated linguistically-motivated features that can capture morphological and syntactic variation between different l1s @xcite @xcite . ## datasets we present a comprehensive account of all available datasets created for and/or used in nli research to date. the list is presented in ## text representation and features most approaches to nli have relied on statistical ml classifiers. these classifiers take a variety of features as input such as characters and, most commonly, words extracted from the training corpora often as part of n-gram models. in addition to character and word-based features, pos tags are widely used in nli along with several additional lexical and syntactic features. these different feature classes have been shown to provide complementary information @xcite . in the following paragraphs we define each feature and provide a list of studies that use each group of feature in lexical features lexical features used in nli include word frequencies, word type frequencies, spelling errors, number of phonemes and syllables, and more. these features provide insight into the linguistic fingerprint of an individual and can therefore be used to identify their native language. @xcite and @xcite used spelling and grammatical errors to predict learners' l1 influences for nli. del río (2020) and malmasi and dras (2014c) used frequencies of letters, phonemes, syllables, morphemes and suffixes respectively for portuguese and finnish nli. @xcite and @xcite explored the use of function words for nli, including the use of stopword lists for multilingual nli. n-grams n-grams are a sequence of n items in texts, most often letters or words, and are subsequently defined at either the character-level or word-level. @xcite utilized wordlevel n-grams and character bigrams and trigrams, and the 100 most frequent unigrams found within their dataset. @xcite also explored word-level unigrams and bigrams both separately and in combination. tydlitátová (2016) employed character n-grams extracted from individual words for @xmath0, 2, 3 for czech. n-grams therefore have been frequently used throughout nli research to identify character or word combinations that are indicative of an individual's native language. ngrams have also been used to model sequences of pos tags as described in this section. pos tags part-of-speech (pos) tags are used to denote a word's grammatical function, such as noun, verb, adjective, etc. pos tags are assigned at the word-level most commonly by an automatic pos tagger. several studies have used pos tag frequencies to help differentiate between speakers of various l1s. @xcite used mixed pos function word n-grams (@xmath1. @xcite and del río (2020) used fine grained pos tags as topic independent features of nli including verbs, nouns and adjectives for portuguese. @xcite and @xcite used pos n-gram (@xmath2..3) and applied alkhalil arabic-specific morphological analyser @xcite to tags word for arabic nli. tf-idf weighting while the aforementioned features can be represented by binary or normalized feature vectors, term frequency-inverse document frequency (tf-idf) weighting takes into account feature frequency across the entire training corpus, helping to identify rare features that might be highly discriminative. tf-idf vectors have been frequently used for information retrieval and various nlp tasks such as text categorization and authorship identification. their ability to characterize a given text along with their predictive capabilities has made their use popular in nli, often leading to improved performance @xcite . syntactic features syntactic transfer from l1 to l2 has been widely studied in various language pairs @xcite . syntactic features for nli range from the use of context free grammar (cfg) structures to the frequency of noun or verb phrases within a text. @xcite explored several syntactic features including the use of the cfg production rule. @xcite proposed a model treating parse tree horizontal slices as cfg production rule sets, using them as binary features. in addition, several studies have investigated topic-independent features alongside cfg rules to capture relationships useful for nli, includ- dataset char-n-grams word-n-grams pos tags tf-idf lexical syntactic reference icle ✓ -✓ -✓ -(koppel et al., 2005) icle ✓ ✓ ----(wong and dras, 2009) icle -----✓ (swanson and charniak, 2012) icle -----✓ (tetreault et al., 2012) icle -✓ ✓ ---(wong et al., 2012) toefl11 -✓ ----(wu et al., 2013) toefl11 ✓ ✓ -✓ ✓ -(lynum, 2013) toefl11 -----✓ (mizumoto et al., 2013) toefl11 ✓ ✓ ✓ ✓ -✓ (gebre et al., 2013) toefl11 ✓ ✓ ✓ ---(jarvis et al., 2013) multiple ✓ ✓ ✓ -✓ ✓ (bykh et al., 2013) toefl11 ✓ ✓ ✓ --✓ (goutte et al., 2013) toefl11 ✓ ✓ ----(henderson et al., 2013) toefl11 --✓ -✓ -(tsvetkov et al., 2013) alc --✓ --✓ (malmasi and dras, 2014a) clc --✓ --✓ (malmasi and dras, 2014b) las2 -✓ ✓ ---(malmasi and dras, 2014c) ask -----✓ (malmasi et al., 2015a) toefl11 ✓ ✓ ✓ --✓ (malmasi et al., 2015b) alc -----✓ (mechti et al., 2016) czesl-sgt --✓ ---(tydlitátová, 2016) toefl11 -✓ --✓ -(mohammadi et al., 2017) multiple --✓ ---(malmasi and dras, 2017a) toefl11 ✓ ✓ -✓ --(zampieri et al., 2017) inli ---✓ --(nayel and shashirekha, 2017) inli ---✓ --(bharathi et al., 2017) multiple ✓ ✓ ✓ ---(ircing et al., 2017) toefl11 ✓ ✓ ✓ ---(vajjala and banerjee, 2017) multiple ✓ ✓ ✓ -✓ -(markov et al., 2017) multiple -✓ ✓ -✓ -(kulmizev et al., 2017) multiple ✓ ✓ ----(rama and çöltekin, 2017) multiple ✓ ✓ --✓ ✓ (li and zou, 2017) multiple ✓ ✓ ✓ ---(goutte and léger, 2017) multiple ✓ ✓ ----(oh et al., 2017) multiple ✓ ---✓ ✓ (bjerva et al., 2017) inli ----✓ ✓ (anand kumar et al., 2017) inli ---✓ ✓ -(jain et al., 2017) inli ✓ ✓ ✓ ---(markov and sidorov, 2018) inli ---✓ ✓ -(gupta, 2018) inli ---✓ --(mondal et al., 2018) inli ---✓ ✓ -(thenmozhi et al., 2018) rlc ✓ ✓ -✓ --(remnev, 2019) nli-pt -✓ ----(del río, 2020) alc ---✓ -✓ (mechti et al., 2020) tlc ✓ ✓ ✓ -✓ -(uluslu, 2023) ## computational models a variety of traditional ml classifiers trained on the features described in section 5 have been explored in nli, most notably svms and logistic regression. the use of multiple base classifiers as part of ensemble or meta-classifiers have also been a popular approach for nli yielding competitive performance in nli-2017. finally, various papers have approached nli using deep learning architectures. unlike in other nlp tasks, the performance of these approaches is, however, not always clearly superior to the performance of traditional ml classifiers. support vector machines linear svm classifiers are the most widely used in nli due to their high performance and ability to deal with large and sparse feature spaces @xcite @xcite . svms have been used in early work @xcite and in the highest performing entries of @xcite @xcite @xcite , inli-2017 @xcite @xcite , and inli-2018 @xcite . svm implementations available at liblinear @xcite and scikit-learn @xcite have been widely used @xcite @xcite @xcite . some studies @xcite have used the svm implementation available in the libsvm package @xcite whereas @xcite applied svms with linear and polynomial kernel functions to nli. finally, logistic regression has also been used in inli-2018 @xcite . ensembles and meta-classifiers an ensemble classifier combines the predictions of multiple individual classifiers, referred to as base classifiers or weak learners, to create more accurate predictive models. this combination is done by aggregating the predictions using methods such as voting. the objective of ensemble classification is to use the diversity among these base classifiers to achieve better prediction than any individual classifier. researchers have employed ensemble methods to integrate a variety of models and features in nli @xcite . in the nli shared task 2017, for example, several teams used ensemble methods with competitive performance taking advantage of the multimodal nature of the dataset that contained both text and speech goutte and léger (2017); oh et al. (2017); bjerva et al. (2017); ircing et al. (2017); anand kumar et al. (2017); thenmozhi et al. (2017); jain et al. (2017). malmasi et al. (2015b) combined the predictions from all nli shared task 2013 systems in a voting ensemble, showing that this approach can achieve very high accuracy results. similar to voting ensembles, meta-classifiers also leverage the prediction of multiple base classifiers. by training a model that uses the base classifier predictions as inputs, such systems are better able to combine this information by learning patterns in base model predictions and thereby yielding more accurate predictions compared to any individual base classifier. malmasi and dras (2017b) presented a thorough examination of meta-classification models for nli, achieving state-of-the-art results on three datasets from different languages. in another study, malmasi and dras (2018) presented a system with 200 meta-classifiers merged in a bagging ensemble, where all models are trained on different subsets of the base classifiers. deep learning deep learning approaches such as deep neural networks (dnn) @xcite have also been applied to nli with various levels of success. examples of architectures used include gated recurrent unit (gru) @xcite , rectified linear unit (relu) @xcite , convolutional neural network (cnn) (ajees and idicula, 2018), and long short-term memory (lstm) @xcite . these architectures typically take an embedding representation of hundreds of dimensions instead of the n-gram models and linguistically-inspired features described in section 5. transformer architectures such as bert @xcite had great impact in nlp @xcite paving the way for the latest generation of pre-trained llms. however, they have not had the same impact on nli with only a few studies proposing the use of transformers for the task @xcite . the study by @xcite showed that the performance of lstms and bert on toefl11 and icle was much lower than the performance of svms and that of gpt-2 as described next. llms and recent advances recently proposed llms (e.g., gpt, mistral, llama-2) have displayed state-of-the-art performance on various nlp tasks @xcite . one of the first papers to explore llms in nli is the work by @xcite . they have fine-tuned a gpt-2 model showing performance slightly superior to svms on both the toefl11 and icle corpora. more recently, a couple studies (uluslu and schneider, 2022; zhang and salle, 2023) evaluated the performance of llms on the toefl11 dataset, including gpt-2, gpt-3.5 @xcite and gpt-4 @xcite . @xcite demonstrated that gpt-4 achieved state-of-the-art performance of 91.7% accuracy on this dataset using zero-shot prompting without any task-specific fine-tuning, surpassing all previously proposed ensemble approaches. zhang and salle (2023) further explored the interpretability of models by prompting gpt-4 to provide explanations for its predictions including linguistic features such as spelling errors, syntactic patterns, and translated phrases as cues. this study highlights the potential of generative models in achieving strong performance in nli while also enabling explainable predictions. however, as this analysis has been done only on a single english l2 dataset, the impact of llms in nli is still unclear, particularly for l2s other than english where llms support is still very limited. ## conclusion and future directions this paper presented the first comprehensive survey on text-based nli. we surveyed over 100 papers and discussed features, models, and datasets used for nli. we collected information about all available datasets created for or used in nli research to date. moreover, we also discussed the nli shared tasks organized in 2013 and 2017 and the inli shared tasks organized in 2017 and 2018. this survey shows that apart from a few studies @xcite @xcite the bulk of nli research has focused on approaches that rely on feature engineering and traditional ml classifiers, most notably svms. traditional ml approaches have shown to deliver competitive performance in nli shared tasks and other available nli datasets. this indicates that text-based nli is essentially a patternmatching task that requires little to no semantic understanding of language. these findings are similar to what we observe in related language identification tasks such as dialect and language variety identification where classical ml approaches often outperform deep learning approaches @xcite @xcite . the similarity between nli and other language identification has been noted in previous work as discussed in @xcite . amidst exciting developments in ai and nlp such as the introduction of pre-trained llms, we believe that this survey is an important information source for future research in text-based nli. we expect it to be a useful resource for new and well-established researchers alike. we conclude by discussing future directions and open challenges in text-based nli including the use of llms, explainability, and low-resource scenarios. low-resource domains as described in section 3, most corpora used in nli are collected within educational settings (e.g., toefl exams, classroom activities). l2 production, however, is not restricted to education. texts on social media, blogs, online reviews, etc. are also written by l2 speakers and can be used in nli (anand @xcite . more research should be carry out to produce suitable datasets that allow us to investigate l2 production in non-educational domains. this is important as prior work has shown that nli models may not generalize across datasets @xcite .