A Deep Generative Approach to Native Language Identification (2020)

Abstract: Native language identification (NLI) – identifying the native language (L1) of a person based on his/her writing in the second language (L2) – is useful for a variety of purposes, including marketing, security, and educational applications. From a traditional machine learning perspective,NLI is usually framed as a multi-class classification task, where numerous designed features are combined in order to achieve the state-of-the-art results. We introduce a deep generative language modelling (LM) approach to NLI, which consists in fine-tuning a GPT-2 model separately on texts written by the authors with the same L1, and assigning a label to an unseen text based on the minimum LM loss with respect to one of these fine-tuned GPT-2 models. Our method outperforms traditional machine learning approaches and currently achieves the best results on the benchmark NLI datasets.

## introduction the goal of native language identification (nli) is to automatically identify the native language (l1) of a writer based solely on his or her texts written in the second language (l2), which can inform marketing, educational, security and forensic applications, e.g., by narrowing down the list of candidate authors of a text under investigation. nli is an interesting example of a task which is hard to perform for humans: a study of human performance in nli @xcite showed that they achieve around 37% accuracy, while automated nli systems perform in the 80%-90% accuracy range, depending on the number of languages being considered, amount of data, etc. nli is usually approached from a machine learning perspective as a multi-class classification problem of assigning class labels representing l1s to texts written in l2, where the main focus (of the traditional machine learning) is to design features that capture the systematic fingerprints of the first language in the second language writing (native language interference @xcite ). numerous feature types that capture various aspects of the interference phenomenon have been explored for nli: spelling errors @xcite ; lexical features, e.g., word and lemma n-grams @xcite , cognates @xcite , etymologically-related words @xcite ; syntactic features, e.g., context-free grammar features (wong and dras, 2011), stanford parser dependency features @xcite ; stylometric features, e.g., punctuation @xcite , character n-gram features @xcite ; emotion-based features @xcite , etc. the combination of such features provides the best results for nli, as shown by the two shared tasks on the nli task organized in 2013 @xcite and 2017 @xcite , where the two top-ranked systems @xcite used support vector machines (svm) with a variety of engineered features. nli has also been approached using deep neural networks: franco-salvador et al. (2017) used word embeddings, reporting a slight increase in performance when they are combined with string kernels in a meta-learning set-up; @xcite and @xcite used convolutional neural networks (cnn) and long short-term memory networks (lstm), concluding that traditional methods, i.e., svm with engineered features, appear to work better for nli. openai's generative pre-trained transformer-2 (gpt-2) @xcite is a unidirectional transformer-based language model pretrained on 40 gb of text data with the objective of predicting the next word, given all of the previous words within a text. gpt-2 is able to generate conditional synthetic texts without task-specific training, achieves the state-of-the-art zero-shot results on a variety of domain-specific language modeling tasks, and was successfully applied to various other nlp tasks such as question answering, summarization, reading comprehension, and machine translation @xcite . we show that these generative models, due to their ability to adapt to the style and content of the conditioning text, are able to capture l1 peculiarities and can be successfully used for identifying the first language of the author. we propose a new approach exploiting the gpt-2 model for text classification and evaluate it on the task of native language identification. our approach consists in fine-tuning a gpt-2 model on texts written by the authors with the same l1, and assigning a label to an unseen text according to the minimum lm loss with one of the fine-tuned gpt-2 models. the contributions of the work presented here are the following: (i) we introduce a novel approach for text classification using generative models, and (ii) we evaluate the approach on the nli task, improving the state-of-the-art results. ## data we evaluate our approach on two datasets commonly used in nli research: toefl11 @xcite : the ets corpus of non-native written english (toefl11) contains 1,100 essays in english for each of the 11 l1s: arabic (ara), chinese (chi), french (fre), german (ger), hindi (hin), italian (ita), japanese (jpn), korean (kor), spanish (spa), telugu (tel), and turkish (tur). in total there are 12,100 essays with on average 348 tokens per essay. the essays were written in response to eight different writing prompts, all of which appear in all 11 l1 groups, by authors with low, medium, or high english proficiency. the dataset is considered a benchmark dataset for nli and was used in two shared tasks on the nli task @xcite . icle @xcite : the iclev2 dataset consists of essays written by highly-proficient nonnative college-level students of english. for comparability, we used a 7-language subset of the corpus normalized for topic and character encoding @xcite (the so-called icle-nli subset @xcite to which we refer as icle). this subset contains 110 essays for each of the 7 languages: bulgarian (bul), chinese (chi), czech (cze), french (fre), japanese (jpn), russian (rus), and spanish (spa) (in total 770 essays with on average 747 tokens per essay). ## methodology instead of training a classifier, we fine-tune a pretrained generative model (gpt-2) on the subset of the training samples written by the authors with the same l1, which provides us with n gpt-2 models (n equals the number of l1s (classes)). each of these models has learned the characteristics of a certain l1 group, which we will use to discriminate between new samples: at inference time, we feed an unseen text to all these models and calculate their lm loss. based on the fine-tuning, we expect to get the least lm loss from the model that is trained on the same class of texts, thus the assigned label is the argmin of all models' losses. an example of assigning a label to an unseen text (turkish l1 in this case) is shown in figure 1 foot_0 . although the lm loss (as a measure of likelihood) has been used before as an evaluation metric @xcite , to the best of our knowledge, it has not been exploited in this way for text classification tasks in general and for nli in particular. ## results and discussion to compare the performance of our approach with the previously published state-of-the-art results @xcite , we report the results in terms of classification accuracy on the toefl11 test set, as well as on the toefl11 dataset under 10-fold cross-validation (10fcv) and on the icle dataset under 5-fold cross-validation (5fcv). to better situate the method, several baseline models have also been selected and evaluated, including bag-of-words (bow), bert @xcite and an lstmbased @xcite language model approach similar to our method but with the difference that models are trained from scratch. the implementation has been done in pytorch @xcite , using the gpt-2 medium model from the huggingface transformers library @xcite , fine-tuned for 2-4 epochs. as for the baselines, the lstms are 2-layer networks with 650 hidden units per layer and a hidden dimension of 650, trained for 25 epochs with a batch size of 20. the bert model is the standard base-uncased version trained for 12-16 epochs with a batch size of 12. toefl11 (test set) toefl11 (10fcv) icle (5fcv) bow (svm) 71.1 68.7 80.6 bert 80.8 76.3 76.8 lstm 77.8 75.9 77.9 best shared task 2017 (svm) 88.2 86.4 -best after (previous sota, svm) 88.6 86.5 93.4 ours 89.0 86.8 94.2 in a detailed, per-language view, presented through the confusion matrices for the toefl11 test set (figure 2 ), we can note that the trends are similar to the svm approach with a large number of engineered features @xcite : german is the easiest l1 to identify, while hindi and telugu, and korean and japanese are the most problematic l1s, with the highest degree of confusion between them. this indicates that, despite the relatively small size of the datasets (for deep learning model standards: around 70k tokens in the icle dataset), fine-tuning allows to focus on and learn the differentiating factors, as also evidenced by the lower lstm results when the models are trained from scratch. our model manages to pick up discriminative characteristics of l1 writing from this limited amount of data. the gpt-2 model represents a language model combining many writing styles and topics. since in our data the topics are nearly-equally distributed across the represented l1 groups @xcite , by fine-tuning, the derived models have learned to focus on the interference cues particular to different l1s. figure 3 illustrates the way the loss of different language models varies when receiving a text token by token (@xmath0; toefl11: 1262.txt). universal surges in the loss value (e.g., first 7 tokens and also around token 45 in this case) usually correspond to typos or non-discriminative grammar errors (in this case, the aforementioned intervals contain 'people have got different qualities from each other ...' and '...take risks rathet than only...'), while singular trends can be attributed to discriminative patterns. in this case, for example, a decisive bifurcation happens around token 80, after which the tur model maintains a winning margin against the other languages. interestingly, the interval corresponds to '...they feel themselves the happiest person in the world . they think the result of this accoplishment...', containing examples of preposition errors, which is one of the most frequent types of errors made by turkish learners of english as l2 @xcite . ## conclusions we proposed a novel method to address the nli task. the method consists in fine-tuning a gpt-2 model with the training data grouped by the l1 group into one model per l1, and selecting the fine-tuned model with the lowest loss to assign the l1 class for an unseen text. this approach, simple and not requiring any feature engineering, nevertheless achieves the best results on the datasets commonly used in nli research. it also improves upon deep learning methods applied directly to the classification task. the method is generic in that many text categorization tasks (both stylistic and content-based ones) can be framed this way. we therefore hypothesize that the approach can be used for any other stylebased classification tasks if the (test) samples are long enough, so that the difference in the lm loss is significant. testing this hypothesis is our main direction for future work.