Native-Language Identification with Attention (2020)

Abstract: The paper explores how an attention-based approach can increase performance on the task of native-language identification (NLI), i.e., to identify an author’s first language given information expressed in a second language. Previously, Support Vector Machines have consistently outperformed deep learning-based methods on the TOEFL11 data set, the de facto standard for evaluating NLI systems. The attention-based system BERT (Bidirectional Encoder Representations from Transformers) was first tested in isolation on the TOEFL11 data set, then used in a meta-classifier stack in combination with traditional techniques to produce an accuracy of 0.853. However, more labelled NLI data is now available, so BERT was also trained on the much larger Reddit-L2 data set, containing 50 times as many examples as previously used for English NLI, giving an accuracy of 0.902 on the Reddit-L2 in-domain test scenario, improving the state-of-the-art by 21.2 percentage points.

## introduction native-language identification (nli) is the task of identifying an author's first language (l1; e.g., spanish) given only information expressed in the author's second language (l2; e.g., english). nli operates under the assumption that an author's l1 will dispose them towards particular language production patterns in their l2 @xcite . knowing what mistakes a learner typically makes when writing or speaking in a foreign language can help educators create custom tutoring experiences and give feedback based on l1s. nli also has applications within forensic linguistics, where identifying what country an unknown author is originally from is useful when detecting, e.g., plagiarism, web fraud and grooming, as well as in web-applications, web-crawling, and data collection to ensure high quality data. containing english learners of 11 different l1s, the toefl11 data set @xcite has been the standard corpus for nli. however, recent advances in auto-generating data based on social media users of high proficiency @xcite have enabled the collection of much larger data sets with more languages. state-of-the-art approaches on toefl11 reach over 0.88 accuracy in identifying the native language of the author using text only @xcite , while the best results on more english-proficient social media users approach 0.69 when evaluated on the same topics as trained on @xcite . however, today's state-of-the-art systems are known for quite substantial drops in performance when tested on documents about topics not seen during training @xcite . while the nli field is evolving, so are advances in deep learning. recently, attention-based models have shown promising results on various nlp tasks, and have become the de facto standard in sequence-to-sequence processing. models such as the transformer @xcite rely solely on attention mechanisms, allowing for heavy parallelisation in the model training, as no recurrence or convolution is required. using a network of transformers, bert (bidirectional encoder representations from transformers) obtained new stateof-the-art results on 11 natural language processing tasks varying from question answering to sentence classification @xcite . given the success of attention-based systems and the need for good nli systems, the paper explores how such an approach can improve nli performance, and investigates how robust attentionbased systems are when tested on different topics than those they were trained on. furthermore, while such systems perform well on their own, the paper also addresses how they can allow for improvements in combination with existing tech-niques, given that all the best nli systems to date include some ensemble or multi-classifier based architecture. the paper is structured as follows: section 2 describes the main data sets used in the field of nli, while section 3 covers related work. section 4 introduces the model architectures, before section 5 provides an overview of the experiments and their results. finally, section 6 sums up the findings and provides suggestions for further study. ## data sets the field of native language identification is in a broader perspective quite young, and only gained serious momentum during the previous decade, most notably after @xcite trained a support vector machine (svm) model on a vast amount of features including part-of-speech (pos) tags, n-grams and grammatical errors on icle, the international corpus of learner's english @xcite . since then, two shared tasks have been dedicated to nli, in 2013 and 2017. the experiments below and the discussed related work all focus on two data sets, toefl11 (the main data set used in the shared tasks) and reddit-l2: the toefl11 data set 1 @xcite consists of english essays written by people with 11 different first languages for the collegeentrance test of english as a foreign language: arabic, chinese, french, german, hindi, italian, japanese, korean, spanish, telugu and turkish. the corpus contains 13,100 essays, with 1,100 essays per language distributed as evenly as possible across 8 different topics. each essay is also labelled with its score (low, medium or high). for the 2013 shared task, the data set was split into a training set consisting of 900 essays per l1, and development and test sets consisting of 100 essays each per l1. the reddit-l2 data set 2 was collected by rabinovich et al. ( 2018 ) from the social media platform reddit, where users can subscribe to areas of interest, called "subreddits" @xcite . these subreddits vary from general topics such as "pictures" to specific niche areas such as "birds with arms" (which is for posting pictures of birds photoshopped to look like they have arms). some subreddits are dedicated to europe, where the users can optionally provide a tag ('flair') indicating their home country. these flairs were utilised by rabi-1 catalog.ldc.upenn.edu/ldc2014t06 2 cl.haifa.ac.il/reports/l2/index.shtml novich et al. to infer the users' l1, with experiments performed to quantify the reliability of the labels. @xcite added further measures to verify the labels' correctness. while toefl11 contains texts written by learners of english, many of the reddit-l2 authors are close to fluent in english, making the task of performing nli on this data set more challenging. the data set consists of texts the users have produced in europe-related subreddits (in-domain), as well as texts written in other subreddits, that can have virtually any topic @xcite . the data is separated into chunks of 100 sentences from one user. after pre-processing and removing users with less than 100 sentences, the reddit-l2 data set consists of roughly 200 million sentences and 3 billion tokens across 29 native countries, from 34,511 unique users. the number of users varies between countries, making the data set imbalanced. hence @xcite downsampled the data, grouping 29 native countries into 23 languages: english, german, dutch, french, polish, romanian, finish, swedish, spanish, greek, portuguese, estonian, czech, italian, russian, turkish, bulgarian, croatian, norwegian, hungarian, lithuanian, slovenian and serbian. the number of users per label was capped at 104, which is the number of users present for the labels with fewest users (lithuania and slovenia). for labels with more than 104 users available, 104 users were selected at random. for each user, the median number of chunks was selected. 3 chunks per user for an in-domain scenario, and 17 chunks per user for an out-of-domain scenario, 3 related work as mentioned in the previous section, two shared tasks have been dedicated to nli, taking place in 2013 and 2017, and with respectively 29 and 19 teams participating. the 2013 nli shared task @xcite introduced the toefl11 corpus and was divided into three sub-tasks: closed-training, open-training-1 and open-training-2. the first was the main task, allowing usage of the training and development sets only. the open-training-1 task allowed the use of any training data except toefl11, while open-training-2 allowed the use of any data, including toefl11. however, open-training-1 showed that training on external corpora while testing on toefl11 caused a significant drop in accuracy. the most common features used were word, character and part-of-speech n-grams. four of the top five teams used at least word 4-grams, and some as high as 7-and 9-grams. the best team achieved an accuracy of 0.846 @xcite , and like the overwhelming majority of the participating teams used svms. @xcite additionally improved performance using ensemble methods. @xcite improved the results further using string kernels. the 2017 nli shared task @xcite was similar to the 2013 task, but in addition to the text utilised toefl11 data from a speech-based nli task included in the 2016 in-terspeech computational paralinguistics challenge @xcite . the raw speech data could not be distributed in that challenge, so the data set consisted of textual transcripts together with so called i-vectors, which are vectors of fixed length (here 800), working as lowerdimensional representations of high-dimensional sequential speech recordings. hence the 2017 shared task was divided into three tracks: essays only, speech only, and a combined 'fusion' track using both text and speech. the fusion track submissions showed that combining written and spoken responses provided a large boost in prediction accuracy @xcite . furthermore, ensemble-based systems were the most effective in all tasks, but typically the same features were used as in 2013, and svms were still the most popular approach, dominating deep learning models. @xcite hypothesise that this could be due to the size of the toefl11 data set, and that more training examples could help deep learning models perform better. unibickernel (ionescu and popescu, 2017) performed best on the fusion track (0.932 accuracy), foot_0 topped the speech-only track, and placed in the top tier in the essay track, by using multiple kernel learning and kernel discriminant analysis, kda. kda is a kernelised version of lda, linear discriminant analysis, that is, a method for finding the best linear combination of features that characterise or separate two or more classes, by projecting the data points down from a feature space where they not are linearly separable to a lower dimensional space where they are. also finishing in the top group was cemi @xcite , using a neural network based metaclassifier of several isolated feed-forward neural network (ffnn) models, each trained on a separate feature type (such as word, character and pos ngrams, plus i-vectors), with the outputs combined using softmax to predict the final label. the italianlp team @xcite topped the essay-only track with 0.882 accuracy, which currently is the highest score achieved on the textual toefl11 test set. they used two stacked svm-classifiers, one trained at sentence level and the other at document level, using the output of the sentence classifier as input, together with features from the documents themselves: text and average word length; function words; as well as character, word, lemma, pos, and linear dependency n-grams. after the shared tasks, @xcite performed a systematic examination of ensemble methods for nli, in addition to evaluating deeper ensemble architectures such as classifier stacks. the experiments included a rigorous application of meta-classification models, achieving state-ofthe-art results on several large data sets, evaluated both intra-corpus and cross-corpus. two important trends were observed: the meta-classification results were better than the ensemble combination methods alone, and meta-classifiers trained on continuous output performed better than their discrete label counterparts. the best performing metaclassifier was lda, both individually and as an ensemble. the ensemble of lda meta-classifiers obtained an accuracy of 0.871 -close to the stateof-the-art accuracy of 0.882 obtained by italianlp. @xcite tried the nli task in a new environment using the larger reddit-l2 data set. a simple regression classifier was trained for all experiments, since the main focus was not to build a new nli classifier, but to explore possible features specific to social media and the reddit-l2 data. they used content features (3-grams and pos 1grams), spelling and grammar features, and contentindependent features such as function words, the most frequent pos 3-grams in the data set, and average sentence length. furthermore, goldin et al. experimented with social network specific features such as votes from other users, average number of submissions and comments, and the most frequent subreddits the user had visited, compared to the most popular subreddits for each country. the most visited subreddits feature performed stunningly, both in and out of the domain. however, this feature is not only specific to the particular data set, but as many users tended to frequent subreddits specific to their country, the feature often contained the correct label itself. hence goldin et al. reported the results of using all features, excluding the subreddit feature, which yielded an accuracy of 0.690 in the in-domain scenario, dropping down to 0.36 when tested out-of-domain. ## architecture and model three model architectures were used in the experiments below: a stand-alone bert model tailored to nli, a meta-classifier architecture, and an ensemble of meta-classifiers. the stacked classification architectures use traditional techniques in combination with bert. the pytorch version of bert was used, foot_1 which is a clone of google's official bert implementation for tensorflow, foot_2 using the same pre-trained models as provided by @xcite . all other classifiers were implemented using the open source scikit-learn (sklearn) machine learning library. 6 for svms the default parameters of sklearn were used, but with a linear kernel, as it was the kernel used for all base classifiers in @xcite . the ffnns (feed-forward neural networks) were run with the default parameters of sklearn, unless specified otherwise. sklearn was also used for feature manipulation (e.g., transforming text into tf-idf weighted vectors), with natural language toolkit (nltk) foot_4 used for basic text-processing, and pandas foot_5 for data manipulation. all experiments were carried out on a two gpu cluster, with 16 gb each. for most jobs, 64 gb ram was sufficient, but for the largest job 128 gb ram was needed in order to keep both bert and all the data in memory. running bert on toefl11 for 5 epochs typically took 2-3 hours, while experiments using millions of reddit-l2 examples took more than 6 days. all the code from the paper is available on github. foot_6 ## conclusion the paper is the first to apply bert to the native language identification task. an empirical exploration showed that bert alone is not able to compete with traditional state-of-the-art approaches on the toefl11 data. however, a meta-classifier architecture combining bert and ten traditional classifiers trained on different features produced an accuracy of 0.853 on the toefl11 test set, closing in on the current state-of-the-art of 0.882. further experiments showed that bert can produce state-of-the-art results on the novel reddit-l2 data set, both in-and out-of-domain, with a 0.902 10-fold cross-validated accuracy for the indomain scenario, with a model trained only on outof-domain data, but tested on in-domain data. this can be compared to the result of 0.690, obtained by @xcite when both training and testing the model on in-domain data. with the same setup, bert achieved an accuracy of 0.805. additionally, when evaluated on the entire indomain test set as a whole (70,000 test cases versus roughly 500 test cases for each fold in the previous in-domain scenario), bert was able to obtain an even higher accuracy of 0.861. this is not directly comparable to any related work, as the entire indomain part of the data set has not before been used for testing; however, this task is potentially more demanding than the original in-domain task, as it contains 140 times as many test cases in each fold. testing the bert model out-of-domain showed that it is more robust to topic differences than previous approaches on reddit-l2, both bert alone and in a meta-classifier stack, which produced a final accuracy of 0.529, a 16.7 point improvement over the state-of-the-art. the performance of bert with more data available indicates that bert's performance increases with more data, regardless of what topics it is trained on. in addition to the document granularity the model is trained on, a further reason for why bert performs so much better on reddit-l2 than on toefl11 can be the number of spelling mistakes found in the toefl11 data. this is due both to the higher proficiency of the reddit-l2 users and to them having spellcheckers and other tools available. a higher frequency of misspelled words can cause the wordpiece representations used by bert to be different from those which it was pre-trained on. this might be mitigated by further training of the pre-trained models from bert's check points on data containing spelling mistakes, providing a custom bert model with embeddings tuned to include spelling mistakes. as bert obtained such promising results on the reddit-l2 in-domain data set, future work should look into how to increase the accuracy on toefl11, the standard data set for nli. in addition of further pre-training of bert to learn embeddings of spelling errors and other domain attributes, this would include using additional information sources, such as the sentiment and emotions reflected in the texts, as done by @xcite , or including information about punctuation @xcite or capitalisation. clearly, training on more relevant data would also improve performance, so including training data from the italki corpus @xcite should be feasible. it contains about 122,000 documents gathered from the language learning site italki, in the same languages as toefl11. with more power available, running bert-large with a sequence length of 512 should be feasible, or alternatively extensions of bert or similar models, such as albert (a lite bert; lan et al., 2020), gpt-3 (generative pre-trained transformer; brown et al., 2020) continuous pre-training ('ernie 2.0'; @xcite or transformers for longer sequences @xcite could be tested on the problem. opening up the attention mechanism and looking at what parts of the input sequence the model is paying attention to could also give new insights, and could potentially help educators.