Native Language Prediction from Gaze: a Reproducibility Study (2023)

Abstract: Numerous studies found that the linguistic properties of a person’s native language affect the cognitive processing of other languages. However, only one study has shown that it was possible to identify the native language based on eye-tracking records of natural L2 reading using machine learning. A new corpus allows us to replicate these results on a more interrelated and larger set of native languages. Our results show that comparable classification performance is maintained despite using less data. However, analysis shows that the correlation between L2 eye movements and native language similarity may be more complex than the original study found.

## introduction research has shown that a speaker's native language can affect their learning and performance in a foreign language @xcite @xcite . the eye movements of a reader, namely fixations and saccades, are a window to the online cognitive processing of text with milliseconds accurateness @xcite . native speakers of different languages may exhibit different eye movement patterns when reading a foreign language, with those reading in their native language making shorter and more frequent fixations while making longer fixations due to the increased cognitive load when reading in other languages @xcite @xcite . several researchers have examined eyemovement patterns across different nationalities, exploring various aspects such as sentence reading times, fixation count, and saccade duration @xcite . @xcite showed that gaze data could be used for examining, e.g., reading processes, second language acquisition, and discourse processing, as well as give relevant insights into fields of second language acquisition and processing. early research in native language identification @xcite focused on the relationship between a person's native language and their writing in a second language, while @xcite for the first time predicted a reader's native language using machine learning across four languages (chinese, japanese, portuguese, and spanish) using only eye-tracking features from natural reading in their second language (l2), english. the study leveraged the knowledge that different languages have unique features, such as word order, grammatical rules, and phonological features, that affect language processing in other languages. despite a general interest in eye-tracking corpora for l2 reading, e.g., @xcite , until recently, there has not been a publicly available dataset with enough languages to reproduce the results of @xcite . @xcite used a subset of the licensed celer dataset @xcite which is the largest eyetracking corpus by the number of l2 readers encompassing five different native language backgrounds. the multilingual eye-movement corpus (meco) l2 dataset @xcite foot_0 comprises english l2 reading by 12 different language backgrounds and allows replication of the findings by @xcite on a different and larger set of languages which is why we employ the meco dataset for this study. in this study, we replicate the study by @xcite and classify the native language of the reader from eye-tracking records of them reading english from another corpus. 2 we include readers from seven different language backgrounds that are more interrelated than the original study; the language iso n participants estonian et 23 english en 21 finnish fi 23 german de 23 hebrew he 18 italian it 20 spanish es 21 ## data the meco data was collected in 12 eye-tracking laboratories around the world. participants were young adults ranging from 18 to 39 years old with high levels of l2 proficiency, which was ensured through english instruction in higher education. for more comprehensive information about the dataset, we refer to the authors' paper @xcite . the meco data set includes eye-tracking input gathered from native speakers of 12 languages recorded during reading an english encyclopedic text. due to an insufficient number of participants in some of the cohorts, we used the subset of seven languages with the most participants. to avoid overfitting, we randomly undersampled 23 participants for the two largest cohorts, equivalent in size to the third largest group within the dataset as shown in we only use the texts read by all the participants (also named "shared regime" in @xcite ). the total amount of words read per participant is 595 words, while the original study used 900 words. the feature set employed comprises three word-based measurements: first fixation duration (ff), first pass duration (fp) which is the sum of all fixations during the first pass reading of the word, and total fixation duration (tf). 3 the calculation is explained in §3.3.1 ## methods in this section, we describe the methods employed to replicate @xcite , giving a detailed description of the steps deviating from the setup of the original study. ## results all variants of the model perform substantially better than the majority class baseline. similarly to the results by @xcite , the model trained on the wfc feature set achieves the highest cross-validation accuracy (81.29%). while the model trained on syntactic and information cluster features improves with additional bi-and tri-grams, the words in the fixed context feature set do not follow this trend which differs from the original paper's results. since the original study was done with a different number of languages, we investigate how the performance changes depending on the number of target classes. figure 1 shows the changes in model performance depending on how many target classes it has. e.g., 3 on the x axis corresponds to a group of all combinations (c 3 7 ) of any three languages in the train set. the y-axis shows the mean performance of all classifiers in that group. the results of each classifier in a group vary, thus, we plot the mean performance. as expected, we see that for all feature sets the performance drops when the number of language increase. ## discussion as evident from we also explore language similarity by looking at the suggested positive correlation between classification uncertainty and linguistic similarities. results from @xcite are included in figure 5 , appendix a for convenience. the plot reproduced in figure 2 does not seem to confirm this hypothesis as no clear trend is visible. we observe that the uncertainty when classifying native speakers vs. l2 reading is substantially lower (mean 0.01) than when distinguishing two groups of l2 readers from those of different native languages (mean 0.11). we also compute a correlation coefficient of 0.06 which does not indicate a significant correlation found by berzak et al. similarly, ward hierarchical clustering for linguistic similarities and classification uncertainty, presented in figure 3 , does not present a closeness between grouping using either of these metrics. the plots have little overlaps on the set of languages we used, contrary to the original finding, see figure 4 , and share a little similarity both in terms of languages in each cluster and the general shape of the tree. this suggests that the relation between the english reading patterns and language similarities of the native language found by @xcite may be more nuanced than the original plot (figure 4 , appendix a) initially suggests. ## conclusion we replicate the finding of @xcite and are the first to confirm their finding that a reader's native language can be predicted from gaze patterns when reading english text. having a larger set of more interrelated languages than the original study, we achieve comparable classification results supporting the suggested cross-linguistic influence from the native language to l2. despite the satisfactory performance of the nlir model, the results of investigating the relationship between reading patterns and linguistic similarity are not as straightforward. we believe the relation to be more nuanced than suggested as we are not able to replicate the same outcomes.