Cross-Lingual Transfer of Cognitive Processing Complexity (2023)

Abstract: When humans read a text, their eye movements are influenced by the structural complexity of the input sentences. This cognitive phenomenon holds across languages and recent studies indicate that multilingual language models utilize structural similarities between languages to facilitate cross-lingual transfer. We use sentence-level eye-tracking patterns as a cognitive indicator for structural complexity and show that the multilingual model XLM-RoBERTa can successfully predict varied patterns for 13 typologically diverse languages, despite being fine-tuned only on English data. We quantify the sensitivity of the model to structural complexity and distinguish a range of complexity characteristics. Our results indicate that the model develops a meaningful bias towards sentence length but also integrates cross-lingual differences. We conduct a control experiment with randomized word order and find that the model seems to additionally capture more complex structural information.

## introduction approximately 7,000 languages are currently spoken in the world, exhibiting differences at almost every level of linguistic organization @xcite . nonetheless, psycholinguistic theories are predominantly supported by evidence from a handful of indo-european languages @xcite . only recently, researchers have started to explore cross-linguistic differences in the neural implementation of language, uncovering both striking similarities across languages and empirical differences that cannot be explained by a unitary account @xcite . in natural language processing, multilingual language models are optimized for tasks such as machine translation or cross-lingual information retrieval @xcite and follow a linguis-tically naïve training regime. they are trained on dozens of languages simultaneously and do not account for typological differences between languages. nevertheless, their cross-lingual transfer performance sets new records, even in zero-shot settings @xcite . the ability to transfer knowledge across languages has been attributed to the shared vocabulary that is used for all languages @xcite because it enables the reuse of common morphological roots for languages from the same family. however, recent studies indicate that vocabulary sharing is not a prerequisite for cross-lingual transfer @xcite and that structural commonalities between languages play a more prevalent role in models @xcite . human sentence processing is sensitive to structural complexity. eye movement data recorded during reading provide insights into cognitive processing patterns with a temporal accuracy of milliseconds @xcite . structural processing difficulty materializes as regressions towards the complex region and an increase of fixations on that region @xcite . for example, sentences with an object-relative structure trigger more regressions than sentences with more common subject-relative clauses @xcite . a classical example of structural complexity are garden-path sentences which initially trigger a simplified interpretation that must be revised when reading the rest of the sentence @xcite . on the surface level, eye movement patterns are language-specific since they are influenced by visual factors such as orthography and word length @xcite . for example, the chinese script is much more visually dense than the alphabetic script, resulting in longer fixations and saccades that move to positions relatively close to the current word @xcite . on a deeper processing level, reading patterns seem to converge across languages. predictability effects have been demonstrated in multiple languages @xcite and sentences that are matched for content are read at a similar speed in chinese, english, and finnish @xcite . @xcite find that the representations of an english pre-trained transformer-based language model encode structural complexity more prominently when they are fine-tuned to predict english eye-tracking patterns. interestingly, @xcite claim that structural similarity between languages is only weakly represented in multilingual models. nevertheless, @xcite show that multilingual models are able to predict eye movement patterns of reading even for languages that are not seen during fine-tuning, which indicates a general learnability of the relationship between structural complexity and eye movement patterns. their results are restricted to four languages (three of them are from the germanic family), and it remains unclear which structural cues are leveraged for the cross-lingual prediction because the test sentences are not aligned across languages. ## related work we introduce recent findings on the role of structural information for cross-lingual transfer in multilingual models and motivate the use of eye-tracking data as a proxy for cognitive processing complexity. ## methodology we fine-tune a pre-trained multilingual transformer model to predict eye-tracking metrics in a setting of zero-shot cross-lingual transfer. ## cross-lingual transfer results figure 1 shows the explained variance and r 2 scores of the fine-tuned model for total fixation duration across languages. in terms of explained variance, we see that the model achieves a similar performance across languages, i.e. it captures 60 to 80 percent of the variance in the original eyetracking signal for all languages. the r 2 scores, on the other hand, vary much more depending on the language. similar results were observed for two of the other eye-tracking metrics, i.e. fixation count and first-pass duration, but the model is worse at predicting regression duration (see figure 8 in the appendix). to better control for spurious correlations, we ran the experiment on permuted input-output pairs, i.e., we paired input sentences with eye-tracking values corresponding to another random sentence and averaged the results over 5 folds. for this random baseline setup, both explained variance and r 2 are always strictly negative for all languages. to better understand the varied r 2 scores for different languages, we show the distribution of the true and predicted values for total fixation duration for two languages with high r 2 (estonian, turkish) and two languages with low r 2 (english, korean) in figure 2 . we see that the low r 2 for english and korean is caused by predictions that are consistently too high. for estonian and turkish, the difference between true and predicted values is clearly smaller, resulting in a higher r 2 . nevertheless, the model is able to predict a significant amount of the variance in the eye-tracking signal of all languages, as expressed by the stable explained variance scores across languages. interestingly, the model performs slightly better for most zero-shot languages than for the finetuning language english. recall that this performance difference cannot be attributed to crosslingual differences in semantics, since all sentences are parallel with respect to content. on the right side of figure 2 , we analyze the predictions with respect to sentence length and find that both the model predictions and the true values for fixation duration correlate with sentence length in all languages. as sentence length is an indicator of structural complexity, we further dissect this phenomenon and conduct an analysis of a range of structural characteristics in the following section. ## sensitivity to structural complexity we explore four categories of sentence-level complexity features: length, frequency, morphosyntactic, and syntactic. word frequencies are obtained as standardized zipf frequencies using the python package wordfreq @xcite . the package combines several frequency resources, including subtlex lists (e.g. brysbaert and new (2009)) and opensubtitles @xcite . the morpho-syntactic and syntactic features are computed using the profiling-ud tool @xcite . ## the role of sentence length to test whether the fine-tuned xlm-r model captures more sophisticated structural information than sentence length, we conduct two additional experiments. first, we probe the final-layer representations of the model for the complexity features from ## conclusion we find that xlm-r can apply cross-lingual transfer to predict cognitive processing difficulty with similar performance across 13 typologically diverse languages, despite being fine-tuned only on english data. we conducted a range of experiments to quantify the model's sensitivity to structural complexity and find that the fine-tuned model prominently encodes sentence length, but also considers more complex structural information such as dependency structure and word order for the prediction of eye-tracking metrics. our analyses suggest that domain differences in training and testing data have a greater impact on model performance than language differences within the same domain. more specifically, xlm-r performs better on in-domain geco data than cross-domain meco data, but within meco, xlm-r shows similar performance across languages. this aligns with the findings of @xcite , who show that the correlation between relative importance metrics and total fixation duration is influenced by text domain. our study highlights the significance of controlling for text domain and size, as it allows to evaluate crosslingual generalization that is independent of dataset characteristics. in future work, we plan to better account for individual differences between readers @xcite and spill-over effects across sentence boundaries @xcite . the modeling approach for learning eye-tracking patterns also needs further exploration. we find that sentence-level prediction of eye-tracking patterns works well for learning about structural complexity, but that it is not optimal for capturing lexical complexity. token-level measures, as predicted in @xcite , are more likely to be informative about lexical phenomena. a joint loss for sentence and token-level eye-tracking metrics might lead to sensitivity to a wider range of linguistic complexity features.