Multilingual Language Models Predict Human Reading Behavior (2021)

Abstract: We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.

## introduction when processing language, humans selectively attend longer to the most relevant elements of a sentence @xcite . this ability to seamlessly evaluate relative importance is a key factor in human language understanding. it remains an open question how relative importance is encoded in computational language models. recent analyses conclude that the cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance @xcite . alternative methods based on salience @xcite , vector normalization @xcite , or subset erasure @xcite are being developed to increase the post-hoc interpretability of model predictions but the cognitive plausibility of the underlying representations remains unclear. in human language processing, phenomena of relative importance can be approximated indirectly by tracking eye movements and measuring fixation figure 1 : from the fixation times in milliseconds of a single subject in the zuco 1.0 dataset, the feature vector described in section 3.2 for the wors "mary" would be [2, 233, 233, 431, 215.5, 1, 1, 1]. contextualized neural language models are less interpretable than conceptually motivated psycholinguistic models but they achieve high performance in many language understanding tasks and can be fitted successfully to cognitive features such as self-paced reading times and n400 strength @xcite . moreover, approaches to directly predict cognitive signals (e.g., brain activity) indicate that neural representations implicitly encode similar information as humans @xcite @xcite . however, it has not been analyzed to which extent transformer language models are able to directly predict human behavioral metrics such as gaze patterns. the performance of computational models can be improved even further if their inductive bias is adjusted using human cognitive signals such as eye tracking, fmri, or eeg data @xcite @xcite . while psycholinguistic work mainly focuses on very specific phenomena of human language processing that are typically tested in experimental settings with constructed stimuli @xcite , we focus on directly generating token-level predictions from natural reading. we fine-tune transformer models on human eye movement data and analyze their ability to predict human reading behavior focusing on a range of reading features, datasets, and languages. we compare the performance of monolingual and multilingual transformer models. multilingual models represent multiple languages in a joint space and aim at a more universal language understanding. as eye tracking patterns are consistent across languages for certain phenomena, we hypothesize that multilingual models might provide cognitively more plausible representations and outperform language-specific models in predicting reading measures. we test this hypothesis on 6 datasets of 4 indo-european languages, namely english, german, dutch and russian. 1 we find that pretrained transformer models are surprisingly accurate at predicting reading time measures in four indo-european languages. multilingual models show an advantage over languagespecific models, especially when fine-tuned on smaller amounts of data. compared to previous psycholinguistic reading models, the accuracy achieved by the transformer models is remarkable. our results indicate that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. as a consequence, it should be possible to adjust the inductive bias of neural models towards more cognitively plausible outputs without having to resort to large-scale cognitive datasets. ## related work using eye movement data to modify the inductive bias of language processing models has resulted in improvements for several nlp tasks (e.g., @xcite . it has also been used as a supervisory signal in multi-task learning scenarios @xcite and as a method to fine-tune the attention mechanism @xcite . we use eye tracking data to evaluate how well transformer language models predict human sentence processing. therefore, in this section, we discuss previous work on probing transformers models as well as on modelling human sentence processing. ## data we predict eye tracking data only from naturalistic reading studies in which the participants read full 2 note that the exact numbers might differ slightly from the original publications due to different preprocessing methods. ## language models we compare the ability to predict eye tracking features in two models: bert and xlm. both models are trained on the transformer architecture @xcite and yield state-of-theart results for a wide range of nlp tasks @xcite . the multilingual bert model simply concatenates the wikipedia input from 104 languages and is optimized by performing masked token and next sentence prediction as in the monolingual model @xcite without any cross-lingual constraints. in contrast, xlm adds a translation language modeling objective, by explicitly using parallel sentences in multiple languages as input to facilitate cross-lingual transfer (lample and @xcite . both bert and xlm use subword tokenization methods to build shared vocabulary spaces across languages. we use the pretrained checkpoints from the hug-gingface repository for monolingual and multilingual models (details in we fine-tune the models described above on the features extracted from the eye tracking datasets. the eye tracking prediction uses a model for token regression, i.e., the pretrained language models with a linear dense layer on top of it. the final dense layer is the same for all tokens, and performs a projection from the dimension of the hidden size of the model (e.g., 768 for bert-en or 1,280 for xlm-100) to the dimension of the eye tracking feature space (8, in our case). the model is trained for the regression task using the mean squared error (mse) loss. training details we split the data into 90% training data, 5% validation and 5% test data. we initially tuned the hyper-parameters manually and set the following values for all models: we use an adamw optimizer @xcite with a learning rate of 0.00005 and a weight decay of 0.01. the batch size varies depending on the model dimensions (see appendix c.2). we employ a linear learning rate decay schedule over the total number of training steps. we clip all gradients exceeding the maximal value of 1. we train the models for 100 epochs, with early stopping after 7 epochs without an improvement on the validation accuracy. evaluation procedure as the features have been standardized to the range 0-100, the mean absolute error (mae) can be interpreted as a percentage error. for readability, we report the prediction accuracy as 100-mae in all experiments. the results are averaged over batches and over 5 runs with varying random seeds. for a single batch of sentences, the overall mae is calculated by concatenating the words in each sentence and the feature dimensions for each word, and padding to the maximum sentence length. the per-feature mae is calculated by concatenating the words in each sentence. for example, for a batch of b sentences, each composed of l words, and g eye tracking features per word, the overall mae is calculated over a vector of b*l*g dimensions. in contrast, the mae for each individual feature is calculated over a vector of b*l dimensions. ## results & discussion tables 3 and 4 show that all models predict the eye tracking features with more than 90% accuracy for english and dutch. for english, the bert models yield high performance on all three datasets with standard deviations below 0.15. the results for the xlm models are slightly better on average but exhibit much higher standard deviations. similar to the results presented by @xcite , we find that more training data from multiple languages improves prediction performance. for instance, the xlm-100 model achieves higher accuracy than the xlm-17 model in all cases. for the smaller non-english datasets, potec (de) and rsc (ru), the multilingual xlm models clearly outperform the monolingual models. for the english datasets, the differences are minor. size effects more training data results in higher prediction accuracy even when the eye tracking data comes from various languages and was recorded in different reading studies by different devices (all-langs, fine-tuning on the data of all four languages together). however, merely adding more data from the same language (all (en), finetuning on the english data from dundee, geco and zuco together) does not result in higher performance. to analyze this further, we perform an ablation study on varying amounts of training data. the results are shown in figure 3 for dutch and english. the performance of the xlm models remains stable even with a very small percentage of eye tracking data. the performance of the bert models, however, drops drastically when fine-tuning on less than 20% of the data. similar to @xcite and @xcite we find that the model architecture, along with the composition and size of the training corpus have a significant impact on the psycholinguistic modeling performance. eye tracking features the accuracy results are averaged over all eye tracking features. for a better understanding of the prediction output, we plot the true and the predicted values of two selected fea-tures (fprop and nfix) for two example sentence in figure 2 . in both examples, the model predictions strongly correlate with the true values. the difference to the mean baseline is more pronounced for the fixpropfeature. figure 4 presents the quantitative differences across models in predicting the individual eye tracking features. foot_4 across all datasets, first pass duration (fpd) and number of re-fixations (nrefix) are the most accurately predicted features. proportions (fprop and reprop) are harder to predict because these features are even more dependent on subject-specific characteristics. nevertheless, when comparing the prediction accuracy of each eye tracking feature to a baseline which always predicts the mean values, the predicted features fprop and reprop achieve the largest improvements relative to the mean baseline. see figure 5 for a comparison between all features for the best performing model xlm-100 on all six datasets. ## data sensitivity for the main experiment, we always tested the models on held-out data from the same dataset. in this section, we examine the influence of dataset properties (text domain and language) on the prediction accuracy. in a second step, we analyze the influence of more universal input characteristics (word length, text readability). ## conclusion while the superior performance of pretrained transformer language models has been established, we have yet to understand to which extent these models are comparable to human language processing behavior. we take a step in this direction by finetuning language models on eye tracking data to predict human reading behavior. we find that both monolingual and multilingual models achieve surprisingly high accuracy in predicting a range of eye tracking features across four languages. compared to the xlm models, bertthe dashed lines show the results of the pretrained language models without fine-tuning on eye tracking data. multi is more robust in its ability to generalize across languages, without being explicitly trained for it. in contrast, the xlm models perform better when fine-tuned on less eye tracking data. generally, fixation duration features are predicted more accurately than fixation proportion, possibly because the latter show higher variance across subjects. we observe that the models learn to reflect characteristics of human reading such as the word length effect and higher accuracy in more easily readable sentences. the ability of transformer models to achieve such high results in modelling reading behavior indicates that we can learn more about the commonalities between language models and human sentence processing. by predicting behavioral metrics such as eye tracking features we can investigate the cognitive plausibility within these models to adjust or intensify the human inductive biases.