Patterns of Text Readability in Human and Predicted Eye Movements (2022)

Abstract: It has been shown that multilingual transformer models are able to predict human reading behavior when fine-tuned on small amounts of eye tracking data. As the cumulated prediction results do not provide insights into the linguistic cues that the model acquires to predict reading behavior, we conduct a deeper analysis of the predictions from the perspective of readability. We try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for English. We compare a range of model configurations to multiple baselines. We show that the models exhibit difficulties with function words and that pre-training only provides limited advantages for linguistic generalization.

## introduction eye movement data of reading provides rich insights into cognitive processes of language understanding. the signal can be used to modulate the inductive bias of machine learning models towards more cognitively plausible processing which can increase model performance @xcite . it has been shown that large multilingual pre-trained language models are able to accurately predict eye tracking patterns when fine-tuned on small amounts of eye tracking data @xcite @xcite . generally, transformer-based language models seem to be better at predicting cognitive signals of human language comprehension (e.g., self-paced reading times, eye movements, or brain activity) than language models based on other architectures @xcite . however, as prediction is not explanation @xcite , we aim to dissect the predicted reading patterns and analyze them in more detail to gain clearer insights into the underlying representation of processing complexity. eye tracking data can be very informative to evaluate sentence comprehension strategies, however, the interdependencies between the eye tracking measures need to be taken into account @xcite . we propose to use the relation between eye movements in reading and text readability in terms of linguistic complexity to better understand procedural patterns of english sentence comprehension in language models. we provide interpretable insights into the prediction errors to investigate the following two questions: ## readability analysis eye movement patterns during reading are known to be influenced by the readability of texts @xcite . @xcite assess text readability with automatically predicted eye tracking features. although their readability assessment model was based only on predicted reading times, it yielded results comparable to models that use extensive syntactic features to compute linguistic complexity. @xcite find that, for english, the accuracy of eye tracking prediction is systematically linked to sentence-level text features that approximate readability. we try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for english. we analyze the correlation between model predictions and readability measures to better understand the processing patterns that the model picks up. ## word-level analysis as lexical aspects seem to be highly relevant, we additionally analyze the influence of the word class on prediction errors. we focus on words that cause a prediction @xmath0. figure 3 shows the aggregated results for all predicted eye movement features. ## methodological limitations in this work, we make a few simplifying assumptions that are common in the field, but severely affect the interpretation of the results. we want to discuss these limitations explicitly and encourage methodological research to better address these open challenges. token-level alignment stimuli for eye tracking are usually pre-processed with high linguistic quality. neural language models, on the other hand, are optimized for engineering objectives that sometimes compete with linguistic and cognitive plausibility. this discrepancy becomes obvious when inspecting the tokenization. the huggingface transformer-based models that we apply expect the use of subword tokenizers. words such as close-knit! that are interpreted as a single unit in the eye tracking data are split into subword tokens (['close', '##-', '##knit']) and it remains an open challenge how to align the different units. in our implementation, we assign the same gaze features to all subtokens and choose to compute the loss only with respect to the first subtoken. it is unclear, however, if this is the best strategy as it complicates fine-grained error analyses. for example, in the eye tracking data, punctuation is not separated from the preceding token as they are usually fixated jointly. if we apply part-of-speech tagging on words with attached punctuation signs they might get assigned incorrect tags leading to skewed results. in this work, we resolved this by manually aligning tokenized text with the eye tracking stimuli for the readability analysis. however, this is laborious and limits the size of the analyzed data. ## conclusion our results show that transformer models yield high accuracy in predicting cognitive language processing signals which confirms tendencies observed in previous work (e.g., @xcite . we go beyond aggregated performance metrics and provide a detailed analysis of the linguistic text complexity factors that underlie the prediction of eye movement patterns. in a detailed analysis of fine-tuned language models that predict eye tracking features from reading, we found that the models learn more from the fine-tuning on psychometric features than from pre-training on textual input. however, the pre-trained models show an advantage over the randomly initialized models in their correlation to text readability measures. although pre-training of large language models is not required to obtain high accuracy in predicting eye tracking feature from reading, it does contribute to a stronger correlation to text readability measures, making the predictions more similar to human reading behavior. the models struggle most to predict accurate eye tracking values for function words, which are exactly the class of words that exhibits large ranges in reading times and skipping rates, together with extreme variability between readers. the next step will be to extend the readability analysis to the other languages and to discern further between syntactic and semantic text difficulty and purely structural complexity. this line of work does not only advance our understanding of language models and allows to compare their output to human language @xcite , but it also furthers research on new readability formulas supported by eye tracking data and machine learning methods (e.g., @xcite . we hope this work can serve as a stepping stone towards a more detailed evaluation setup for eye movement prediction from reading.