Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns (2022)

Abstract: There is a growing interest in the combined use of NLP and machine learning methods to predict gaze patterns during naturalistic reading. While promising results have been obtained through the use of transformer-based language models, little work has been undertaken to relate the performance of such models to general text characteristics. In this paper we report on experiments with two eye-tracking corpora of naturalistic reading and two language models (BERT and GPT-2). In all experiments, we test effects of a broad spectrum of features for predicting human reading behavior that fall into five categories (syntactic complexity, lexical richness, register-based multiword combinations, readability and psycholinguistic word properties). Our experiments show that both the features included and the architecture of the transformer-based language models play a role in predicting multiple eye-tracking measures during naturalistic reading. We also report the results of experiments aimed at determining the relative importance of features from different groups using SP-LIME.

## introduction extensive studies using eye-trackers to observe gaze patterns have shown that humans read sentences efficiently by performing a series of fixations and saccades (for comprehensive overviews, see, e.g. @xcite , @xcite , and @xcite ). during a fixation, the eyes stay fixed on a word and remain fairly static for 200-250 milliseconds. saccades are rapid jumps between fixations that typically last 20-40 ms and span 7-9 characters. in addition, when reading, humans do not fixate one word at a time, i.e. some saccades run in the opposite direction, and some words or word combinations are fixed more than once or skipped altogether. much of the early work in this area was concerned with the careful construction of sentences to model human reading behavior and understand predictive language processing @xcite . the use of isolated, decontextualized sentences in human language processing research has been questioned on ecological validity grounds. with the growing awareness of the importance of capturing naturalistic reading, new corpora of eye movement data over contiguous text segments have emerged. such corpora serve as a valuable source of data for establishing the basic benchmarks of eye movements in reading and provide an essential testing ground for models of eye movements in reading, such as the e-z reader model @xcite and the swift model @xcite . they are also used to evaluate theories of human language processing in psycholinguistics: for example, the predictions of two theories of syntactic processing complexity (dependency locality theory and surprisal) were tested in the dundee corpus, which contains the eye-tracking record of 10 participants reading 51,000 words of newspaper text @xcite . subsequent work has presented accounts where the ability of a language model to predict reading times is a linear function of its perplexity @xcite . more recent work has employed transformer-based language models to directly predict human reading patterns across new datasets of eye-tracking and electroencephalogra-phy during natural reading @xcite , for more details see the related work section below). while this work has made significant progress, there is limited work aimed at determining the role of general text properties in predicting eye movement patterns in corpora of naturalistic reading. to date, research has addressed this issue only peripherally @xcite @xcite , examining the role of text features only on the basis of a small number of linguistic features. in this paper, we conduct a systematic investigation of the effects of text properties on eye movement prediction: we determine the extent to which these properties affect the prediction accuracy of two transformer-based language models, bert and gpt-2. the relationship between these properties and model performance is investigated in two ways: (a) building on the approaches in @xcite and @xcite , by investigating the sensitivity of model predictions to a wide range of text features, and (b) by incorporating text features into the transformerbased language models. with respect to the latter, we examine the effects of the preceding sentence on gaze measurement within the sentence of interest. this was motivated by psycholinguistic literature that has demonstrated "spillover" effects, where the fixation duration on a word is affected by linguistic features of the preceding context @xcite , see also @xcite for a reference to the utility of information about preceding input). computational reading models have not addressed linguistic concepts beyond the level of the fixated word much, with a few exceptions, e.g. spillover effects related to previewing the next word n+1 during the current fixation on word n @xcite . here we extend the study of spillover effects to the effects of textual features of the preceding sentence. to our knowledge, this is the first systematic attempt to investigate the effects of textual features on the prediction of eye-tracking measures in a corpus of naturalistic reading by considering a large number of features spanning different levels of linguistic analysis. ## related work in this section, we provide a brief overview of the available literature that has used transformerbased language models to predict human reading patterns, as well as the literature that has investigated the role of text properties on word predictability during naturalistic reading. @xcite evaluated a broad range of language models on the match of their internal representations to three datasets of human neural activity (fmri and ecog) during reading. their results indicated that transformer-based models perform better than recurrent networks or wordlevel embedding models. they also found that the models with the best match with human language processing were models with unidirectional attention transformer architectures: specifically the generative pretrained transformer (gpt-2) @xcite , consistently outperformed all other models in both fmri and ecog data from sentence-processing tasks. @xcite presented the first study analyzing to what extent transformer language models are able to directly predict human gaze patterns during naturalistic reading. they compare the performance of language-specific and multilingual pretrained and fine-tuned bert and xlm models to predict reading time measures of eye-tracking datasets in four languages (english, dutch, german, and russian). their results show that both monolingual and multilingual transformer-based models achieve surprisingly high accuracy in predicting a range of eye-tracking features across all four languages. for the english geco dataset, which is also used in the current study, the bert and xlm models yielded prediction accuracies (100 -mean absolute error (mae)) ranging between 91.15% (bert-en) and 93.89% (xlm-ende). to our knowledge, the first study to investigate the role of textual characteristics on word predictability during naturalistic reading is an experimental study conducted by @xcite . this study implemented a large-scale cumulative cloze task to collect word-by-word predictability data (surprisal and entropy reduction scores) for 40 text passages which were subsequently read by 32 participants while their eye movements were recorded. @xcite found that surprisal scores were associated with increased reading times in all eye-tracking measures. they also observed a significant effect of text difficulty, measured by flesch-kincaid grade level of each paragraph @xcite , such that increases in text difficulty were associated with increased reading times. crucially, their study yielded evidence of interactions between predictability (surprisal scores) and paragraph difficulty. in the abovementioned computational study, @xcite also investigated the influence of textual characteristics (word length, text readability) on model performance. text readability was measured using flesch reading ease scores @xcite . their results indicated that the models learned to reflect characteristics of human reading, such as sensitivity to word length. they also found that model accuracy was higher in more easily readable sentences. ## modeling approach deep neural transformer-based language models create contextualized word representations that are sensitive to the context in which the words appear. these models have yielded significant improvements on a diverse array of nlp tasks, ranging from question answering to coreference resolution. we compare two such models in terms of their ability to predict eye-tracking features: 'bidirectional encoder representations from transformers' (bert) @xcite and 'generative pre-trained transformer 2' (gpt-2) @xcite . bert is an autoencoder model trained with a dual objective function of predicting masked words and the next sentence. it consists of stacked transformer encoder blocks and uses self-attention, where each token in an input sentence looks at the bidirectional context, i.e. tokens on left and right of the considered token. in contrast, gpt-2 is an autoregressive model consisting of stacked transformer decoder blocks trained with a language modelling objective, where the given sequence of tokens is used to predict the next token. while gpt-2 uses selfattention as well, it employs masking to prevent words from attending to following tokens, hereby processing language fully unidirectionally. bert is trained on the bookscorpus (800m words) and wikipedia (2,500m words), whereas gpt-2 is trained on webtext, an 8-million documents subset of commoncrawl amounting to 40 gb of text. we chose the bert base model (cased) because it is most comparable to gpt-2 with respect to number of layers and dimensionality (bert base model (cased) has 110m trainable parameters, gpt-2 has 117m). we evaluate the eye-tracking predictions of the models both on within-domain text, using an 80/10/10 split of the much larger geco dataset (representing fiction language), as well as on outof-domain text using the complete, much smaller provo dataset (comprising also online news and popular science magazine language). furthermore, since overly aggressive fine-tuning may cause catastrophic forgetting @xcite , we perform all experiments both with 'frozen' language models, where all the layers of the language model are frozen and only the attached neural network layers are trained, and also 'fully fine-tuned' language models, where the error is back-propagated through the entire architecture and the pretrained weights of the model are updated based on the geco training set. for all models we explored in this paper, we apply a dropout rate of 0.1 and a l2 regularization of 1×10 -4 . we use adamw as the optimizer and mean squared error as the loss function. we use a fixed learning rate with warmup. during warmup, the learning rates are linearly increased to the peak learning rates and then fixed. for bert with a 'frozen' language model, the peak learning rate is 5 × 10 -4 with 5 warmup steps and for gpt-2 with a 'frozen' language model, it is 0.001 also with 5 warmup steps. models with 'fully fine-tuned' language models are trained with two phases. in the first phase, the weights of the lan-guage models are frozen and only regression layers are trained. during this phase, peak learning rates of 3 × 10 -4 for bert and 0.001 for gpt-2 are used. for both models, the first phase is performed over 12 epochs with 5 warmup steps. in the second phase, we unfreeze the weights of language models and fine-tune the language models together with the regression layers. during this phase, the bert-based model is trained with a peak learning rate of 5 × 10 -5 while gpt-2-based model is trained with a peak learning rate of 5 × 10 -4 . the number of warmup steps for training both models in this phase is 3. we adopted a two-phase training procedure since preliminary experiments showed that this procedure yields same results as training the entire models from the first epoch, yet it can speed up model convergence. all hyper-parameters are optimized through grid search. ## results & discussion we use sentence-level accuracy (100-mae) and coefficients of determination (r2) as metrics to evaluate the performance of all models. ## conclusion in this paper we conducted the first systematic investigation of the role of general text features in predicting human reading behavior using transformer-based language models (bert & gpt-2). we have shown (1) that model accuracy is systematically linked to sentence-level text features spanning five measurement categories (syntax, complexity, lexical richness, register-specific n-gram frequency, readability, and psycholinguistic properties), and (2) that prediction accuracy can be improved by using hybrid models that con-sider spillover effects from the previous sentence.