Fine-Grained Prediction of Reading Comprehension from Eye Movements (2024)

Abstract: Can human reading comprehension be assessed from eye movements in reading? In this work, we address this longstanding question using large-scale eyetracking data. We focus on a cardinal and largely unaddressed variant of this question: predicting reading comprehension of a single participant for a single question from their eye movements over a single paragraph. We tackle this task using a battery of recent models from the literature, and three new multimodal language models. We evaluate the models in two different reading regimes: ordinary reading and information seeking, and examine their generalization to new textual items, new participants, and the combination of both. The evaluations suggest that the task is highly challenging, and highlight the importance of benchmarking against a strong text-only baseline. While in some cases eye movements provide improvements over such a baseline, they tend to be small. This could be due to limitations of current modelling approaches, limitations of the data, or because eye movement behavior does not sufficiently pertain to fine-grained aspects of reading comprehension processes. Our study provides an infrastructure for making further progress on this question.

## introduction reading comprehension is an indispensable skill for successful participation in modern society. consequently, many efforts and resources are invested in the development of reading comprehension assessments by educational institutions and commercial companies. the standard, and to date the only practical way to assess reading comprehension is through behavioral tasks, most commonly reading comprehension questions. however, despite its clear value and ubiquitous use, this approach is extremely time-consuming and costly, which severely limits the volume and public availability of reading comprehension tests. further, this testing methodology relies on offline behavioral signals -the end responses to a few select reading comprehension questions, and has no ability to trace the rich online reading comprehension processes as they unfold over time. an alternative vision for assessing reading comprehension has been emerging in psycholinguistics and the psychology of reading. it posits that reading comprehension may be decoded in realtime directly from eye movements in reading. this vision is rooted in literature that suggests a tight correspondence between eye movements and real time language comprehension @xcite @xcite . with the rise of modern machine learning and nlp, multiple studies over the past decade attempted to use eye movement data to predict reading comprehension @xcite @xcite . this line of work suggests that in some cases various aspects of reading comprehension can be predicted from eye movements with above-chance performance. however, despite the advances so far, predictive modeling of reading comprehension from gaze is still in its infancy. a number of factors have been hindering progress in this area. one is the paucity and small size of reading comprehension data paired with eye movements. second, the task of reading comprehension prediction has thus far been predominantly formulated as prediction of aggregated scores across multiple questions rather than prediction of comprehension at the resolution of an individual question. further, reading comprehension has been primarily studied when the reader has no specific goals with respect to the text beyond general comprehension, a regime that we refer to as ordinary reading. many other reading regimes common in daily life, such as explicit information seeking, remain largely unaddressed. finally, despite the dramatic progress in machine learning and nlp in recent years, effective joint modeling of text and eye movements remains a nascent and challenging domain of investigation. in this work, we take a step forward in advancing the state-of-the-art in eye movement-based prediction of reading comprehension by combining new models, new data, and systematic evaluations. our primary contributions are the following: our study contributes to an existing body of work on the prediction of reading comprehension from eye movements in reading. to address various aspects of this task, prior studies used a wide range of models, including linear models @xcite , kernel methods @xcite , feed-forward networks (e.g. @xcite , cnns @xcite and rnns (e.g. @xcite . these were typically applied to the prediction of aggregated comprehension scores over multiple items. in this work, we evaluate multiple models from prior work on the single-item reading comprehension task. while transformer models @xcite , have been used for joint modeling of eye movements and text (e.g. @xcite , they have not been applied to the problem of reading comprehension prediction from eye movements. in this work we introduce three new transformer models which draw on multimodal transformers, in particular mag @xcite which integrated text, speech and vision for sentiment analysis, and language vision models such as visualbert ( li et al., 2019) most prior studies on reading comprehension prediction from eye movements relied solely on eye movement features @xcite @xcite , while a few combined eye movements with properties of the underlying text @xcite @xcite . in the current work, we take the latter, under-explored approach. the importance of combining eye movements with attributes of the text is motivated by a large literature in the psychology of reading which points to systematic effects of linguistic properties of the text on reading times @xcite @xcite @xcite , in particular in the context of reading comprehension @xcite and linguistic proficiency @xcite . while highly informative, existing work is critically limited by small data, especially with respect to the number of available questions and participants. for example, @xcite have 9 text pages, 18 questions and 39 participants. sb-sat @xcite , the only publicly available eyetracking dataset for reading comprehension, has 22 text pages, 20 questions, and 95 participants. the small size of previously used datasets severely limits the potential of nlp and machine learning approaches for reading comprehension prediction. at the same time, the reading comprehension component of broad coverage eyetracking datasets such as meco @xcite and celer @xcite comprises only simple comprehension questions that serve as attention checks, and as such are not well suited for studying reading comprehension. onestop, used here, has a large number of items, participants and questions, enabling meaningfully addressing item-level prediction of comprehension. prior work varies in experimental designs. in several studies, multiple questions are presented after reading a multi-screen text without the ability to return to the text @xcite @xcite . this design is advantageous in the separation of text reading and question answering, but can lead to loose relations between eye movements and question-answering behavior due to memory limitations. in other studies, such as @xcite , participants can switch back and forth between the text and the questions. this creates a complex mix of ordinary reading and information seeking components which are difficult to disentangle. in onestop, a single question appears immediately after reading a single text page, setting a middle ground between the two primary existing approaches for question presentation, and alleviating their main disadvantages. at the same time, it includes a question preview manipulation which allows to systematically compare reading comprehension in ordinary reading and question guided information seeking. an additional limitation of prior work is the scope and nature of the evaluations. with the exception of @xcite , both training and evaluation were previously carried out over aggregated responses across multiple questions, and in some cases also across multiple texts. these approaches, which focus on measuring overall comprehension, do not enable testing direct links between eye movements and understanding specific aspects of the text. in several studies @xcite @xcite , an additional step was taken, binning comprehension scores into two binary categories, high versus low comprehension, thus further simplifying the task. a second important evaluation limitation in prior work is evaluations in which eyetracking data for both the test participants and items is used in the training set. to our knowledge, except for @xcite , no work has evaluated reading comprehension prediction when neither the participant nor the item appears in the training data. this evaluation regime is needed to fully characterize model generalization ability. importantly, even in less challenging regimes and with aggre-gated scores and binning, model performance in prior work is typically only modestly higher than chance level. more stringent evaluations without binning comprehension scores @xcite , or with held-out participants and/or items @xcite tend to exhibit chance level performance. these results suggest that generalization in reading comprehension prediction is highly challenging. ## eyetracking data we use onestop, an extended version of the dataset collected by @xcite over the textual materials of onestopqa @xcite . onestop is the largest english l1 eyetracking for reading corpus to date. the data was collected using an eyelink 1000+ eyetracker at a sampling rate of 1000hz. in this dataset, 360 adult native english participants read newswire articles from the guardian, and answer a multiple-choice reading comprehension question about each paragraph. the dataset includes 30 articles divided into 162 paragraphs. the average paragraph length is 109 words. each paragraph has 3 possible questions, corresponding to a total of 486 questions. ## experimental setup we evaluate the models in three evaluation regimes that test different aspects of model generalization. we further report aggregated results across all three regimes. we perform model training, hyperparameter tuning, and evaluation separately for the ordinary reading and information seeking parts of the data, with 10-fold cross-validation. figure 3 presents schematically one of the 10 data splits for a 10article 60-participant batch. a full data split for a reading regime (ordinary reading or information seeking) is the union of three such splits. in each split, approximately 64% of the data is allocated for training, 17% for validation, and 19% for testing. the test data is further divided into 9% in the new participant, 9% new item, and 1% new item & participant regimes. in total across the 10 splits, approximately 90% of the trials in the dataset appear in each of the new participant and new item evaluation regimes, and 10% in the new item & participant regime. items are assigned to the train, validation and test portions of each split at the article level, such that no article is split across different data portions, ensuring generalization to items whose content is unrelated to items seen in training. see appendix c for further information on the splits. because the data is unbalanced across classes, we use balanced accuracy as the evaluation metric. as prior work has shown considerable differences in reading behavior between the ordinary reading and information seeking reading conditions @xcite @xcite , we train and evaluate the models on each type of trials separately. we perform hyperparameter tuning for each split, and report balanced accuracy results on the aggregation of the predictions across the 10 test sets. we assume that at test time the evaluation regime of the trial is unknown. model hyperparameter tuning is therefore based on the entire validation set of the split. as prior models from the literature were developed for different tasks and on different datasets, we run a hyperparameter search for each model over a search space that includes the original parameter settings. hyperparameters are also optimized for the text-only roberta baseline. to address the ## summary and discussion this paper presents a systematic evaluation of the ability to predict reading comprehension from eye movements in reading at the level of a single question over a single paragraph. we address this task using a range of existing and new models, applied to large scale data across several task variants and evaluation regimes. our experiments indicate that the task at hand is highly challenging, and further highlight the importance of text-only baselines for assessing the added value of eye movements information. however, we do find that small improvements over a strong text-only baseline are achievable with the proposed and some of the past modeling approaches. given the presented results, the extent to which specific aspects of reading comprehension can be reliably decoded from eye movements signal remains an open question. it is possible that eye movements simply do not contain sufficient information for decoding comprehension at high accuracy rates for the examined level of granularity. alternatively, it may be the case that current modeling techniques do not represent or process eye movements data effectively enough for this task. another factor whose role in task difficulty needs to be investigated in more detail is the imbalanced nature of the data, where only a relatively small fraction of the responses are incorrect. additional work on eye movement data analysis, new model architectures, feature representations and training regimes is needed for making further progress on this task. additionally, new datasets with other task variants and other populations such as children and l2 readers are required to study the problem in a more comprehensive manner. we envision that the models, tasks, evaluation protocols, and data presented here will serve as a stepping stone for such work, as well as a broader scientific investigation of the relations between eye movements and reading comprehension.