Bridging Information-Seeking Human Gaze and Machine Reading Comprehension (2020)

Abstract: In this work, we analyze how human gaze during reading comprehension is conditioned on the given reading comprehension question, and whether this signal can be beneficial for machine reading comprehension. To this end, we collect a new eye-tracking dataset with a large number of participants engaging in a multiple choice reading comprehension task. Our analysis of this data reveals increased fixation times over parts of the text that are most relevant for answering the question. Motivated by this finding, we propose making automated reading comprehension more human-like by mimicking human information-seeking reading behavior during reading comprehension. We demonstrate that this approach leads to performance gains on multiple choice question answering in English for a state-of-the-art reading comprehension model.

## introduction much of the work in nlp strives to develop systems that are able to perform linguistic tasks similarly to humans. to achieve this goal, one typically provides nlp systems with human knowledge about the task at hand. this knowledge can come in the form of linguistic annotations, hand-crafted rules and access to linguistic databases, as well as various model design choices. in this work, we study the possibility of providing the model with an inductive bias by using human behavioral signals based on eye movements in reading as an additional source of information which can guide nlp models to adequately process linguistic input and solve linguistic tasks. as a case study, we examine reading comprehension, a task of central importance for probing both human and machine understanding of text. to enable this study, we collect eye movement data from 269 participants who engage in a reading comprehension task using the materials of onestopqa @xcite . we argue that reading comprehension is a particularly well-suited task for linking human eye movement information to nlp modelling due to the substantial correspondence between reading times and the relevance of the text segment for answering the question. @xcite have shown this correspondence by establishing increased reading times on the correct answer in a question answering task where answers are named entities. our study generalizes this result to an arbitrary qa setting, and demonstrates longer reading times for portions of the text which are most pertinent for answering the question correctly. building on this observation, we develop a new approach to machine reading comprehension in which the model is directed to mimic human fixation times over the text, given the question. the idea behind this approach is to encourage the model to focus on question-relevant information. specifically, we introduce a multi-task reading comprehension architecture in which a state-of-the-art transformer model jointly performs question-answering and prediction of the human reading time distribution over the text. our modelling framework is behavioral, treating the reading comprehension model itself as a blackbox. this leads to both theoretical and practical advantages. from a theoretical perspective, this approach is appealing as it creates a direct parallel to human reading, in which eye movements are an external behavior. practically, our approach has the advantage of being modular, allowing swapping our model with other reading comprehension models, and the task with other nlp tasks. our experiments demonstrate that our approach leads to consistent gains in question-answering performance across different training regimes, model variants, and on both in-and out-of-domain evalu-ations. in particular, our model outperforms baseline models with gaze from human reading without exposure to the question. it also performs better than using manual annotations of the textual span critical for answering the question. to summarize, we present three contributions: while this work is a proof of concept and uses a relatively costly data collection procedure, as eye-tracking technology continues to become more ubiquitous and affordable, it will be feasible to perform large scale data collection and deployment of similar approaches for qa and other nlp tasks. our work contributes to two areas of research. the first is how human gaze is conditioned on the reading task. this question was previously investigated in the domain of question answering by @xcite , who collected eye-tracking data in an experimental setup similar to ours for materials from the cnn and daily mail corpus @xcite . they demonstrate that reading times on the named entity which is the correct answer to the question are longer if participants are shown the question before reading the passage as compared to ordinary reading. our work builds on this result, introducing a more general qa setup which is not restricted to questions whose answer is a named entity. crucially, we further leverage this information for improving machine question answering. the second research area to which or work contributes is augmenting nlp models with gaze data. in this area, gaze during reading has been used for tasks such as syntactic annotation @xcite @xcite ), text compression @xcite ), text readability (gonzález-garduño and søgaard, 2017), named entity recognition @xcite , and sentiment classification @xcite @xcite . work on the first four tasks used task-independent eye-tracking corpora, primarily the dundee corpus @xcite and geco @xcite . for the task of sentiment classification, the authors used task specific eye-tracking corpora in which the participants were asked to perform sentiment classification. our study differs from this literature in several aspects. first, we address the previously unexplored task of reading comprehension, which has established theoretical and empirical connections to eye movements in reading @xcite @xcite . also differently from these studies, we cover and directly compare both a task specific reading condition (hunting) and a task-independent condition (gathering), as well as both external (dundee) and corpus specific (onestopqa) eye-tracking data. our qa task can be viewed as a generalization of the work in @xcite @xcite , where instead of being asked about the sentiment of a paragraph, subjects are presented with arbitrary questions. our multitask approach for jointly performing the qa task and predicting gaze is similar to @xcite and @xcite . in particular, in equation 4 we use the same loss term as @xcite which consists of a linear combination of an nlp task loss and gaze prediction loss. our approach differs from @xcite in that their model uses the gaze predictions as input attention weights for the nlp task, while our model treats gaze only as an output. our approach provides a parallel to human reading, in which eye movements are an external behavior rather than an input to language processing tasks. our work differs from @xcite in the model and the use of a single auxiliary objective based on gaze. finally, we note that in @xcite eye-tracking data from esl learners was collected for 4 articles from the same source of onestopenglish articles (vajjala and lučić, 2018) used here, and utilized to study the influence of text difficulty level on fixation measures and reading comprehension. our work focuses on a different task and a different population of readers. a large body of work exists on qa, including span prediction (e.g. bidaf @xcite ), cloze (e.g. @xcite ), and multiple choice qa (e.g. stanford attentive reader @xcite ). here, we focus on multiple choice qa due to its prevalence in human evaluations of reading comprehension, and use roberta due to its state-of-the-art performance on this task. further, neural models for qa deploy various notions of internal attention. the study of nlp model internal attention has drawn much interest in recent years @xcite @xcite @xcite . in this work we abstract away from model internal dynamics due to their complexity, and the theoretical justification for treating gaze as an external behavior rather than an internal model property. examination of internal model attention and its relation to human gaze is however an intriguing research direction that we intend to pursue in future work. ## question conditioned gaze in human reading comprehension we motivate our approach by demonstrating that human gaze distributions are strongly conditioned on the reading comprehension task. this conditioning has been previously established for the case of named entities @xcite , and we examine it here in a more general qa setting. specifically, we consider speed-normalized total fixation duration; for each subject, we take the total fixation duration (i.e. sum of all the fixation times) on a word and normalize it by the subject's total reading time for the passage. consider the example in figure 2 , where we visualize the speednormalized gaze on each word averaged across subjects for the same question -paragraph pair in the hunting (with question preview) and gathering (without question preview) conditions. as can be seen from the heatmaps, the gaze distributions are fundamentally different between these conditions. in particular, in the hunting condition we observe a noticeable correspondence between gaze and the annotated critical span. although the degree of correspondence between gaze and the critical span in the hunting condition depends on the specifics of the question and the text, the presented example is representative of a large portion of our items. to further substantiate this observation, in figure 1 we compare the average total fixation duration within versus outside the critical span in both the hunting and gathering conditions. we observe that in the hunting condition, reading times are significantly longer within the critical span compared to outside of the critical span (@xmath0, foot_0 while in the gathering condition they are slightly shorter within the critical span (@xmath1. the difference between within-span vs outside-of-span reading times between hunting and gathering conditions is also significant (@xmath2. we further note that the total reading time for the passage is shorter in the hunting condition (@xmath3, consistent with more targeted reading as compared to the gathering condition. while our analysis provides evidence for an increased concentration of gaze time around text that is critical for answering the question, the potential what will result from an increase in human population in the future? in the next 30 years, the planet's human population will increase to nine billion. already one billion people do not get enough food. the increase will put more pressure on agricultural land, water, forests, fisheries and resources, and also food and energy supplies. the cost of meat is increasing it costs more money now, but also people have to destroy a lot of rainforest to make fields or to grow food for cows. cows also make methane. the farming of cows, pigs and sheep makes very large amounts of greenhouse gases -35% of the planet's methane, 65% of its nitrous oxide and 9% of the carbon dioxide. what will result from an increase in human population in the future? in the next 30 years, the planet's human population will increase to nine billion. already one billion people do not get enough food. the increase will put more pressure on agricultural land, water, forests, fisheries and resources, and also food and energy supplies. the cost of meat is increasing it costs more money now, but also people have to destroy a lot of rainforest to make fields or to grow food for cows. cows also make methane. the farming of cows, pigs and sheep makes very large amounts of greenhouse gases -35% of the planet's methane, 65% of its nitrous oxide and 9% of the carbon dioxide. utility of human gaze is not limited to this aspect alone. human gaze can be viewed as a soft form of text annotation that relates the entire text to cognitive load during processing. in particular, it can in principle provide valuable fine-grained information within the critical span. ## method: joint question answering and human gaze prediction to test the effectiveness of utilizing human gaze data for enhancing the performance of a reading comprehension system, we trained a reading comprehension model to perform the same multiple choice task as the human subjects. we then conducted a series of controlled experiments to assess how the accuracy of the model is affected by providing it with human eye movements information. ## experimental results our results are summarized in fine-tuning the model for qa on onestopqa is most beneficial in the two resource-lean regimes when the model has not been previously fine-tuned on race (@xmath5, wald test). similarly, adding auxiliary loss to the qa model in these two regimes has a substantially larger impact on model performance compared to performing prior fine-tuning on race (@xmath6. in our within-domain evaluations on onestopqa, we observe improvements of our model over all the baselines in all evaluations, except for the case of the large model without race fine-tuning where our model comes second. we also observe improvements in the out-ofdomain evaluations on race. when the large model is fine-tuned for qa only on onestopqa, it obtains an accuracy of 53.0, reflecting a 0.4 improvement over the strongest baseline. the base model comes second in this evaluation. when first fine-tuning the model for qa on race, then performing additional fine-tuning on onestopqa, the base model obtains an improvement of 0.1 over the strongest auxiliary loss baseline. for the large model we observe a similar improvement when using gaze, with the same performance in the hunting and gathering conditions. interestingly, we do not observe a consistent ordering in the performance of the baselines. in particular, we do not observe a clear advantage of using gaze in the gathering condition over question-passage similarity. we also obtain comparable performance when gaze data in the gathering condition comes from onestopqa and dundee. notably, in nearly all the evaluations our model performs better compared to the manual critical span annotation baseline. we hypothesize that this may be because the annotated spans do not capture potential inter-annotator variation in span annotations, as well as within-span information which is informative for our task. we note that while the gains over the strongest baselines are not statistically significant at the .05 level, the overall consistent pattern across evaluation regimes suggests the promise of using hunting gaze data as the target of the auxiliary loss objective over any other single baseline. finally, we note that an α of 0.2 -0.4 was most often chosen. ## conclusion we present a framework for performing automated reading comprehension in a human-like fashion, yielding performance gains for a state-of-the art reading comprehension model. our work also contributes to the study of human reading, providing evidence for a systematic conditioning of human reading on the reading comprehension task. in the future we intend to study the relation between gaze and internal model attention, and further explore the relation between gaze, task and task performance in qa and well as other tasks.