Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding (2023)

Abstract: Human gaze data offer cognitive information that reflects natural language comprehension. Indeed, augmenting language models with human scanpaths has proven beneficial for a range of NLP tasks, including language understanding. However, the applicability of this approach is hampered because the abundance of text corpora is contrasted by a scarcity of gaze data. Although models for the generation of human-like scanpaths during reading have been developed, the potential of synthetic gaze data across NLP tasks remains largely unexplored. We develop a model that integrates synthetic scanpath generation with a scanpath-augmented language model, eliminating the need for human gaze data. Since the model’s error gradient can be propagated throughout all parts of the model, the scanpath generator can be fine-tuned to downstream tasks. We find that the proposed model not only outperforms the underlying language model, but achieves a performance that is comparable to a language model augmented with real human gaze data. Our code is publicly available.

## introduction and related work when humans read, they naturally engage in the cognitive process of comprehending language, which, in turn, is reflected in their gaze behavior @xcite . in a nutshell, a scanpath (i.e., sequence of consecutive fixations) on a stimulus text approximates the reader's attention, which can be exploited to inform natural language processing (nlp) tasks. gaze data has been shown to be beneficial in various nlp tasks, such as part-of-speech-tagging @xcite , named entity recognition @xcite , generating image captions @xcite ) and question answering @xcite . researchers have explored 1 https://github.com/aeye-lab/ emnlp-syntheticscanpaths-nlu-pretrainedlm . language model (bert) the use of aggregated word-level gaze features to regularize neural attention mechanisms @xcite . moreover, non-aggregated scanpaths, which capture the complete sequential ordering of the reader's gaze behavior, have also demonstrated promise in nlp tasks @xcite @xcite . however, collecting gaze data is a resourceintensive endeavor, even for very small text corpora. hence, human gaze data is scarce, and nlp taskspecific gaze recordings are even scarcer. moreover, applying a language model that additionally consumes gaze data requires gaze data to be available for the input text at deployment time-which is unrealistic for most use cases. to overcome these limitations, researchers have proposed a multi-task learning approach for nlp tasks such as sentence compression @xcite , sentiment analysis @xcite , and predicting text readability @xcite . in this approach, labeled data for the specific nlp task is used as the primary task, while a separate eye-tracking corpus is utilized as an auxiliary task. while this approach helps mitigate the need for task-specific gaze data during training and testing, the problem of general scarcity of gaze samples remains and hinders effective supervision for dataintensive architectures. in this paper, we propose an alternative approach by using synthetic gaze data, which can be generated easily for any given text, to provide cognitive signals across nlp tasks. the seminal work of @xcite , which integrates eye movement data generated by a computational cognitive model of eye-movement-control-during-reading for tasks such as sentence compression and paraphrase generation, demonstrated the potential of synthetic eye-gaze data. @xcite explored a proof-of-concept model that integrated synthetic gaze data across multiple nlp tasks, but their results did not reach the performance of a fine-tuned bert model @xcite without eye gaze on the general language understanding evaluation (glue) benchmark. in our work, we build on recent advances in the development of machinelearning models for generating human-like scanpaths during reading @xcite @xcite . we develop a model that combines synthetic scanpath generation with a scanpath-augmented language model, eliminating the need for human gaze data. the model allows for fine-tuning the scanpath generator to downstream tasks by propagating the error gradient through the entire model. our approach not only outperforms the underlying language model in multiple tasks on the glue, especially in low-resource settings, but even reaches a performance comparable to an eye-gaze augmented model that uses real, rather than synthetic, eye movement data in sentiment classification. ## model we develop a model that combines a scanpath generation model with a scanpath-augmented language model to perform nlp downstream tasks. figure 1 depicts the proposed model architecture. ## experiments in this section, we describe the data and present the evaluation results of our model for a wide range of nlp tasks. further details about training and hyperparameter tuning can be found in appendix b. ## discussion and conclusion we developed a model that integrates synthetic scanpath generation into a scanpath-augmented language model. we observe that the model achieves results that are comparable to a language model augmented with human scanpaths, which eliminates the need for human scanpaths during both training and testing. human gaze data are only available for a very limited number of nlp tasks and data sets. at application time, under any standard use case scenario of nlp tasks, no gaze recordings are available. synthetic gaze data not only open the possibility to train high-capacity gaze-augmented models across tasks, which would otherwise require the collection of an impractical large volume of gaze data, but also allow for the exploitation of eye gaze signals as model input at application time. using the glue benchmark, we observe that gaze signals show benefits not only for sentiment classification tasks (sst-2), as reported in previous research, but also for entailment classification tasks (mnli, rte) and a sentence similarity task (sts-b). this highlights the potential of integrating cognitive signals from eye gaze into a wider range of nlp tasks in the future. nevertheless, it is evident that not all tasks derive equal benefits from gaze data. it remains up to future research to explore which types of tasks benefit most from gaze signals. our results further show that the potential benefit of augmenting language models with gaze data is higher for low-resource settings. hence, we believe that the augmentation with gaze data might be particularly interesting for low-resource languages. two ongoing multi-lab efforts to collect large multilingual eye-tracking-while-reading corpora (meco foot_2 and multipleye foot_3 ) include a range of low-resource languages, which will allow for training scanpath generators and augmenting language models with synthetic eye gaze for these languages in the near future.