Synthesizing Human Gaze Feedback for Improved NLP Performance (2023)

Abstract: Integrating human feedback in models can improve the performance of natural language processing (NLP) models. Feedback can be either explicit (e.g. ranking used in training language models) or implicit (e.g. using human cognitive signals in the form of eyetracking). Prior eye tracking and NLP research reveal that cognitive processes, such as human scanpaths, gleaned from human gaze patterns aid in the understanding and performance of NLP models. However, the collection of real eyetracking data for NLP tasks is challenging due to the requirement of expensive and precise equipment coupled with privacy invasion issues. To address this challenge, we propose ScanTextGAN, a novel model for generating human scanpaths over text. We show that ScanTextGAN-generated scanpaths can approximate meaningful cognitive signals in human gaze patterns. We include synthetically generated scanpaths in four popular NLP tasks spanning six different datasets as proof of concept and show that the models augmented with generated scanpaths improve the performance of all downstream NLP tasks.

## introduction integrating human signals with deep learning models has been beginning to catch up in the last few years. digital traces of human cognitive processing can provide valuable signals for natural language processing @xcite . various approaches for integrating human signals have been explored. for example, human feedback for better decisioning @xcite , nlp tasks @xcite , and most recently language modeling using reinforcement learning with human feedback (rlhf) based reward @xcite . rlhf involves explicit human feedback and is expensive and hard to scale. on the other hand, previous studies have also tried to use implicit human ## related work when reading a text, humans do not focus on every word and often do not read sequentially @xcite . a series of studies in psycholinguistics have shown that the number of fixations and the fixation duration on a word depend on several linguistic factors. the linguistic factors can also be determined given the cognitive features @xcite . though advances in ml architecture have helped bring machine comprehension closer to human performance, humans are still superior for most nlp tasks @xcite . it has been shown in the literature that integrating explicit @xcite and implicit (cognitive processing) human feedback signals in traditional ml models is expected to improve their performance @xcite . however, the cost of explicit feedback (e.g., using mturk) and implicit feedback (e.g., eye tracking) at scale is excessively high. similarly, privacy-invasive eye-tracking processes limit the scope of this idea. one way to address this problem is to use generated eye movements to unfold the full potential of eye-tracking research. hence, the idea is to architect scantextgan, a scanpath generator for text reading, and test its usefulness in downstream nlp tasks. more precisely, this work builds upon previous works on 1) human attention modeling and 2) gaze integration in neural network architectures, which are described as follows: human attention modeling: predicting what people visually attend to in images (saliency prediction) is a long-standing challenge in neuroscience and computer vision, the fields have seen many data-based models @xcite . in contrast to images, most attention models for eye movement behaviors during reading are cognitive process models, i.e., models that do not involve machine learning but implement cognitive theories @xcite . key chal-lenges for such models are a limited number of parameters and hand-crafted rules. thus, it is difficult to adapt them to different tasks and domains and use them as part of end-to-end trained ml architectures @xcite . in contrast, learning-based attention models for text remain under-explored. within that, all eye tracking models are saliency prediction models with non-existent work in predicting scanpaths. on the other hand, visual scanpaths generation for imagebased eye tracking data has been recently explored for both traditional @xcite and 360 • images @xcite . @xcite presented the first fixation prediction work for text. they built a person-independent model using a linear conditional random fields (crf) model. @xcite designed the neural attention tradeoff (neat) language model, which was trained with hard attention and assigned a cost to each fixation. other approaches include sentence representation learning using surprisal and part of speech tags as proxies to human attention @xcite . our work differs from previous studies as we combine cognitive theory and data-driven approaches to predict scanpaths and further show its application in downstream nlp tasks @xcite . integrating gaze in network architecture: integration of human gaze data into neural network architectures has been explored for a range of computer vision tasks such as image captioning, visual question answering, and tagging @xcite @xcite . hence, recent research has utilized features gleaned from readers' eye movement to improve the performance of complex nlp tasks such as sentiment analysis @xcite , sarcasm detection @xcite , part-of-speech tagging @xcite , ner @xcite , and text difficulty @xcite . while in recent years, eye tracking data has been used to improve and evaluate nlp models, the scope of related studies remains limited due to the requirement of real-time gaze data at inference time. @xcite reported that there exists no automated way of generating scanpaths yet in the literature. with high-quality artificially generated scanpaths, the potential of leveraging eyetracking data for nlp can be unfolded. additionally, generating scanpaths that mimic human reading behavior will help advance our understanding of the cognitive processes behind language understanding. hence, we propose scantextgan; researchers can use that to generate scanpaths over any text without worrying about collecting them from real users. ## proposed model in this section, we define the scanpath generation task, describe the scantextgan model architecture, and provide details on loss functions and model training. task definition: the task of scanpath generation is to generate a sequence s(t ) representing a scanpath over the text @xmath0, w 2 , ..., w n } composed of a sequence of words, can be defined as follows: ## performance evaluation we quantify the performance of scantextgan in two regimes 1 ; first, scanpath generation with three datasets, and second, nlp tasks with six datasets. similar to prior computer vision studies @xcite @xcite , we evaluate the scant-extgan model over the scanpath generation task. for this, we use the test split of the celer dataset, @xcite . in addition, unlike the computer vision studies, we also evaluate the scantextgan model for improvement in nlp tasks. the hypothesis is that the human eyes (and consequently the brain) process many language comprehension tasks unconsciously and without visible effort. the next logical step is to capture (or, in our case, generate) this mental representation of language understanding and use it to improve our machine-learning systems. for evaluation, we use four tasks from the glue benchmark and two from the tasks proposed by @xcite . while the scantextgan model is trained over news text from the celer dataset, with the help of the other datasets, we expand our testing to other domains, including reviews, quotes, tweets, and wikipedia text. @xcite comprises eye movements and reading difficulty data recorded for 32 paragraphs on 16 different topics, viz. history, science, literature, etc. for each topic, comparable paragraphs were extracted from wikipedia 2 and simple 1 all results are calculated with five random seeds and reported as the mean of those five runs 2 https://en.wikipedia.org/ wikipedia foot_0 . the participant's eye movements are tracked using an sr-research eyelink-1000 plus eye tracker. using the ground truth scanpaths over the text corpora, we evaluate the quality of generated scanpaths. mishra et al. (2016a ) contains eye fixation sequences of seven participants for 994 text snippets annotated for sentiment and sarcasm. these were taken from amazon movie corpus , twitter, and sarcastic quote websites. the task assigned to the participants was to read one sentence at a time and annotate it with binary sentiment polarity labels (i.e., positive/negative). the same datasets were used in several studies @xcite to show improvements in sarcasm and sentiment analysis. we use the datasets to evaluate both the generation quality and potential improvements in nlp tasks. ## conclusion in this work, we make two novel contributions toward integrating cognitive and natural language processing. (1) we introduce the first scanpath generation model over text, integrating a cognitive reading model with a data-driven approach to address the scarcity of human gaze data on text.