Fine-Tuning Pre-Trained Language Models with Gaze Supervision (2024)

Abstract: Human gaze data provide cognitive information that reflect human language comprehension and has been effectively integrated into a variety of natural language processing (NLP) tasks, demonstrating improved performance over corresponding plain text-based models. In this work, we propose to integrate a gaze module into pre-trained language models (LMs) at the fine-tuning stage to improve their capabilities to learn representations that are grounded in human language processing. This is done by extending the conventional purely text-based fine-tuning objective with an auxiliary loss to exploit cognitive signals. The gaze module is only included during training, retaining compatibility with existing pre-trained LM-based pipelines. We evaluate the proposed approach using two distinct pre-trained LMs on the GLUE benchmark and observe that the proposed model improves performance compared to both standard fine-tuning and traditional text augmentation baselines.

## introduction as humans read, the unconscious cognitive processes that unfold in their minds while comprehending the stimulus text are reflected in their eye movement behavior @xcite . these gaze signals hold the potential to enhance nlp tasks. research has focused on using aggregated word-level gaze features to enrich text features @xcite @xcite or to regularize neural attention mechanisms, making their inductive bias more human-like @xcite @xcite . moreover, there has been growing interest in adopting non-aggregated scanpaths (i.e., sequences of consecutive fixations) to augment lms. these scanpaths capture the complete sequential ordering of a reader's gaze behavior and approximate their attention. @xcite and @xcite employed neural networks to independently encode scanpaths and text, followed by the fusion of the features extracted from both modalities. @xcite proposed rearranging the contextualized token embeddings produced by pre-trained lms based on the order in which the reader fixates on the words, followed by applying sequence modeling to the reordered sequence. to tackle the issue of gaze data scarcity, @xcite explored the possibility of augmenting lms using synthetic scanpaths, generated by a scanpath generation model. remarkably, synthetic scanpaths demonstrated advantages across various nlp tasks, particularly in settings with limited labeled examples for the downstream task. in contrast to previous studies that concentrated on learning joint cross-modal representations of text and scanpath, we start from a different perspective and explore utilizing gaze data to improve on the learned text representations of pre-trained lms during the fine-tuning stage, without incurring additional computational effort when using the model at application time. to this end, we extend the standard pre-trained lm fine-tuning objective with an auxiliary loss by integrating a scanpath module, which serves a dual purpose. first, the auxiliary loss can effectively incorporate human-like gaze signals generated using a scanpath generation model and thus provide informative gradients to guide the lm towards more representative local minima. second, reordering the token-embedding sequence based on the fixation sequence can diversify textual information, potentially improving generalization performance @xcite . this stands in contrast to heuristic text augmentation strategies, like random word insertion, replacement, swapping, and deletion @xcite . scanpaths inherently contain cognitive information that better aligns with and complements textual information. notably, our proposed gaze module is only active during training (fine-tuning), ensuring alignment with the standard usage of lms after this stage. this offers two key benefits. first, it facilitates seamless integration with existing lmbased pipelines. second, at deployment time, it eliminates the need to either collect real-time gaze recordings, which is costly and impractical for most use-cases, or generate synthetic gaze data, which is often computationally challenging for devices with limited computational resources. on the general language understanding evaluation (glue) benchmark, our proposed gazeaugmented fine-tuning outperforms both standard text-only fine-tuning and traditional text augmentation baselines, without incurring additional computational effort at application time. ## method in this section, we start out with a brief description of the conventional fine-tuning procedure for transformer-based encoders on downstream tasks. subsequently, we introduce our method, and explain how it incorporates synthetic scanpaths into this fine-tuning procedure to enhance representation learning of transformer-based encoders. the overall model architecture is illustrated in figure 1 . preliminaries our learning objective is to solve standard multi-class classification or regression problems. we assume access to a transformer-based pre-trained lm like bert @xcite or roberta @xcite . in the conventional fine-tuning approach for downstream tasks, the pre-trained lm is adapted to a specific task by fine-tuning all the parameters end-to-end using task-specific inputs and outputs. the final hidden state of the "[cls]" token typically serves as the aggregated sentence representation, which is then fed into a newly initialized (series of) dense layer(s) with output neurons corresponding to the number of labels in the task. we minimize the standard cross-entropy loss for classification and mean-squared-error loss for regression, denoted as l standard in figure 1 . ## conclusion our work contributes to the broad effort of enriching nlp models by grounding them in various domains of experience. specifically, we focus on the use of scanpath data, demonstrating its vital role in enhancing textual representation learning. by extending the standard pre-trained lm finetuning objective with a scanpath-integrated loss, we ground the lm in human language processing. finally, our experiments show that the proposed method surpasses standard fine-tuning and eda baselines on the glue benchmark, pointing to the potentially promising future direction of enriching textual representations with gaze data, especially for low-resource tasks and languages @xcite . however, it should be noted that the performance gains achieved by incorporating gaze supervision vary across different nlp tasks. future work may include further analysis of the impact of incorporating cognitive information into language models on specific downstream tasks.