A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Models (2023)

Abstract: There is growing interest in incorporating eye-tracking data and other implicit measures of human language processing into natural language processing (NLP) pipelines. The data from human language processing contain unique insight into human linguistic understanding that could be exploited by language models. However, many unanswered questions remain about the nature of this data and how it can best be utilized in downstream NLP tasks. In this paper, we present EyeStyliency, an eye-tracking dataset for human processing of stylistic text (e.g., politeness). We develop an experimental protocol to collect these style-specific eye movements. We further investigate how this saliency data compares to both human annotation methods and model-based interpretability metrics. We find that while eye-tracking data is unique, it also intersects with both human annotations and model-based importance scores, providing a possible bridge between human- and machine-based perspectives. We propose utilizing this type of data to evaluate the cognitive plausibility of models that interpret style. Our eye-tracking data and processing code are publicly available.

## introduction human perception and understanding of text is critical in nlp. typically, this understanding is leveraged in the form of ground-truth human annotations in supervised learning pipelines, or in the form of human evaluations of generated text. however, human language understanding is complex; multiple cognitive processes work together to enable reading, many of which occur automatically and unconsciously @xcite . because of the complexity, disciplines concerned with understanding and modeling how humans read -e.g., psycholinguistics and cognitive science -heavily utilize implicit measures of the bert: @delta. are you kidding? delayed from 7:30pm to 11:00 -only to cancel it because the pilot is overtime. annotation: @delta. are you kidding? delayed from 7:30pm to 11:00 -only to cancel it because the pilot is overtime. eye tracking: @delta. are you kidding? delayed from 7:30pm to 11:00 -only to cancel it because the pilot is overtime. figure 1 : salient words for impoliteness from three different perspectives. we find that eye tracking data contains some overlap between machine and human-annotated salience. recent research in nlp has demonstrated the feasibility of incorporating various types of eye movement data into nlp models in order to improve performance on a number of tasks (see to address these questions, we conduct an eye tracking case study in which participants read texts the hummingbird dataset @xcite . we choose this dataset because it contains lexicallevel human annotations indicating which words contribute to the text's style and because its domain (textual styles) has not to our knowledge been widely explored for eye tracking applications -although prior work investigates eye tracking and sentiment analysis, it does not extend to other linguistic styles such as politeness. we collect style-specific eye movements through a carefully designed experiment (see section 3 for details), and we use these eye movements to derive saliency scores over the text. we compare this eye-based saliency to human annotations as well as two large language model (llm)-derived importance scores: integrated gradient scores from a bert model fine-tuned on style datasets @xcite , and word-surprisal scores from gpt-2 @xcite (see figure 1 for an example). our findings indicate that eye-trackingbased saliency highlights some unique areas of the text, but it also intersects with both saliency from model-based metrics and saliency from human annotations, making a bridge of sorts between the human-and machine-based perspectives. we discuss some implications of these findings for nlp research. specifically, our contributions are: • an experimental paradigm for obtaining eye tracking-based signals for specific features of text (in our case, textual style). ## related work eye tracking has been a staple of psycholinguistic investigations of reading for decades @xcite . eye movement data is compelling because it provides realtime information about how people process language in a natural, ecologically valid setting (i.e., there is no nlp area h m learning from eye data ours textual style ✓ ✓ ✗ kuribayashi et al. (2021) perplexity ✗ ✓ ✗ malmaud et al. (2020) qa ✗ ✗ joint learning bolotova et al. (2020) qa ✗ ✓ ✗ sood et al. (2020b) qa ✗ ✓ ✗ sood et al. (2020a) paraphrasing ✗ ✗ joint learning hollenstein et al. (2019) sentiment clf., ner ✗ ✗ joint learning barrett et al. (2018) pos tagging ✗ ✗ hmm tokunaga et al. (2017) ner ✗ ✗ ✗ klerke et al. (2015) summarization ✓ ✗ ✗ green (2014) parsing ✗ ✗ ✗ eye data provides insight into cognitive processes through the eye-mind assumption, which posits that (1) our eyes fixate on whatever our brains are currently processing, and (2) as cognitive effort to process an item increases, the amount of time that the eyes fixate on that item also increases @xcite . analysis of eye data under this framework has led to important insights into many unconscious phenomena in human language comprehension, e.g. the mechanisms involved in ambiguity resolution during reading @xcite . eye tracking in nlp. due to the eye-mind assumption, eye-tracking data is particularly wellsuited to inferring patterns of reader attention, or saliency, over text. this saliency information has so far shown promising results when integrated into nlp models for question answering (e.g. @xcite ; @xcite ; @xcite ). however, this is still a developing research area: there is limited available data, and there is little consensus regarding how to effectively collect data and incorporate it into nlp pipelines. to our knowledge there is no previous research that investigates saliency for style via eye tracking, nor any previous research that compares saliency from eye tracking to human annotations ( outside of textual saliency, eye-tracking data has been leveraged for a variety of nlp tasks. @xcite quantify the difficulty of sentences in machine translation tasks using eye movement data; @xcite determine whether a reader understands sarcasm in text, and søgaard (2016) evaluate the quality of word embeddings and text generations, respectively. other work uses existing datasets, sometimes augmenting the data with a learned gaze predictor model, and uses this eye movement data as an additional signal when training models for various nlp tasks, including named entity recognition @xcite , paraphrasing @xcite , part-of-speech tagging @xcite , and sentiment analysis (see also @xcite for a review). saliency in linguistic styles. people apply styles to language in order to express attitudes, reflect interpersonal intentions or goals, or convey social standings of the speaker or listener. (note that while many sociolinguistics theories distinguish between textual style and textual attributes, in this work, we follow the common convention in recent nlp papers of broadly using 'style' to encompass both of these ideas @xcite .) the meaning expressed by textual styles can be significant; in fact, there is strong evidence that effective communication requires an understanding of both style and literal semantic meaning @xcite . although bert @xcite based fine-tuned models show strong performance on style classification, there are notable differences between how bert perceives style at the lexical level and how humans perceive it, and that using data about these differences during training improves model performance @xcite . ## eyestyliency: a dataset of eye movement for textual saliency we describe the data collection procedure for eye-styliency dataset from 20 participants and methods for computing saliency scores over text. ## key findings and discussion here we discuss the relationship between our results and our research questions: rq1: does eye tracking data for saliency meaningfully differ from simply gathering word-level human annotations, or from model-based word importance measures? our data show a substantial difference between eye-tracking-based saliency, model-based saliency, and human annotations. it is perhaps unintuitive that reading behavior would differ from self-reports after reading, but this is consistent with findings in psycholinguistics that establish strong distinctions between explicit measures (i.e., human annotations) and implicit measures (i.e., eye tracking) of human language processing. interestingly, there is some intersection between eye tracking-based saliency and model-based saliency that is not shared with human annotators. this suggests that some automatic aspects of human language processing, accessible through eye tracking but not necessarily survey methods, may be shared with large language models. rq2: how can we measure eye movements specific to a high-level textual feature like style, and which eye tracking metrics and data processing methods are best suited to capturing textual saliency? the results from our experiment indicate that our experimental paradigm exploiting congruency effects may be effective in finding eye movements specific to certain text features. in a linear mixed effect model analyzing the data, we find significant effects of the congruency condition on dwell time and pupil size (see appendix a.2). this suggests that the congruency effect does impact reading patterns -whether this impact is directly linked to the textual style is difficult to definitively answer, but given the overlap between eye-tracking-based style saliency and other style saliency measures, it seems reasonable to believe that the experimental manipulation resulted in an implicit measure of style perception. experiments based on congruency effects may be a promising route for capturing eye movements related to other high-level textual features such as sarcasm and metaphor. we find that dwell time appears to be the strongest eye-tracking metrics for capturing textual saliency, as it has both the highest overlap with human-and machine-based saliency and most strongly responded to the experimental manipulation. using the same criteria, we also find that using participant-level z-scores to represent the eye movement data yields the best results. ## limitations in this exploratory study, our dataset and sample size are both small, limiting the possibilities for a more thorough evaluation of the data e.g. by fine-tuning a language model. we also note that by design, our experiment presents incongruent items rarely, and consequently we have considerably more congruent datapoints than incongruent datapoints -an inherent limitation of the proposed experimental paradigm. in light of our results, which suggest that eye-tracking data contains useful and unique information, we plan to develop methods for collecting this kind of real-time human reading data at scale -i.e., without the constraints of costly in-person eye tracking -in future work. finally, eye tracking analysis in general is limited by the eye-mind assumption, which holds that the eye fixates on what the mind is currently processing. while there is strong evidence supporting the eye-mind assumption during reading, there is a notable exception: retrieval processes (i.e. accessing memory) are not reflected in eye movements @xcite . annotations model grads surprisal dwell time reread time go past time first run dwell first fixation duration pupil size annotations model grads surprisal dwell time reread time go past time first run dwell first fixation duration pupil size 1 0.17 1 0.058 0.038 1 -0.0055-0.013 0.064 1 -0.019 -0.042 0.04 0.19 1 -0.027 -0.034 0.041 0.23 0.98 1 -0.056 0.015 0.038 0.22 -0.012 0.14 1 -0.037 -0.035 0.065 0.076 0.01 0.046 0.28 1 -0.019 -0.053 -0.066 -0.072 -0.021 -0.023 -0.031 -0.013 1 figure 12: correlations (pearson's r) between the saliency scores derived from each method.