Describing Images \textit{Fast and Slow} : Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes (2024)

Abstract: There is an intricate relation between the properties of an image and how humans behave while describing the image. This behavior shows ample variation, as manifested in human signals such as eye movements and when humans start to describe the image. Despite the value of such signals of visuo-linguistic variation, they are virtually disregarded in the training of current pretrained models, which motivates further investigation. Using a corpus of Dutch image descriptions with concurrently collected eye-tracking data, we explore the nature of the variation in visuo-linguistic signals, and find that they correlate with each other. Given this result, we hypothesize that variation stems partly from the properties of the images, and explore whether image representations encoded by pretrained vision encoders can capture such variation. Our results indicate that pretrained models do so to a weak-to-moderate degree, suggesting that the models lack biases about what makes a stimulus complex for humans and what leads to variations in human outputs.

## introduction humans can capture the gist of an image usually incredibly fast -100 msec could be enough @xcite ; however, they would need more time to act on an image. for instance, human behavior while describing images illustrates the intricacies of visuo-linguistic processes. there may be repetitions, silent intervals and disfluencies, with considerable degrees of variation in what is uttered across speakers. the period prior to the utterance involves perceiving the image, conceptualizing the message, retrieving the labels of the entities to mention, formulating and preparing to articulate a grammatical and relevant utterance @xcite . as a result, we observe variations in speech onsets, as in figure 1 , which could be indicative of the min: 1.69 sec max: 7.07 sec figure 1 : the images with the minimum and maximum mean speech onsets across speakers in the dataset. the image with the maximum onset also elicits the highest variation in the first nouns of the descriptions. in addition, different speakers might start their utterances with different words (starting points, see @xcite , continuing to produce a varied set of image descriptions (linguistic variation) with variation in gaze. these signify the intricate cross-modal relation between visual and linguistic processes in humans (griffin and @xcite . although human data can be rich in behavioral signals, current pretrained multimodal models virtually never receive information about such signals during training. the models generate descriptions without necessarily modeling how human processes unfold. for instance, deep neural networks can output words at the same rate even for images that would result in diverse speech behavior by humans due to complexity or ambiguity. moreover, there is a gap between the manner in which humans perceive stimuli as compared to how large models process them. model-predicted surprisal values for linguistic input can be lower than human surprisal, possibly due to the massive size of the training data and the number of model @xcite @xcite . models also display different patterns of visual attention compared to humans @xcite . we argue that it is essential to consider human signals such as speech onsets and looking times, as they reflect the complexity and ambiguity of visuo-linguistic tasks @xcite @xcite @xcite . it is therefore desirable if models encode what leads to variations in such signals to help generate image descriptions in a way that is aligned with human processing and with types of variation observed in human data @xcite . to this end, several applications have exploited human gaze to enhance image captioning and visual question answering models @xcite @xcite @xcite . still, the relation between gaze on images and language is not widely researched in nlp @xcite . we first explore the natural dynamics in visuolinguistic processes using the dutch image description and eye-tracking corpus (didec; @xcite . this corpus provides gaze and speech data concurrently collected while participants describe images depicting real-life scenes. we preprocess the didec dataset extensively, and propose metrics to quantify the variation in visual and linguistic modalities. we reveal for the first time significant correlations between speech onsets, variation in starting points, descriptions and gaze. we hypothesize that this variation is partly due to the properties of the images, and that similar images would elicit similar amounts of variation. given the superior performance of pretrained encoders that are widely used in multimodal models, we investigate whether visual encoders such as clip @xcite and @xcite capture information regarding the variation in visuo-linguistic signals. 1 this is akin to probing pretrained models for meaningful syntactic and semantic information; see @xcite . using a similarity-based prediction method @xcite , we find that the pretrained encoders capture variation in signals to a limited extent. our findings suggest that underlying factors leading to variation are encoded rather weakly by pretrained models. with our work, we aim to direct attention towards the importance of the information contained in such signals and the variation thereof when crowdsourcing data as well as during model development. ## background we first give an overview of visuo-linguistic processes in humans in section 2.1, and then, in models in section 2.2. ## data we aim to explore the variation in human signals in visuo-linguistic processes and whether pretrained models can capture such variation in a realistic setup. a dataset consisting of simultaneous language production and eye movements over complex images would enable such an exploration. therefore, we opt for using the didec corpus @xcite instead of other existing image description datasets with eye-tracking, as this corpus allows us to delve into the dynamics of visual and linguistic processes in parallel. there exist few datasets containing such information, which we did not opt for utilizing, as they differ in their tasks (narratives; @xcite , or the processing steps the authors have taken, e.g., only a small subset of the captions were checked manually @xcite ; the authors sample one gaze point every 4 points @xcite . the didec dataset comprises manually checked descriptions of high quality, and the gaze data is provided in a raw format enabling custom processing. we use the 'production viewing' subset of didec, which contains spoken descriptions for 307 real-life images originating from the ms coco dataset @xcite , with high-quality eye-tracking data. foot_1 45 participants describe ≈ 102 images without a time limit. on average, each image has 15 descriptions (4604 in total). next, we explain how we extract features corresponding to human signals in visuo-linguistic processes from this dataset, to obtain 4586 descriptions with speech onsets, starting points, and fixated regions. ## variation in human signals we first delve into the nature of the variation across humans per image in the didec dataset. our focus is on uncovering potential correlations between the variations in human signals in visuo-linguistic processes. we first explain how we quantify each signal and its variation, see figure 2 for an example image with all of its variation scores. then, we conduct pairwise correlation analyses between the 4 variables. if there exist correlations between variations across signals, one can speculate that at least part of the correlation stems from the image, with the rest being potentially due to factors such as viewing order, priming and cognitive load. ## similarity-based prediction in light of the correlation findings in section 4, we expect image features to be predictive of the variation in visuo-linguistic signals to some extent. we explore if the similarity scores between image features encoded by pretrained models would be meaningful when capturing variation in human signals. in particular, we hypothesize that the signals that are more internal to the pretrained models' training objectives would be captured better. for instance, clip was trained with respect to an image-to-text alignment objective @xcite ; hence, it would be reasonable to expect that signals that are more inherent to the visual and language data could be encoded better compared to speech onsets, which are never seen by the model. ## conclusion we quantified the variation in speech onsets, starting points, descriptions and gaze using a dutch dataset of image descriptions with eye-tracking data. our findings revealed the extent of variation in the process of describing images, and that min: 0.529 max: 0.541 min: 23.666 max: 24.308 figure 6: variation in gaze as predicted by the similaritybased approach. human and machine processing have differences, and we are motivated by the potential benefits of making the models increasingly knowledgeable about the multimodal landscape of human data. although the impact of fine-tuning an already powerful pretrained model on a small-scale dataset with human signals could be modest, we hope that our work motivates the collection of more signals during crowdsourcing. for instance, it would be beneficial to take into account how long it took participants to complete a task given a certain stimulus, indicating the relative complexity and the uncertainty induced by the task as well as the stimulus. by inducing biases based on human signals, models can further take advantage of the information contained within such signals. although it would be difficult to capture the full extent of the intricacies of human processing, this could help, for instance, a model interacting with human users to generate responses more aligned with human expectations.