Image-conditioned human language comprehension and psychometric benchmarking of visual language models (2024)

Abstract: Large language model (LLM)s’ next-word predictions have shown impressive performance in capturing human expectations during real-time language comprehension. This finding has enabled a line of research on psychometric benchmarking of LLMs against human language-comprehension data in order to reverse-engineer humans’ linguistic subjective probability distributions and representations. However, to date, this work has exclusively involved unimodal (language-only) comprehension data, whereas much human language use takes place in rich multimodal contexts. Here we extend psychometric benchmarking to visual language models (VLMs). We develop a novel experimental paradigm, \textit{Image-Conditioned Maze Reading} , in which participants first view an image and then read a text describing an image within the Maze paradigm, yielding word-by-word reaction-time measures with high signal-to-noise ratio and good localization of expectation-driven language processing effects. We find a large facilitatory effect of correct image context on language comprehension, not only for words such as concrete nouns that are directly grounded in the image but even for ungrounded words in the image descriptions. Furthermore, we find that VLM surprisal captures most to all of this effect. We use these findings to benchmark a range of VLMs, showing that models with lower perplexity generally have better psychometric performance, but that among the best VLMs tested perplexity and psychometric performance dissociate. Overall, our work offers new possibilities for connecting psycholinguistics with multimodal LLMs for both scientific and engineering goals.

## introduction human language comprehension is highly incremental. our minds integrate linguistic input with context very rapidly: words within sentences, and even phonemes or letters within spoken or writ-ten words, to update our understanding of linguistic input @xcite . this process involves the rapid update of expectations about the interpretation of what has already been said and predictions about what might be said next. these predictions affect how we process the language we encounter, helping us to recognize and correct errors @xcite and to analyze input more rapidly. the fundamental operation of large language models (llms) is similar: llms put probability distributions over the next tokens given the preceding context. this convergence has made it natural to compare llm distributions with human linguistic behavior. in unimodal language processing, llm predictions have been shown to align fairly well with those generated by humans in the cloze task @xcite . furthermore, there is a linear relationship between the surprisal of a word in linguistic context (negative log-probability; @xcite ) and how long comprehenders take to read it @xcite . these findings have generated interest in psychometric benchmarking of language models (lms): comparing lms in terms of how well their autoregressive probabilities predict human reading times or other types of linguistic behavior @xcite @xcite @xcite . psychometric benchmarking of llms has exclusively involved unimodal, language-only data and models. however, human language use generally involves a rich multimodal context. for this reason, there is growing interest in multimodal language models. the most advanced such type of model is vision-language models (vlms), which relate visual content (most commonly static images) to linguistic content. for example, models like blip-2 @xcite can generate text associated with an image; to do this, it autoregressively places con-ditional probability distributions over next linguistic tokens given an image in context plus preceding linguistic context. however, evaluation techniques for vlms are less developed than for unimodal llms, and we are aware of no work to date on psychometric benchmarking for vlms. here we present a framework and experimental results on psychometric evaluation of visual language models using a novel yet simple psycholinguistic experimental paradigm. in an experimental trial, a participant first previews an image, then reads a sentence describing an image, with wordby-word reading times measured (figure 1 ). the image may be the one that the sentence describes (the correct image condition), a different image that the sentence does not describe (the wrong image condition), or simply a black screen (the no image condition). intuitively, previewing the correct image should prepare the participant for the sentence description and facilitate them reading it more quickly and accurately. however, there are different forms that this facilitation could take, corresponding to different theoretical accounts of how visual context shapes language processing. additionally, we can compare vlms in terms of how well they capture how different image contexts influence the participant's reading behavior. we can thus use this experimental paradigm both to gain insight into the role of visual context in language processing in the human mind and to psychometrically benchmark visual language models. all the experiment codes, analysis, and datasets used in the project are made available at the linked repository. 1 . ## experimental methodology we developed an image-conditioned maze experimental paradigm which is like the original maze, but participants preview an image before reading each text passage. we chose 108 images and their corresponding descriptions from the validation split of microsoft coco @xcite . in each experimental trial, participants were first shown an image for 5 seconds, and then the image disappeared from the screen and they read an image description word by word in the maze task. we generated distractor words using the a(uto)-maze software of @xcite , which uses an lstm rnn based model @xcite to generate contextually unlikely words. reaction time and response for each word choice (correct vs. distractor) were recorded. we recruited 69 us native english speaker participants (a quantity determined using power analysis based on a pilot study with a different set of images and descriptions) on prolific, showed them some examples, and paid them 12$/hour for their participation. each of them participated in 36 trials, 12 in each of the three conditions described before in figure (1), with trial order randomized for each participant. no participant saw the same image description twice. in a separate study with different participants, we collected groundedness ratings for each word in each description in the context of the correct image associated with the description (figure 2 ). we recruited 42 us native english speaker participants on prolific for this study. each sentence was rated by 7 participants on average. participants used a slider to indicate how "present" each word was in the image, ranging from -10 (not present) to +10 (surely present). ## psycholinguistic hypotheses under wide circumstances, visual input automatically activates corresponding linguistic representations; a famous example is the stroop effect, where a word naming one color but presented in another, such as blue, is difficult to say due to the interference between the words activated by the color versus orthographic information. we thus hypothesize that previewing the image will tend to activate at least some of the linguistic content in the image's description, so that reaction times will be faster and accuracy higher more quickly and accurately in the correct image condition than in the wrong image and no image conditions. we also hypothesize that the wrong image condition may slow reaction times and reduce accuracy relative to the no image condition, since the linguistic content that the image activates may conflict with the content in the subsequent text. we distinguish between two versions of these hypotheses. one possibility is that activation of linguistic content may be restricted to content that is straightforwardly grounded in the image. for example, in the correct image example of figure 1 , the words woman, red, and dress are straightforwardly grounded: the meaning of each word is prominent in the image without extensive reasoning or search for complex linguistic descriptions. in contrast, the rest of the words in that description are less straightforwardly grounded. our lexicalgrounding hypothesis is that linguistic facilitation or interference effects from the image will be limited to relatively straightforwardly grounded words. in cognitive terms, objects, properties, events, and states in the scene are visually identified, and the corresponding lemmas are activated so that when those lemmas are encountered in the image description, they are processed more effectively. we operationalize groundedness in two different ways: first as open-class (generally more grounded) versus closed-class (generally less grounded) parts of speech; second, through our grounding study as described in section 3. the second possibility, the comprehensivegrounding hypothesis, is that images evoke expectations over complete possible descriptions. this hypothesis predicts that facilitation or interference will affect all types of words in the sentence, regardless of part of speech or groundedness. a particularly strong version of the comprehensivegrounding hypothesis is that all facilitation and interference effects from the image will be mediated by this change in linguistic expectations. if this strong version of the hypothesis is correct, and if visual language models do a good job of capturing this shift in expectations, then visual language model surprisal should fully account for the effect of experimental conditions in the human behavioral data in our experiment. ## modelling approach we created a set of predictor variables including condition_id, frequency, word length, groundedness, open vs. closed part of speech, and surprisals from six transformer-based llms: four visual language models with a variety of objectives regarding language-vision alignment (blip2, @xcite @xcite and two language only models (gpt2, @xcite . condition_id indicates whether a certain image description was seen in correct, wrong, or no image condition, which could be extracted from the experiment setup on ibexzehr and schwarz, 2018. for length, we used the length in characters excluding end punctuation. we obtain word frequencies from subtlex_us @xcite ; for the words not in the database, we use the minimum frequency of any word in that database. groundedness comes from our norming study. for open versus closed class part of speech, we ran the stanford pos tagger on our image descriptions and considered all nouns, adjectives, adverbs, and non-auxiliary verbs, as open-class, and the rest as closed-class. surprisal does not vary across conditions for llms, but does so for vlms for image conditioning. ## perplexity and psychometric accuracy in recent years, there has been an effort to study the increase of log-likelihood for including llm surprisal estimate from models as a function of perplexity @xcite . to investigate what traits in a vlm give them better predictive power for human rt, we ran a similar analysis with different-sized open-sourced versions of all the models we used in the work -two versions of all the vlms except for kosmos-2 and a new vlm that improved upon llava, llava-next. the baseline regression model was considered with all baseline predictors such as main effects of helmert encoded condition_id and sum encoded pos and interaction between them, frequency, length and full regression models additionally contained each lm surprisal predictor. both the baseline and full regression models had the same random effects structure; a random intercept and slope for condi-tion_id within each subject, sentence, word, and word token type was included. after fitting the regression models, we determined the increase in log-likelihood (∆ll) for each model by subtracting the log-likelihood of the baseline model from that of the full model. finally, the perplexity of each model type was calculated in our dataset of all items. note that the increase of log-likelihood for adding surprisals from different-sized versions of the same model isn't very different, however different models can have very different predictive power regardless of the size, consider llava and llava-next for example, both versions considered for these models have the same sizes(7b and 13b parameter) but llava-next has a lot more predictive power compared to llava. this strongly indicates that training diet and objective are more important than the model size when it comes to psychometric predictive power. however, all the smaller-size versions except for llava-next are better than the bigger-size versions. although this needs further exploration, the observations indicate that for each type of training objective and diet, there is possibly an optimal number of parameters that make the model most aligned with human expectations, and beyond that alignment decreases. ## conclusion in this work, we have developed a novel experimental paradigm, image-conditioned maze reading, to study human linguistic expectations during real-time language comprehension when a visual context is involved. our results demonstrate a substantial facilitatory effect of correct image context on language comprehension. this effect is evident not only for concrete nouns, adjectives, or verbs directly present in the image but also extends to words not explicitly grounded in the visual context. we extended psychometric benchmarking to visual language models and found that vlm surprisals capture most to all of the facilitator effect that occurs due to the presence of a relevant visual context. we discovered that as one goes from text based model surprisal to vlm surprisal, the effect of surprisal on reading time doesn't change much, but the huge condition_id related effects mostly disappear for vlm surprisal based models. so, the explanation is in how the nature of the surprisal changes. we also found a strong correlation between the human judgment of a word's degree of grounding in the image and the reduction of that word's surprisal for the presence of that image. we showed empirical support indicating that heightened contextual surprisal significantly contributes to errors in maze tasks. finally, our findings reveal compelling evidence that the training objectives and diet of vision-language models (vlms) significantly impact their psychometric predictive power, more so than their size. however, this observation warrants further investigation.