A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender (2024)

Abstract: This study employs the classical psycholinguistics paradigm, the visual world eye-tracking paradigm (VWP), to explore the predictive capabilities of LLAVA, a multimodal large language model (MLLM), and compare them with human anticipatory gaze behaviors. Specifically, we examine the attention weight distributions of LLAVA when presented with visual displays and English sentences containing verb and gender cues. Our findings reveal that LLAVA, like humans, can predictively attend to objects relevant to verbs, but fails to demonstrate gender-based anticipatory attention. Layer-wise analysis indicates that the middle layers of the model are more related to predictive attention than the early or late layers. This study is pioneering in applying psycholinguistic paradigms to compare the multimodal predictive attention of humans and MLLMs, revealing both similarities and differences between them.

## introduction recent psycholinguistic research has shown that human language processing involves multimodal predictions, especially between language and vision (e.g., altmann & @xcite ; see @xcite , for a review). for instance, numerous visual world paradigm (vwp) studies have demonstrated that when people hear an utterance, they predict upcoming mentions, which direct their looks to the visual objects. for example, in @xcite , participants heard a sentence featuring either male or female characters and looked at the visual display of four objects at the same time (figure 1 ). they found that: (1) participants used verb semantics to predict upcoming mentions (e.g., looking at wearable objects such as a tie or dress at hearing tonight, james/kate will wear …); (2) they further used the gender of the subject to refine their prediction (e.g., more looks to a tie than a dress following james, and more looks to a dress than a tie following kate ). the finding that humans use linguistic (verb and gender) information to make predictive fixations of a visual scene led us to ask whether llava @xcite , a multimodal large language model (mllm), exhibits similar cross-modal predictive behaviors. previous studies have found parallels between model attention weights and human attention (measured by eye-tracking movements) in text reading @xcite @xcite . @xcite recently provided tentative evidence that multimodal models like clip @xcite may also resemble human predictive visual attention in video viewing. however, there is a gap in our understanding of whether mllms like llava can predictively "look at" a target object ## discussion this study uses the vwp to investigate the predictive capabilities of llava, a specific mllm. the findings reveal that the model exhibits human-like behavior in using verb information to predict the upcoming object in a visual display. this aligns with previous research demonstrating that both humans and models can utilize multimodal information to predictively attend to relevant features @xcite . however, unlike humans, the model does not predictively attend to relevant objects based on gender information, consistent with the lack of gender bias in clip, which is the basis for llava's vision encoder @xcite . however, attributing this lack of gender prediction solely to clip's characteristics requires further investigation. future studies should conduct more fine-grained comparisons between unimodal (text-only) and multimodal models to isolate the source of this behavior and better understand the interplay between linguistic and visual information in gender-based predictions. the difference between the model and humans may be explained by the nature of the stimuli, as our study used cartoon-like images while llava is mainly trained and evaluated on real-world objects @xcite . to investigate this hypothesis, we replaced the cartoon-like objects with real-world ones. as shown in figure 7 in appendix c, we observed a main effect of gender in the verb segment (β = 0.01, @xmath0.00, @xmath1.12, @xmath2.001), suggesting that the model processes real-world objects in a more human-like way than cartoon objects. this is consistent with the idea that models lack the perceptual flexibility of humans, leading to lower performance in recognizing atypical objects @xcite . the study also found that the middle layers play a significant role in multimodal predictions, aligning with previous studies showing that attention weights in middle layers better fit neural signals @xcite . however, the discrepancy with some studies showing that late layers correlate most significantly with human eyetracking data @xcite may be attributed to task differences: comprehension tasks (as in our and lamarre et al.'s studies) require more high-level semantic processing in middle layers, while production tasks (as in @xcite focus more on low-level features of individual words in later layers. further detailed experiments are needed to explore this hypothesis. ## conclusion in conclusion, our study utilizes the vwp from psycholinguistics to probe whether llava shows similar multimodal predictive patterns to humans. we found that llava can predictively attend to verb-relevant objects in visual displays similar to humans, but they do not show the same predictive attention for gender-relevant objects. these verbrelated predictive behaviors are predominantly driven by the middle layers of the model.