Exploring Spatial Schema Intuitions in Large Language and Vision Models (2024)

Abstract: Despite the ubiquity of large language models (LLMs) in AI research, the question of embodiment in LLMs remains underexplored, distinguishing them from embodied systems in robotics where sensory perception directly informs physical action.Our investigation navigates the intriguing terrain of whether LLMs, despite their non-embodied nature, effectively capture implicit human intuitions about fundamental, spatial building blocks of language. We employ insights from spatial cognitive foundations developed through early sensorimotor experiences, guiding our exploration through the reproduction of three psycholinguistic experiments. Surprisingly, correlations between model outputs and human responses emerge, revealing adaptability without a tangible connection to embodied experiences. Notable distinctions include polarized language model responses and reduced correlations in vision language models. This research contributes to a nuanced understanding of the interplay between language, spatial experiences, and the computations made by large language models.Project Website: https://cisnlp.github.io/Spatial_Schemas/

## introduction large language models (llms) excel in varied nlp tasks like text generation, sentiment analysis, or summarization. nonetheless, an underexplored facet in the study of llms pertains to the concept of embodiment. unlike embodied systems in robotics, where the physical form plays a central role in shaping the system's abilities, llms lack a direct connection between sensory perception and physical action. within this context, we investigate the extent to which llms, despite their lack of direct embodiment, might capture the implicit, often sensory-derived, conceptual structures that underlie human language and cognition. in our analysis, we figure 1 : overview of the three experiments make use of image schema theory, which provides a set of spatio-temporal cognitive building blocks that are learned in early infancy based on reoccurring sensorimotor experiences @xcite @xcite . for example, the image schema support is learned by observing and experiencing objects like tables or chairs supporting other objects. according to theory, the same image schema is reused to structure our language and thought, even in regard to abstract topics. for instance, when talking about emotional support, we say to support a friend. given that llms lack grounding, our research centers around a fundamental question: can llms encode people's intuitions about the image schematic basis of words and phrases? in the subsequent sections, we present the rationale behind our inquiry and the methodologies for investigating the intricate relationship between language, embodiment, and the core aspects of human cognition. in summary, we make the following contributions: ## method this section provides an overview of the general model and prompt selection, while following subsections detail the setup for individual experiments. ## discussion we explored if llms capture human intuitions about image schematic basis in language through three psycholinguistic experiments comparing llms and vlms. model responses often correlate with people's, especially in larger models, although discrepancies exist for certain image schemas. nevertheless, the models reflect spatial primitive intuitions, potentially stemming from their ability to model words, their contextual use, and their relation to schema definitions. another possibility is that the original experiment papers, serving as training data, might contribute to the models' reproduction of observed patterns, although parsing the original papers' results effectively is unlikely. at the same time, one might wonder why, in some cases, the model answers are so far apart from human answers. besides the explanation of the lack of embodied experience, one aspect to consider is that the llms only had access to one item at a time when rating stimuli. thus, models are unable to rate items relative to each other -a strategy a human participant is likely to adopt. for instance, given the two stimuli "stand in awe" and "the clock stands on the mantle" separately, gpt-4 gives scores of 2 and 1 for the image schema bal-ance. people, on the other hand, tend to relate balance much more strongly to a clock standing on a mantle. this intuition is captured when prompting the model with both stimuli at once, thus allowing the model to provide relative scores; in this case, the model provides a score of 2 for the first phrase but a score of 4 for the clock on the mantel, thus being much closer to human scores. however, prompting the model to provide all answers at once is currently not feasible, as described in section 4. this missing comparison between items is also reflected in the fact that the models tend to use the extremes of the scale, i.e., rating items with 1 or 7. another possible answer for the differences is related to the partially small participant pool used in the original experiments. the three experiments recruited 27, 24, and 173 participants. therefore, the originally recorded human answers might not be very robust, and future experiments should rerun similar setups not only with novel stimuli but also with larger participant pools to enable a more robust data set for comparison. the overall weakest correlations are observed for vlms, particularly with open-source model idefics, displaying meaningless responses. for instance, idefics-80b inst consistently selects the left-arrow image, disregarding the item, image order, or randomized labels. this failure is attributed to vlms being trained on natural images, unsuitable for interpreting the highly abstract line drawings. even gpt-4 vision , despite showing moderate correlations (0.56-0.57), occasionally provides random answers, deviating from human consensus (see fig. 3 ). this discrepancy raises questions about the models' alignment with human intuitions. ## conclusion this exploratory study is the first of its kind to reproduce psycholinguistic experiments in order to explore spatial schema intuitions. moreover, it provides evidence that llms are able to reflect those intuitions in different tasks and setups. the results also point out that despite the duo-modality, vlms do not encode the spatial understanding as effectively as their textual origins. future research aims to assess models using novel, collected stimuli, ensuring no prior exposure during training. our cur-rent findings provide a foundation for formulating precise, testable hypotheses in subsequent experiments. additionally, we would like to extend this line of research to include a multilingual analysis. in conclusion, our study not only sheds light on the disparities between llms and human cognition but also paves the way for new research perspectives in understanding and refining language models. by providing empirical evidence of these disparities, we advocate for a deeper exploration into the limitations of current models and the development of novel approaches to bridge the gap between artificial intelligence and human intelligence.