Do Vision and Language Models Share Concepts? A Vector Space Alignment Study (2024)

Abstract: Large-scale pretrained language models (LMs) are said to “lack the ability to connect utterances to the world” (Bender and Koller, 2020), because they do not have “mental models of the world” (Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT, and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy, and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).1

## introduction the debate around whether lms can be said to understand is often portrayed as a back-and-forth between two opposing sides @xcite , but in reality, there are many positions. some researchers have argued that lms are ''all syntax, no semantics'', i.e., that they learn form, but not meaning @xcite @xcite ). 2 1 code and dataset: https://github.com /jiaangli/vlca . 2 the idea that computers are ''all syntax, no semantics'' can be traced back to german 17th century philosopher leibniz's mill argument @xcite . the mill argument states that mental states cannot be reduced to physical states, so if the capacity to understand language requires mental states, this capacity cannot be instantiated, merely imitated, by machines. in 1980, searle introduced an even more popular argument against the possibility of lm understanding, in the form of the so-called chinese room thought experiment @xcite . the chinese room presents an interlocutor with no prior knowledge of a foreign language, who receives text messages in this language and follows others have argued that lms have inferential semantics, but not referential semantics @xcite @xcite , 3 whereas some have posited that a form of externalist referential semantics is possible, at least for chatbots engaged in direct conversation @xcite @xcite . most researchers agree, however, that lms ''lack the ability to connect utterances to the world'' @xcite ), because they do not have ''mental models of the world'' @xcite . this study provides evidence to the contrary: language models and computer vision models (vms) are trained on independent data sources (at least for unsupervised computer vision models). the only common source of bias is the world. if lms and vms exhibit similarities, it must be because they both model the world. we examine the representations learned by different lms and vms by measuring how similar their geometries are. we consistently find that the better the lms are, the more they induce representations similar to those induced by computer vision models. the similarity between the two spaces is such that from a very small set of parallel examples we are able to linearly project vms representations to the language space and retrieve highly accurate captions, as shown by the examples in figure 1 . contributions. we present a series of evaluations of the vector spaces induced by three families of vms and four families of lms, i.e., a total of fourteen vms and fourteen lms. we show that within each family, the larger the lms, the more their vector spaces become structurally similar to those of computer vision models. this enables retrieval of language representations of images (referential semantics) with minimal supervision. retrieval precision depends on dispersion of image and language, polysemy, and frequency, but consistently improves with language model size. we discuss the implications of the finding that language and computer vision models learn representations with similar geometries. ## related work inspiration from cognitive science. computational modeling is a cornerstone of cognitive science in the pursuit for a better understanding of how representations in the brain come about. as such, the field has shown a growing interest in computational representations induced with self-supervised learning @xcite . cognitive scientists have also noted how the objectives of supervised language and vision models bear resemblances to predictive processing @xcite @xcite (but see @xcite for a critical discussion of such work). studies have looked at the alignability of neural language representations and human brain activations, with more promising results as language models grow better at modeling language @xcite . in these studies, the partial alignability of brain and model representations is interpreted as evidence that brain and models might process language in the same way @xcite . ## methodology our primary objective is to compare the representations derived from vms and lms and assess their alignability, i.e., the extent to which lms converge toward vms' geometries. in the following sections, we introduce the procedures for obtaining the representations and aligning them, with an illustration of our methodology provided in figure 2 . ## experimental setup in this section, we discuss details around bimodal dictionary compilation ( § 4.1), evaluation metrics, as well as our baselines ( § 4.2). ## analysis here, we test whether our findings extend to different parts of speech, as well as how alignment precision is influenced by factors such as dispersion, polysemy, and frequency, or by the size of the seed used for procrustes analysis. for all experiments in § 6, we use the largest model per model family for both vms and lms and the exclude-1k bimodal dictionary, unless stated otherwise. ## discussion having established that language and vision models converge towards a similar geometry, we discuss the implications of this finding from various practical and theoretical perspectives within ai-related fields and beyond. implications for the lm understanding debate. bender and koller (2020) have been cited for their thought experiment about an octopus listening in on a two-way dialogue between two humans stuck on deserted islands, but bender and koller (2020) also present the following more constrained version of their thought experiment: for the text data, the training task is purely one of predicting form. for the image data, the training task could be anything, so long as it only involves the images. at test time, we present the model with inputs consisting of an utterance and a photograph, like how many dogs in the picture are jumping? or kim saw this picture and said ''what a cute dog'' what is cute? this second thought experiment highlights the importance of relating word representations to representations of what they refer to. it also shows what their argument hinges on: if unsupervised or very weakly supervised alignment of lms and vms is possible, their argument fails. the question then is whether such alignment is possible? bender and koller deem lms unable to 'connect their utterances to the world' (or images thereof), because they assume that their representations are unrelated to representations in computer vision models. if the two representations were structurally similar, however, it would take just a simple linear mapping to make proxy inferences about the world and to establish reference. thought experiments are, in general, fallible intuition pumps @xcite , and we believe our results strongly suggest that the second thought experiment of bender and koller (2020) is misleading. ## conclusion in this work, we have studied the question of whether language and computer vision models learn similar representations of the world, despite being trained on independent data from independent modalities. we evaluated the structural similarity of the representations learned by these models for different sizes and architectures. we found that the geometries of these spaces are surprisingly similar, and that similarity increases with model size. these results seem to challenge the second thought experiment in @xcite . in our experiments, our baseline never goes beyond 1% at p@100, but our linear maps exhibit p@100 scores of up to 64%-an inferential ability which, in our view, strongly suggests the induction of internal world models, something which many previously have deemed impossible. we have discussed various implications, but not all: in the past, researchers have speculated if image representations could act as an interlingua for cross-lingual knowledge transfer @xcite @xcite . our results suggest this is viable, and that the quality of such transfer should increase log-linearly with model size. clip results. we also investigate the effects of incorporating text signals during vision pre-training by comparing pure vision models against selected clip @xcite vision encoders (resnet50, resnet101, resnet50×60, vit-base-patch32, and vit-large-patch14). the results align with our expectations, indicating that the clip vision encoders exhibit better alignment with lms. the findings also support our previous observation that larger lms tend to demonstrate better alignment. however, it would be unfair to directly compare the results from clip with pure vision models, as the pretraining datasets they utilize differ significantly in scale and scope. detailed results are presented in figure 9 and figure 10 .