Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era (2025)

Abstract: Human learning and conceptual representation is grounded in sensorimotor experience, in contrast to state-of-the-art foundation models. In this paper, we investigate how well such large-scale models, trained on vast quantities of data, represent the semantic feature norms of concrete object concepts, e.g. a ROSE is red, smells sweet, and is a flower. More specifically, we use probing tasks to test which properties of objects these models are aware of. We evaluate image encoders trained on image data alone, as well as multimodally-trained image encoders and language-only models, on predicting an extended denser version of the classic McRae norms and the newer Binder dataset of attribute ratings. We find that multimodal image encoders slightly outperform language-only approaches, and that image-only encoders perform comparably to the language models, even on non-visual attributes that are classified as “encyclopedic” or “function”. These results offer new insights into what can be learned from pure unimodal learning, and the complementarity of the modalities.

## introduction multimodal models depend on vision encoders to provide information about the objects that are depicted, including their properties, spatial configuration, lighting, and scene information. recent work has highlighted a degree of linear alignment between neural network representations of the vision and language modalities @xcite @xcite . this implies that the respective representation spaces have similar configurations, in terms of the local organisation (nearest neighbours) of concepts. however, there remains an open the question of how the different modalities "understand" or represent the concepts: which attributes are salient for 1 code, datasets and results are available at: https:// danoneata.github.io/seeing-what-tastes-good . figure 1 : given a dataset of concrete concepts, depicted using either visual or linguistic data, that are paired with semantic norms, we train linear probes on frozen modality-specific representations of to understand how well conceptual attributes can be extracted from models. a concept? in other words, how similar, in terms of underlying attributes: is a chair as seen by a vision encoder similar to a chair as encoded by a language model? this question concerns the complementarity of vision and language: are different modalities distinct, or in fact convergent @xcite ? is a single modality, such as text, sufficient, or are multiple knowledge sources necessary? early work on distributional representations, in text-only @xcite @xcite @xcite and multimodal @xcite ) models of static word embeddings, studied this question extensively. recent advances in representation learning calls for revisiting this question to understand the relative representational power of each modality in modern models. in this paper, we investigate how vision, language, and vision-and-language models represent concrete object concepts in terms of their associ-ated attributes (semantic norms). we use a linear probing methodology to test whether model representations make distinctions corresponding to attributes associated with concepts, depicted visually or in text. figure 1 presents an overview of our approach. the semantic norms cover many types of attributes, from visual-perceptual is green, to the functional is eaten, to the encyclopedic grows on trees. our first question is whether different encoders, from different modalities, capture particular attribute types more or less well. secondly, the models we evaluate correspond to a set of hypotheses about the role of language and labelling in conceptualization and category learning, a hotly debated topic in cognitive and neuroscience @xcite @xcite . at one extreme are pure vision encoders (vit-mae, dinov2) trained without any language or category label supervision. at the other, models like clip and siglip learn to represent the visual input by aligning it to text as batch-wise nearest neighbours: a form of language-steered world learning. we also evaluate text-only models that get categories for free (via word labels) but have to infer perceptual and other attributes from distributional semantics. inasmuch language "carves up the world", visual encoders with more language input should be better aligned with semantic norms for english concepts. we test these hypotheses using two concept attribute datasets. the first dataset links the semantic norms from the mcrae dataset @xcite to the concepts of the things project @xcite , with an additional expansion step, to create the new mcrae×things dataset. the second is a dataset of neuro-cognitive attribute ratings from @xcite , which has been used to investigate language model representations (utsumi, 2020; @xcite , but not, to our knowledge, visual or multimodal representations. our results demonstrate strong conceptual awareness in multimodal visual encoders across all attribute types. moreover, while single-modality models behave most similarly (i.e. vision models and language models correlate most strongly within-modality), all performant models are highly correlated, indicating a degree of convergence, given exposure to sufficient data of either modality. the main contributions of this paper include: ## related work understanding and evaluating the lexical semantics learned by language models via co-occurrence patterns is a long-standing concern in distributional semantics. a popular method for evaluating vector representations of lexemes is the correlation between the cosine similarity of two words in model space compared to human ratings of word similarity (e.g. using men @xcite and sim-lex @xcite ). however, cosine similarity cannot uncover the underlying dimensions of meaning space, or how the space distinguishes between human-meaningful attributes. in contrast, testing for specific semantic attributes, by predicting semantic norms, can inform us about the underlying organisation of a model's representation space. @xcite were the first to use the mcrae semantic norm dataset to evaluate the correspondence between early models of distributional semantics and cognitive concepts, using nearestneighbors procedures. using prediction models similar to linear probing, @xcite ; @xcite find that static word embeddings encode taxonomic properties significantly more accurately than other types of properties, a finding we replicate. however, @xcite find that embeddings also reliably encode attributes that cut across taxonomic classes, such as is dangerous. @xcite show that semantic norms can be predicted from word embeddings for unseen concepts. contextual representation models outperform static word embeddings @xcite . @xcite use semantic norms to explicitly probe for taxonomic generalization across concepts. conceptual attributes (either in the form of mcrae norms directly or very similar data) have also been used to investigate the complementar-ity of representations learned from language and vision. while silberer et al. (2013); derby et al. (2018); derby (2022) show that multimodal representations can improve norm prediction, i.e. that two modalities are better than one, @xcite ; @xcite find only slight patterns of differences when they examine the differences between vision and language representations in predicting different attribute types. this latter finding (which we confirm for more recent models) is in line with more recent work by @xcite ; @xcite which posits a linear relationship between vision and language encodings. these works also compare across different vision architectures with more or less supervision. @xcite connect frozen visual encoders to frozen language models with a trained linear transform, and find that the performance on image captioning correlates with the amount of language supervision of the visual encoder: clip, trained with full captions, performs better than a model trained on category labels, and self-supervised beit, trained on image data alone, performs worst. alternatively, @xcite perform procrustes analysis (a linear mapping) between image representations from imagenettrained vision models and language model representations for the same concepts, and find better alignment with larger language models, and with vision models that have been trained on supervised classification tasks, rather than self-supervised learning. there is less work on the semantic alignment of vision model representations with human conceptual knowledge. in the computer vision literature, @xcite ; @xcite has investigated the alignment between vision model representation spaces and human visual similarity judgements, using the things dataset @xcite . this work is directly analogous to evaluating pairwise similarities of language model representations against semantic similarity judgements, and as such, doesn't separate out individual concept attributes. moreover, it assesses representations corresponding to instances (single images), rather than concepts (collections of instances). @xcite compare sparse representations of human and model similarities, finding that while core dimensions overlap, humans use more semantic cues, and vision models rely more on visual cues, as well as many humanuninterpretable cues. in a study of several vision encoders, @xcite find that mod-els trained on larger datasets and language supervision (clip) are more aligned with human similarity than smaller label-and self-supervised models. finally, @xcite show that image encoders trained to predict object attributes, rather than object classes, are more aligned with humans. ## concept attributes: datasets understanding concepts via a core set of distinctive attributes is a long-standing quest in cognitive science @xcite @xcite . one method of discovering which attributes are important for human categorisation is semantic norm elicitation: participants are asked to write down the "characteristics and attributes" @xcite or "properties" @xcite they associate with a particular concept. pooled over many participants, semantic norms thus represent a concept as a set of frequently mentioned salient attributes. while commonly used, semantic norm data have two important weaknesses. firstly, they are not complete: less-salient, but nonetheless present, attributes of concepts are often missing (e.g. tiger but not cat has teeth). to remedy this first issue, we synthetically "complete" the attribute values from @xcite across a large set of concepts. secondly, norms are biased towards attributes that are easily lexicalised. we thus also explore a recent dataset of ratings across a fixed set of attributes related to sensory and neurological dimensions that are not based on elicited lexicalised norms @xcite . since we are exploring visual and linguistic representations, the concepts we consider are concrete objects, corresponding to english nouns. we use the set of object concepts from things @xcite , which also includes a set of qualitycontrolled images for each concept. ## models we primarily study the performance of image encoder models using vision transformers (vit) backbones @xcite , trained with different amounts of linguistic supervision. ## methodology we use trained linear probes @xcite @xcite to measure the extent to which conceptual attributes (mcrae feature norms or binder attribute ratings) are evident in image and text representations. this evaluation requires generalizing attributes to unseen concepts, based on a small set of positive examples. following standard methodology, the linear probes are trained on top of frozen representations, which allows us to estimate what is captured in the representations directly. each attribute is learned with a separate probe. for mcrae×things, we train a linear classifier for each attribute that maps a concept representation to a binary label, using a simple logistic regression. 4 for the binder ratings dataset, we train a linear regression on each attribute to predict the mean rating for each concept-attribute pair. 5 for both datasets, we generate 10 train-test splits for each attribute using 5-fold stratified cross-validation repeated twice, and report the average performance. ## conclusion this linear probing analysis on two datasets shows that multimodally-trained vision encoders represent conceptual attributes better than singlemodality vision-only or text-only encoders. however, the single-modality encoders still perform well. in particular, the self-supervised swin-v2, and to a lesser extent dinov2 models, have learned a large amount of conceptual attribute knowledge, comparable to modern llms, and more than static word embeddings. this result is particularly surprising given that these vision models have encyclopaedic (48) function (55) smell (1) sound (6) tactile (12) taste (3) taxonomic (29) visual colour (17) visual form & surface (92) visual motion (14) 0 20 40 60 f1 selectivity vit-mae max vit (in-21k) dinov2 swin-v2 clip (image) paligemma siglip glove 840b fasttext clip (text) deberta v3 gemma figure 5: results (f 1 selectivity) per attribute (norm) type on the mcrae×things data. the number below each type indicates the number of norms belonging to that type. the error bars denote 95% confidence intervals using bootstrapping. vision models are in reddish colours, while language models are in greenish colours. there is a long-held belief that we need multimodally-grounded representations to overcome the limitations of learning from only linguistic data. our results suggest that vision and language encoders encode (somewhat) complementary views of concepts inasmuch samemodality models correlate stronger than differentmodality models. however, overall correlations are high, indicating a level of convergence. previous claims of modality convergence have used nearestneighbours measures @xcite ; here we show similar convergence results using a very different linear probing methodology. we expect models with conceptual knowledge organised in human-like ways, that are aware of the semantic attributes that underlie category memberships, would, in turn, achieve better downstream performance in language processing tasks. in future work, we will investigate the predictive power and utility of our probing tasks for multimodal training. this will also require going beyond simple object concepts to investigate more abstract, situational and configurational, concepts, in order to cover a larger proportion of the human conceptual repertoire.