Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024)

Abstract: The visual representation of a concept varies significantly depending on its meaning and the context where it occurs; this poses multiple challenges both for vision and multimodal models. Our study focuses on concreteness, a well-researched lexical-semantic variable, using it as a case study to examine the variability in visual representations. We rely on images associated with approximately 1,000 abstract and concrete concepts extracted from two different datasets: Bing and YFCC. Our goals are: (i) evaluate whether visual diversity in the depiction of concepts can reliably distinguish between concrete and abstract concepts; (ii) analyze the variability of visual features across multiple images of the same concept through a nearest neighbor analysis; and (iii) identify challenging factors contributing to this variability by categorizing and annotating images. Our findings indicate that for classifying images of abstract versus concrete concepts, a combination of basic visual features such as color and texture is more effective than features extracted by more complex models like Vision Transformer (ViT). However, ViTs show better performances in the nearest neighbor analysis, emphasizing the need for a careful selection of visual features when analyzing conceptual variables through modalities other than text.

## introduction language and vision play a crucial role for the understanding of the world surrounding us. among the five senses, vision is considered the primary source of perceptual information for our mental representations when experiencing the real world @xcite . based on these premises, computational studies have leveraged the strong interaction between visual and textual information to uncover the latent relationships between these two modalities and to build richer and more precise representations. in most cases, the contribution of these two very different modalities is asymmetric, with the textual modality having a stronger influence on model performance; for example, when investigating the concreteness of a concept, its compositionality, or its semantic representation @xcite @xcite . the exact reasons behind such asymmetry are still unclear, and especially the role of the visual elements has been explored significantly less. therefore, this paper focuses explicitly on the nature and contribution of the visual component. to this end, we analyze the different characteristics of concrete and abstract concepts to determine whether and how visual information can help distinguish between them. our analysis is particularly important when addressing the complex task of modeling abstract concepts, which often lack a distinctive visual component, unlike their concrete counterpart. for example, concrete concepts like banana and chariot evoke vivid mental images anchored to objects that are easy to visualize. in contrast, abstract concepts like accountability and allegiance are more challenging and subjective to visualize @xcite . various studies have successfully attempted to predict the concreteness score of a concept by exploiting the visual information extracted from multiple images associated with it in combination with more traditional textual representations @xcite @xcite . a building assumption of these visual models is a certain degree of visual coherence that facilitates the construction of stable visual representations. while images of concrete concepts are generally expected to show greater consistency, a notable variability is still present in both concrete and abstract concepts, i.e., the properties of these images, including color, shape, size, and other visual details, may vary significantly, thus reflecting the diversity of the intrinsic nature of the concept. figure 1 illustrates the variability in the images associated with abstract and concrete concepts. images can be highly representative of a concept and be visually similar (e.g., affordability, waterfall) or be rather different from one another (e.g., chariot). conversely, images of a concept can be very similar but not informative representations of the concept (e.g., allegiance). finally, they can be highly different yet individually all strongly associated with the same target concept (e.g., banana, accountability). these degrees of variation highlight some of the inherent challenges computational methods face in constructing a comprehensive visual representation of a concept and mapping it to its labels. these challenges are orthogonal to previously raised issues regarding depictions of (mostly concrete) semantic concepts such as variability of prototypicality @xcite @xcite , and will be explored further in the course of this study (see rq3 below). our research targets the challenges of precisely quantifying the contribution of visual information in describing concrete versus abstract concepts, using interpretable representations to explore the following three research questions: rq1: can visual diversity differentiate between concrete and abstract concepts? rq2: how consistent are visual attributes across multiple images of the same concept? rq3: what are inherent yet plausible failure categories for unimodal visual representations? in study 1, we address rq1 by classifying approximately 500 concrete and 500 abstract concepts based on the diversity in visual features extracted from images associated to each concept. this approach helps us identify the most salient visual features that distinguish between concepts based on their concreteness. in study 2, we address rq2 and analyse the consistency of these features across multiple concept images, by performing a nearest-neighbor analysis of image representations. finally, study 3 targets rq3 by qualitatively analyzing the failures in study 2 and manually determining categories of problematic issues. to our knowledge, this is the first large-scale study conducting a detailed quantitative and qualitative investigation into how visual features contribute to representing abstract and concrete concepts. by focusing exclusively on the visual component, we can systematically identify the strengths and weaknesses of using such extremely rich source of information. additionally, compared to previous studies, our methodology highlights cases that are particularly challenging because they are equally plausible rather than erroneous. ## related work the distinction between abstract and concrete words is highly relevant for natural language processing and has been exploited for metaphor @xcite @xcite @xcite , lexicography @xcite , and embodied agents and robots @xcite @xcite , among others. most studies addressing this distinction have primarily focused on the textual modality alone @xcite @xcite @xcite @xcite . first extensions to further modalities explored free associations and imageability @xcite @xcite . since the primary distinction between degrees of abstractness is influenced by the strength of sense perception, with vision being considered the main source of perceptual information @xcite , later studies began to explore bimodal approaches that combine text and images @xcite . compared to text, the visual component has provided less definitive insights, and it is unclear whether this is due to architectural choices or to the inherent challenge triggered by depicting abstract concepts. @xcite analyzed the mechanism behind this indirect grounding of abstract concepts by collecting word association data and pairs of images and abstract words. @xcite discuss the visual variety of a dataset using mean shift clustering, where the dataset is designed to contain images in the same ratio of sub-concepts as in real life. @xcite performed a regression study to predict the imageability of concepts using the yfcc100m dataset; our feature selection builds on their results. @xcite and @xcite postulated that concreteness in images varies across datasets and is not directly connected to the underlying linguistic concept. @xcite evaluated the alignment of semantic representations learned by multimodal transformers with human semantic intuitions, finding that multimodal representations have advantages with concrete word pairs but not with abstract ones. @xcite argue that there are multiple notions of "image similarity" and that models should adapt dynamically. for example, models trained on im-agenet tend to prioritize object categories, while a user might want the model to focus on colors, textures, or specific elements in the scene. they introduce the genecis benchmark, assessing models' adaptability to various similarity conditions in a zero-shot evaluation setting. they observe that even robust clip models struggle to perform well, and performance is only loosely connected to ima-genet accuracy. most recently, @xcite examined to which degree siglip, a state-of-the-art vision-language model (vlm), predicts labels for images of abstract and concrete concepts that are semantically related to the original labels in various ways: synonyms, hypernyms, co-hyponyms, and associated words. the results show that not only abstract but also concrete concepts exhibited significant variability in semantically appropriate label variants. ## experimental design in the following sections, we present the resources used in our analyses. we introduce the target concepts under investigation, their abstractness scores, and the associated images. subsequently, we describe the algorithms employed to extract the visual attributes from the images. ## study 1: classifying concepts using visual information this first study aims to identify the visual features that are most useful for discriminating between images of concrete vs. abstract nouns. we utilize three different classifiers: support vector machine (svm) with rbf kernel, random forest (rf), and logistic regression (lr) with hyper-parameter tuning, while using the eigenvalues of the combined visual features described above as predictors. 2 in the main text, we report the performance of the rf model as, overall, it outperforms the other two classifiers (the results for lr and svm are reported in figures 5 and 6 in the appendix). we evaluate the predictive power of our features independently and by concatenating them. to account for data skewness between classes, we apply 5-fold cross-validation. ## study 2: inspecting visual nearest neighbors in our second study, we directly build on the evidence from study 1 and perform a nearest neighbors analysis to inspect the consistency of visual attributes across multiple images of the same concept. we compute the cosine similarity of each image of a concept with all other images of all concepts in the same dataset, represented by using the same features as before. we then inspect the top n (where @xmath0, . . . , 500]) most similar images and compute the percentage of neighbors associated with the same concept; e.g., how many nearest neighbor images of an image of banana are also images of banana. ## study 3: exploring factors behind visual diversity as discussed in section 5 when analyzing nearest neighbors, the biggest challenge in using images of a concept comes from the diversity of the images associated with it. the same concept, whether abstract or concrete, can be depicted in many different yet plausible ways, thus relating to previously discussed issues regarding the variability of prototypical attributes in depictions of the semantic concepts @xcite @xcite . in our final analysis, we provide a manual classification of the critical factors influencing the nearest neighbors of our target concepts. we identified five primary reasons for visual diversity, as exemplified in figure 4 . for concepts like accuracy, generation, and cone, the words used as a proxy to our concepts may be lexically ambiguous and have multiple senses. according to wordnet @xcite , 650 out of the 918 concepts used in our studies have more than one sense, and 248 concepts have four or more senses. a further source of variability is physical context, manifesting itself as different background information, objects, etc. in our example, both images depict bananas, but they differ visually: in the bottom image, the bananas are still hanging on a banana tree, which dominates the scene. another form of visual diversity is triggered by subjective representations: concepts like equality and paper show very high variability. people have different visual interpretations and realizations of these concepts, even when the underlying conceptual meaning is understood in the same way. popular culture often associated with films and books represents a kind of variability that introduces visual representations often completely disjoint from the original meaning of the concept: for example, images of the concepts inception and office contain images which are from the movie "inception" and the tv show "the office", respectively. finally, primarily abstract concepts like intention and idealist lack distinctive visual representations and are hard to depict. these concepts are often represented by writing the associated word into an image. another orthogonal source of variability in visual representations comes from the selection process in the source dataset. for example, the yfcc dataset contains images from flickr that are uploaded by users, resulting in a lot of variability and bias toward specific senses of a tagged concept. ## conclusion we performed three empirical studies to understand how abstract and concrete concepts are depicted in images. compared to existing studies, we focused exclusively on the role of variability in the visual information. after automatically generating nine different feature representations for the images, we tested their reliability in a classification study to distinguish between concrete and abstract concepts. we showed that, overall, combining low-level features produces good results. we then investigated the consistency of the visual attributes across multiple images of the same concept by looking at the nearest neighbors of each image in the two datasets. the results across feature types, datasets, and concreteness scores were very low; overall, abstract concepts showed considerably higher cases where none of the most similar images were associated with the same concept. the results also showed that both concrete and abstract concepts lack a univocal visual representation in terms of objects depicted and, in general, basic visual properties. finally, in an error analysis study with human participants, we highlighted the five most frequent reasons explaining visual diversity among images associated with the same concept. overall, our research significantly advances the understanding of the role of the visual component in tasks that heavily rely on the integration of multiple types of information beyond just text.