Evaluating Semantic Relations in Predicting Textual Labels for Images of Abstract and Concrete Concepts (2024)

Abstract: This study investigates the performance of SigLIP, a state-of-the-art Vision-Language Model (VLM), in predicting labels for images depicting 1,278 concepts. Our analysis across 300 images per concept shows that the model frequently predicts the exact user-tagged labels, but similarly, it often predicts labels that are semantically related to the exact labels in various ways: synonyms, hypernyms, co-hyponyms, and associated words, particularly for abstract concepts. We then zoom into the diversity of the user tags of images and word associations for abstract versus concrete concepts. Surprisingly, not only abstract but also concrete concepts exhibit significant variability, thus challenging the traditional view that representations of concrete concepts are less diverse.

## introduction concrete concepts, such as apple and dog, are easily perceivable through our senses, whereas abstract concepts such as happiness and justice lack physical referents and are not directly linked to our sensory experiences @xcite . these differences have played a crucial role in various applications involving both textual and multi-modal inputs @xcite @xcite @xcite @xcite . recent advances in multi-modal learning with vision-language models (vlms) like clip @xcite , align @xcite , and siglip @xcite , have improved the alignment of textual and visual data to generate context-aware representations. however, the ability of vlms to capture semantic relationships between concepts and their visual representations remains underexplored. for example, the concept idea is semantically related to the synonym thought and the hyponym belief, and associated with invention. this example highlights the range of possible labels for the visual representation of a concept and the importance of including not only human-assigned labels but also their semantically related counterparts to enhance applications like image retrieval and visual question answering. inspired by this, we evaluate how well siglip, a stateof-the-art vl model, predicts image labels that are generated by users, as well as their synonyms, hypernyms, co-hyponyms, and associative words. assessing the impact on model performance, we aim to determine if integrating these relations into vlm training can potentially improve the representation of abstract and concrete concepts. people use a variety of cues, including visual and linguistic, to perceive and understand concepts @xcite . traditionally, concrete concepts, which are directly related to sensory experiences, are considered less diverse in their visual representations compared to abstract concepts @xcite , and are expected to have more consistent tags and associations. in contrast, abstract concepts, being inherently diverse, are expected to have varied word associations and user tags, reflecting the complexity of understanding these concepts across modalities. to evaluate these differences, we pose the following questions: rq1: how do different semantic relations of user tags affect the prediction of image labels for abstract and concrete concepts? rq2 how do user tags given a visual cue (image), and word associations given a linguistic cue, differ in characterizing abstract versus concrete concepts? our findings show that siglip often predicts semantically related labels (such as hypernyms) instead of the original user tags. our analysis of association data and user tags reveals that concrete concepts, like abstract ones, invoke a diverse range of descriptors, challenging the traditional view of less diversity. 2 experimental design 2.1 materials target concepts & concreteness norms: we selected concrete and abstract nouns using the concreteness ratings from @xcite . the ratings range from 1 (abstract) to 5 (concrete) and were collected via crowdsourcing. we utilized the filtered dataset of 5, 438 nouns from schulte im @xcite to reduce ambiguity through frequency thresholds and pos tagging. to better understand the differences between the two extremes of the concreteness spectrum, we focused on the most concrete and most abstract nouns. at the same time, we wanted to ensure a sufficient number of nouns from both extremes with at least 300 images available for each concept. however, acquiring a sufficient number of images (300) was challenging for many abstract nouns. therefore, we selected concrete nouns rated from 4.5 -5, and used a broader range for abstract nouns from 1 -2.5. from these, we excluded all nouns which occur in the 1, 000 classes of the ilsvrc-2012 imagenet dataset @xcite , since many vision models are trained or evaluated on these classes. we also filtered out nouns that could lead to images depicting explicit content, as well as the nouns camera, picture and photo which were very common user tags because of the nature of the dataset. image dataset and user tags: we used images from the yfcc100m multimedia commons dataset (yfcc; @xcite ), the largest publicly available user-tagged dataset containing ≈ 100 million media objects from flickr. each image has tags provided by users (user tags) when uploading the image. for example, figure 1 is an image with the corresponding possible user tags: wa-terfall, nature, vacation, rock, forest, hiking, peace. we only retained user tags that consisted solely of english characters. we randomly selected 300 images where the target concept appeared among the user tags and consider them as relevant images of that concept. this resulted in 1, 278 nouns (371 abstract and 907 concrete) with 300 images each. ## rq1 -impact of semantic relations on model predictions we first analyze the number of user tags associated with each image and the model's performance in predicting these tags as labels. then, we evaluate whether the model also predicts synonyms, hypernyms, and co-hyponyms of user tags as possible labels. we hypothesize that synonyms will provide alternative labels that capture variations in naming, potentially improving prediction accuracy for both abstract and concrete concepts. hypernyms are expected to offer more general category labels. co-hyponyms will highlight sibling relationships between concepts, improving label prediction by capturing related yet distinct categories. ## rq2 -relationship between association norms and user tags we analyzed the overlap between user tags for images and word associations. we expected abstract concepts to show higher diversity in both visual associations (user tags of images) and word associations due to their inherently diverse nature, while we expected concrete concepts to show a larger overlap between user tags and word associations. we analyzed 682 concepts (527 concrete and 155 abstract) for which we had 300 images and 100 annotators. from de deyne et al. (2019), we selected association words with a frequency @xmath0. for each image where the target concept was one of the user tags, we evaluated how well siglip predicts the associated words of the target concept as possible labels. these findings point towards the need to further explore whether and how current models trained on user-generated tags might fail to capture the full range of human conceptual associations. models might benefit from integrating these broader associative data to enhance their understanding and representation of concepts. ## conclusion our study highlights the potential of integrating diverse semantic relationships in improving the representations in vision-language models (vlms), particularly siglip, for abstract and concrete concepts. the results demonstrate that for images associated with both abstract and concrete concepts, siglip often predicts semantically related words such as synonyms, hypernyms, and co-hyponyms of a user tag even when the user tag itself is not predicted as a label. furthermore, the distinction between visual and linguistic associations revealed differences in how these concepts are perceived and described. our findings suggest that leveraging semantic relationships and associations should be further explored to enhance representations of abstract and concrete concepts in vlms, aligning them more closely with human cognitive variation.