From Speed to Car and Back: An Exploratory Study about Associations between Abstract Nouns and Images (2022)

Abstract: Abstract concepts, notwithstanding their lack of physical referents in real world, are grounded in sensorimotor experience. In fact, images depicting concrete entities may be associated to abstract concepts, both via direct and indirect grounding processes. However, what are the links connecting the concrete concepts represented by images and abstract ones is still unclear. To investigate these links, we conducted a preliminary study collecting word association data and image-abstract word pair ratings, to identify whether the associations between visual and verbal systems rely on the same conceptual mappings. The goal of this research is to understand to what extent linguistic associations could be confirmed with visual stimuli, in order to have a starting point for multimodal analysis of abstract and concrete concepts.

## introduction in the last years, the debate over abstract and concrete conceptual representations has gained more attention from a cognitive, psycholinguisitc and, recently, from a computational point of view too. explaining the nature of abstract and concrete concepts is very challenging, and a generally agreed definition is still lacking. we can refer to them as a internal mental representations @xcite , or as units of information and relationships @xcite , or as unit of knowledge for specific categories @xcite . recently, differences between concrete and abstract concepts have been studied in relation to concreteness ratings @xcite @xcite , underling that a dichotomic distinction does not take into account existing relationships among them. in the context of grounded theories of cognition, the general assumption is that our conceptual representations are strictly linked to our sensorimotor experience. this assumption seems to be fairly explanatory for concrete concepts (e.g., dog, church, car), but when it comes to analyse abstract concepts (e.g., freedom or knowledge), we have prima face no references in the real and physical world that could activate any kind of sensorimotor experience. it is in the process called multimodal simulation @xcite that situated conceptualizations for a given concept arise. one of the open question in the debate, is how a concept could exploit grounding if no direct sensorimotor experience is available. many studies have been conducted to explore the issue @xcite @xcite . @xcite , for example, assume that through visual experience our cognitive system infers meaning from the linguistic experience, transferring it in a perceptual experience thanks to language-to-vision relations. this means that an abstract concept such as knowledge could be linguistically associated to a concrete referent book or university, or the concept idea could be metaphorically conceptualized as a lightbulb. this process is called indirect grounding. in fact, in the symbol interdependency hypothesis @xcite highlights that language comprehension is symbolic and mediated by interdependencies of amodal linguistic symbols and it is indirectly embodied through linguistic symbols to perceptual representations. in other words, if concrete concepts exploit direct grounding in perceptual representations, abstract concepts anchor their meaning to different referents mediated both by linguistic associations and figurative, metaphorical and analogical associations. the aim of this study is to investigate what are the mechanisms involved in the indirect grounding of abstract concepts, both in their visual and linguistic anchoring. furthermore, we are also interested in exploring the figurative interpretation mechanism of concrete images. in particular, our research questions are: q1 what kind of linguistic associations emerge between abstract and concrete concepts, accordingly also to the degree of concreteness? q2 are these associations grounded in sensorimotor experience? q3 these associations are confirmed also when the stimulus proposed is an image picturing the concrete concept? q4 if these associations could be confirmed also in image-abstract noun pairs associations, could we learn something about visual features that contribute to the indirect grounding processes? in the exploratory study we propose, we conducted five psycholinguistic tests via crowdsourcing -the first three with linguistic data, and the others with multimodal data -to investigate the link between abstract and concrete concepts and images. we selected 130 abstract nouns that we used as stimuli in our tests. in the linguistics tests, we collected word associations with three different elicitation methods. subjects have been instructed to provide respectively only concrete nouns in response to abstract stimuli (test 1), both concrete and abstract nouns in response to abstract stimuli (test 2), the mental mental images in response to abstract stimuli (test 3). results of linguistic tests show an interesting response pattern in the three tests, with respect to the concreteness degree of the answers and the elicitation methods. in the multimodal association tests, we presented image and abstract nouns pairs, and we asked subjects to rate the strongest image-nouns association (test 4). finally, we replaced the images with the concrete nouns they represent, and we asked to rate the strongest concrete-abstract nouns association (test 5). in many cases, results confirm the associations of the word association tasks, but other interesting issues emerge, since the contextual features of images impact on the multimodal associations. ## related work a large body of studies have been conducted to discuss the relation that interconnects language, nonlinguistic meaning, and perception @xcite . abstract concepts have been always at the centre of debate. due to the high degree of interdependency between concreteness and abstractness, scholars debated about the best method to classify these two semantic macrocategories @xcite . the aim of this research line is to define how meaning arises, considering the large variety of internal and external stimuli that humans use to create conceptual associations to understand the world. among the most promising approaches to address the problem of abstract vs. concrete concepts is to consider them as forming a conceptual continuum, rather than a dichotomic relation. in the experiment conducted by @xcite , in fact, abstractness and concreteness are evaluated like a scale, providing evidences about the perception of different degrees of concreteness of 40,000 english words. these data confirm, to some extent, the idea that concrete and abstract concepts rely on different information @xcite @xcite . however, despite the fact that abstract and concrete concepts form distinct cognitive domains, when it comes to explore the interconnection among different dimensions and modalities, it is unclear what features emerge and contribute to meaning formation and comprehension. abstract concepts, in fact, are characterised by an intrinsic complexity related to events, situations, physical and mental states, and they are much more variable in their realization of intra and extra-linguistic meaning @xcite . moreover, abstract concepts are also directly connected to metaphorical thinking, events, and affective states @xcite . lexico-semantic theories underlines the contribution that language brings to the meaning formation, with the respect to the context (louwerse, 2011). for example, according to the context availability theory @xcite , concrete concepts are associated to more specific contexts, compared to the ones of abstract concepts. the dual coding theory @xcite , instead, assumes that all concepts are rooted in the verbal system, while only concrete concepts have a direct connection with images. multimodal approaches gained more and more interest both in linguistics, cognitive science, neuroscience and computational studies, and new semantic models combining linguistic and visual information have been proposed. computational models of semantics make use of linguistic and perceptual information, to obtain complementary data to explore our conceptual system @xcite . there have been several studies aiming to address the multimodal mechanisms of abstract/concrete grounding @xcite . recently @xcite proposed a multi-modal context-based approach to learn word embeddings, observing that visual surroundings of objects are informative and could be exploited to build word representations, jointly with the visual appearance of the object themselves. interesting attempt to model abstractness and concreteness with the respect to figurative language and multimodality can be found in @xcite . while abstractness has mostly been modeled at word-level without paying attention to contextualization, @xcite explore the dynamics of meaning interchange between texts and images for visual metaphors, by using different degrees of concreteness. ## word association data selection we collected 130 english nouns and we divided them as low abstract, medium abstract and high abstract (1). abstract noun level concreteness rating belief high 1,19 democracy high 1,78 love medium 2,07 anxiety medium 2,21 sight low 3,21 speed low 3,62 since our aim was to explore the interconnection between abstract nouns and images, we performed preliminary investigation to control the availability of images related to our stimuli. we used unsplash.com as a reference website to collect images, firstly because we want to make sure to use royalty free pictures, and secondly because this portal offers a large variety of user generated content labelled with tags chosen directly by users. then we use the concreteness rating data contained in @xcite to perform a more precise division based on the concreteness ratings. we obtained 47 low abstract nouns, 42 medium abstract nouns, 38 high abstract nouns. ## image abstract associations data collection the next step of the work was dedicated to the multimodal association collection. we identified the most frequent concrete nouns appearing in c, nd and imagery and we selected 44 concrete nouns appearing at least in 2 tests. each of these concrete nouns was paired with 5 abstract nouns, selected among the most frequent abstract nouns associated to each of 44 concrete nouns. then, we used unsplash.com to collect 4 images for each concrete noun. image-abstract nouns rating task we administered to 30 native english speakers a rating test (test 4 "img") in which participants were asked to rate the strongest image -abstract nouns association on a scale from 1 to 5, choosing from a set of 5 abstract nouns. for each image-stimulus we presented 4 different pictures. participants rated 180 image-noun pairs in total. only the images were shown and not the concrete noun they represent, as we wanted to avoid that the human judgments were biased by the linguistic clues. a second test (test 5 "wrd") was designed in which participants were asked to rate the strongest concrete -abstract nouns association a scale from 1 to 5, choosing from a set of 5 abstract nouns. in this test, the images of test 4 were replaced by the corresponding concrete noun. analysis we computed the rating means for the image-noun pairs and for the concrete-abstract noun pairs and the standard deviation for the images in test 4. the image -abstract nouns and the word abstract nouns ratings show a very high correlation (spearman ⇢ = 0.77). we then computed the correlation between the mean ratings obtained by the test 4 (e.g., mean rating given to speed in relation to the 4 images selected for car) and the mean frequencies of the abstract-concrete noun associations (e.g., the value of frequency for the association speed -car in test 1, 2 and 3) (see the same correlation was calculated between the mean frequencies of the abstract-concrete noun associations and the rating means obtained in the test 5 (e.g., mean rating given to speed in relation to the linguistic stimulus of car). in this case, the correlation of ⇢ = 0.55 shows that associations like speed-car and back (car -speed) are quite solid in both directions. the results reveal an higher correlation for the linguistic associations, but still a good correlation within the image-noun pairs. in order to evaluate if the prototypical associations highlighted by word associations testing phase were confirmed also when images were proposed, we calculated the correlation between the rating means in the test 4 and the highest frequency value of abstract-concrete association test (e.g., the association knowledge-book obtained a maximum frequency value of 11 in word associations tests, see ## discussion in our exploratory study we were interested in understanding q1 what kind of linguistic associations emerge between abstract and concrete, accordingly also to the degree of concreteness. q2 whether these associations are grounded in sensorimotor experience. to answer this first two questions, we conducted three word associations tasks, by adopting three different elicitation methods. a1 we observed that interesting prototypical answers were provided by subjects, confirming tendencies in association paths. these prototypical answers were mainly derived from processes grounded in sensorimotor experiences (anxiety - • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • • • 0 2 4 6 8 10 12 1 2 3 4 5 freq. mean wrd rating mean sweat), metaphorical and conventionalized association (idea -lightbulb), and also by cultural references (freedom -america). beside the prototypicality of answers we also observed that the distribution of abstract and concrete linguistic associations vary based on the concreteness degree and the elicitation methods. in fact, we can observe a variability in the typology of the responses: if the elicitation method does not specify to provide concrete nouns/association or mental representations to the abstract stimulus, subjects tend to produce an abstract-to-abstract association, especially providing associations of similar linguistic distribution (e.g., synonyms). on the other hand, elicitation methods that clearly specify to produce concrete association or mental images associations provide concrete and grounded answers. interestingly, we observed also that based on the degree of concreteness of abstract stimuli, a low abstractness is associated more with concrete concepts, and high abstractness is associated more with abstract concepts. a2 since we observed that in test c and imagery more concrete associations were provided, we found a great variety of associations referable to different situational context and sensorimotor experiences. (e.g., apology -flower; flowwater; attention -eyes; consideration -nurse). once we obtained these data, we wanted to understand whether q3 these associations are confirmed also when the stimulus proposed is an image picturing the concrete concept; q4 if these associations could be confirmed also in image-abstract noun pairs associations, could we learn something about visual features that contribute to the indirect grounding processes? a3 in the image-abstract association tasks we explored a reverse schema of associations, and we found out that the prototypicality in general is confirmed also when the stimulus is an image. notwithstanding the predominant prototypicality also with the visual stimuli, some exceptions arise. there are cases in which the strongest linguistic association is not confirmed in the preference of image-abstract nouns (e.g., apology is strongly associated to flower in the linguistic tasks, and strongly associated to beauty in visual tasks). this could be explained by the degree of prototypicality of the associations: the mean association of apology and flower (3.68) is weaker than the mean association of book and learning (7). a4 since we showed 4 images for the same concrete concept, we observed that the semantic of the visual scene have an effect on the preference of the abstract concept associated. these results show indeed a strong connection between visual context and the scene interpretation. grounding of the anchoring mechanisms has to be found especially in the situation depicted. if the visual context change (e.g.: from a picture in which a car is moving to a parked car) different conceptual features are taken into account to define the strongest association between the abstract noun and the visual stimulus. in fact, the correlation between frequency and rating means in the linguistic associations is higher than the correlation of image-abstract associations. this result could be justified by the variety of the visual stimuli proposed. moreover, the highest correlation values among linguistic and visual norming can be found in imagery test, demonstrating also that contextual clues may help in finding better associations between abstract and concrete concepts, as subjects are asked to imagine situations/events to connect two conceptual domains. ## conclusions the study we proposed aimed to explore the role of linguistic associations in the mechanisms that link them to images and abstract concepts. we were particularly interested in studying the indirect grounding processes that connect an abstract concept such as speed to a concrete such as car, and in the same way how car could lead to the idea of speed. in order to gain data about these indirect connections of indirect grounding, we collected norming data of word-to-word associations and image-to-word associations. in the word association tasks, we exploited different elicitation methods to first understand what kind of associations emerge if an abstract noun is provided as a stimulus, considering also the degree of concreteness. our analysis reveals that the degree of concreteness of the abstract stimulus impacts on the distribution of abstract vs. concrete concepts in the three different elicitation methods. furthermore, the elicitation method also impacts on the kind of the produced associations (abstract/concrete). to some extent this confirms that conceptual systems do not rely only on linguistic information, but context plays an important role in defining the link between concrete vs. abstract concepts (e.g., in the imagery test, where subjects were asked to "imagine" the abstract stimuli, mostly concrete association arise). with regards to the image-abstract tasks, we observed that when an image is provided, the grounded anchoring offered by the visual scene confirms similar linguistic associations with abstract concepts. in general, our results show that linguistic associations correlate with image-abstract noun ratings. in this view, concepts such as speed leads both via linguistic associations and visual associations to the concept car. however in the case of image-abstract noun pairs we observed differences of preference in image-abstract pairs, confirming that even the visual content has an important role in the construction of multimodal meaning and the indirect, figurative grounding of images and abstract concepts. for example, despite the association carspeed is confirmed in verbal and visual data, the association carasset arises in one of the 4 images proposed to subjects. in this case the car was parked and not moving on the road. since this work has been conducted as an exploratory study, we are interested in analysing more in depth the semantic associations that arise from both perspectives. in fact, we qualitatively observed that situational semantic relations are the most prominent in the prototypical associations @xcite , but the degree of concreteness. furthermore, other information may have an impact on the abstract-concrete association both in visual and verbal systems. in the future we would like to explore the differences arising from different languages to detect cultural and linguistic influences in the grounding processes. if it is true that convenzionalizations in linguistic associations may occur in cases such as idealightbulb across several languages, it is not clear whether association such as curiosity -cat could be confirmed in other languages, or if the connection that brings together beliefchurch make use of cultural or linguistic influences, both in word associations and in image-abstract association. despite the explorative nature of our study, we think that our results could bring new insights about the many modes in which verbal and visual system continuously interact. more studies are needed to investigate this interconnection, due to the large variety of information that our conceptual systems uses to build meaning. this is particularly important also for computational approaches aiming in exploiting more features of multimodal data.