What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models (2024)

Abstract: Humans have clear cross-modal preferences when matching certain novel words to visual shapes. Evidence suggests that these preferences play a prominent role in our linguistic processing, language learning, and the origins of signal-meaning mappings. With the rise of multimodal models in AI, such as vision-and-language (VLM) models, it becomes increasingly important to uncover the kinds of visio-linguistic associations these models encode and whether they align with human representations. Informed by experiments with humans, we probe and compare four VLMs for a well-known human cross-modal preference, the bouba-kiki effect. We do not find conclusive evidence for this effect but suggest that results may depend on features of the models, such as architecture design, model size, and training details. Our findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of VLMs that align well with human cross-modal associations.

## introduction the development of machine understanding and generation of natural language has benefited immensely from the introduction of transformerbased architectures @xcite . these architectures have since then been adapted and extended to handle multimodal data, leading to the creation of various types of multimodal models, including vision-and-language models. these models can potentially revolutionize how ai systems understand the world and interact with humans. however, we lack direct access to the exact representations and associations they encode. how vlms integrate representations in the two modalities and whether associations between modalities are made in a human-like way is still being ac-* equal contribution figure 1 : which of these two shapes is kiki? images from @xcite tively investigated @xcite @xcite . here, we use a well-known paradigm from the field of cognitive science to probe into a specific cross-modal association between speech sounds and visual shapes: the bouba-kiki effect. when humans see two figures, one with jagged and one with smooth edges, and are told one is a kiki and the other a bouba, 95% will name the jagged figure @xcite . this effect was initially discovered and described anecdotally by wolfgang köhler @xcite , using the two images shown in figure 1 with the labels maluma and takete. since then it has been widely studied (as reviewed in section 2), and expanded with many other cross-modal preferences in human processing of (speech) sounds and visual imagery. moreover, a wealth of evidence suggests that such preferences widely influence patterns we see in human languages (e.g., @xcite @xcite @xcite . even though non-arbitrariness in language is often still regarded as an exception in some disciplines, in fields such as language evolution, and sign language linguistics, iconic formmeaning mappings are considered omnipresent @xcite . given the central role crossmodal preferences play in human visio-linguistic representations and their effects on language, it is pertinent to investigate whether vlms associate non-words and visual stimuli in a human-like way. examining universal human cross-modal preferences in vlms can help us gain key insights across disciplines. first, it may reveal whether vlms process multimodal information in a human-like way and whether similar biases drive their understanding of visual-auditory form-meaning mappings. overlap in cognitive biases can potentially increase mutual understanding and improve interactions between humans and machines @xcite . second, it may help pinpoint what is missing to make vlms more suitable for realistic simulations of human language emergence. increasingly, vlms are used in emergent communication settings, where agents communicate with each other and develop a novel language @xcite @xcite . these models are used to improve machine understanding of human language @xcite @xcite , but also to simulate and study human language evolution processes @xcite . while the influence of cross-modal associations on the emergence of language has been studied extensively in language evolution experiments with humans @xcite @xcite , the phenomenon is still absent from current emergent communication paradigms. evidently, cognitively plausible vlms are more suitable for simulating aspects of the evolution of meaning in language. finally, the actual origin of the bouba-kiki effect is still being debated within cognitive science and linguistics, with proposed explanations ranging from attributing it to similarities between shape features and features of either orthography @xcite , acoustics and articulation @xcite @xcite , affective-semantic properties of human and non-human vocal communication @xcite , or physical properties relating to audiovisual regularities in the environment @xcite . if the bouba-kiki effect can be reproduced in a vlm, it can help reveal the crucial ingredients for this effect, potentially leading to models better aligned with human representations. to the best of our knowledge, only one previous paper discussed the bouba-kiki effect in vlms. @xcite tested two models, clip @xcite and stable diffusion @xcite , and reported to find strong evidence for the effect in these models. this is somewhat surprising given the way these models are trained and the absence of relevant data sources such as auditory information and experience with physical object properties. therefore, we introduce nuance in this discussion and show, contrary to this previous finding, that the bouba-kiki effect does not occur consistently in vlms, and the presence of this cross-modal preference may depend on the way it is tested and properties like model architecture, attention mechanism, and training details. ## methods to test for the presence of a bouba-kiki effect in vlms we employ previously used as well as newly generated images ( §3.1) and use a method for constructing pseudowords (3.2) that is directly borrowed from @xcite . probing ( §3.3) was used to obtain image-text scores and responses were analyzed in two ways ( §3.4). ## results the findings are analyzed in two ways. first, we compare the results of vlm probing to the performance of human participants @xcite . for blip2, clip and vilt this means we first only consider the syllable or pseudoword with the highest probability for each image. these are then analyzed similarly to those selected by humans or generated by gpt-4o. second, we examine the probabilities for each possible syllable or pseudoword from blip2, clip and vilt, to obtain a more comprehensive measure of cross-modal associations. for the gpt-4o results reported below, one image in the jagged shape condition is consistently missing since it (top right image in figure 7 in appendix a) was flagged as 'content that is not allowed by our safety system'. ## discussion our findings partly contradict previous work, which found that sound-symbolic associations are present in clip and stable diffusion @xcite . we use a different method, focusing on image-to-text probabilities, which is more similar to how the effect has been tested with humans. we show that it is too early to conclude that vlms understand sound-symbolism or map visio-linguistic representations in a human-like way since the results depend heavily on which specific model is tested and how the task is formulated. this is unsurprising given that cnn-based models often classify based on superficial textural rather than shape features @xcite @xcite ) and, albeit less so, this texture bias is also present in vision transformers @xcite . moreover, @xcite identified that, during inference, vit networks create artifacts at low-informative background areas of images that are used for computations rather than describing visual information. both findings are in stark contrast with what, at its core, is required for sound symbolism. however, the fact that some evidence for a bouba-kiki effect could be found in two of the four models tentatively suggests that real-world physical experience with different object properties may not be needed to develop this cross-modal preference but that it can, to some extent, be learned from statistical regularities in data containing text and images. human language on its own already contains many non-arbitrary regularities between speech sounds and meaning @xcite , and these regularities, like phonesthemes @xcite , can be detected and interpreted by models such as word embeddings @xcite and lstm based language models @xcite . no visual input is needed for this, and perhaps this is also what caused the appearance of the bouba-kiki effect in the work by @xcite . in our work, we gave more prominence to the visual input and found much less convincing evidence for the effect. regarding the design features of the models we tested, we see that the model with the best boubakiki alignment to human preferences, clip, is also trained on the largest amount of data (comparing the three models we have information on, not including gpt-4o). this finding aligns with previous work showing that dataset properties affect alignment with human representations @xcite . however, despite having much more parameters than clip, blip2 does not show the effect. in addition, while both blip2 and clip use dual-stream architectures, only clip, which uses modality-specific attention mechanisms, displays some evidence of a bouba-kiki effect. despite impressive performance on vision-language tasks, the q-former in blip2 apparently does not promote sound-symbolic associations. this is important knowledge for developing models with vision-language representations that align with those of humans. more aligned models show more robust few-shot learning (sucholutsky and griffiths, 2023) and promote more natural interactions between humans and machines @xcite . although we find modest evidence for a bouba-kiki effect in gpt-4o, we cannot know the origin of this effect as model details are unknown. ## conclusion given the pervasive role cross-modal associations play in human linguistic processing, learning and evolution, we tested for the presence of a boubakiki effect in four vlms that differ along various dimensions such as architecture design, training objective, number of parameters, and input data. evidence for this effect is limited, but not entirely absent, in the tested vlms and these findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of vlms that align well with human cross-modal associations.