With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models (2024)

Abstract: Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to “hear” via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, and more task information may be required for VLMs versus their human counterparts for in silico experimentation. We additionally see through higher maximum agreement levels that Magnitude Symbolism is an easier pattern for VLMs to identify than Shape Symbolism, and that an understanding of linguistic iconicity is highly dependent on model size.

## introduction sound symbolism refers to a perceived similarity between speech sounds and the conceptual meanings of the words they comprise. evidence of this can be found in linguistic devices such as onomatopoeia (e.g., "bang", "shriek", and "bellow"), where a word imitates the concept it describes via its phonetic form. the ability of large language models (llms) and vision language models (vlms) to reflect a sense of sound symbolism would therefore suggest that these models * corresponding author figure 1 : illustration of the 3 main experiments we perform. firstly, shape symbolism is a binary choice between two pseudowords to best describe an object that is spiky or rounded. magnitude symbolism involves a binary choice between two pseudowords to best describe an object that is small or large. finally, iconicity involves rating the perceived iconicity of words, or how much their written/phonetic form is representative of what they describe. by extending these experiments to llms/vlms, our study aims to shed light on the processes underlying multimodal perception in language models. 1 moreover, the presence of sound symbolism in such models could inform the development of more effective natural language processing algorithms, aiding tasks such as sentiment analysis, emotion recognition, and content generation that take into account more abstract layers of human reasoning and perception such as abstract connotations between words rather than semantics alone @xcite ). an understanding of sound symbolism also has the potential to have a profound effect on creative generation, including language forms such as poetry and narratives (and their accompanying illustrations). additionally, sound symbolism is a prevalent strategy used in marketing products to create desirable associations in potential customers, and llms capable of understanding this phenomenon could be used as pilot testing before using focus groups to reduce time and monetary costs @xcite @xcite . we summarise our main contributions as follows: foot_1 ## related works the work of early linguists, such as saussure, touched upon the topic of whether or not the link between the "sign" (i.e., a word) and the "signified" (i.e., the entity/concept to which the sign relates) is arbitrary, with there being nothing more "boat"-esque about the word "boat" than any other combination of phonotactically legal sounds (de @xcite . however, there are many types of language where this association is seen to be non-arbitrary such as in the onomatopoeia commonly used in literary works (e.g., "bang" for a loud noise, or "shriek" for a high-pitched wail), where the phonetic realisation mirrors the concept it describes. these phenomena as a whole are known as sound symbolism, where there is thought to be a non-arbitrary link between the sign and the signified, in contrast with the popular stance of early linguistics. outside of onomatopoeia, sound symbolism is believed to have a range of effects on human perception, including applying to nonsense pseudowords. for example, even if a word was not created to explicitly denote a known concept or entity (and therefore has no true denotative meaning), it is nevertheless able to manifest a connotative meaning in the mind of the reader based on its phonological representation and/or phonetic realisation. the first identification of these patterns is frequently attributed to @xcite , who gave 10 participants a series of pseudowords alongside drawings and found a higher-than-chance level of agreement between evaluators for which nonsense word best described which drawing. perhaps the most famous example of this is in the kiki-bouba effect @xcite @xcite which concerns the allocation of the name "kiki" to sharp, hard-edged entities, and "bouba" to more soft and round-edged entities (which in the original works consisted exclusively of 2d shapes). a similar relationship has also been noted between the words "mil" and "mal", where the changing vowel in the phonological min-imal pairs 3 has a relationship to perceived size, in a phenomenon known as magnitude symbolism (where vowels with higher frequency content are associated with smaller entities due to the relationship between vocal tract length and vocal productions) @xcite . extensive research has been performed in the area of sound symbolism, demonstrating interesting findings such as these patterns being largely language agnostic @xcite as well as being weaker in neurodivergent individuals @xcite and not being yet developed in very early childhood @xcite . other research has also investigated the exact requirements and limits of the effects @xcite @xcite . recently, alper and averbuch-elor (2023) have investigated the ability of language models to exhibit the kiki/bouba effect using clip and stable diffusion by generating images from sound-symbolic prompts and provide positive evidence for this association to be present. we build upon this work in §3 by introducing a wider range of vlms in a forced naming task for "real" entities as opposed to abstract shapes, and additionally extend this to magnitude symbolism via the mil-mal test in §4. we further differentiate our work by focussing on the task of assigning pseudowords to provided visual stimuli, in contrast to alper and averbuch-elor (2023) who investigate the effects of different pseudowords on the outputs of image generation models. additionally, in recent times, large-scale efforts have been made to collect ratings of linguistic iconicity (i.e., the level of symbolism a particular word has), with @xcite collecting ratings of over 14k english words. some effort has been made to analyse whether or not similar ratings would be assigned by an llm, where trott (2024) used gpt-4 and reports a moderate positive correlation across ratings. we build upon this work in §5 by introducing a wider range of vlms, including open-source alternatives. numerous computational works in nlp have investigated other aspects of iconicity and sound symbolism, with @xcite investigating the word embeddings of different aspects of morphology in relation to symbolism (see also @xcite . additionally, @xcite investigated the emotional intensity of nonsense words using nlp methods to determine which phoneme combinations were most responsible. 3 a minimal pair refers to a pair of words that differ only in one phonological segment, such as "cat" /kat/ versus "bat" /bat/. several works in similar areas have also exemplified the ability of llms to demonstrate perceptual behaviour akin to humans and the potential for these models to replace human participants in pilot studies, as well as facilitating the scaling of evaluation in-silico @xcite @xcite @xcite . ## shape symbolism in this section, we perform a replication of the classic kiki-bouba effect experiment (ramachandran and hubbard, 2001) using a range of multi-modal llms. within the traditional set-up for the kiki/bouba test, human participants are presented with either a rounded soft-edged shape or a spiky sharp-edged shape and asked to assign one of two pseudowords to either. in numerous experiments @xcite @xcite , words such as "bouba" and "maluma" are preferred for the latter rounded shapes, whilst "kiki" and "takete" are preferred for the spiked shapes. these findings are thought to demonstrate a non-arbitrary link between particular speech sounds and the physical characteristics of the shapes to which they refer. ## magnitude symbolism whilst the kiki-bouba effect demonstrates sound symbolism in relation to the perceived spikiness/roundness of an object, magnitude symbolism refers to the non-arbitrary relationship between certain vowels and the perceived physical size of the entity they refer to and is commonly demonstrated through the names "mil" and "mal", where the high front vowel in "mil" is associated with small entities, and the low back vowel of "mal" is associated with larger entities. ## iconicity ratings in this section we investigate whether llms demonstrate human-like associations between word forms and the entities/concepts they symbolise. 6 @xcite present a dataset of 14k+ human judgements of iconicity to which we compare our llm ratings (where, on a 7-point scale, "how" scored 1.3, whilst "woof" scored 6.8 due to being onomatopoeic). ## discussion the source of sound symbolism across our experiments, we see evidence that llm/vlms are capable of making decisions that are similar to those of humans in sound symbolism tasks, whilst only having access to textual and visual modalities, while human decisions are believed to be grounded in sound. we hypothesise several reasons for the emergence of sound symbolism in llm/vlms. firstly, due to human languages exhibiting mostly regular orthography, auditory information in speech is moderately reflected in the spellings of words via grapheme sequences (a characteristic that grapheme-to-phoneme conversion models have long exploited). through this, text-based models are able to learn associations between grapheme sequences and semantics, based on more abstract characteristics than morpheme combinations alone, such as phonaesthemes @xcite . whilst such models have no embodied understanding of sound, such statistical patterns pose a viable signal for the implicit learning of sound-based phenomena. secondly, such associations between sounds (or grapheme combinations) and physical characteristics are naturally present in language, such as in poetry, narratives, or descriptions of entities that are cute, scary, small, or large, and are consequently paired with relevant visual stimuli in image captions when training vision modules for multimodal systems. however, such associations are subtle and not entirely ubiquitous. for example, whilst the high front vowel /i/ typically associated with small entities is present in "tiny" and "mini", the word "small" itself possesses a low back vowel /o:/. as a result, the relatively weak performance of our tested models could also be explained by the relative lack of sound-symbolism-heavy language in the models' training data which is overshadowed by more prosaic language forms that do not exploit these phenomena as readily. this in turn would explain why the closed-source models we tested (e.g. gpt-4/gemini) outperform open-source models due to the significant (assumed) differences in parameter size, allowing the larger closed models to retain more information regarding sound symbolism within the weights, in addition to being continuously updated with rlhf. the results of our multimodal experiments additionally demonstrate that under certain conditions, vlms show systematic disagreement with human labels, indicating the potential interference of additional knowledge contained within language model training data that influences the associations made between pseudowords and images that are not present in humans. however, it is important to note that in our experiments we compare language model selections against the majority vote or mean scores assigned by humans. consequently, this results in a comparison to an "ideal" human by necessity, overlooking individual differences in perception across humans (where for a decision to be "human-like", it has to match a choice made by any human, rather than the majority). consequently, higher agreement levels can be observed when compared to the choices of individual humans, as inter-human agreement is not perfectly aligned in these tests. future directions as a result of llm/vlms not being able to fully reflect human preferences in tasks regarding sound symbolism, it remains a promising future direction to explicitly pre-train language models on more sound-symbolism heavy datasets or explicitly include sound-symbolism-related tasks into the training or finetuning of these models for use on related downstream tasks (such as creative writing and marketing). additionally, investigating the reason behind model predictions is a promising direction, such as through additional prompting to generate textual justifications, or investigating the visual attention of vlms to investigate whether they are attending to characteristics closely associated with the concepts being tested (e.g., spikes). ## conclusion we have shed light on the processes underlying multimodal perception and understanding in language models. to do so, we performed a series of tests on modern vlm/llms regarding their ability to exhibit an understanding of sound symbolism. through comparison with human judgements, we see that vlms are able to approximate human perception in sound symbolism tests under certain conditions, such as when informed of the nature of the study (via the informed prompts), but struggle overall. we additionally see that magnitude symbolism potentially presents an easier pattern for vlms to recognise than shape symbolism, with selections having a higher agreement with humans on magnitude symbolism tests than shape symbolism. we also see that the ability of llms to emulate human judgements of iconicity scales more linearly with model size. these findings indicate room for future research on more explicit inclusion of abstract perceptual properties into language model training in order to facilitate better in silico experimentation and improve performance on other downstream tasks.