Visually Grounded Speech Models Have a Mutual Exclusivity Bias (2024)

Abstract: When children learn new words, they employ constraints such as the mutual exclusivity (ME) bias: A novel word is mapped to a novel object rather than a familiar one. This bias has been studied computationally, but only in models that use discrete word representations as input, ignoring the high variability of spoken words. We investigate the ME bias in the context of visually grounded speech models that learn from natural images and continuous speech audio. Concretely, we train a model on familiar words and test its ME bias by asking it to select between a novel and a familiar object when queried with a novel word. To simulate prior acoustic and visual knowledge, we experiment with several initialization strategies using pretrained speech and vision networks. Our findings reveal the ME bias across the different initialization approaches, with a stronger bias in models with more prior (in particular, visual) knowledge. Additional tests confirm the robustness of our results, even when different loss functions are considered. Based on detailed analyses to piece out the model’s representation space, we attribute the ME bias to how familiar and novel classes are distinctly separated in the resulting space.

## introduction when children learn new words, they employ a set of basic constraints to make the task easier. one such constraint is the mutual exclusivity (me) bias: when a learner hears a novel word, they map it to an unfamiliar object (whose name they don't know yet), rather than a familiar one. this strategy was first described by @xcite over 30 years ago and has since been studied extensively in the developmental sciences @xcite @xcite . with the rise of neural architectures, recent years saw renewed interest in the me bias, this time from the computational modeling perspective: several studies have examined whether and under which conditions the me bias emerges in machine learning models @xcite @xcite . the models in these studies normally receive input consisting of word and object representations, as the me strategy is used to learn mappings between words and the objects they refer to. object representations vary in their complexity, from symbolic representations of single objects (e.g., @xcite to continuous vectors encoding a natural image (e.g., vong and lake, 2022). word representations, however, are based on their written forms in all these studies. for example, the textual form of the word fish has an invariable representation in the input. this is problematic because children learn words from continuous speech, and there is large variation in how the word fish can be realized depending on the word duration, prosody, the quality of the individual sounds and so on; see, e.g., @xcite on how the me bias affects atypical pronunciations such as @xcite instead of @xcite . as a result, children face an additional challenge compared to models trained on written words. this is why it is crucial to investigate the me bias in a more naturalistic setting, with models trained on word representations that take into account variation between acoustic instances of the same word. recently, there has been considerable headway in the development of visually grounded speech models that learn from images paired with unlabeled speech @xcite @xcite @xcite @xcite . several studies have shown, for instance, that these models learn word-like units when trained on large amounts of paired speech-vision data @xcite @xcite @xcite . moreover, some of these models draw inspiration from the way infants acquire language from spoken words that co-occur with visual cues across different situations in their environments @xcite @xcite . however, the me bias has not been studied in these models. in this work we test whether visually grounded speech models exhibit the me bias. we focus on a recent model by @xcite , as it achieves state-of-the-art performance in a few-shot learning task that resembles the word learning setting considered here. the model's architecture is representative of many of the other recent visually grounded speech models: it takes a spoken word and an image as input, processes these independently, and then relies on a word-to-image attention mechanism to learn a mapping between a spoken word and its visual depiction. we first train the model to discriminate familiar words. we then test its me bias by presenting it with a novel word and two objects, one familiar and one novel. to simulate prior acoustic and visual knowledge that a child might have already acquired before word learning, we additionally explore different initialization strategies for the audio and vision branches of the model. to preview our results, we observe the me bias across all the different initialization schemes of the visually grounded speech model, and the bias is stronger in models with more prior visual knowledge. we also carry out a series of additional tests to ensure that the observed me bias is not merely an artefact, and present analyses to pinpoint the relationship between the model's representation space and the emergence of the me bias. in experiments where we look at different modeling options (visual initialization and loss functions, in particular), the me bias is observed in all cases. the code and the accompanying dataset are available from our project website. foot_0 2 related work visually grounded speech models learn by bringing together representations of paired images and speech while pushing mismatched pairs apart. these models have been used in several downstream tasks, ranging from speech-image retrieval @xcite and keyword spotting @xcite to word @xcite and syllable segmentation @xcite . in terms of design choices, early models used a hinge loss @xcite , while several more advanced losses have been proposed since @xcite @xcite . a common strategy to improve performance is to initialize the vision branch using a supervised vision model, e.g., @xcite @xcite and @xcite used clip. for the speech branch, self-supervised speech models like wav2vec2.0 and hubert have been used for initialization @xcite . other extensions include using vector quantization in intermediate layers @xcite and more advanced multimodal attention mechanisms to connect the branches @xcite @xcite . in this work we specifically use the few-shot model of @xcite that incorporates many of these strategies (section 5). we also look at how different design choices affect our analysis of the me bias, e.g., using different losses (section 7.4). as noted already, previous computational studies of the me bias have exclusively used the written form of words as input @xcite @xcite . visually grounded speech models have the benefit that they can take real speech as input. this better resembles the actual experimental setup with human participants @xcite . concretely, since the models in @xcite and vong and lake (2022) are trained on written words, which are discrete by design, they need to learn a continuous embedding for each of the input classes. however, this makes dealing with novel inputs difficult: if a model never sees a particular item at training time, its embeddings are never updated and remain randomly initialized. as a result, the me test becomes a comparison of learned vs random embeddings instead of novel vs familiar. to address this issue, @xcite use novel examples in their contrastive loss during training, while vong and lake (2022) perform one gradient update on novel classes before testing. these strategies mean that, in both cases, the learner has actually seen the novel classes before testing. such adaptations are necessary in models taking in written input. in contrast, a visually grounded speech model, even when presented with an arbitrary input sequence, can place it in the representation space learned from the familiar classes during training. we investigate whether such a representation space results in the me bias. ## mutual exclusivity in visually mutual exclusivity (me) is a constraint used to learn words. it is grounded in the assumption that an object, once named, cannot have another name. the typical setup of a me experiment @xcite involves two steps and is illustrated in figure 1 . first, the experimenter will ensure that the learner (usually a child) is familiar with a set of specific objects by assessing their ability to correctly identify objects associated with familiar words. in this example, the familiar classes are 'clock', 'elephant', and 'horse', as illustrated in the top panel of the figure. subsequently, at test time the learner is shown a familiar image (e.g., elephant) and a novel image (e.g., guitar) and is asked to determine which of the two corresponds to a novel spoken word, e.g., guitar (middle panel in the figure). if the learner exhibits a me bias, they would select the corresponding novel object, guitar in this case (bottom panel). our primary objective is to investigate the me bias in computational models that operate on the audio and visual modalities. these models, known as visually grounded speech models, draw inspiration from how children learn words @xcite , by being trained on unlabeled spoken utterances paired with corresponding images. the models learn to associate spoken words and visual concepts, and often do so by predicting a similarity score for a given audio utterance and an input image. this score can then be used to select between competing visual objects given a spoken utterance, as required in the me test. bottom: if a learner has a me bias, then when prompted with a novel spoken query, the novel object (guitar) would be selected. in section 4 below we describe how we set up our test of the me bias. in section 5 we then present the visually grounded speech model that we use in this study. ## constructing a speech-image test for mutual exclusivity to construct our me test, we need isolated spoken words that are paired with natural images of objects. we also need to separate these paired word-image instances into two sets: familiar classes and novel classes. a large multimodal dataset of this type does not exist, so we create one by combining several image and speech datasets. for the images, we combine ms coco @xcite and @xcite . ms coco contains 328k images of 91 objects in their natural environment. caltech-101 contains 9k google images spanning 101 classes. ground truth object segmentations are available for both these datasets. during training, we use entire images, but during evaluation, we use segmented objects. this resembles a naturalistic learning scenario in which a learner is familiarized with objects familiar bear, bird, boat, car, cat, clock, cow, dog, elephant, horse, scissors, sheep, umbrella novel ball, barrel, bench, buck, bus, butterfly, cake, camera, canon, chair, cup, fan, fork, guitar, lamp, nautilus, piano, revolver, toilet, trumpet by seeing them in a natural context, but is presented with individual objects (or their pictures) in isolation at test time. for the audio, we combine the faac @xcite , buckeye @xcite , and librispeech @xcite datasets. these english corpora respectively span 183, 40, and 2.5k speakers. to select familiar and novel classes, we do a manual inspection to make sure that object segmentations for particular classes are of a reasonably high quality and that there are enough spoken instances for each class in the segmented speech data (at least 100 spoken examples per class). as an example of an excluded class, we did not use curtain, since it was often difficult to reliably see that curtains are depicted after these are segmented out. the final result is a setup with 13 familiar classes and 20 novel classes, as listed in during training (figure 1 , top panel), a model only sees familiar classes. we divide our data so that we have a training set with 18,279 unique spoken word segments and 94,316 unique unsegmented natural images spanning the 13 familiar classes. these are then paired up for training as explained in section 5.1. during training we also use a development set for early stopping; this small set consists of 130 word segments and 130 images from familiar classes. for me testing (figure 1 , middle panel) we require a combination of familiar and novel classes. our test set in total consists of 8,575 spoken word segments with 22,062 segmented object images. to implement the me test, we sample 1k episodes: each episode consists of a novel spoken word (query) with two sampled images, one matching the novel class from the query and the other containing a familiar object. we ensure that the two images always come from the same image dataset to avoid any intrinsic dataset biases. there is no overlap between training, development, and test samples. ## a visually grounded speech model we want to establish whether visually grounded speech models exhibit the me bias. while there is a growing number of speech-image models (section 2), many of them share the same general methodology. we therefore use a visually grounded speech model that is representative of the models in this research area: the multimodal attention network (mattnet) of @xcite . this model achieves top performance in a few-shot word-object learning task that resembles the way infants learn words from limited exposure. most useful for us is that the model is conceptually simple: it takes an image and a spoken word and outputs a score indicating how similar the inputs are, precisely what is required for me testing. ## mutual exclusivity results our main question is whether visually grounded models like mattnet (section 5) exhibit the me bias. to test this, we present the trained model with two images: one showing familiar and one showing a novel object. the model is then prompted to identify which image a novel spoken word refers to (section 3). we denote this me test as the familiar-novel test. with this, we also introduce our notation for specific tests: <image one @xmath0, with the type of the audio query underlined. the class of the audio query will match the one of the underlined image, unless explicitly stated. before we look at our target familiar-novel me test, it is essential to ensure that our model has successfully learned to distinguish the familiar classes encountered during training; testing for the me bias would be premature if the model does not know the familiar classes. we therefore perform a familiar-familiar test, where the task is to match a word query from a familiar class to one of two images containing familiar classes. we now turn to the me test. the results are given in the familiar-novel column of to investigate whether the reported accuracies are stable over the course of learning, we consider mattnet's performance over training epochs on the two tests: familiar-familiar and familiar-novel. we use the model variation with the strongest me bias, i.e., with both the audio and vision branches initialized. figure 3 shows that the me bias (familiar-novel, green solid line) is stronger early on in training and then decreases later on. the pattern is similar for the familiar-familiar score (red dashed line), but the highest score in this case is achieved later in training than the best familiar-novel score. the scores stabilize after approximately 60 epochs; at this epoch, the model's accuracy is 84.38% on the familiar-familiar task and 56.78% on the familiar-novel task (numbers not shown in the figure). this suggests that the results reported above for both tests are robust and do not only hold for a particular point in training. in summary, we found that a visually grounded speech model, mattnet, learns the familiar set of classes and thereafter exhibits a consistent and robust me bias. this bias gets stronger when the model is initialized with prior visual knowledge, although the results for the audio initialization are inconclusive. whereas the strength of the me bias slightly changes as the model learns, it is consistently above chance, suggesting that this is a stable effect in our model. ## further analyses we have shown that our visually grounded speech model has the me bias. however, we need to make sure that the observed effect is really due to the me bias and is not a fluke. in particular, because our model is trained on natural images, additional objects might appear in the background, and there is a small chance that some of these objects are from the novel classes. as a result, the model may learn something about the novel classes due to information leaking from the training data. here we present several sanity-check experiments to show that we observe a small leakage for one model variant, but it does not account for the strong and consistent me bias reported in the previous section. furthermore, we provide additional analyses that show how the model structures its audio and visual representation spaces for the me bias to emerge. ## conclusion and future work mutual exclusivity (me) is a constraint employed by children learning new words: a novel word is assumed to belong to an unfamiliar object rather than a familiar one. in this study, we have demonstrated that a representative visually grounded speech model exhibits a consistent and robust me bias, similar to the one observed in children. we achieved this by training the model on a set of spoken words and images and then asking it to match a novel acoustic word query to an image depicting either a familiar or a novel object. we considered different initialization approaches simulating prior language and visual processing abilities of a learner. the me bias was observed in all cases, with the strongest bias occurring when more prior knowledge was used in the model (initializing the vision branch had a particularly strong effect). in further analyses we showed that the me bias is strongest earlier on in model training and then stabilises over time. in a series of additional sanity-check tests we showed that the me bias was not an artefact: it could not be explained away by possible information leakage from the training data or by trivial model behaviors. we found that the resulting embedding space is organized such that novel classes are mapped to a region distinct from the one containing familiar classes, and that different familiar classes are spread out over the space to maximize the distance between familiar classes. as a result, novel words are mapped on to novel images, leading to a me bias. lastly, we showed that the me bias is robust to model design choices in experiments where we changed the loss function and used a supervised instead of self-supervised visual initialization approach. future work can consider whether using a larger number of novel and familiar classes affects the results. another interesting avenue for future studies resolves around multilingualism. following on from the original me studies with young children, @xcite and @xcite , among others, have looked at how multilingualism affects the use of the me constraint. this setting is interesting since in the multilingual case different words from the distinct languages are used to name the same object. these studies showed that in bi-and trilingual