Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024)

Abstract: Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension, and their internal representations are remarkably well-aligned with representations of language in the human brain. But to achieve these results, LMs must be trained in distinctly un-human-like ways — requiring orders of magnitude more language data than children receive during development, and without perceptual or social context. Do models trained more naturalistically — with grounded supervision — exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition. We train a diverse set of LM architectures, with and without auxiliary visual supervision, on datasets of varying scales. We then evaluate these models’ learning of syntactic categories, lexical relations, semantic features, word similarity, and alignment with human neural representations. We find that visual supervision can indeed improve the efficiency of word learning. However, these improvements are limited: they are present almost exclusively in the low-dataregime, and sometimes canceled out by the inclusion of rich distributional signals from text. The information conveyed by text and images isnot redundant—models mainly driven by visual information yield qualitatively different from those mainly driven by word co-occurrences. However, our results suggest that current multimodal modeling approaches fail to effectively leverage visual information to build human-like word representations from human-scale data.

## introductions neural language models (lms) have achieved remarkable success across a wide variety of language processing tasks @xcite @xcite . they have also proven useful for predicting aspects of human language processing, both in behav-ioral models of production @xcite and comprehension @xcite as well as models of neural responses to linguistic inputs @xcite @xcite . lms are therefore strong candidates as computational models of core aspects of human language processing. at present, however, these models are profoundly implausible as models of human cognitive development. the amount of training data required by effective lms greatly exceeds the amount of linguistic input that human language learners receive during development @xcite : modern lms are typically trained on tens of billions of sentences, whereas children only receive around a million sentences in the first three years of their lives @xcite . techniques for building lms that learn like humans would immediately enable richer computational models of language acquisition and child language learning dynamics, and perhaps offer a path toward more sample-efficient learning in lms targeted at nlp applications. one of the most significant differences between how humans and lms learn language is that humans ground language in perceptual signals spanning many different modalities, including vision, touch, and hearing @xcite @xcite . in sighted learners, vision is hypothesized to play a central role, as it delivers detailed information that is often directly coupled to linguistic input @xcite . researchers in the natural language processing community have argued that multi-modal training might offer a path toward more human-like language learning @xcite . promisingly, recent years have seen the introduction of a profusion of multi-modal models and learning algorithms, mostly targeted at tasks that require reasoning about data in both modalities simultaneously @xcite @xcite @xcite . however, the extent to which these models actually acquire more human-like learning efficiency of language itself has received limited attention. in this paper, we investigate whether visual grounding can improve a key aspect of language understanding-word learning-in neural lms. we study a variety of visually grounded models, including clip @xcite , git @xcite , and flamingo @xcite , which represent drastically different ways of fusing vision and language data. while training these models, we carefully control, and systematically vary, both dataset size and the amount of withinlanguage distributional information provided by word co-occurrence statistics. we then characterize these models with a suite of tasks designed to benchmark various facets of word knowledge, via prediction of syntactic categories, lexical relations, semantic features, semantic similarity, and human neural representations. we find that grounded word learning indeed can yield better performance than the control languageonly models in capturing word similarity and semantic features. however, this benefit is only observed when training on datasets that are small even by the standards of human learning. more surprisingly, it is observed only when limiting models' exposure to word co-occurrence information within language: in some cases, models actually exhibit reduced sample efficiency when learning from images and captions rather than images accompanied by single words. although further analyses show that visual and distributional information are partially complementary, none of the models we study can integrate both perceptual and textual contexts to yield improved word representations. our results suggest that current models and training procedures remain far from serving as models of visually grounded language acquisition. in these models, learning of some, but not all, aspects of semantics can be facilitated by grounding, but distributional information contained within language can override (and perhaps interfere with) visually grounded learning. ## background word learning in children. the present study investigates how visual grounding can help acquire knowledge of words and their meanings. research has shown that children can correctly understand or produce words at a very young age, possibly even before the age of one @xcite @xcite . @xcite measured children's attention to visual inputs when prompted with words and found that meanings of several common words are known by children from the age of 6 months onward. using a different approach to measure word learning, @xcite collected the responses of parents to questionnaires about whether their children can correctly understand or produce words. given the small amount of data required by children to exhibit these behaviors, word learning thus offers an appealing test-bed for comparative studies of lms in a low-data regime. multi-modality learning. in recent years, multimodality learning has seen significant advancements. for example, the clip model @xcite is trained contrastively on 300m noisy (image, caption) pairs. it yields transferable visual representations as well as word representations that perform well on some tests of words similarity @xcite . git @xcite , by contrast, is a generative model, conditioning next-word predictions using visual inputs. it achieves state-of-the-art performance on multiple visual-language tasks, including image captioning and visual question answering. as a final example, flamingo @xcite uses visual representations to modulate attention in a transformer language model, obtaining similar results. as clip, git, and flamingo all differ significantly from each other in how vision and language are fused, we test all of them in this work to explore what algorithm designs best benefit grounded language learning. grounded and ungrounded learning algorithms as models of language acquisition. chang and bergen (2022) investigate the word-acquisition trajectories in language-only models. however, their trajectories are computed only by measuring the change of model surprisal towards one word throughout training, which is less relevant to knowing word meanings. @xcite and @xcite aim to improve language models trained on small datasets but focus more on grammar learning. within grounded models, @xcite train image captioning models on first-person videos children receive @xcite and claim that visual information helps predict words in context. @xcite and @xcite also propose using multi-modality algorithms to understand word acquisition. @xcite specifically study acquisition of word categories, while @xcite focus on learning function words from simplified visual stimuli. our experiments aim at a significantly more general model of lexical knowledge, via a diverse collection of model architectures, more naturalistic visual supervision, and tests of numerous facets of word knowledge. foot_0 3 methods ## conclusion and discussion we have shown that language learning grounded in visual information helps current neural models acquire some aspects of word knowledge more efficiently than they can from text alone. grounded models also learn qualitatively different representations from language-only models. but with reasonably large training data, ungrounded models become comparable to or even outperform grounded models. even in low-data regimes, this benefit from visual information requires limited exposure to distributional information, as current learning algorithms struggle to integrate visual and distributional information. the strong performance of visual + word (clip) models in low-data regimes provides a compelling hypothesis about how children acquire language: they may also learn through mapping each visual input to single words that are "loosely" tied to the input. however, more work is needed to test this hypothesis through evaluating the models on the benchmarks or experiments where data from children are also available. our results show that visual information can boost the learning of word meanings according to multiple measures. future grounded lms might thus serve as candidate models of how visual input can be leveraged to acquire language.