World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models (2023)

Abstract: The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words. While humans demonstrate fast mapping in new word learning, it remains unclear whether modern vision-language models can truly represent language with their grounded meanings, and how grounding may further bootstrap new word learning. To this end, we introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in open-world language learning. As an initial attempt, we propose World-to-Words (W2W), a novel visually-grounded language model by pre-training on image-text pairs highlighting grounding as an objective. Through extensive experiments and analysis, we demonstrate that W2W is a more coherent and fast grounded word learner, and that the grounding ability acquired during pre-training helps the model to learn unseen words more rapidly and robustly.

## introduction language is learned through sensorimotor experience in the physical world @xcite . the ability to connect language units to their referents in the physical world, referred to as grounding, plays an important role in learning and understanding grounded meanings of words @xcite . as shown in figure 1 , a human reader would easily ground noun phrases to the corresponding entities captured in the image. even when the term "incinerator" is new to human learners, they can still locate the object of interest through the language and visual context, and acquire its meaning. in fact, this ability to bootstrap new word learning with only minimal information, known as fast mapping, is demonstrated abundantly in cognitive a lady wearing a navy blue stripe tank top is getting ready to burn glass in front of an incinerator. literature on human language acquisition @xcite @xcite . recently, there has been a substantial effort on pre-training vision-language models (vlms) @xcite . despite the exciting performance of these models on a variety of downstream vision and language tasks, it remains unclear whether these models can truly understand or produce language with their grounded meanings in the perceived world, and how grounding may further bootstrap new word learning. these questions are of interest from both a scientific and an engineering point of view. from a scientific perspective, grounding is crucial to language learners, as children attend to intended objects in the environment when producing @xcite and comprehending @xcite utterances. from an engineering perspective, even with the availability of grounded vision language datasets (image-text pairs with fine-grained wordobject mappings) @xcite , the costly grounding annotation can hardly cover the whole vocabulary space during the training time. building upon the pre-trained models, it's important for the agent to have the ability to learn grounded new words in a few shots of raw image-text pairs without word-object mappings. to this end, we introduce grounded open vocabulary acquisition (gova), a scalable formulation to examine grounding and bootstrapping in openworld language learning. in this formulation, lan-guage learning is a combination of learning to predict a word in a linguistic context as well as learning to ground the word in the physical world. under this formulation, we explore the framework in which the model first acquires the grounding ability during pre-training, and then transfers this ability to learn unseen words without grounding supervision. as an initial step, we developed world-to-words (w2w), a novel visually grounded language model motivated by recent advances in detection transformers (detr) @xcite . compared to many existing vlms, w2w performs language modeling upon explicit object representations. the model first acquires the ability to ground during pre-training, and then transfers this intrinsic ability to learn unseen words when grounded supervision is no longer available. our empirical results show that learning to map words to their referents plays a significant role in grounded word acquisition. by pre-training with fine-grained word-object mappings, w2w demonstrates stronger performance in learning grounded meanings of words, both seen and unseen, yet with orders of magnitude fewer data compared to other competitive vlm baselines. the pre-trained model can further provide a foundation for efficient learning of new grounded words with a few examples. we further present an in-depth analysis to understand potential predictors of w2w in word learning, which demonstrates intriguing behaviors in comparison to human language learning. our findings will provide a stepping stone for future work on grounded language learning in an open world. ## grounded open vocabulary acquisition (gova) we start by introducing the settings of grounded word acquisition and few-shot learning of new words tasks, which are two key components of the grounded open vocabulary acquisition (gova) task formulation. we further present a unified evaluation protocol and introduce the dataset we curated for this problem. ## conclusion and future work the connection between language and their referents captures the grounded meaning of words, and an explicit treatment is key to empowering efficient open-world language learning abilities in humans and ai agents. this work introduces grounded open vocabulary acquisition (gova), a scalable formulation to examine grounding and fast mapping in open-world grounded language learning. we propose world-to-words (w2w), a novel visually grounded language model to investigate a paradigm where the model initially acquires grounding ability during pre-training and subsequently applies this ability to quickly learn new words without explicit grounding supervision. our empirical findings highlight the significance of visual grounding in neural word acquisition. especially, we find that pre-trained w2w can serve as a foundation for fast mapping of novel grounded words via fewshot learning. we also conduct a comprehensive analysis to explore potential predictors influencing the performance of vision-language models, revealing both consistent and surprising behaviors with respect to human language learning patterns. these insights pave the way for future research in grounded language learning in the open world.