A Visuospatial Dataset for Naturalistic Verb Learning (2020)

Abstract: We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal child is likely to have access: That is, naturalistic, spontaneous speech paired with richly grounded visuospatial context. We use the collected data to compare several distributional semantics models for verb learning. We evaluate neural models based on 2D (pixel) features as well as feature-engineered models based on 3D (symbolic, spatial) features, and show that neither modeling approach achieves satisfactory performance. Our results are consistent with evidence from child language acquisition that emphasizes the difficulty of learning verbs from naive distributional data. We discuss avenues for future work on cognitively-inspired grounded language learning, and release our corpus with the intent of facilitating research on the topic.

## introduction while distributional models of semantics have seen incredible success in recent years @xcite , most current models lack "grounding", or a connection between words and their referents in the non-linguistic world. grounding is an important aspect to representations of meaning and arguably lies at the core of language "understanding" @xcite . work on grounded language learning has tended to make opportunistic use of large available corpora, e.g. by learning from web-scale corpora of image @xcite or video captions @xcite , or has been driven by particular downstream applications such as robot navigation @xcite . in this work, we take an aspirational look at grounded distributional semantics models, based on the type of situated contexts and weak supervision from which children are able to learn much of their early vocabulary. our approach is motivated by the assumption that building computational models which emulate human language processing is in itself a worthwhile endeavor, which can yield both scientific @xcite and engineering (linzen, 2020) advances in nlp. thus, we aim to develop a dataset that better reflects both the advantages and the challenges of humans' naturalistic learning environments. for example, unlike most vision-and-language models, children likely have the advantage of access to symbolic representations of objects and their physics prior to beginning word learning @xcite . however, also unlike nlp models, which are typically trained on image or video captions with strong signal, children's language input is highly unstructured and the content is often hard to predict given only the grounded context @xcite . we make two main contributions. first ( §2), using a virtual reality kitchen environment, we collect and release foot_0 the new brown corpus foot_1 : a dataset containing 18k words of spontaneous speech alongside rich visual and spatial information about the context in which the language occurs. our protocol is designed to solicit naturalistic speech and to have good coverage of vocabulary items with low average ages of acquisition according to data on child language development @xcite . second ( §3), we use our corpus to compare several distributional semantics models, specifically comparing mod-els which represent the environment in terms of objects and their physics to models which represent the environment in terms of pixels. we focus on verbs, which have received considerably less attention in work on grounded language learning than have nouns and adjectives @xcite . more so than nouns, verb learning is believed to rely on subtle combinations of both syntactic and grounded contextual signals @xcite and thus progress on verb learning is likely to require new approaches to modeling and supervision. in our experiments, we find that strong baseline models, both featureengineered and neural network models, perform only marginally above chance. however, comparing models reveals intuitive differences in error patterns, and points to directions for future research. ## data the goal of our data collection is to enable research on grounded distributional semantics models using data that better resembles the type of input young children receive on a regular basis during language development. doing this fully is ambitious if not impossible. thus, we focus on a few aspects of children's language learning environment that are lacking from typical grounded language datasets and that can be emulated well given current technology: 1) spontaneous speech (i.e. as opposed to contrived image or video captions) and 2) rich information about the 3d world (i.e. physical models of the environment as opposed to flat pixels). we develop a virtual reality (vr) environment within which we collect this data in a controlled way. our environment data is described in section 2.1 and our language data is described in section 2.2. our collection process results in a corpus of 152 minutes of concurrent video, audio, and ground-truth environment information, totaling 18k words across 18 unique speakers performing six distinct tasks each. the current data is available for download in json format at https:// github.com/dylanebert/nbc . the code needed to implement the described environment and data recording is available at https://github.com/ dylanebert/nbc_unity_scripts . ## experiments using the above data, we now compare several grounded distributional semantics models (dsm) in terms of how well they encode verb meanings, focusing in particular on differences in how the environment is represented when put in to the dsm. our hypothesis is that models will perform better if they represent the environment in terms of 3d objects and their physics rather than pixels, since work in psychology has shown that children learn to parse the physical world into objects and agents very early in life (spelke and kinzler, 2007), long before they show evidence of language understanding. we also explore how models vary when they have access to linguistic supervision early in the pipeline, during environment encoding, in addition to later, during language learning. we note that the models explored are intended as sim- ## discussion we compare two types of models for grounded verb learning, one based on 2d visual features and one based on 3d symbolic and spatial features. our analysis suggests that these approaches favor in different aspects of verb semantics. one open question is how to combine these differing signals, and how to design training objectives that encourage models to chose the right sensory inputs and time scale to which to ground each verb. we evaluated on a small set of verbs that are acquired comparably early by children. nonetheless, our models perform only marginally better than random. this disconnect highlights an important challenge to be addressed by work on computational models of grounded language learning: can statistical associations between words and contexts result in more than simple noun-centric image or video captioning, eventually forming general-purpose language models? while that question is still wide open, research from psychology could better inform work on grounded nlp. for example, @xcite argues that verb learning in particular is not learned from purely grounded signal, but rather is "scaffolded" by earlier-acquired knowledge of nouns and of syntax. from this perspective, the models we explored here, which are similar to what is used for noun-learning, are far too simplistic for verb learning. more research is needed on ways to combine linguistic and grounded signal in order to learn more abstract semantic concepts. ## related work we contribute to a large body of research on learning grounded representations of language. grounded representations have been shown to improve performance on intrinsic semantic similary metrics @xcite as well as to be better predictors of human brain activity @xcite . much prior work has explored the augmentation of standard language modeling objectives with 2d image @xcite @xcite @xcite and video @xcite data. recent work on detecting fine-grained events in videos is particularly relevant @xcite @xcite . especially relevant is the data collected by @xcite , in which human subjects were asked to play simple games with a physical robot and narrate while doing so. our data and work differs primarily in that we focus on the ability to ground to symbolic objects and physics rather than only to pixel data. past work on "situated language learning", inspired by emergence theories of language acquisition (macwhinney, 2013), has trained ai agents to learn language from scratch by interacting with humans and/or each other in simulated environments or games @xcite @xcite @xcite , ## conclusion we introduce the new brown corpus, a dataset of spontaneous speech aligned with rich environment data, collected in a vr kitchen environment. we show that, compared to existing corpora, the distribution of vocabulary collected is more comparable to that found in child-directed speech. we analyze several baseline distributional models for verb learning. our results highlight the challenges of learning from naturalistic data, and outlines directions for future research.