Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations (2023)

Abstract: We present a novel method for using agent experiences gathered through an embodied simulation to ground contextualized word vectors to object representations. We use similarity learning to make comparisons between different object types based on their properties when interacted with, and to extract common features pertaining to the objects’ behavior. We then use an affine transformation to calculate a projection matrix that transforms contextualized word vectors from different transformer-based language models into this learned space, and evaluate whether new test instances of transformed token vectors identify the correct concept in the object embedding space. Our results expose properties of the embedding spaces of four different transformer models and show that grounding object token vectors is usually more helpful to grounding verb and attribute token vectors than the reverse, which reflects earlier conclusions in the analogical reasoning and psycholinguistic literature.

## introduction a common critique of modern large language models (llms) is that they lack understanding in the sense of being able to link an utterance to a specific communicative intent @xcite . this shortcoming is often characterized as being due to a lack of ability to ground or link lexical items to real-world entities such as classes of objects, or associated properties or actions. for instance, a modern generative llm like chatgpt foot_0 may be able to generate coherent text describing an object (e.g., "a coconut has a hard, often hairy outer shell"), without any inherent underlying conceptualization of what the item actually is. crucially, these underlying conceptualizations necessarily invoke other modalities. existing approaches to grounding in nlp typically treat the problem as one of making the correct kind of link between text and another modality, usually images @xcite @xcite . however, still images do not capture the wealth of information humans receive when interacting with objects or experiencing events, and video data requires orders of magnitude more data and computational power to effectively process. additionally, humans do not use vision alone as their only non-linguistic modality. as humans develop object concept representations and map them to associated nouns, they are also learning to individuate objects from the perceptual flow not just based on visual features but also based on experience that includes interacting with them in real time @xcite @xcite . @xcite argues that @xcite 's findings on variability in verbal semantics helped to explain why nouns are typically learned before words for verbs or other properties. concrete nouns are more easily "groundable" not just because of their visual manifestations but also because of their physical presences that leave traces in the world, and these physical properties provide a scaffold on which to build representations of related concepts that are supervenient upon understanding of objects. in this paper we take an embodied simulation approach to grounding, using a virtual environment to create experiences for an agent interacting with objects. we show that similarity learning over data gathered during the agent's experience in the virtual world can not only make comparisons between objects, but also appears to learn information pertaining to more abstract properties of the objects. fig. 1 shows a schematic view of our overall approach. we map token vectors from different transformer-based llms into the resulting representation space, and show that with just a few samples, grounding noun representations alone is helpful for subsequent grounding of verbal tokens, abstract properties, and attributive terms, but that grounding verbal or attributive token representations is less helpful for subsequent grounding of object concepts. ## related work multiple works in cognitive science have identified contrastive mechanisms, and the ability to analogize by applying previous experiences to novel scenarios, as a cornerstone of problem-solving @xcite @xcite @xcite . in visual analogy, @xcite created analogies by contrasting relational structure. for solving raven's progressive matrices (rpms) @xcite , @xcite applied a generalization of the noise contrastive estimation (nce) algorithm @xcite . @xcite performed feature learning using visual similarity via unsupervised learning at the instance-level with nce. oh @xcite used deep feature embedding based on lifted structure loss, and evaluated their method via clustering and retrieval tasks on images from unseen classes. @xcite trained a siamese cnn with contrastive loss @xcite to learn an embedding space of interior design images and applied the embeddings to image search applications like finding visually similar products across categories. since evaluating ai agents in physical environ-ment can be expensive, many works have used both embodied and non-embodied simulations to explore language learning. hermann et al. ( 2017 ) combined reinforcement and unsupervised learning to teach agents to correlate linguistic symbols with physical percepts and action sequences. however, this still-computationally-expensive method required millions of training episodes. the snare benchmark @xcite was evaluated on grounding to objects but not in context or under interaction. @xcite demonstrated an emergent clustering of semantic tokens from a (non-embodied) continuous representation space and @xcite extended that method with an application of an information bottleneck. our work integrates concepts from the above areas: embodied simulation environments, language grounding using both situated and linguistic context, and emergent semantic categorization. @xcite examined 2d and 3d visual and interactive data for learning object affordances and found that 3d and interactive data performed better. @xcite extracted verbal semantics from object trajectories in 3d space, but focused only on verbs whereas we examine nouns, verbs, and attributes. we show that objects and properties can also be encoded by object trajectories or behavior in 3d space, using a stacking task that exposes richer correlations between object properties and behaviors. like us, @xcite investigated word grounding but they evaluated on within-domain concepts (e.g., learning "left" to help ground "right" where we investigate how, say, learning "sphere" can help ground "round") in a grid world (our world representation is continuous), and where their transformation passed input through a whole llm, our transformation is a simple affine map between embedding spaces. @xcite mapped vision embeddings to language via ridge regression, but their multimodal skip-gram used static word vectors, not contextualized vectors from transformers. @xcite evaluated the representations of transformer models and found that multimodal representations better align with human judgments in the domain of concrete nouns, but not abstract terms. our work arrives at a related conclusion using cross-model transfer. ## methodology our methodology comprises two primary components: similarity learning to create a representation space of objects by making comparisons between geometric properties, and linear projection to ground language representations to this space. ## results of conceptual grounding for illustrative purposes, let us first examine the concepts of "flat" and "round" using word embeddings drawn from xlm, the best-performing model in this domain when "hinting" is used. further results from other models are given in the appendix. in fig. 4 , we see word embeddings for "flat" and "round" transformed into the object embedding space. at first (the left figure), when only information about cubes, spheres, and eggs are used to compute the mapping, there is only a slight separation between the two transformed embedding clusters and neither term clusters cleanly with either flat or round objects. when information about all objects is used to compute the transformation (center), the two word embedding clusters distinctly separate, with most "flat" embeddings clearly overlapping with the flat-sided objects, mutatis mutandis "round" embeddings and the round objects. finally (right), the "hint" is provided, by explicitly pairing a small set of 5 "flat" or "round" word embeddings to object embeddings whose type appears in a generated sentence collocated with the target word (e.g., "the cubes were flat on all sides, making it easy to stack them neatly."). with this hint we see that the "flat" and "round" word embeddings more completely overlap with the objects that have the respective attributes. when little information about related object concepts is provided when computing the mapping from llm space to object embedding space, the transformed clusters of contrasting terms share a high level of similarity in object space, but as more information about related object concepts is introduced into the transformation, the separation of the transformed novel concept clusters start to cleanly separate and become distinguished from each other. fig. 5 shows the mean similarity of the transformed clusters of attributive, verbal, and object synonym terms as different object terms are mapped into the object space, using embeddings drawn from the four different llms. fig. 6 shows the same change in the similarity between cluster centers, but this time evaluated over the transformed object terms when the transformation is computed using attributive and verbal terms. in both plots, dashed lines show where an explicit "hint" is given about specific concepts. we see that just by using a few samples of each concept and projecting them into object space using an affine transformation, grounding object terms is helpful in distinguishing the meaning of terms denoting related properties, attributes, and verbs, but grounding the more abstract concept vocabulary first does not usually cause the transformed clusters of object terms to separate before explicit hinting is provided, reflecting the psycholinguistic hypothesis of @xcite . ## discussion separation of conceptual clusters in fig. 5 , we can see that for object concept vectors from certain models, as information about certain other concepts is included in the transformation from llm space to object space, the centers of the conceptual clusters in question start to organically separate. this is particularly true for albert object word vectors and to some extent xlm and bert vectors. in flat/round is the easiest of the attributive or verbal concepts to distinguish, through affine transformations that include information about flat and round objects. stable/unstable is a particularly hard distinction for most model representations, in part because of the low support for these terms in the training corpus but also because in the scenario captured in the simulation data and described in training sentences, the terms refer to properties not of the objects themselves, but of the objects in the context of the stacking task (i.e., spheres are not inherently "unstable" but are if someone attempts to stack them). this suggests data gathered from either stacking more objects, or from tasks involving more complex balancing acts would be useful to learn a robust interpretation of such terms. inverse trends are observable in fig. 6 , where we see that when the transformation includes only information about attributes and verbs, transformed bert and xlm object word vectors for contrasting objects do not meaningfully separate until explicit hints are provided (and even then sometimes they don't separate much). some of the roberta object word clusters do appear to appreciably separate as more attribute and verb information is added to the transformation, and albert object word clusters, actually at first grow closer as related conceptual information is added to the transformation, until suddenly separating at the provision of an explicit hint. this suggests that albert, perhaps due to its smaller training size and architecture, learns vocabulary representations that are more "entangled" or that representations of flat-sided or round object words carry with them a bias toward object-related interpretations of "flat," "round," and associated terms. meanwhile xlm and other representations of abstract vocabulary are perhaps less correlated with concrete nouns, making them less easy to ground but also in principle more compositional with less bias toward certain interpretations. figure 5 : separation of cluster centers for transformed (in order) bert, roberta, albert, and xlm embeddings for verb and property concepts, as more information about other concepts is progressively added to compute the transformation. dashed lines show where a "hint" is given about the concept to be grounded (denoted by the similarly-colored solid line). ## conclusion and future work in this paper, we have presented evidence that similarity learning over rich object behavior and trajectory data from an embodied simulation environment can create a representation space that not only successfully classifies concrete objects but can make analogical comparisons between them based on abstract properties that inhere across multiple object types. we used the resulting representation to conduct investigations into the properties of token embeddings from different llms by mapping them into the object space using a linear ridge regression technique. we found that computing a mapping using representations of objects/object terms correlated with increased ability to distinguish and assign related conceptual vocabulary to the right categories, but that representations from different llms behaved quite differently. we also observed that computing mappings using information about abstract properties was less useful for distinguishing and classifying object terms. this reflects earlier arguments from psycholinguistics and analogical reasoning, e.g., @xcite 's hypothesis that names for concrete objects should be learnable by humans very early but that associated verbs and attributes are harder. our approach uses numerical data that situates and embodies an agent's positioning in the environment relative to the objects it interacts with. this method allows us to build a model over rich information without visual artifacts like occlusion or perspective distortion, prior research, e.g., krishnaswamy and @xcite ; @xcite has demonstrated that embodiment is also influenced by other factors like events and habitats, and that purely linguistic representations of objects, attributes, and activities may not capture these types of information. in fact, the corpus generated using chatgpt, an unembodied language model trained solely over text, is likely not entirely representative of these aspects beyond cooccurrences between object terms, habitats, and affordances in the training data. what our embodied approach brings is a way to correlate representations extracted from unembodied models to representations learned from embodied data, and provides evidence that the ability to ground realworld entities, properties, or actions to lexical items could enable llms to simulate the human ability to link utterances to specific communicative intents. however, further investigation is necessary. since the primary objective of this research is to provide a method that achieves human-like "understanding" of communicative intents, we should note that we do not argue that human learners use the same mathematical transformations we use herein, but just that we can use them to make ai systems behave similarly. directions for future work include 1) investigating the effects of intra-class order when grounding tokens, e.g., introducing object concepts to the affine mapping in a different order; 2) using similarity learning over images, or images combined with the embodied data, to create the representation space; 3) using data gathered in other embodied tasks to investigate other concepts like concavity or directedness, that are not captured in this stacking task; 4) evaluating token representations directly from a decoder like a gpt-style model; and 5) directly operationalizing analogical comparisons in a real-time embodied simulation, e.g., by making an agent solve problems using analogical reasoning in a live environment. of this paper. this research was supported in part by a grant from the u.s. army research office (aro) to colorado state university, under award #w911nf-23-1-0031. the positions expressed herein do not reflect the official position of the u.s. department of defense or the united states government. any errors or omissions are the responsibility of the authors.