Pretraining on Interactions for Learning Grounded Affordance Representations (2022)

Abstract: Lexical semantics and cognitive science point to affordances (i.e. the actions that objects support) as critical for understanding and representing nouns and verbs. However, study of these semantic features has not yet been integrated with the ?foundation? models that currently dominate language representation research. We hypothesize that predictive modeling of object state over time will result in representations that encode object affordance information ?for free?. We train a neural network to predict objects? trajectories in a simulated interaction and show that our network?s latent representations differentiate between both observed and unobserved affordances. We find that models trained using 3D simulations outperform conventional 2D computer vision models trained on a similar task, and, on initial inspection, that differences between concepts correspond to expected features (e.g., roll entails rotation) . Our results suggest a way in which modern deep learning approaches to grounded language learning can be integrated with traditional formal semantic notions of lexical representations.

## introduction much of natural language semantics concerns events and their participants-i.e., verbs and the nouns with which they compose. evidence from cognitive science @xcite and neuroscience @xcite suggests that grounding such words in perception is an essential part of linguistic processing, in particular suggesting that humans represent nouns in terms of their affordances @xcite , i.e., the interactions which they support. affordance-based representations have been argued to form the basis of formal accounts of compositional syntax and semantics @xcite . as such, prior work in formal semantics has sought to build grounded lexical semantic representations in terms of objects and their interactions play with these objects! seen unseen now play with this one! can it roll? can it contain other objects? figure 1 : we investigate whether observing interactions with an object in a 3d environment encodes information about their affordances and whether this generalizes in the zero shot setting to unseen object types in 3d space. for example, @xcite and @xcite represent verbs like roll as a set of entailed positional and rotational changes specified in formal logic, and @xcite argue that nouns imply (latent) events-e.g., that cups generally hold things-which should be encoded as telic values within the noun's formal structure. such work provides a compelling story of grounded semantics, but has not yet been connected to the types of large scale neural network models that currently dominate nlp. thus, in this work, we ask whether such semantic representations emerge naturally as a consequence of self-supervised predictive modeling. our motivation stems from the success of predictive language modeling at encoding syntactic structure. that is, if neural language models trained to predict text sequences learn to encode desirable grammatical structures @xcite , perhaps similar models trained to predict event sequences will learn to encode desirable semantic structures. to test this intuition, we investigate whether a transformer @xcite trained to predict the bottle1 wine bottle 04 bottle2 plate02_flat coin plate02 balls apple splitmetalball atomball soccer ball wheelball spikeball buckyball bombball eyeball boxes cardboardbox_01 cardboardbox_02 cardboardbox_03 bowls bowl01 bowl02 bowl03 book_0001c book_0001b book_0001a cans and mugs meat can box can small mug03 can spam can cola can mug02 disks old_ussr_lamp_01 lamp lamps vase_volutekrater gas bottle vase_amphora vase_hydria vases bottles ladle ladle pen books pen black roll bounce contain stack w-grasp slide future state of an object given perceptual information about its appearance and interactions will latently encode affordance information of the type thought to be central to lexical semantic representations. in sum: ## models we consider two primary variants of the representation learning task described in section 2.2 which differ in how they represent the world state-i.e., using 2d visual data ( §3.1) vs. using 3d visualspatial data ( §3.2). to provide additional insights into performance differences, we also consider two ablations in the 3d model ( §3.2.2), one that removes visual information and one that further removes pretraining altogether. these models are summarized in figure 3 . ## results figure 4 shows our primary results. overall, the 3d visual-spatial model substantially outperforms the 2d vision-only model across all affordances, often by a large margin (4-11 percentage points). we also see, perhaps unexpectedly, that the 3d pretrained representations encode information about affordances even when the associated actions are not explicitly observed. for example, the model differentiates objects that can stack and objects that can contain other objects from those that cannot, even though the model has not directly observed objects being stacked or serving as containers during training. this result points to the richness of the physical information that is required to perform the pretraining task of next-state prediction. 5 looking more closely at the ablated variants of the 3d model, we see that most of the gains are from the 3d input representation itself. that (a) 3d blind model (b) 3d visual-spatial model (c) 3d no-training (d) 2d computer-vision model (rpin). is, the 3d no-training model-which does not include visual information and does not even include pretraining-outperforms the cv baseline in all cases, and often substantially. pretraining on top of the 3d inputs often (but not always) yields performance gains. pretraining with visual information does not provide a clear benefit over pretraining on the spatial data alone-i.e., visual information leads to performance gains on three affordances (slide, roll, and stack) and losses on the other three (contain, w-grasp, and bounce). ## qualitative analysis in order to better understand the nature of the affordance-learning problem, we run a series of qualitative analyses on the trained models. we focus our analysis on the pair of affordances roll vs. slide. these are verbs have received significant attention in prior literature @xcite since they exemplify the types of manner distinctions that we would like lexical semantic representations to encode. we first compare the 2d video vs. 3d simulation variants of our pretraining objective. figure 5 shows a t-sne projection of the sequence representations from all four models, labeled based on if the object affords rolling or sliding. we find that object representations from the 3d blind model cluster strongly according to the distinction between these two concepts. the trend is notably not apparent in the no-training model. figure 6 demonstrates why spatial data pretraining may encourage this split. in the example shown, we take two thrown objects from our dataset-one round and one not round-and track the height of the center point of the object and one of the corners of the object bounding box. when they hit the ground the center point stays relatively constant as it moves across the floor in both, but in the rolling action, the corner point moves up and down as it rotates around the center point. since this is so distinguishable given the input representation, the model is better able to differentiate these concepts. it may be that the next state prediction task facilitates learning the slide vs. roll distinction in the 3d blind setting. however, the same pattern is not present in the 3d visual-spatial model (which also predicts next state). one possibility is that the presence of visual information competes with the 3d information, and as a result the joint space does not encode this distinction as well as the 3d space alone. designing more sophisticated models that incorporate visual and spatial information and preserve the desirable features of both is an interesting area for future work. ## conclusion we propose an unsupervised pretraining method for learning representations of object affordances from observations of interactions in a 3d environment. we show that 3d trajectory data is a strong signal for grounding such concepts and performs better than a standard computer vision approach for learning the desired concepts. moreover, we show through counterfactual analyses that the learned representations can encode the desired entailmentse.g., that roll entails axial rotation. our work contributes to an existing line of work that seeks to develop lexical semantic representations of nouns and verbs that are grounded in physical simulations. we advance this agenda by offering a way in which modern "foundation model" approaches to visual and linguistic processing can in fact be corroborative of traditional feature-based approaches to formal lexical semantics. our results suggest a promising direction for future work, in which pretraining objectives can be augmented to include richer notions of embodiment (e.g., planning, agent-agent interaction) and consequently encoder richer lexical semantic structure (e.g., presuppositions, transitivity).