Do Trajectories Encode Verb Meaning? (2022)

Abstract: Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world. Grounded language models have had success in learning to connect concrete categories like nouns and adjectives to the world via images and videos, but can struggle to isolate the meaning of the verbs themselves from the context in which they typically occur. In this paper, we investigate the extent to which trajectories (i.e. the position and rotation of objects over time) naturally encode verb semantics. We build a procedurally generated agent-object-interaction dataset, obtain human annotations for the verbs that occur in this data, and compare several methods for representation learning given the trajectories. We find that trajectories correlate as-is with some verbs (e.g., fall), and that additional abstraction via self-supervised pretraining can further capture nuanced differences in verb meaning (e.g., roll and slide).

## introduction while large distributional language models such as bert @xcite and gpt @xcite have had empirical success in deriving representations of words and sentences from large text corpora, most of these models lack grounding, or a connection between the words and their real-world referents. grounding, in addition to being necessary for multimodal tasks like video recognition, has been argued to lie at the core of language understanding @xcite . work on grounded language learning associates language with the non-linguistic world, typically by learning from large-scale image @xcite or video @xcite datasets. much prior work on language grounding has focused on concrete nouns (objects) and adjectives (attributes), which are captured well by patterns of pixels. verbs, however, have received less attention, despite being essential for building models that can interact in realistic 3d environments @xcite . verbs are especially challenging to model, given that they take place over time. image and video data alone is insufficient to fully capture verb semantics, as demonstrated by prior work @xcite , in many cases failing to isolate the meaning of the verb from context in which it typically occurs. for example, @xcite show that an image of a person laying in the snow next to a snowboard is labeled "standing on a snowboard". moreover, recent work has introduced datasets and benchmarks based on situated 3d environments @xcite @xcite ) that demonstrate the challenges of learning task-oriented behavior, which demands a combination of object and verb grounding. in this paper, we test the hypothesis that the semantics of (concrete) verbs are grounded in the 3d trajectories of objects: i.e., the absolute and relative paths objects take through 3d space. we investigate if and when verb meanings appear to be a product of raw perception of objects in 3d space, and when differentiating verb meanings requires additional abstraction and representation beyond what is available via direct perception. to study this, we collect a clean dataset of 3d object trajectories in simulation. we collect human descriptions of these perceived world dynamics, i.e., to determine whether or not a given event constitutes a fall or a tumble. we then propose a self-supervised pretraining approach, whereby we train a time-series prediction model to obtain representations of trajectories in a 3d environment without any linguistic input. we evaluate the learned representations on how well they encode verb semantics for specific verbs. we show that the pretrained model learns to represent events in a way that aligns well with the meaning of english verbs, e.g. differentiating slide from roll. in summary, our primary contributions are: ## related work grounded language with deep learning. our contributions add to a large body of work on grounded representation learning. much of this work augments language modeling objectives with images @xcite @xcite and videos @xcite . in this work, we focus on representations that encode verb semantics. prior work on verb learning has been conducted in the computer vision community, typically described as "humanobject interactions" @xcite @xcite . most closely related to our approach, which focuses on trajectory data, is work on learning affordances for human-robot communication. for example, @xcite ; @xcite learn affordance representations based on the state changes of objects, but do not encode the full trajectory between states. also related is work in grounded language in text-only models which investigates models ability to reason about objects through space and time @xcite . outside of nlp, models have been trained on trajectory data for applications like human motion path forecasting @xcite and human activity recognition @xcite . our work lies at the intersection of grounded language learning and spatiotemporal machine learning, using representations of trajectory data to study verb semantics. grounding and lexical semantics. prior work in formal semantics attempts to build feature-based representations of verb meaning in terms of the 3d trajectories and state transitions entailed by those verbs @xcite @xcite . such work is related more generally to the idea of mental simulation as a means for representing and reasoning about linguistic concepts @xcite @xcite . we view our contribution as consistent with and complementary to this formal semantics program. while the prior work has sought to codify the precise truth conditions of motion verbs, we investigate whether such representations could emerge organically from datadriven processes. while we focus on concrete verbs in this paper, other work has argued that motor processing and mental simulation plays a more general role in language processing. for example, @xcite makes a case for grounded distributional "conceptual spaces" as the foundation for modeling linguistic concepts. @xcite discusses the role of metaphor in modeling abstract uses of words like push. borghi and riggio (2009) argues for the notion of a "motor prototype" as a key component of recognizing and processing objects, and @xcite presents evidence that the sensorimotor system (in particular the interactive aspects) drive acquisition of abstract concepts. ## dataset analysis in this section, we analyze trends in the dataset annotations, including worker agreement, and comparisons between semantically related verbs. ## experiments our hypothesis is that representation learning in the 3d visuospatial world (without language supervision) can yield concept representations that align to english verb semantics-i.e. can the representations capture nuanced distinctions like throw vs. toss or slide vs. roll? to test this, we pretrain a self-supervised model on a time-series prediction task, and then use a perceptron classifier to evaluate its learned representations. we evaluate four approaches, described in detail below. first, we train a simple perceptron to evaluate the representational capacity of the trajectory data as-is, as a comparative baseline. second, we train a fully supervised model to determine a soft upper bound on the task without pretraining. third, we evaluate our self-supervised model. and finally, we fine-tune the self-supervised model to determine an upper bound with pretraining. ## results we report mean average precision on unseen test data for each approach in perceptron. the perceptron approach evaluates the representational capacity of raw trajectory data as-is, with a lower bound of random stratified and soft upper bound of fully supervised. the perceptron performs relatively well for its simplicity, being only 7 points below the fully supervised upper bound. this suggests that the trajectory data itself encodes a significant amount of verb meaning, but leaves plenty of room for improvement. self-supervised pretraining. the pretraining + probe approach evaluates the ability of selfsupervised models to encode verb meaning from trajectory data. this is equivalent to the perceptron approach, but with learned hidden representations as input rather than raw trajectory data. the pretrained model does outperform the perceptron, as well as the fully supervised approach. fine-tuning only improves on this slightly, highlighting that self-supervised pretraining can yield representations that successfully encode verb meaning. ## discussion and conclusion we test the hypothesis that verb meanings can be grounded in 3d trajectories, i.e., the position of objects over time. specifically, we investigate the extent to which representations of object trajectories, learned without any linguistic supervision, naturally encode concepts that align to english verb semantics. our primary contributions are twofold. first, we build a procedurally generated agent-object-interaction dataset for which we collect crowdsourced annotations. this is the first dataset of its kind, and provides a rich inventory of human judgments about the extensions of 24 verbs of motion. second, we compare a variety of representation learning approaches, specifically contrasting approaches which operate directly on perceptual inputs to approaches which learn abstractions over the raw perception (via pretraining). we find that some verbs meanings (e.g., fall and push) are captured easily by the raw state information, while others (e.g., roll and turn) require additional processing to be represented well. this work is a first step toward exploring ways to capture fine-grained distinctions in grounded verb semantics that are trivial for humans, but challenging for models. recent benchmarks at the intersection of nlp, vision and robotics @xcite illuminate unsolved challenges in ai that demand a more robust understanding of verb semantics and spatial reasoning. as these benchmarks continue to be developed, and rich multimodal datasets from technologies like virtual reality become increasingly abundant, we envision that future work in this vein will be especially relevant. in the future, we plan to explore more sophisticated models for self-supervised pretraining, and evaluate how well these models transfer to more naturalistic language learning settings @xcite . beyond this, there is a large body of related research questions to be explored. for example, can representations of trajectory data be fused with visually-grounded representations to yield better encodings of verb semantics? collaborative efforts will be key to addressing these next milestones in natural language understanding.