Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding (2024)

Abstract: When connecting objects and their language referents in an embodied 3D environment, it is important to note that: (1) an object can be better characterized by leveraging comparative information between itself and other objects, and (2) an object’s appearance can vary with camera position. As such, we present the Multi-view Approach to Grounding in Context (MAGiC) model, which selects an object referent based on language that distinguishes between two similar objects. By pragmatically reasoning over both objects and across multiple views of those objects, MAGiC improves over the state-of-the-art model on the SNARE object reference task with a relative error reduction of 12.9% (representing an absolute improvement of 2.7%). Ablation studies show that reasoning jointly over object referent candidates and multiple views of each object both contribute to improved accuracy. Code: https://github.com/rcorona/magic_snare/

## introduction to distinguish a "thin handled mug" between two mugs, we must contextually reason about the object with the relatively thinner handle. such grounded language can connect to machine representations of the world @xcite . considering pragmatic context @xcite in grounded natural language can assist applications in vision and robotics @xcite @xcite @xcite . additionally, object features like mug handles may be occluded from certain viewpoints, requiring multiple views or 3d information @xcite . in the real world, language use is situated in a 3d environment and must consider a rich context of alternatives. however, for tasks like object disambiguation, some models score referring expression * denotes equal contribution figure 1 : left: previous methods for identifying object referents of language expressions in the snare benchmark consider target and distractor objects independently and pool multiple views before grounding. right: by contrast, magic jointly reasons over target and distractor objects and their views from different angles to identify the correct referent with higher accuracy than the previous state-of-the-art model. we introduce multi-view approach to grounding in context (magic) magic jointly reasons over candidate referent objects and considers each object from multiple possible vantage points (figure 1 ). we evaluate magic via the shapenet annotated with referring expressions (snare) benchmark @xcite . in snare, candidate objects are always of the same highlevel category, such as chair or mug, and language references uniquely identify one target referent object in contrast to the distractor object of the same category. embodied agents operating in realworld environments analogously need to disambiguate between similar objects, such as mugs in a kitchen, parts on a conveyor belt, or rocks on the seafloor. by reasoning about both objects and their views, magic achieves a relative error reduction of 12.9% (improved accuracy by 2.7%). our contributions include: ## background and related work embodied agents increasingly operating alongside humans must understand the relationships between natural language and the objects they reference. to best capture these relationships, our method synthesizes the comparative context afforded by reasoning over multiple objects and considering each one in multiple views. ## reference grounding we define a reference grounding task where, given one or more visual views of candidate objects and a natural language description, the reference grounding task is to select the object identified by the contrastive referring expression. formally, a model must use a given language description l to predict a target object o l that is aligned with the language description from among a set of m objects @xmath0, o c 1 , o c 2 , ..., o c m-1 }. besides object o l , there are m-1 distractor objects o c i that contribute to the context in which a model needs to reason about. for each object o, the model is able to perceive n views for each object o 1 , ..., o n . these objects are unordered, and we do not assume access to the relative positions between each view. the goal of the task is to learn a classifier function f (o, l) → [0, 1] m such that a higher probability is assigned to the target object. previous approaches @xcite @xcite learn f for single objects, then each object @xmath1, l) → [0, 1]. while classifying only individual objects simplifies the implementation, it limits the model's ability to comparatively reason about objects in context. also, previous image-based methods for reference grounding tasks @xcite aggregate each object's n views without reasoning about each view's relationship to the language description. to overcome these limitations, a model needs to address two key challenges: 1) reasoning about the contextual relationships between objects, and 2) reasoning about multiple views of each object in relation to the language description. we propose magic, a transformer-based architecture that enables joint reasoning over object-specific and view-specific contextual dependencies for 3d language grounding. ## magic we introduce multi-view approach to grounding in context (magic) for language grounding of 3d objects (figure 2 ). in contrast to previous work that individually score each object, magic considers both the language and the objects, along with their views, simultaneously. the design of our model is guided by principles in 3d language grounding and pragmatics. in snare, a model should consider information about comparative differences between two objects to identify the correct referent of the language expression. to enable a model to more effectively ground language to these visual dissimilarities, we focus on two key elements of the model formulation: (1) object context, which involves jointly reasoning over both objects and the referring expression, and (2) multi-view context, where multiple views of the object representation are explicitly utilized throughout the model without aggregating their representation as a preprocessing step. we adopt this paradigm in 3d language grounding and design our model to concurrently process features from both objects and the referring expression to leverage context-dependent information. with the context established by considering both objects and the referring expression, our model can leverage context-dependent information effectively. more concretely, consider the scenario of the model being asked to choose between two chairs given the referring expression "the tall, skinny chair". the model can exploit context-dependent information, such as using the descriptor "tall" to reason over both objects comparatively to ascertain which is taller. additionally, by incorporating features from multiple views of the object, our model benefits from the additional 3d perspective, ensuring that important object information, even if initially rotated out of view, is captured and utilized. a transformer architecture is well-suited for context-based 3d language grounding due to its wide receptive field and low inductive bias @xcite . unlike cnn-based architectures that have a spatial locality bias, transformers have a wide receptive field that includes all input features after just one transformer layer. this architecture enables our model to attend to all inputs and effectively leverage both object and multi-view context for 3d language grounding. moreover, the low inductive bias of transformers makes the design choice suitable for 3d language grounding, as the transformers are particularly good at handling multiple modalities @xcite . ## evaluation we evaluate the effectiveness of our method on the snare @xcite benchmark, a language grounding task that draws from a subset of items in the shapenetsem @xcite dataset, specifically those included in the acronym @xcite robot grasping dataset. the snare benchmark adversarially selects similar target and distractor objects to challenge 3d language grounding approaches. in the object reference task, the model is presented with a natural language description l and must correctly identify the target object o l from a set of @xmath2, o c 1 }. in sec-tion a.4, we investigate varying m. each object o in the benchmark is accompanied by @xmath3, capturing the object from different perspectives at 45-degree intervals. as both target and distractor objects are from the same shapenetsem category, the snare benchmark aims to evaluate a model's contextual reasoning abilities. the snare benchmark encompasses two types of object descriptions: visual and blindfolded. visual descriptions are generated by annotators who are guided to include the object's name, shape, and color. these visual descriptions aim to capture a comprehensive understanding of the object, providing relevant visual cues to guide the grounding process (e.g., "the red mug"). on the other hand, blind descriptions predominantly focus on the object's shape and specific distinguishing attributes, intentionally omitting color and other visual characteristics that might aid identification (e.g., "the one with a tapered lip"). the snare benchmark is split into training, validation, and test splits. the train/validation/test sets are split over (207 / 7 / 48) shapenetsem object categories, containing (6,153 / 371 / 1,357) unique object instances and (39,104 / 2,304 / 8,751) object pairings, each accompanied by a referring expression. the validation and test sets include unseen object categories that were not encountered during the model training phase, thus evaluating the generalizability and robustness of different methods. ## results in this section, we present the test set performance of our model and compare it with the previous state-of-the-art models. additionally, we report the average performance and standard deviation of our model and various ablations on the validation set, calculated over 10 different seeds. ## discussion in this work, we present magic, which demonstrates significant improvements in language grounding accuracy on an object reference task by reasoning jointly over objects and their multi-view contexts when scoring their compatibility with referring expressions. we find that comparatively reasoning over multiple objects is central to capturing contextual relationships that enhance the model's ability to ground object descriptions, with added multi-view context also contributing to better language-to-object grounding. the experimental results from the snare object identification benchmark highlight the effectiveness of magic, which outperforms all methods on both the validation and test sets. ## limitations magic heavily relies on having access to multiple views of objects. while using multiple views allows for capturing richer context and improving performance, it also requires obtaining and processing multiple images for each object, which may not always be feasible or practical in certain scenarios. future work could consider actively selecting views that promote the most information gain. additionally, our experiments focus on a single distractor object. we provide preliminary multiple distractor experiments in the appendix to showcase the practicality of magic in the real world, which is provides a foundation for future work on comparatively reasoning over multiple objects. additionally, magic uses clip embeddings for encoding visual information. while clip provides powerful pre-trained image and text encoders, its representations may not fully capture the intricacies and characteristics of 3d objects. this limitation could potentially impact the model's ability to discriminate between visually similar objects or capture fine-grained details crucial for accurate language grounding. ## potential negative societal impact magic was designed to ground language to 3d household objects. however, magic has direct potential uses for sensitive applications such as face identification and surveillance. for instance, law enforcement agencies may use magic with vague witness testimony to discern a suspect given two sets of mugshots with multiple views. in these high-stakes applications, our model could generate harmful and discriminatory identifications that would further negatively impact historically minoritized peoples. furthermore, our model uses a clipbackbone, and previous literature has shown that clip reinforces malignant sexist and racist stereotypes @xcite and exhibits gender bias @xcite which are part of broader patterns of marginalization in society. vision-and-language models have also been shown to compound gender biases that exist separately in language and vision @xcite . therefore, these models must account for the ways in which language and perception reflect social norms. demonstrate magic's ability to generalize to multiple distractors, as may be the case in a more realistic use case. snare provides adversariallyselected pairs with language annotations. to have multiple distractor objects, we randomly select an object in the same train/val set. there is no guarantee for new distractor objects will be in the same category of as the initial two objects since the language might not differentiate additional objects of the same category. our results in figure 7 show that while overall performance decreases, magic generally retains its strong performance over an architecture without object context. magic without object context is similar to the mlp-based match model, as they score each object individually. we find that reasoning over all objects generally outperforms scoring the objects individually. we note that performance clearly degrades as more objects are added, and we show a line depicting random chance to show that our model has generally high performance. due to snare being a dataset for loading 2 objects at a time, implementation constraints limited us from scaling up these experiments efficiently. thus each variant is trained only on 1 seed, which makes it clear that this result becomes noisy as more distractor objects are added. we also note that for magic, for additional distractor objects, magic is trained and evaluated on the same number of distractor objects.