Language Grounding with 3D Objects
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
Seemingly simple natural language requests to a robot are generally underspeciﬁed,for example Can you bring me the wireless mouse?
Flat images of candidate micemay not provide the discriminative information needed for wireless .
The world,and objects in it, are not ﬂat images but complex 3D shapes.
If a human requests anobject based on any of its basic properties, such as color, shape, or texture, robotsshould perform the necessary exploration to accomplish the task.
In particular,while substantial effort and progress has been made on understanding explicitlyvisual attributes like color and category, comparatively little progress has been madeon understanding language about shapes and contours.
In this work, we introducea novel reasoning task that targets both visual and non-visual language about 3Dobjects.
Our new benchmark S hape N et A nnotated with R eferring E xpressions(SNARE) requires a model to choose which of two objects is being referencedby a natural language description.
We introduce several CLIP-based [1] modelsfor distinguishing objects and demonstrate that while recent advances in jointlymodeling vision and language are useful for robotic language understanding, it isstill the case that these image-based models are weaker at understanding the 3Dnature of objects – properties which play a key role in manipulation.
We ﬁnd thatadding view estimation to language grounding models improves accuracy on both SNARE and when identifying objects referred to in language on a robot platform,but note that a large gap remains between these models and human performance.
Introduction
Choose the mug with …
How?
A B a wide handle no logo
Rotate(B)
Rotate(A)Rotate(B) return A return B
Rotate them!
Joint language and vision models are often trained on image captionswhich have a bias towards canonically “visual” attributes of objects,such as color, rather than functional ones like shape.
Image captionsomit properties that require understanding objects as 3D, ratherthan ﬂat, concepts.
In this work, we show how the effect of thisdisconnect is that while robotics research has beneﬁted greatly fromadvances in computer vision, vision-and-language models cannotalways be directly applied to robotics.
People use geometric andphysical properties of objects when describing them.
For example,a robot carrying out a request to Bring me the mug with the widehandle needs to be able to spot the wide handle , even if the mug restson a countertop with the handle oriented out of view, but a robotcan rotate such an object to investigate in further detail.
Generally,identifying objects based on natural language is more challengingfrom non-canonical viewpoints, which are common for robots inhome, ofﬁce, and industrial environments.
The codiﬁcation and commodiﬁcation of ResNet [2] architectures pretrained on ImageNet [3] hasyielded immense progress in off-the-shelf computer vision techniques.
Such representations providea strong foundation for visual reasoning.
With the introduction of pretrained Transformer [4] architectures like BERT [5], natural language processing has similar off-the-shelf tools for representing language.
Modern visual language grounding is done through massive, internet-scale pretrainingto combine these strengths: aligning images with their captions and content descriptions in naturallanguage using Transformer models [6, 1, 7, 8].
However, internet images suffer from a type of reporting bias .
These images are captured by sightedhumans, from human-centric vantage points [9].
There is a domain shift between such canonicaldepictions of objects in images and objects as seen through a robot camera [10, 11], as studied incomputer vision between such images and those captured by the blind [12].
For example, images ofmugs online almost exclusively feature a full or three-fourths view of the handle, while a mug in thewild can easily be oriented such that the handle is occluded from the camera by the mug body.
A person requesting a mug with a wide handle has a mental model of the referent object that includesits full 3D spatial information; humans do not imagine objects only in 2 dimensions.
Further, someonetrying to retrieve said mug will differentiate it from other mugs on the countertop by viewing themugs from different angles to inspect the handles.
Our goal is to similarly imbue robots with three dimensional notions of language grounding, byencouraging them to rotate an object when selecting the referent of a language description.
Weintroduce the S hape N et A nnotated with R eferring E xpressions (SNARE) dataset, which providesdiscriminative natural language descriptions of 3D ShapeNet [13] object models.
SNARE includesboth visual descriptions that focus on colors and object categories and blindfolded descriptions thatfocus on shapes and parts.
We ﬁnetune large off-the-shelf models to select the correct object given itsdescription, and ﬁnd that estimating the input views as an auxiliary prediction improves accuracy.
We introduce La nguage G rounding through O bject R otation (L A GOR) which performs view estimation as an auxiliary loss to encourage 3D object understanding (Section 4) during the SNAREtask.
We ﬁnd that using multiple views to select the correct object improves over single-view objectselection, using the large-scale vision-and-language CLIP [1] model as a backbone to score howwell an image and language M ATCH .
We show that models taking in multiple views of objectsobjects, while performing auxiliary view estimation, can result in accuracy approaching that ofmodels consuming panoramic views of objects, while also being more realistic for a robot.
Our key contributions are:
• SNARE, a benchmark dataset for identifying 3D object models given natural language referringexpressions including visual and blindfolded descriptions;• Baseline models for SNARE demonstrating both zero-shot and ﬁne-tuned performance of stateof-the-art vision-and-language models; and• L A GOR, an initial SNARE model that looks at two random views of an object, estimates theangle of those views, then makes a referent prediction, which we demonstrate using a robot.
Related Work
Using natural language to work with robot partners is a long-standing goal in robotics [14].
Weargue that internet-scale, pretrained vision-and-language models offer a powerful starting point forhuman-robot collaboration.
Unlike 2D internet images, physical objects can be picked up and movedby robot agents.
Agents can perform information-seeking behaviors on objects to improve theirworld [15, 16, 17] and language understanding [18, 19, 20].
Such world interaction is inextricablefrom language grounding [21], motivating language annotations for higher-ﬁdelity referents thanstatic images.
We introduce SNARE, comprised of language referring expressions for 3D objects, andthe L A GOR model to combine language and multiple object views to achieve better 3D understanding.
SNARE extends lines of work tying language to static images, 3D object models, and even physicalobjects.
We demonstrate that L A GOR generalizes to physical objects manipulated by a tabletoprobot §5.2. L A GOR is inspired by models that perform information-seeking actions informed bylearned world models to better achieve goals.
Image-based Object Identiﬁcation Object classiﬁcation [3, 2] is the ﬁrst step towards image captioning and visual question answering [22].
Particular object instances can be found with region segmentation models such as MaskRCNN [23], enabling object referent tasks such as GuessWhat!?
Combining visual recognition with Transformer-based language understanding to jointly attend tolanguage and visual tokens leads to improved downstream performance on many vision-and-languagetasks [6, 25, 8].
Joint embedding approaches that learn a shared subspace for representing language and vision tokens [26] also achieve state-of-the-art performance when trained at scale [1].
We showthat these large-scale, pretrained models fall short of human performance on the SNARE task, andthat synthesizing 2D views from multiple vantage points improves object identiﬁcation performance.
Language Grounding in 3D Prior works have associated single-word attributes with 3D object models based on latent representations of 3D meshes [27].
To learn spatial language, vision-and-languagenavigation (VLN) [28] models infer navigation actions from instructions and visual observations in3D simulated worlds.
Such tasks can be extended to include simulated world interactions such aspicking and placing objects [29] and using appliances and tools [30].
Models for these tasks extend Transformer representations to include action taking [31, 32] and invoke object classiﬁcation methodsto create a semantic understanding of the world [33].
SNARE poses a complementary challenge,providing data for selecting referent objects in the presence of distractors by taking into account themultiple views possible of objects in 3D space.
[Table]
The work most similar to ours is ShapeGlot [34], a collection of referring expressions for ShapeNet objects to discriminatebetween two distractors.
While ShapeGlot is similar in spirit, it contains training data only for chair objects, and testson four additional categories.
By contrast,
SNARE spans 262 distinct object categories (Table 1).
ShapeGlot focuses on amodel’s abilities to learn particular parts ofobjects, such as chair arms and legs, while
SNARE aims to ground language moregenerally to 3D features of objects spanning color, shape, category, and parts.
Further, ShapeGlot models take in a pointcloud representationof objects along with visual features, where our L A GOR model does not assume access to the 3Dmodel of an object and instead operates solely on different 2D views achieved through object rotation.
Physical Object Identiﬁcation Modeling the connections between language to physical objectsenables robots to identify objects for manipulation by color, shape, and category [35, 36].
Thephysical forces and sounds objects make during manipulation actions can also be associated withwords such as rattling and heavy for multimodal understanding beyond vision [19, 37].
Prior workhas gathered language annotations for the YCB Benchmark object set [38] to explore how languagedescriptions provide priors on object affordances [39].
In our tabletop robot experiments, we usecamera views of novel objects to evaluate zero shot transfer of L A GOR to robot camera views of realobjects, which are processed by an off-the-shelf segmentation algorithm.
Information-Seeking Actions Embodiment, whether in simulation or the physical world, affordsagents the opportunity to seek out new information to help with a given task.
By predicting the newinformation that can be gotten from different actions, agents can prospect over potential futures forplanning [40, 41, 42].
In the aforementioned VLN task, predicting what will be seen along differentpotential routes facilitates more efﬁcient navigation [43].
Our L A GOR model estimates the angles ofobject input views, encouraging 3D understanding in the learned object representations.
ShapeNet Annotated with Referring Expressions (SNARE)
We introduce SNARE, as a new benchmark for grounding natural language referring expressions todistinguish 3D objects.
The annotations are collected to complement the ACRONYM 1 [44] graspingdataset and include language that targets both visual and tactile attributes of objects.
Our goal isto enable future research to ground both from multi-view vision, as in this work, and directly fromrobotic grasp contact point data (Section 6).
To construct SNARE, we select a subset of 7 , 897 ACRONYM [44] object models from ShapeNetSem [45, 13].
We obtain over 50,000 natural language referring expressions in English for these object [Figure] models via Amazon Mechanical Turk (AMT)
2 Figure 2 gives an example of referring expressionscollected for two mug objects used to distinguish between the two.
To elicit referring expressions with high speciﬁcity, we frame the annotation as a discriminative task.
AMT workers were presented with two object models side-by-side from the same ShapeNet category,for example two different OutdoorTable object meshes, and asked to complete sentences like
• The way to tell Object A from Object B is that Object A looks like a(n)
• Blindfolded, the way to tell Object A from Object B is that Object B is a(n)
with referring expressions.
We displayed the object models as GIFs rendered to give a 360 rotatingview of each object.
The resulting SNARE task asks models to take in one such referring expressionand decide whether it applies to Object A or
Object B.
Priming in the annotation prompts invokes visual features ( .. .looks like. .. ) and non-visual features( Blindfolded, ... ) in the referring expressions.
By pairing objects from the same ShapeNet category,referring expressions must go beyond categorical information like brown dresser to differentiate oneobject from the other with higher speciﬁcity, similar to ShapeGlot’s choice to use distractor objectsfrom a single training category [34].
We collected six referring expressions per object—three primedto be visual and three primed to be blindfolded.
Priming for visual expressions mentioned the nameof the object, shapes, and colors , while for blindfolded mentioned shapes and parts .
Each referringexpression was vetted through a secondary task on AMT where, given the language expression,workers had to correctly select the referent object.
Every referring expression in SNARE wascorrectly associated to its referent object by a majority of such annotators.
SNARE referring expressions average 4 .
27 words, with blindfolded expressions longer (4 . 95) thanvisual expressions (3 . 63).
We estimated that blindfolded expressions use more shape words (14%vs 5% of all words) while visual use more color words (22% vs 1% of all words), by traversing the WordNet [46] hierarchy for each word and noting whether it is a hyponym of color or shape . Thisdistribution difference is consistent with the priming used to elicit visual and blindfolded expressions.
Each SNARE instance is a tuple of (referring expression, Object A, Object B), and models mustselect which of Object A or B the referent of the natural language expression.
We split these datainstances into train, validation, and test folds by ShapeNet category.
We ensure that closely relatedcategories such as 2Shelves and 3Shelves or DiningTable and AccentTable are withinthe same fold.
For example, all shelf-related and bed-related categories are sorted into the train andtest folds, respectively, so that inter-category information does not leak across folds.
More details canbe found Section 7.3.
Table 1 summarizes the overall data statistics.
Methods
There has been a recent proliferation of off-the-shelf, increasingly large-scale pretrained vision andlanguage alignment models applicable for language grounding in robotics [1, 6, 7, 8, 47].
We set outto answer two questions about such models, using SNARE as a testbed:
• Can existing language and vision models ground language in the SNARE benchmark?
[Figure]
• Do SNARE models generalize to a robot object selection task better than off-the-shelf models?
To answer the ﬁrst question, we ﬁrst train a M ATCH module that learns a predictive head on topof a CLIP [1] backbone, and evaluate this module on single and multiple views of 3D objects.
Toanswer the second question, we introduce the L A GOR model, which uses view estimation as anauxiliary loss while predicting language expression referent objects for SNARE, and evaluate onboth SNARE §5.1 and a physical robot platform §5.2.
L A GOR examines an initial 3D object viewand an additional, post-rotation view, striking a balance between single-view models and those thatattempt to capture each object’s entire 3D structure.
4.1 Language-View Match (M ATCH ) Module
The M ATCH module takes in a language expression L and single view of an object, V i , and produces amatch score s ( L , V i ) .
To decide whether Object A or Object B is the referent of a language expressionon SNARE, we interpret argmax O ∈{ A , B } s ( L , V i , O ) as the model’s prediction.
CLIP [1] serves as a backbone on which we add additional, learnable layers for the SNAREtask.
We use CLIP’s transformer-based sentence encoder to extract language features l : R 1 × 512 .
We use CLIP ViT-B/32 to extract visual features for image V i , A , v i,A : R 1 × 512 , and image V i , B ,v i,B : R 1 × 512 .
These modality-speciﬁc feature vectors are concatenated: [ v i,A ; l ] : R 1 × 1024 and[ v i,B ; l ] : R 1 × 1024 and independently run through a learnable, multi-layer perceptron that graduallyreduces the dimensionality from 1024 to 512, then 256, and ﬁnally to a single-dimensional scores .
This ﬁnal score comparison training mirrors the multiple-choice formulation used in existingunimodal [5] and multimodal transformers [7].
We keep the CLIP encoders frozen during training.
ShapeNet objects are 3D meshes, incompatible with the 2D input expected by off-the-shelf vision andlanguage models.
We sample eight rendered viewpoints around each object at 45 degree increments,and compare off-the-shelf CLIP and M ATCH module accuracies when considering single or multipleof these views. To aggregate multiple views, we maxpool over view embedding vectors v i,A ... v j,A .
M ATCH is trained with cross-entropy-loss, L s , to predict a binary label of whether the referringexpression matches the object represented in the view image. M ATCH is trained for 50 epochs on SNARE, with object views chosen at random during each step. Baselines.
We compare our ﬁne-tuned M ATCH against a zero-shot CLIP classiﬁer. Instead of ﬁnetuning CLIP, we use the cosine distance between visual and language features to pick the referredobject. That is, we select argmax O ∈{ A , B } l · v i,O . We also evaluate against a trained ViLBERT baselinethat consumes multiple object views at once, but ﬁnd that it does not perform as well as the M ATCHmodule; more details about the ViLBERT baseline can be found in Section
CLIP Lang. CLIP ViT-B/ 32 “the chair with skinny legs” CLIP ViT-B/ 32 Init ial A B {+2 3 4 5 6 7 2 3 4 5 6 7view = 2 view = 8 Init ial CLIP ViT-B/ 32 CLIP ViT-B/
+}
CLIP Lang. CLIP ViT-B/ “the chair with skinny legs”
CLIP ViT-B/
Initial
A
B                view =view =
Initial
CLIP ViT-B/
CLIP ViT-B/   view =view = can be thought of as a ﬁne-tuned CLIP model; rather than back-propagating through the CLIPrepresentation itself we learn additional tuning layers on top
4.2 Language Grounding through Object Rotation (L A GOR)
A robot tasked with retrieving an object given a natural language expression should not have togather a 360 degree view of each candidate referent object to make a decision.
We have the intuitionthat estimating the input viewing angles can enable a model to develop a global reference frame forlanguage grounding against 3D objects (Figure 3).
Thus, we propose L A GOR, which takes in twoviews of each candidate object, performs view estimation on those views as an auxiliary prediction(Figure 4), then predicts the language referent.
In addition to predicting the object referent using a pre-trained M ATCH module that considers twoviews, L A GOR predicts the input views of each object using a cross-entropy loss, L v , against a1-hot vector representing a discrete set of 8 views at 45 degree offsets.
L A GOR learns a multi-layerperceptron that takes in a visual embedding V i and reduces dimensionality from 512 to 256 to 128 to64 to a vector of 8 logits representing the 8 discrete object views.
The M ATCH loss, L s , is combinedwith the view estimation loss for a ﬁnal loss function L =
L v + 0 .
2 ∗ L s .
4.3 Robot Demonstration
We evaluate L A GOR trained on SNARE on a robot platform, taking two random views of objectsand evaluating the robot’s ability to select which is the correct referent of a language expression(Section 5.2).
We compare L A GOR to the off-the-shelf CLIP model that considers a single view ofthe object only, before performing any view-gathering rotations.
We set two objects on the workspace of a Franka Emika Panda with a wrist-mounted Intel Realsense
D414.
3 We capture an initial view of the scene, then segment the objects from the workspace usingthe Unseen Object Clustering [48] algorithm.
We feed the segmented objects as the initial views to L A GOR and CLIP.
We move the arm to a second vantage point above the workspace and captureanother image of the objects and repeat segmentation to obtain the second views for L A GOR. [Table] [Table]
well above random chance, and adding more object views improves all models’ performance. L A GOR, which needs only two views, making it reasonable to deploy on a robot platform, outperforms two-view M ATCH statistically signiﬁcantly on both the validation and training set, by adding a level of 3D object understanding in the form of view estimation auxiliary losses.
Human accuracy is conservatively calculated as the number of SNARE instances for which the correct referent object was identiﬁed by every voting annotator ( U nanimous); for all SNARE instances a majority of voting annotators correctly selected the referent ( M ajority).
Results
In this section we show that CLIP [1] and ViLBERT [6] select correct 3D object referents in SNAREsubstantially less often than the M ATCH and L A GOR models.
Then, we deploy CLIP and L A GORon a robot tasked with selecting objects conditioned on natural language referring expressions, andﬁnd that view estimation and gathering additional object viewpoints improves referent identiﬁcation.
5.1 Performance on SNARE
We train all models for 50 epochs on SNARE, with object views chosen at random during each step,and report the best validation performance across those epochs, as well as the test performance atthat best validation epoch checkpoint.
We train each model 10 times with a different random seed toestimate performance variance across training runs.
Table 2 gives L A GOR accuracy compared to other two-view alternatives, as well as models taking inall views or single views.
We compare to existing models trained (ViLBERT) and zero-shot (CLIP), asrecommended in their respective papers.
We do not re-run ViLBERT training because it is extremelytime-consuming; the same reason we build on CLIP for our model development. To compare models,we perform an unpaired, Welch’s two-tailed t -test on overall accuracy considering both visual andblindfolded subsets. We perform ﬁve such tests in all: for validation and test, L A GOR against twoview M ATCH , L A GOR against all-view M ATCH , and one additional pooling test (Section 7.5). Weperform a conservative Bonferroni multiple-comparison correction applied to threshold of p < 0 .
L A GOR statistically signiﬁcantly outperforms two-view M ATCH on both the validation ( p = 0 .
0008)and test ( p = 0 . 0055) sets.
The modelling difference between L A GOR and M ATCH with two viewsonly the view estimation auxiliary losses.
L A GOR averages higher accuracy than the M ATCH thatuses all views on every metric; however, these differences are signiﬁcant for neither the validation( p = 0 . 0844) or test ( p = 0 . 0315) set.
While no models achieve human level accuracy, the performance difference is particularly strikingon the blindfolded referring expression subset of data, which lags 5-15% behind visual referringexpressions across models.
That gap supports our intuition that blindfolded referring expressionscapture a complementary and challenging linguistic space currently understudied for vision-languagemodels but key to everyday manipulation.
Thereby, SNARE opens interesting avenues for futurework exploring the shape and grasp-points of objects. [Figure]
Notably, zero-shot CLIP performance on SNARE, even when considering only one random view ofeach object, is well above random chance.
While M ATCH ﬁne-tuning and L A GOR auxiliary lossesimprove accuracy over zero-shot CLIP, the zero-shot performance is a good indicator that massivelypre-trained language and vision alignment like CLIP could be useful for robotics tasks for whichthere is little or no in-domain data.
5.2 Robot Results
We performed a set of experiments on a real robot (Figure 5).
We run single view, zero-shot CLIPagainst L A GOR on 11 referring expressions and pairs of objects.
Two objects at a time were placedon the table, and we selected two arbitrary viewing angles for each object. We used unknown objectinstance segmentation [48] to segment out each object.
Zeroshot CLIP correctly selects the referentobject 7 out of 11 times, while L A GOR selects the correct referent 9 out of 11 times.
L A GORcorrects CLIP’s mistakes on expressions like Mug with the thin metal handle where multiple viewsexpose potentially occluded parts like handle , as well as when information gathering makes decisionsmore conﬁdent, such as Orange box when both objects have orange components, but one object isorange from all views. These experiments introduce a number of new sources of noise not presentin the simulated setting, for example changes in camera angle, shifting lighting during rotation,and automatic segmentation to account for cluttered scenes. For example, segmentation leads tothe creation of voids inside of objects not present in SNARE.
View extraction details, referringexpressions, and extracted views are shown in Section 7.6.
Conclusions
We introduce S hape N et A nnotated with R eferring E xpressions (SNARE), a challenge task to groundlanguage to 3D object models.
We show that ﬁne-tuned, massively pretrained vision and languagemodels fall short of human performance at identifying object referents of natural language expressionsby a wide margin, while still achieving well above random chance performance (Table 2).
Modelsimprove as more 2D views of 3D objects are available, but a robot can only capture one view at atime.
We introduce La nguage G rounding through O bject R otation (L A GOR), a model that considerstwo views of candidate referent objects trained with current view estimation as an auxiliary loss, andbuilds on a pre-trained CLIP model.
We transfer the trained L A GOR model to a physical robot tostudy its generalization to noisy images of physical objects.
In the future, a rotation policy could be learned to examine up to N views across M candidate objectssubject to a referring expression, rather than performing a random rotation to a novel view.
Byassigning rotation actions expected discriminative values using a trained rotation module (thoughthat could introduce a cyclic dependency with the M ATCH module), and penalties based on thetime it takes a particular hardware to achieve an object rotation, one could learn a POMDP policyaggregating object views seen so far to decide whether to next obtain a new view or make a guessabout the referent object.
Obtaining different object views could be done either by picking up androtating the object or by moving the robot camera to see the object from another angle.
Further, since3D meshes are available for each object, it may be possible to extract shape-level information using a PointNet [49], similar to experiments attempted by ShapeGlot [34].
Because we target physical robotapplications, we may be able to tie language to sets of graspable points encoding gripper orientationand position [44] for each object.