LanguageRefer: Spatial-Language Model for 3D Visual Grounding
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
To realize robots that can understand human instructions and perform meaningful tasks in the near future, it is important to develop learned modelsthat can understand referential language to identify common objects in real-world3D scenes.
In this paper, we develop a spatial-language model for a 3D visualgrounding problem.
Speciﬁcally, given a reconstructed 3D scene in the form ofa point cloud with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model identiﬁes thetarget object from a set of potential candidates.
Our spatial-language model uses atransformer-based architecture that combines spatial embedding from boundingbox with a ﬁnetuned language embedding from DistilBert [1] and reasons amongthe objects in the 3D scene to ﬁnd the target object.
We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D [2].
Weprovide additional analysis of performance in spatial reasoning tasks decoupledfrom perception noise, the effect of view-dependent utterances in terms of accuracy, and view-point annotations for potential robotics applications. Projectwebsite: https://sites.google.com/view/language-refer .
Introduction
For a robot to communicate seamlessly with humans to perform meaningful tasks in indoor environments, it should understand the natural language utterances and ground it to the elements in the realworld.
Recently, several advancements have been made to combine language with visual elementsresulting methods for tasks such as visual question and answering (VQA) involving spatio-temporalreasoning tasks [3, 4, 5], embodied QA [6], and pre-training for visual recognition tasks with lan- guage descriptions [7].
Embodied agents are able to follow visually grounded language instructions to perform embodied tasks [8, 9].
To enable a real robot to perform these intelligent tasks, it is critical to use 3D representation from raw sensor data.
ReferIt3D [2] proposes a benchmarking datasetof language utterances referring to objects in 3D scenes from ScanNet [10] dataset.
Our long-termgoal is to enable robot to visual navigate in an indoor environment based on referential instructions.
In this paper we take a step towards it by exploiting the ReferIt3D dataset to learn a model that canidentify the 3D object referred in a language utterance.
Referential language to identify an object in real-world 3D scenes is a challenging problem.
Consider an example utterance - “Facing the foot on the bed, the bed on the right” along with a 3D sceneas shown in Figure 1.
It is easier for a human to follow the clues in the language, infer the point ofview of the speaker, locate all the elements referred to, and spatially reason to locate the bed in thescene despite of two instances of beds .
However, view-point prediction, object identiﬁcation, andspatial reasoning are still open-ended research problems in robotics and vision.
The 3D referencetask as proposed by ReferIt3D is challenging because of four reasons: a) reconstructed 3D scenesof the real world in ScanNet dataset are noisy and lacks ﬁne details in comparison to 2D images orrendered 3D scenes, b) ﬁne-grained class labels and expressions in natural language utterances arenot exactly matched and diverse, c) view-dependent utterances often require guessing the originalview-points which hinders model to properly learn spatial concepts, and d) combined complexity ofmultiple challenges makes it hard to analyze what model learns or understands.
We are inspired by the success of the methods [11, 12] on CLEVR and CLEVRER domains inspatial reasoning task and hypothesize that in the 3D reference task, if the spatial-reasoning task isdecoupled from the perceptual task of identifying the objects in the 3D scene, we can achieve betterperformance and clearly track the contribution of perception noise in performance.
More specifically, instead of developing integrated multi-modal perception system, we assume a pre-trainedinstance classiﬁcation model or ground-truth classes for the objects that can inform the spatial reasoning task.
We focus on how the language model with spatial information can handle the referringtask.
The ReferIt3D dataset includes two sub-datasets containing natural and synthetic languageutterances, namely Nr3D and Sr3D respectively.
In our experiments, our model is able to achievedcomparable scores on Nr3D and Sr3D with predicted instance class labels.
We observe high accuracy with ground-truth labels in Sr3D which indicates that the model has better understanding fromtemplate-based language data.
Our pipeline is modular with multiple models (perceptual model, spatial embedding model, pre-trained language embedding model, and spatial-language model).
Thismodular approach gives ﬂexibility and can adapt to environments with different appearance, andobject entities.
Another aspect of the 3D reference task is the view-point prediction.
The ReferIt3D dataset containsutterances that can be grouped into view-independent (VI) and view-dependent (VD).
An exampleof VI utterance is “The lamp closer to the white armchair” , and VD utterance is “The lamp onthe right in between the beds” .
The VD utterance requires view-point prediction.
This is crucialfor an application in robotics where the agent has to infer view-point with respect to which thespeaker is referring to.
In the ReferIt3D dataset, some of VD utterances do not have information toguess the valid orientation of the agent (who utters the language description).
It prevents a modelin understanding the signiﬁcance of spatial relationships such as ‘left of.’
This is due to annotationprocess of ReferIt3D datasets where both a speaker and a listener can freely rotate the scene to inferthis ill-deﬁned orientation in the utterance.
Human annotators who are aware of spatial conceptstend to validate arbitrary orientations.
However, a data-driven model will suffer from degradedlearning when it is not capable of doing veriﬁcation just like annotators.
Viewpoint-free annotationsmay not well ﬁt to most robotics applications; but still it is important to predict or verify whetherthe given statement and the viewpoint match.
For better training and investigation of the agent inview-dependencies, we provide an extra collection of annotation of orientations for VD utterancesand compare the models with and without the view-point correction from annotations.
Overall, the contributions of the this paper include: 1) a novel transformer based spatial-languagemodel in a modular pipeline for better understanding of spatial relationships in a 3D visual groundingtask.
We show that our model achieves comparable performance with the state-of-the-art methodson Nr3D and Sr3D datasets, 2) we provide analysis of the ReferIt3D dataset with viewpoint orientation annotations to remove potential artifacts from implicit orientations, 3) ablation and additionalexperiments with ground-truth classes which decouple the impact of the perception noise in spatialreasoning task.
Related Work2.1 Vision-and-Language Navigation (VLN) and Robot Navigation
Vision-and-language navigation (VLN) has been extensively studied and achieved remarkableprogress ([13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]).
Given a language instruction in the simulation environment, the goal is to make an agent reach the desired node on the pre-deﬁned traversalgraph from images.
A few extensions were proposed in the literature.
ALFRED [8] proposes anextended VLN task that grounds a sequence of sub-tasks to achieve a higher-level task in AI2Thorenvironment [9].
In ALFRED, navigation sequences have (implicit) goals which are often close toobjects of interest in next sub-tasks.
For instance, the agent is asked to move in front of the sinkbecause it is going to clean the cup in the sink.
With high-level semantic tasks, object-centric spatialunderstanding is more of importance.
In another direction, recent approaches have relaxed the constraint of discrete traversal in VLN intocontinuous space ([24, 25, 26, 27, 28]).
Here, the agent needs to deal with higher task complexity involving time and space.
Thus expanding the space representation to 3D can be an effective solution.
In context of visual language navigation, we consider the 3D visual grounding task as a proxy of3D indoor navigation with fully observability assumption and goal-oriented language descriptions.
Especially, our approach focuses on understanding spatial relationships among objects which playskey role in VLN and robot navigation.
2.2 2D and 3D Visual Grounding
2D visual grounding is a task to localize an object or region in the image given a language descriptionabout the object or the region ([29, 30, 31]).
Most methods are two-stage approaches where they ﬁrstgenerate proposals and choose the grounded proposal by comparing the proposals to the languagedescription ([32, 33, 34, 35, 36]).3D visual grounding is to localize a 3D bounding box from the pointcloud of a scene given a language description.
Recently, ReferIt3D [2] and ScanRefer [37] were proposed datasets for 3D visualgrounding task with language annotation on ScanNet [10] dataset.
Most 3D grounding approachesare ([2, 37, 38, 39, 40]) following two-stage schemes as many approaches do in 2D visual groundingtask; ﬁrst, propose multiple bounding boxes or use the ground-truth bounding boxes and second,features from proposals are combined or compared with features from language description.
InstanceRefer [38] extracts a few attribute features both from pointclouds and utterance and comparethem to select the best matching object.
FFL-3DOG [39] does the feature matching from languageand the pointclouds in guidance with a language and a visual scene graph.
These methods relyon speciﬁc designs such as bird-eye-view mapping with different types of operations, intensivelanguage preprocessing for graph generation.
In contrast, our approach leverages the language embedding space from the pre-trained language model. As we follow the pipeline and most of thearchitecture of the language model, it requires minimal manual design in comparison to previousworks. SAT [40] is similar to our approach that relies on Transformer [41] models. It learns afused embedding of multi-modal inputs and used auxiliary 2D images.
Instead, our approach takesa semantic classiﬁer to predict object class labels and use these labels as input. Our approach hasa marginal cost of learning fused embeddings in comparison to training multiple BERT [42] models from scratch in SAT [40]. Even only with semantic information from pointcloud, our modelachieved comparable performance with the state-of-the-art methods on Nr3D and Sr3D datasets. Inaddition, the decoupled perception module makes our approach modular and transferable to differentdata.
Problem Statement & Methodology
Given a 3D scene S with a list of objects ( O 1 , · · · , O M ) and a language utterance U , the problem isto predict the target object O T , T ∈
I M
= { 1 , · · · , M } referred to in the language.
A single object
O i comprises a bounding-box B i ∈ B
= R 6 and corresponding pointcloud P i ∈ P =
R N i × 6 (xyzpositions and RGB values) within the bounding-box with N i number of points.
We propose a language model based approach, LanguageRefer , to solve a 3D visual grounding task.
Our model focuses on understanding spatial relationship between objects from language description and 3D bounding-box information due to 1) high dependency on spatial relationship description inlanguage and 2) holistic nature of spatial relationship understanding different from unary attributessuch as color and shape.
As shown in the Figure.
2, we use two-stage approach: 1) determine theclass labels of objects in the scene, 2) determine the referred object based on inputs and predictionswith spatial-language embedding.
In the following subsections we will describe these steps in detail.
Semantic Classiﬁcation Model and Tokenization: For semantic classiﬁcation of the pointcloudin a bounding-box, we employed PointNet++ [43] and it achieved 69% accuracy on average in thetest dataset.
In training and inference, we use a sampled pointcloud P (cid:48)
i ∈ R 1024 × 6 and PointNet++predicts the semantic label ˆ l ∈ L where L is a set of class labels in text.
Each scene has pairsof predicted class labels and a bounding-box values ((ˆ l i , B i ) :
ˆ l i ∈ L , B i ∈ B ) from objects.
Predicted labels are concatenated to the utterance U with a separator [SEP], then split into a list ofindices of tokens by a tokenizer: U becomes ( u 1 , · · · , u t ) and each predicted class label ˆ l i becomes( o i 1 , · · · , o in 1 ) where each token index is in I D and D is the size of the dictionary.
Language Model and Token Embedding Generation:
Our model uses pre-trained languagemodel, DistilBert, for the reference task.
Transformers consider relationship among all pairs ofelements through attention and it can be effectively leveraged in understanding spatial relationshipsbetween objects as discrete entities.
In our formulation, predicted class labels are considered assentences so they are concatenated to the utterance with separation by [SEP].
Therefore, the ﬁnalsequence of token indices would be V =
([CLS], u 1 , · · · , u t , [SEP], o 11 , · · · , o 1 n 1 , [SEP], · · · , [SEP],o M 1 , · · · , o Mn M , [SEP] ) .
The token index sequence complies with the speciﬁcation of DistilBert butwith the violation of the number of sentences.
Then we transform V into token embedding sequence
W = ( w 1 , · · · , w T ) , w i ∈ R 768 by the word embedding in DistilBert.
For concise notation, we deﬁne the indices of utterance tokens in V or W as a mask M U =
(2 , · · · , t + 1) and the indicesof ﬁrst tokens from objects ( o 11 , · · · , o M 1 ) as a mask M O .
We also deﬁne mask operator [ · ] to manipulate speciﬁc elements in the sequence.
For instance, W [ M U ] = 0 will empty all the utteranceembeddings in W .
Spatial-Language Model and Spatial Embedding:
In order to combine spatial information fromraw bounding-box value to the token embedding W , we employ sinusoidal positional encoding PE ( · ) from [41] to transform bounding-box vector (center position and size) B i ∈ B ⊂ R 6 to b i =
PE ( B i ) ∈ R [-
-]768 and they are added to W [ M O ] 1 .
Then the token embedding W combined withspatial information is ﬁnally transformed into the output embedding X =
( x 1 , · · · , x T ) , x i ∈ R 768by the reference model ﬁnetuned from the pre-trained DistilBert.
The ﬁnal reference task is done bythe reference classiﬁer from X [ M O ] as explained in the following subsection.
Loss Functions :
In training, we use three tasks for training and corresponding loss values.
First, weuse the reference loss, L ref , following the original proposal in [2].
We ask the model to choose oneobject as the target instance among M candidates.
We collect scalar values from X [ M O ] by a linearlayer and take the argmax on those values to choose the target instance.
Second, we add a binary target classiﬁcation loss L clf on X [ M O ] to determine whether given anobject belongs to the target class.
Last, we employ mask loss from language model pre-training, L mask .
We randomly replace thetokens from nouns in the utterance with probability of 15 %.
The noun token is replaced by [MASK]with chance of 80 %, by a random token with chance of 10 %, or remain the same with probabilityof 10 %.
Then the model is asked to recover the original token index.
We expect the model to ﬁllin the replaced tokens in the utterance by understanding the relationship between objects.
We usecross entropy loss for all tasks.
We compute the ﬁnal loss as follows:
[Equation]
At inference, we followed the approach of InstanceRefer [38] to ﬁlter out objects that do not belongto the predicted target class.
Further, we use top- k predictions for instance class labels in the ﬁlteringapproach to prevent the true target instance from getting rid of.
We use k = 4 through out the experiments.
An accuracy of the target classiﬁcation model was 94%.
For masking loss computation,we extracted nouns in the utterance.
We used ﬂair [44] for part-of-speech (POS) tagging.
Experiments
4.1 Dataset
We evaluated our model on the referring task from ReferIt3D [2].
Our model was evaluated withtwo datasets Nr3D (Natural Reference in 3D) and
Sr3D (Spatial Reference in 3D, contains spatialdescriptions only).
Both datasets augment ScanNet [10], a reconstructed 3D indoor scene dataset,with language descriptions.
Nr3D has 41,503 natural language utterances and Sr3D contains 83,572template-based utterances on 707 scenes following the ofﬁcial splits of ScanNet [10].
The datasetshave 76 target classes and they are design to have multiple same-class distractors in the scene.
4.2 Experiment Settings
ReferIt3D [2] provides the ground-truth bounding-boxes of objects in the scene, pointcloud of thescene, utterances and corresponding target objects.
We measure the accuracy of the model by comparing the selected object from M candidates to the ground-truth target object.
When the number ofsame-class distractors is larger than two, we classify the instance “hard” according to [2].
The othercases are classiﬁed as “easy.”
We trained models with learning rate of 0.0001 with AdamW, warm-up and linear scheduling.
Weinitialize our model with a pre-trained model of distilbert-base-cased [1] from huggingface implementation [45].
We compared the performance of our model with state-of-the-art methods on ReferIt3D ([2, 40, 37,38, 39]) from the reported numbers on the challenge website [46] and corresponding papers.
Since SAT [40] uses extra 2D image dataset in their training, we separate non-SAT, their baseline modelthat is not trained with the extra dataset, from SAT. [Table]
4.3 Evaluation on ReferIt3D
Table 1 shows the accuracy on Nr3D and Sr3D. Our model outperformed other models (withoutextra training dataset) on both Nr3D and
Sr3D. Our model achieved 43 .
9% in Nr3D with +8 .
3%improvement over the baseline method of ReferIt3D [2] and 2 . 2% increase from the best accuracyfrom other models in Nr3D. In Sr3D, our model achieved 56 . 0% which is 15 .
2% and 8 . 0% increasefrom the baseline and the best accuracy in the Sr3D section, respectively.
It is also comparable tothe accuracy of SAT (with 1 . 9% difference) which was trained with additional 2D image trainingdata.
It proves that our model is able to accurately reason spatial relationship from the spatiallanguage embedding and outperforms in both datasets despite the loss of information in appearance.
In addition, less diverse language expression and utterances only about spatial reasoning can explainoutperform in Sr3D.
4.4 Evaluation with Ground-Truth Class Labels
We further investigate the model’s ability in spatial reasoning by removing noise in class labels.
Itwas done by replacing predicted class labels by ground-truth class labels and it was possible due tothe explicit usage of class labels in our model.
Table 2 shows the result with and without ground-truth class labels.
The ﬁrst and the second columndemonstrates the type of dataset in training and evaluation, respectively.
For instance, the secondrow shows the evaluation result of the model trained in Nr3D dataset with ground-truth class labelsand evaluated in Nr3D dataset but with predicted class labels.
When we train and evaluate ourmodel with ground-truth labels in Nr3D and Sr3D (row 4 and 8), we get accuracy of 54 .
3% and91 .
1% , respectively.
In comparison to the accuracy of models trained and evaluated with noisylabels, improvement is 10 .
4% and 35 .
1% for each dataset.
When we use the models trained withnoisy labels and evaluate them with ground-truth labels, they get 9 .
7% and 24 .
2% performanceincrease.
We believe multiple reasons may cause the difference in performance gaps on Nr3D and
Sr3D; diversity in natural language dataset, higher portion of view-dependent utterances or viewdependent utterances without hint of the orientation, description other than spatial relationshipssuch as color or appearance.
In addition, we examine the accuracy in switched dataset; train a model in Nr3D and evaluate in
Sr3D, and vice versa.
Last two rows in Table 2 show the results of two switched evaluation results:40 .
0% and 37 .
6% on Sr3D and Nr3D. Accuracy of 37 .
6% on Nr3D from the model trained on Sr3Dis comparable to accuracy of non-SAT [40] and InstanceRefer [38].
It shows generalizability ofspatial reasoning in our model and potential in transfer to different perception modules and datasets. [Table]
4.5 Ablation on Loss terms
We have trained models with different combinations of loss terms and Table 3 shows the results.
Inaddition to three tasks, we examined the effect of text classiﬁcation task.
The task is to predict thetarget class label l ∈ L from the tokens from utterance, X
[ M U ] .
We found that only the binary classiﬁcation loss shows its effect clearly ( +3 .
5% from Ref.
to Ref.-
Clf., +3 .
9% from Ref.-Mask to Ref.-Mask-Clf.
, +1 .
0% from Ref.-Text to Ref.-Text-Clf.).
Themask loss was not effective and the text classiﬁcation loss degraded the accuracy.
One hypothesison this is that text classiﬁcation loss is not helping the model to transform the initial languageembedding to spatial-language embedding.
The model may not require strong language constraintbecause it already has a good initial language embedding and the text classiﬁcation loss is not usefulin shaping spatial-language embedding space because it is independent to spatial conﬁguration ofthe scene.
Due to the performance drop, we choose our model without the text classiﬁcation loss.
Ref.
Clf.
[Table]
4.6 View-point Annotation
View-dependent utterances without information about original view-point make the reference taskin ReferIt3D [2] more challenging.
For instance, utterances such as “ The door is wood with thehandle on the left side. ” assume speciﬁc orientations of the agent and it is impossible to recoverthe true orientation without knowing the referred object, not like view-dependent utterances withexplicit view-point information such as “ Facing the foot of the bed. ” However, the original dataset Correction Overall
Easy Hard View [Table]
dep [Table] View [Table]
indep. Training Evaluation
[Table] [Figure] of ReferIt3D [2] does not distinguish the utterances without orientation information from those withthe orientation information.
Therefore, we split the view-dependent (VD) utterance category intotwo subcategories,VD-explicit and VD-implicit, where VD-explicit has explicit view-point information in the utterance.
Then we collected orientations that make the utterances valid from humanannotators.
We set four standard orientation assuming the agent is in the room (around the center ofthe scene) and ask the annotators to select all of orientations that can be considered valid from utterance.
Figure 3 shows examples of four orientations.
With the assumption of the agent being insideof the room, we found that four orientations are good enough to recover the original view-pointsof the speakers.
In total, 12,680 view-dependent utterances of the Nr3D dataset were annotated.
From those, 5,942 utterances are classiﬁed as VD-expclit.
For train and test split, 10,206 and 2,474utterances were annotated.
From the orientation for view-dependent utterances, we revised the dataset with the corrected orientation.
For view-dependent utterances, we rotate the scene with respect to the annotation in orderto make all the scenes remain valid at the canonical orientation.
For view-independent utterances,we randomly rotate the scenes since they are valid in any directions.
Table 4 shows the accuracyvalues from models trained with and without corrected orientations.
At inference, we evaluated eachmodel with and without corrected orientations as well.
The second row shows the accuracy of themodel which is trained and evaluated with corrected orientations.
Its overall accuracy was improvedby +5 .
1% from the ﬁnal model without the correction which was used in Table 1 (the last row in
Table 4). Note that the improvement on view independent utterances was marginal ( +1 . 4% ), butthe improvement on view dependent utterances was signiﬁcant ( +12 .
The ﬁrst row describesthe accuracy of a model trained with corrected orientations and evaluated on the test data withoutcorrection. This model achieved comparable accuracy to the ﬁnal model ( − 0 .
It implies thecorrection helped the model to correctly interpret the view-dependent scene when the orientation isconsistently aligned without introducing unwanted bias.
Conclusion
We have proposed LanguageRefer, a spatial-language model for 3D visual grounding in a referring task.
LanguageRefer combines language embeddings from the utterance and class labels withpositional-encoded spatial information for efﬁcient learning the spatial-language embedding spacewithout different modules for individual modality.
Experimental results show that LanguageReferoutperformed state-of-the-art models on ReferIt3D without additional training data.
Analysis andablations demonstrate effects of 1) noisy class labels, 2) arbitrary viewpoints in view-dependentutterances, 3) and transfer of model to different datasets for future robotics applications.