LanguageRefer: Spatial-Language Model for 3D Visual Grounding
Abstract: For robots to understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that comprehend referential language to identify common objects in real-world 3D scenes.
In this paper, we introduce a spatial-language model for a 3D visual grounding problem.
Speciﬁcally, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identiﬁes the target object from a set of potential candidates.
Speciﬁcally, LanguageRefer uses a transformer-based architecture that combines spatial embedding from bounding boxes with ﬁne-tuned language embeddings from DistilBert [1] to predict the target object.
We show that it performs competitively on visio-linguistic datasets proposed by ReferIt3D [2].
Further, we analyze its spatial reasoning task performance decoupled from perception noise, the accuracy of view-dependent utterances, and viewpoint annotations for potential robotics applications.
Project website: https://sites.google.com/view/language-refer.
Introduction
For robots to communicate seamlessly with humans to perform meaningful tasks in indoor environments, they must understand natural language utterances and ground them to real-world elements.
Several recent advances have combined language and visual elements, producing methods for tasks such as visual question and answering (VQA) involving spatio-temporal reasoning tasks [3, 4, 5], embodied QA [6], and pre-training for visual recognition tasks with language descriptions [7].
Further, embodied agents can follow visually grounded language instructions to perform embodied tasks [8, 9].
However, for real robots to intelligently perform these tasks, we need 3D representations from raw sensor data; ReferIt3D [2] proposes a benchmarking dataset of language utterances referring to objects in 3D scenes from the ScanNet [10] dataset.
Our long-term goal is to enable robots to visually navigate indoor environments based on referential instructions.
In this paper we take a step towards this goal by leveraging the ReferIt3D dataset to build a model that can identify the 3D object referred to in a language utterance.
Referential language to identify an object in real-world 3D scenes poses a challenging problem.
Consider the sample utterance “Facing the foot of the bed, the bed on the right” along with a 3D scene as shown in Figure 1.
Humans can easily follow the language clues, infer the point of view of the speaker, locate all the referenced elements, and spatially reason to locate the bed in the scene despite two instances of beds .
However, viewpoint prediction, object identiﬁcation, and spatial reasoning remain open-ended research problems in robotics and vision.
The 3D reference task proposed by ReferIt3D is difﬁcult because: (1) reconstructed 3D scenes of the real world in the
ScanNet dataset are noisy and lack ﬁne details compared to 2D images or rendered 3D scenes, (2) ﬁne-grained class labels and expressions in natural language utterances are diverse and not exactly matched, (3) view-dependent utterances often require guessing the original viewpoints, which deters the model from properly learning spatial concepts, and (4) the combined complexity of multiple challenges complicates efforts to analyze what the model learns or understands.
Inspired by the success of the methods in [11, 12] on CLEVR and CLEVRER domains for the spatial reasoning task, we hypothesized that for the 3D reference task, decoupling the spatial-reasoning from the perceptual task of identifying the objects in the 3D scene would improve performance and clearly track the role of perception noise in performance.
More speciﬁcally, instead of developing an integrated multi-modal perception system, we assumed a pre-trained instance classiﬁcation model or ground-truth classes for the objects that informed the spatial reasoning task.
We focused on how the language model with spatial information could handle the reference task.
The ReferIt3D dataset includes two sub-datasets containing natural and synthetic language utterances, namely Nr3D and Sr3D, respectively.
In our experiments, our model achieved comparable scores on both with predicted instance class labels.
We observed high accuracy with ground-truth labels in Sr3D, which indicates that our model better understood template-based language data.
Since our pipeline is modular and features multiple models (perceptual, spatial embedding, pre-trained language embedding, and spatial-language), our approach is ﬂexible and adaptable to different environments and object entities.
Another aspect of the 3D reference task is viewpoint prediction.
The ReferIt3D dataset contains utterances that can be grouped into view-independent (VI) and view-dependent (VD) categories.
An example of a VI utterance is “The lamp closer to the white armchair” and a VD utterance is “The lamp on the right in-between the beds” .
The VD utterance requires viewpoint prediction.
This distinction is crucial for robotics applications where the agent must infer the viewpoint to which the speaker is referring.
In the ReferIt3D dataset, some VD utterances lack information to guess the valid orientation of the agent (who utters the language description), which prevents a model from understanding the signiﬁcance of spatial relationships such as ‘left of.’
This is due to the annotation process of ReferIt3D datasets, where both a speaker and a listener can freely rotate the scene to infer an ill-deﬁned orientation in the utterance.
Human annotators who are aware of spatial concepts tend to validate otherwise arbitrary orientations.
However, a data-driven model will suffer from degraded learning when it cannot verify orientations as well as human annotators.
Viewpoint-free annotations may suit most robotics applications; nonetheless, predicting or verifying whether a given statement and the viewpoint match remains important.
To better train and investigate the agent in view dependencies, we provide an extra collection of orientation annotations for VD utterances and compare the models with and without viewpoint correction from them.
To summarize, the paper contributes: (1) a novel transformer-based spatial-language model in a modular pipeline that better understands spatial relationships in a 3D visual grounding task.
We show that our model achieves comparable performance with state-of-the-art methods on the Nr3D and Sr3D datasets.
(2) analysis of the ReferIt3D dataset with viewpoint orientation annotations to remove potential artifacts from implicit orientations.
(3) ablation and additional experiments with ground-truth classes that decouple the impact of perception noise in the spatial reasoning task.
Related Work 2.1 Vision-and-Language Navigation and Robot Navigation
Vision-and-language navigation (VLN) has been extensively studied and made remarkable progress over the last few years ([13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]).
Given a language instruction in the simulation environment, the goal is for an agent to reach the desired node on the pre-deﬁned traversal graph using images as input.
The literature offers several extensions.
ALFRED [8] proposes an extended VLN task that grounds a sequence of sub-tasks to achieve a higher level task in the AI2Thor environment [9].
ALFRED navigation sequences have (implicit) goals that are often close to objects of interest in the subsequent sub-tasks, e.g., when an agent is asked to move in front of the sink because it is going to clean a cup there.
With high-level semantic tasks, object-centric spatial understanding is of even greater importance.
In another direction, recent approaches have relaxed the constraint of discrete traversal in VLN into continuous space ([24, 25, 26, 27, 28]).
Here, an agent encounters more complex tasks involving time and space.
Thus, expanding the space representation to 3D can be an effective solution.
In the context of VLN, we consider the 3D visual grounding task as a proxy for 3D indoor navigation that includes the full observability assumption and goal-oriented language descriptions.
In particular, our approach focuses on understanding spatial relationships among objects, which plays a key role in VLN and robot navigation.
2.2 2D and 3D Visual Grounding
The 2D visual grounding task localizes an object or region in an image given a language description about the object or region ([29, 30, 31]).
Most methods use two-stage approaches: they ﬁrst generate proposals and then compare the proposals to the language description to choose the grounded proposal ([32, 33, 34, 35, 36]).
The 3D visual grounding task localizes a 3D bounding box from the point cloud of a scene given a language description.
Recently, ReferIt3D [2] and ScanRefer [37] were proposed as datasets for 3D visual grounding tasks, with language annotation on the ScanNet [10] dataset.
Most 3D grounding approaches ([2, 37, 38, 39, 40]) follow a two-stage schemes similar to many 2D visual grounding tasks.
First, multiple bounding boxes are proposed or the ground-truth bounding boxes are used, and then features from the proposals are combined or compared with features from the language description.
InstanceRefer [38] extracts attribute features both from point clouds and utterances and compares them to select the object that best matches.
FFL-3DOG [39] matches features from language and point clouds with guidance from language and a visual scene graph.
These methods rely on speciﬁc designs, e.g., bird-eye-view mappings with different types of operations or intensive language pre-processing for graph generation.
In contrast, our approach leverages the language embedding space from the pre-trained language model.
Following the pipeline and general architecture of the language model therefore requires minimal manual design compared to previous works.
SAT [40], like our approach, relies on transformer [41] models; it learns a fused embedding of multi-modal inputs and uses auxiliary 2D images.
In contrast, our approach uses a semantic classiﬁer to predict object class labels and takes these labels as input.
It has a marginal cost of learning fused embeddings compared to training multiple BERT models [42] from scratch in SAT [40].
Even given only semantic information from point clouds, our model still achieved comparable performance with state-of-the-art methods on Nr3D and Sr3D datasets.
In addition, the decoupled perception module makes our approach modular and thus transferable to different data.
Problem Statement and Methodology
Given a 3D scene S with a list of objects ( O 1 , · · · , O M ) and a language utterance U , the problem is to predict the target object O T , T ∈
I M
= { 1 , · · · , M } referred to in the language.
A single object
O i consists of a bounding box B i ∈ B
= R 6 and corresponding point cloud P i ∈ P =
R N i × 6 (xyz positions and RGB values) in the bounding box with N i number of points.
We propose an approach based on language models, called LanguageRefer , to solve a 3D visual grounding task.
Our model focuses on understanding spatial relationships between objects from language descriptions and 3D bounding box information.
We chose this approach due to (1) the high dependency on spatial relationship descriptions in language, and (2) the holistic nature of spatial relationship information, which differs from unary attribute information such as color and shape.
As shown in Figure.
we use a two-stage approach.
First, we determine the class labels of objects in the scene.
Second, we use spatial-language embedding to identify the referenced object.
The following subsections describe these steps in detail.
Semantic Classiﬁcation Model and Tokenization.
For semantic classiﬁcation of the point cloud in a bounding box, we employed PointNet++ [43], which achieved 69% accuracy on average in the test dataset.
In training and inference, we use a sampled point cloud P (cid:48) i ∈ R 1024 × 6 and PointNet++ predicts the semantic label ˆ l ∈ L , where L is a set of class labels in text.
Each scene has pairs of predicted class labels and bounding box values ((ˆ l i , B i ) :
ˆ l i ∈ L , B i ∈ B ) from objects.
Predicted labels are concatenated to the utterance U with a separator [SEP] and then split by a tokenizer into a list of indices of tokens: U becomes ( u 1 , · · · , u t ) , and each predicted class label ˆ l i becomes ( o i 1 , · · · , o i n 1 ) , where each token index is in I D and D is the size of the dictionary.
Language Model and Token Embedding Generation.
Our model uses a pre-trained language model, DistilBert, for the reference task.
Transformers consider relationships among all pairs of elements through attention, and they can be effectively leveraged to explain spatial relationships between objects as discrete entities.
In our formulation, predicted class labels are considered to be sentences, so they are concatenated to the utterance with separation by [SEP].
Therefore, the ﬁnal sequence of token indices would be V =
([CLS], u 1 , · · · , u t , [SEP], o 11 , · · · , o 1 n 1 , [SEP], · · · , [SEP], o M 1 , · · · , o Mn M , [SEP] ) .
Though the token index sequence complies with the speciﬁcation of DistilBert, it violates the number of sentences.
Then, we transform V into the token embedding sequence
W = ( w 1 , · · · , w T ) , w i ∈ R 768 using DistilBert’s word embeddings.
For concise notation, we deﬁne the indices of utterance tokens in V or W as a mask M U =
(2 , · · · , t +1) and the indices of ﬁrst tokens from objects ( o 11 , · · · , o M 1 ) as a mask M O .
We also deﬁne a mask operator [ · ] to manipulate speciﬁc elements in the sequence; for instance, W [ M U ] =
empties all utterance embeddings in
W .
Spatial-Language Model and Spatial Embeddings.
To combine spatial information from raw bounding box values into the token embedding W , we employ sinusoidal positional encoding PE ( · ) from [41] to transform the bounding box vector (center position and size) B i ∈ B ⊂ R 6 to b i =
PE ( B i ) ∈ R 768 , which is then added to W [ M O ] .
The token embedding W is then combined with spatial information and ﬁnally transformed into the output embedding X
= ( x 1 , · · · , x T ) , x i ∈
Rby the reference model, which is ﬁne-tuned from the pre-trained DistilBert.
The ﬁnal reference task is performed by the reference classiﬁer from X [ M O ] , as explained in the following subsection.
Loss Functions .
We use three tasks for training and corresponding loss values.
First, we use the reference loss, L ref , following the original proposal in [2].
We ask the model to choose one object as the target instance from M candidates.
We collect scalar values from X [ M O ] by a linear layer and take the argmax on those values to choose the target instance.
Second, we add a binary target classiﬁcation loss L clf on X [ M O ] to determine whether a given object belongs to the target class.
Last, we employ mask loss from language model pre-training, L mask .
We randomly replace the tokens from nouns in the utterance with a probability of 15 %.
The noun token is replaced by [MASK] with an 80 % chance, by a random token with a 10 % chance, or it remains the same with a probability of 10 %.
Then, the model is asked to recover the original token index.
We expect the model to ﬁll in the replaced tokens in the utterance by understanding the relationship between objects.
We use cross entropy loss for all tasks and compute the ﬁnal loss as
[Equation]
At inference, we followed the approach of InstanceRefer [38] to ﬁlter out objects that do not belong to the predicted target class.
We used an extra DistilBert-based target classiﬁcation model of% accuracy that takes the language utterance as input to predict the target class.
To reduce the chance of removing the true target instance in the ﬁltering process, the top- k class predictions (from the semantic classiﬁer) for each object are compared to the predicted target class.
We use k =throughout the experiments.
For masking loss computation, we extracted nouns in the utterance.
We used ﬂair [44] for part-of-speech (POS) tagging.
Experiments 4.1 Datasets
We evaluated our model on the reference task from ReferIt3D [2] with two datasets, Nr3D (Natural reference in 3D) and Sr3D (Spatial reference in 3D, which contains spatial descriptions only).
Both datasets augment ScanNet [10], a reconstructed 3D indoor scene dataset with language descriptions.
Nr3D has 41,503 natural language utterances, and Sr3D contains 83,572 template-based utterances, on 707 scenes following the ofﬁcial splits of ScanNet [10].
The datasets have 76 target classes and are designed to have multiple same-class distractors in the scene.
4.2 Experiment Settings
ReferIt3D [2] provides the ground-truth bounding boxes of objects in the scene, point clouds of the scene, utterances and corresponding target objects.
We measured the accuracy of the model by comparing the object selected from M candidates to the ground-truth target object.
When the number of same-class distractors exceeded two, we classiﬁed the instance as “hard” according to [2].
The other cases were classiﬁed as “easy.”
We trained models with a learning rate of 0.0001 using AdamW optimization, warm-up and linear scheduling.
Our model was initialized with a pre-trained model of the cased Distilbert base [1] from the Hugging Face implementation [45].
We compared the performance of our model with state-of-the-art methods on ReferIt3D ([2, 40, 37, 38, 39]) based on the reported numbers on the challenge website [46] and corresponding papers.
Since SAT [40] uses an extra 2D image dataset in their training, we separated it from non-SAT, their baseline model that is not trained with the extra dataset.
[Table]
4.3 Evaluation on ReferIt3D
Table 1 shows the accuracy on Nr3D and Sr3D. Our model outperformed other models (without extra training datasets) on both Nr3D and
Sr3D. It achieved 43 .
9% on Nr3D with an +8 .
3% improvement over the baseline method of ReferIt3D [2] and a 2 . 2% increase over the best accuracy from other models in Nr3D. For Sr3D, our model achieved 56 . 0% , which is a 15 . 2% and 8 . 0% increase over the baseline and the best accuracy in the Sr3D section, respectively.
It is also comparable to the accuracy of SAT (with a 1 . 9% difference), which was trained with additional 2D image training data.
These results prove that our model can accurately reason about spatial relationships from spatial-language embeddings and outperforms other models on both datasets despite the loss of information in appearance.
In addition, less diverse language expressions and utterances only about spatial reasoning can explain our model’s strong performance on Sr3D.
4.4 Evaluation with Ground-Truth Class Labels
We further investigated the model’s spatial reasoning ability by removing noise in class labels.
We replaced predicted class labels with ground-truth class labels, which was possible due to the explicit usage of class labels in our model.
Table 2 shows the result with and without ground-truth class labels.
The ﬁrst and second columns demonstrate the type of dataset in training and evaluation, respectively.
For instance, the second row shows the evaluation result of the model trained with the Nr3D dataset with ground-truth class labels and evaluated on the Nr3D dataset with predicted class labels.
When we trained and evaluated our model with ground-truth labels on Nr3D and Sr3D (row 4 and 8), we achieved 54 .
3% and 91 .
1% accuracy, respectively.
Compared to the accuracy of models trained and evaluated with noisy labels, we realized an improvement is 10 .
4% and 35 .
1% , respectively, for each dataset.
When we evaluated with ground-truth labels the models trained with noisy labels, their performance increase was 9 .
7% and 24 .
2% , respectively.
Multiple reasons account for the difference in performance on Nr3D and Sr3D: diversity in the natural language dataset, a higher portion of view-dependent utterances or view-dependent utterances with no mention of orientation, and descriptions other than spatial relationships, such as color or appearance.
In addition, we examined the accuracy given switched datasets, namely, when we train a model on
Nr3D and evaluate it using Sr3D, and vice versa.
The last two rows in Table 2 show two switched evaluation results: 40 .
0% and 37 .
6% on Sr3D and Nr3D, respectively.
The 37 . 6% accuracy on Nr3D from the model trained on Sr3D is comparable to the accuracy of non-SAT [40] and InstanceRefer [38].
It shows our model’s generalizability of spatial reasoning and potential to transfer to different perception modules and datasets.
[Table]
4.5 Ablation on Loss Terms
We trained models with different combinations of loss terms, and Table 3 shows the results.
In addition to three tasks, we examined the effect of a text classiﬁcation task that predicts the target class label l ∈ L from the tokens from utterance X
[ M U ] .
We found that only the binary classiﬁcation loss shows its effect clearly ( +3 .
5% from Ref.
to Ref.-
Clf., +3 .
9% from Ref.-Mask to Ref.-Mask-Clf.
, +1 .
0% from Ref.-Text to Ref.-Text-Clf.).
The mask loss was not effective, and the text classiﬁcation loss degraded accuracy.
One hypothesis here is that the text classiﬁcation loss does not help the model to transform the initial language embedding to a spatial-language embedding.
The model may not require strong language constraints because (1) it already has a good initial language embedding, and (2) the text classiﬁcation loss is independent of the scene’s spatial conﬁguration.
Due to the performance drop, we chose our model without the text classiﬁcation loss.
[Table]
4.6 Viewpoint Annotation
View-dependent utterances without information about original viewpoint make the reference task in
ReferIt3D [2] more challenging.
For instance, utterances such as “ The door is wood with the handle on the left side. ” assume speciﬁc orientations of the agent, and it is impossible to recover the true orientation without knowing the referred object; this differs from view-dependent utterances with explicit viewpoint information, such as “ Facing the foot of the bed. ”
However, the original dataset
Correction
[Table]
Easy
Hard
[Table]
dep
[Table]
Training
Evaluation
[Table]
[Figure]
of ReferIt3D
[2] does not distinguish the utterances without orientation information from those with it.
Therefore, we split the view-dependent (VD) utterance category into two subcategories, VDexplicit and VD-implicit, where VD-explicit has explicit viewpoint information in the utterance.
We then collected orientations from human annotators that validated the utterances.
We set four standard orientations assuming the agent is in the room (around the center of the scene) and asked annotators to select all orientations that could be considered valid from the utterance.
Figureshows examples of the four orientations.
We found that four orientations were sufﬁcient to recover the original viewpoints of the speakers.
In total, 12,680 view-dependent utterances of the Nr3D dataset were annotated; from these, 5,942 utterances were classiﬁed as VD-explicit.
For train and test split, 10,206 and 2,474 utterances were annotated, respectively.
From the orientation for view-dependent utterances, we revised the dataset with the corrected orientation; we rotated the scene with respect to the annotation so all scenes remained valid at the canonical orientation.
For view-independent utterances, we randomly rotated the scenes since they were valid in any direction.
Table 4 shows the accuracy values for models trained with and without corrected orientations.
At inference, we evaluated each model with and without corrected orientations, as well.
The second row shows the accuracy of the model that was trained and evaluated with corrected orientations.
Its overall accuracy was improved by +5 .
1% from the ﬁnal model without the correction that was used in Table 1 (the last row in Table 4).
Note that the improvement on view-independent utterances was marginal ( +1 . 4% ), but the improvement on view-dependent ones was signiﬁcant ( +12 .
The ﬁrst row shows the accuracy of a model trained with corrected orientations and evaluated on the test data without correction.
This model achieved accuracy comparable to the ﬁnal model ( − 0 .
This implies the correction helped the model to accurately interpret the view-dependent scene when the orientation was consistently aligned, introducing no unwanted bias.
Conclusion
We proposed LanguageRefer, a spatial-language model for 3D visual grounding in a reference task.
LanguageRefer combines language embeddings from utterances and class labels with positionalencoded spatial information for efﬁcient learning of the spatial-language embedding space without different modules for individual modality.
Experimental results show that LanguageRefer outperformed state-of-the-art models on ReferIt3D with no additional training data.
Analysis and ablations we performed demonstrate the effects of 1) noisy class labels, 2) arbitrary viewpoints in view-dependent utterances, 3) and the transfer of our model to different datasets for future robotics applications. Acknowledgments
We would like to thank to Xiangyun Meng and Mohit Shridhar for discussion and feedback.
This work is in part supported by NSF IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA
W911NF-15-1-0543, Honda Research Institute as part of the Curious Minded Machine initiative, and gifts from Allen Institute for Artiﬁcial Intelligence.