Differentiable Parsing And Visual Grounding of Verbal Instructions for Object Placement
Abstract: Grounding spatial relations in natural language for object placing could have ambiguity and compositionality issues.
To address the issues, we introduce P ARA G ON , a PARsing And visual GrOuNding framework for languageconditioned object placement.
P ARA G ON leverages object-centric relational representations for the visual grounding of natural language.
It parses language instructions into relations between objects and grounds those objects in visual scenes.
A particle-based GNN then conducts relational reasoning between grounded objects for placement generation.
P ARA G ON encodes all of those procedures into neural networks for end-to-end training.
Our approach inherently integrates parsing-based methods into a probabilistic, data-driven framework.
It is data-efﬁcient and generalizable for learning compositional instructions, robust to noisy language inputs, and adapts to the uncertainty of ambiguous instructions.
Introduction
[Figure]
Human-robot-interaction tasks, such as object placement, navigation, and assembly, often require detailed descriptions with spatial relations.
Natural language provides a rich and intuitive interface for human-robot-interaction tasks [1].
Therefore, managing to learn and ground language-described spatial relations enables robots to assist us better.
This research focuses on object placement tasks instructed by natural language.
Humans verbally instruct robots to pick up an object and put it to a speciﬁc place.
The robot generates object placements conditioned by language description and visual observation.
However, spatial relations in natural language can be ambiguous and compositional , causing issues in language grounding.
We focus on two types of ambiguity: positional ambiguity and referential ambiguity .
Positional ambiguity arises from the phenomenon that people tend to describe the directional relations without the precise distance (e.g., “to the left side”).
In addition, we usually need a reference object to describe spatial relations (e.g., “next to the plate”).
When placing an object next to reference objects, the connection between the reference object and placement is indirect, causing it to be difﬁcult to learn object reference grounding and placement generation simultaneously.
Furthermore, reference expressions of objects can be ambiguous, resulting in a reference expression being grounded to multiple semantically identical objects.
It makes the distribution of correct placement multimodal.
We refer to this issue as referential ambiguity.
The compositional structure of language-described spatial relations comes from the compositional nature of the visual scene and natural language.
A complex scene contains multiple basic objects.
To describe the desired state of a complex scene, one can compose many simple sentences for referents and their relations to form a complex language sentence (e.g., the instruction in Fig 1).
This property increases the data required for learning compositional language instructions.
To address the issues, we introduce P ARA G ON , a PARsing And visual GrOuNding framework for language-conditioned object placement.
The core idea behind P ARA G ON is to leverage structures 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand. in linguistic and visual inputs to extract object-centric relations for reasoning and placement generation, and encode those procedures in neural networks for end-to-end training.
It learns to parse language into relational triplets from the grammatical structure.
The triplets consist of subject, relation, and object phrases, e.g., (“plate”, “upper side”, “knife”) in Fig 1.
It then grounds the mentioned objects to the regions in the visual scene.
A graph formed by grounded triplets is fed into a GNN for relational reasoning and generating placements.
The GNN encodes a mean-ﬁeld message passing algorithm to minimize the reverse KL-Divergence for the target distribution.
We further develop a particle version of GNN to capture multimodal distributions.
P ARA G ON integrates parsing-based methods into a probabilistic, data-driven framework.
It exhibits robustness from its data-driven property, as well as generalizability and data-efﬁciency from parsing.
It adapts to the uncertainty of ambiguous instructions using particle-based probabilistic techniques.
The experiments show that P ARA G ON outperforms the state-of-the-art method in languageconditioned object placement tasks in the presence of ambiguity and compositionality.
Related Work
Many works [2, 3, 4, 5, 6, 7, 8, 9, 10] developed solutions for language-instructed object manipulations of robotic systems.
Our research focuses on object placement instructed by language.
In contrast to picking [2, 3], which needs only a discriminative model to ground objects from reference expressions, placing [4, 5, 6, 7, 8] requires a generative model conditioned on the relational constraints of object placement.
Speciﬁcally, it requires capturing complex relations between objects in natural language, grounding reference expression of objects, and generating placement that satisﬁes the relational constraints in the instructions.
Parsing-based robot instruction following [4, 5, 7, 11, 12, 13, 14, 15] parse natural language into formal representations using hand-crafted rules and grammatical structures.
Those hand-crafted rules are generalizable but not robust to noisy language [1].
Among these studies, those focusing on placing [4, 5, 7] lack a decomposition mechanism for compositional instructions and assume perfect object grounding without considering referential ambiguity.
Recently, [9, 10, 8] used sentence embeddings to learn a language-condition policy for robot instruction following, which are not dataefﬁcient and hard to generalize to unseen compositional instructions.
We follow [16] to integrate parsing-based methods into the data-driven framework.
It is robust, data-efﬁcient, and generalizable for learning compositional instructions.
P ARA G ON has a GNN for relational reasoning and placement generation, which encodes a meanﬁeld inference algorithm similar to [17].
Moreover, our GNN uses particles for message passing to capture complex and multimodal distribution.
The idea is to approximate a distribution as a set of particles [18], which provides strong expressiveness for complex and multimodal distribution.
It is useful in robot perception [19, 20, 21], recurrent neural networks [22], and graphical models [23, 24].
Our approach employs this idea in GNN for particle-based message passing.
Overview
We focus on the language grounding for object placement in tabletop object manipulation tasks.
In this task, scenes are composed of a ﬁnite set of 3D objects on a 2D tabletop.
Humans give natural language (cid:96)
∈ L to guide the robot to pick an object and put it at the desired position x ∗ tgt .
The language instruction is denoted as a sequence (cid:96) =
{ ω l } 1 ≤ l ≤ L where ω l is a word, e.g., in Fig.6, (cid:96) = { ω 1 = put , ω 2 = a , . . . } .
A language instruction should contain a target object expression (e.g., “a plate”) to specify the object to pick and express at least one spatial relation (e.g., “next to a silver mug”) for placement description.
The robot needs to ﬁnd the distribution of the target object’s placement p ( x ∗ tgt | (cid:96), z ) conditioned on the language instruction (cid:96) and visual observation z .
We propose P ARA G ON to solve the problem.
It extracts object-centric relations from linguistic and visual inputs for relational reasoning and placement generation; it encodes those procedures in neural networks for end-to-end training.
The pipeline of P ARA G ON is in Fig 2.
P ARA G ON ﬁrst uses the soft parsing module to convert language inputs “softly” into a set of relations, represented as triplets.
A grounding module then aligns the mentioned objects in triplets with objects in the visual scenes.
The triplets can form a graph by taking the objects as the nodes and relations as the edges. [Table]
/JxX7ZPSi7Z5VStTKKo0DWyQbZIi45JFVyQk5JjXByRx7IE3m27q1H68V6/WqdsEYza+QXrLdPXCOgEA==</latexit> p ( x
[Figure] | `, o ) The resulting graph is fed into a GNN for relational reasoning and generating placements.
The GNN encodes a mean-ﬁeld inference algorithm for a conditional random ﬁeld depicting spatial relations in triplets.
P ARA G ON is trained end-to-end to achieve the best overall performance for object placing without annotating parsing and object-grounding labels.
See Appendix B for implementation details.
Soft Parsing
The soft parsing module is to extract spatial relations in complex language instructions for accurate placement generation.
The pipeline is in Fig 4.
Dependency trees capture the relations between words in natural language, which implicitly indicate the relations between the semantics those words express [25, 26].
Thus, we use a data-driven approach to explore the underlying semantic relations in the dependency tree for extracting relations represented as relational triplets.
It takes linguistic input and outputs relational triplets, where the triplets’ components are represented as embeddings.
We ﬁrst introduce the core concepts of triplets and dependency tree, then demonstrate the algorithm.
4.1 Preliminaries
[Figure]
Triplets.
A triplet consists of two entities and their relation, representing a binary relation.
Triplet provides a formal representation of knowledge expressed in natural language, which is widely applied in scene graph parsing [25], relation extraction [27], and knowledge graph [28].
The underlying assumption of representing natural language as triplets is that natural language rarely has higher-order relations, as humans mostly use binary relations in natural language [29].
For spatial relations, two triplets can represent ternary relations (e.g., “between A and B” equals “the right of A and left of B” sometimes).
As such, it is sufﬁcient to represent instructions as triplets for common object-placing purposes. [Table]
/IHz+QO0G5EA</latexit> { }
W e i g h t
e d
S u m Subject Object Relation <latexit sha1_base64="WfTIqXgPiVpvuCRGyIdyZFi5sHk=">AAAB8HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8cK9kOaUDbbTbt0dxN2N0IJ9U948aCIV3+ON/+NmzYHbX0w8Hhvhpl5YcKZNq777ZRWVtfWN8qbla3tnd296v5BW8epIrRFYh6rbog15UzSlmGG026iKBYhp51wfJP7nUeqNIvlvZkkNBB4KFnECDZWevCzpxz+tF+tuXV3BrRMvILUoECzX/3yBzFJBZWGcKx1z3MTE2RYGUY4nVb8VNMEkzEe0p6lEguqg2x28BSdWGWAoljZkgbN1N8TGRZaT0RoOwU2I73o5eJ/Xi810VWQMZmkhkoyXxSlHJkY5d+jAVOUGD6xBBPF7K2IjLDCxNiMKjYEb/HlZdI+q3sXde/uvNa4LuIowxEcwyl4cAkNuIUmtICAgGd4hTdHOS/Ou/Mxby05xcwh/IHz+QO0G5EA</latexit> { } <latexit sha1_base64="WfTIqXgPiVpvuCRGyIdyZFi5sHk=">AAAB8HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8cK9kOaUDbbTbt0dxN2N0IJ9U948aCIV3+ON/+NmzYHbX0w8Hhvhpl5YcKZNq777ZRWVtfWN8qbla3tnd296v5BW8epIrRFYh6rbog15UzSlmGG026iKBYhp51wfJP7nUeqNIvlvZkkNBB4KFnECDZWevCzpxz+tF+tuXV3BrRMvILUoECzX/3yBzFJBZWGcKx1z3MTE2RYGUY4nVb8VNMEkzEe0p6lEguqg2x28BSdWGWAoljZkgbN1N8TGRZaT0RoOwU2I73o5eJ/Xi810VWQMZmkhkoyXxSlHJkY5d+jAVOUGD6xBBPF7K2IjLDCxNiMKjYEb/HlZdI+q3sXde/uvNa4LuIowxEcwyl4cAkNuIUmtICAgGd4hTdHOS/Ou/Mxby05xcwh/IHz+QO0G5EA</latexit> { }
Semantic Features
Dependency Tree
[Figure]
QNkmmGW+xRCa6E1LDpVC8hQIl76Sa0ziU/CEc3Uz9h0eujUjUPY5THsR0oEQkGEUr+X7+VMCf9Ko1t+7OQJaJV5AaFGj2ql
[Figure] Dependency Tree. A dependency tree (shown in Fig. 3) is a universal structure that examines the relationships between words in a phrase to determine its grammatical structure [30].
It uses part-ofspeech tags to mark each word and dependency tags to mark the relations between two words.
A part-of-speech tag [31] categorizes words’ correspondence with a particular part of speech, depending on the word’s deﬁnition and context, such as in Fig.
3, “cup” is a “NOUN”.
Dependency tags mark two words relations in grammar, represented as Universal Dependency Relations [32].
For example, in Fig.
3, the NOUN “cup” is the “direct object (Dobj)” of the VERB “place”.
Those relations are universal. A proper dependency tree relies on grammatically correct instructions, whereas noisy language sentences may result in imperfect dependency trees. Thus, we use the data-driven method to adapt to imperfect dependency trees.
4.2 Method
To make the parsing differentiable, the soft parsing module “softens” the triplet as the attention to the words of linguistic inputs { a γ,l } 1 ≤ l ≤ L , (cid:80) l a γ,l = 1 , γ ∈ { subj , obj , rel } .
We compute the embeddings of components in triplets by the attention-weighted sum of the word embeddings ϕ γ = (cid:80) l a γ,l f CLIP ( ω l ) .
The word embeddings are evaluated using pre-trained CLIP [33] f CLIP .
We use a GNN [34] to operate over the dependency tree from spaCy [35] to encode the structural information into the node features.
For the GNN, the part-of-speech tags are node features, and the dependency tags are the edge attributes.
Then, we feed the node features into three modules to evaluate each word’s weights to indicate how much this word contributes to the components in a triplet.
We use a single layer MLP to compute the attention of subject words { a subj ,l } 1 ≤ l ≤ L . Slot attention is a learnable module that can extract a set of task-dependent features called slots, which is used in discovering multiple objects in a visual scene. To detect multiple spatial relations in compositional instructions, we use slot attention module [36] to get N features and attentions { a n obj ,l } 1 ≤ n ≤ N, 1 ≤ l ≤ L of object phrases, where N is the total number of triplets.
We feed the features of object phrases as the query, and structural features as the key and value of attention, into a multi-head attention layer [37] to get the attention for corresponding relational phrases { a n rel ,l } 1 ≤ n ≤ N, 1 ≤ l ≤ L .
We com- pute the attention-weighted sum of word embeddings from pre-trained language models to get the embeddings of N possible relational triplets, denoted as Ω = {(cid:104)
ϕ subj , ϕ n rel , ϕ n obj (cid:105)}
Nn =1 .
Ideally, each triplet should contain information about single objects and their relations.
However, we discovered that the features of multiple objects and their relations could be entangled in one triplet.
As the disentangled triplets can provide more precise essential information and be more accurate in recovering the sentence, we add sentence recovery to guide extracting disentangled triplets.
We ﬁrst compute the embedding of each token using attention: ˜ ϕ
l = (cid:80) Nn =1 (cid:80) i ∈{ rel , obj } a ni,l ϕ ni + a subj ,l ϕ subj .
We then feed the embedding of tokens to an LSTM to produce the recovered word embeddings: ˜ (cid:96) =
{ ˜ ω l } Ll =1 = f LSTM ( { ˜ ϕ l } Ll =1 ) .
We minimize the L2 loss between generated sequence l and ˜ l as an auxiliary task to improve the soft parsing component.
Visual Grounding
After relation extraction, we ground the mentioned objects to the visual scenes and reason about their spatial relations for placement generation.
We align the visual features of objects and the embeddings of their reference expressions into the same embedding space for object grounding.
We then represent relations in conditional random ﬁelds and encode them into GNNs for reasoning and placement generation.
5.1 Object Reference Grounding
We align the object and subject phrase embeddings into the objects in the scene for visual grounding.
Given the visual inputs, we use a pre-trained object detector (Mask-RCNN [38] or SPACE [39]) to get object bounding boxes { y m } Mm =1 and encode their visual features { z m } Mm =1 using a pretrained CLIP [33].
We then project the object visual features z m and linguistic features in triplets ϕ i , ϕ i ∈ { ϕ subj }∪{ ϕ n obj } Nn =1 into the same feature space via learnable projecting matrices Φ , Ψ .
We evaluate the cosine similarity d cos ( · , · ) with a learnable scaling factor α between the visual and text feature to get the grounding belief: b im =
Softmax m ( α · d cos (Φ z (cid:62) m , Ψ ϕ (cid:62) i )) .
As such, the grounding results of object i in the triplets is a set of object-centric regions with belief { ( b im , y m ) }
Mm =1 .
5.2 Relational Reasoning for Placement Generation
Spatial relations in Conditional Random Field.
The triplets essentially build up a relational graph
G =
( V , E ) with objects as the nodes V and their spatial relations as the edges E .
We use the conditional random ﬁeld (CRF) speciﬁed using G to represent the relations between positions of the context and target objects in language instruction.
The variable for the vertices X =
{ x i | x i ∈ R} i ∈V denotes the grounded position of each context object and the placement position of the target object.
We formulate the CRF as: p ( X | z , Ω) ∝ (cid:81)
( i,j ) ∈E ψ ij ( x i , x j | r ij ) (cid:81) i ∈V / { subj } φ i ( x i , z ) .
ψ ij describes the spatial relations between two objects based on the spatial relation r ij ∈ { ϕ n rel } Nn =1 .
φ i denotes the probability of the context object’s position conditioned on observation, except for the target object (subj phrase).
It is because the grounded position of the target object is only useful for picking rather than placing.
Mean-ﬁeld variational inference approximates the CRF into a meanﬁeld p ( X | z , Ω)
≈ q ( X ) =
(cid:81)
i ∈V q i ( x i ) and optimizes the reversed KL divergence KL( q || p ) to converge to the multimodal distribution [40].
Mean-ﬁeld q factorizes all the variables to simplify the
CRF for more efﬁcient inference.
The equation of Mean-ﬁeld variational inference is: log q ti ( x i ) = [Equation]
+ [Equation]
i
[Equation] i ,
[Equation] +
[Equation]
j ∈ N ( i ) m tji ( x i ) , where q ti ( x i ) receives the message m tji ( x i ) from its neighboring nodes j ∈ N ( i ) and the message from observed node m t obs ,i ( x i ) , forming a message passing algorithm for 1 ≤ t ≤ T .
As such, the mean-ﬁeld q ( X ) will iteratively converge to p ( X | o , Ω) .
Relational Reasoning in Mean-ﬁeld GNN.
To represent complex spatial distributions conditioned on complex linguistic features, we follow [17] to map the spatial variables into high-dimensional feature spaces and learn an approximate message-passing function between these variables. It forms a GNN to conduct reasoning on the conditional random ﬁeld by approximating the mean-ﬁeld message passing algorithm.
To handle multimodal output distributions better, we develop a method based on [22] to represent the factor of mean-ﬁeld q i as particles { ( h ti,k , w ti,k ) } in message passing, where each particle encodes the position of the corresponding object.
The GNN module takes the embeddings of spatial relations r ij = ϕ n rel as the edge attributes to compute the message.
We initialize the particles according to a normal distribution with uniform weights: h 0 i,k ∼ N ( 0 , I ) , w
i,k = 1 /K .
In message passing, we uses a message network f ψ to output new message by the previous inputs h t − 1 j,k and edge attributes r ij : m tji,k
= f ψ ( h t − 1 j,k , r ij ) .
We then computes the embedding of observation by deepset [41]: m t obs ,j = f φ ( (cid:80)
Mm =
b jm f pos ( y m )) , where f φ and f pos are MLPs.
Following the form of mean-ﬁeld inference, we compute the weights of the message w tji,k by the weights of particles and scores from observations computed by a weighting
[Equation]
Then, we aggregate the message from neighboring nodes and observations to compute the node features and beliefs.
First, we sum up the messages and computes the new particles by an aggregate
MLP [Equation]
[Equation] [Equation] ,i ) . We follow the mean-ﬁeld inference to evaluate new particle weights by reweighting
[Equation]
Next, we use resampling to avoid particle degeneracy problems.
Particle degeneracy refers to the weights of all but one particle being close to zero, resulting in extremely high variance.
We resample after each iteration step to build a new set of particles with new weights: { ( h (cid:48) ti,k , w (cid:48) ti,k ) }
Kk =1 =
SoftResamp ( { ( h ti,k , w ti,k ) } Kk =1 ) .
However, this operation is not differentiable.
Soft resampling [22] makes this process differentiable via sampling from a mixture of distribution q ( j ) =
αw j + (1 − α ) 1 K , and the new weights are evaluated using importance sampling, resulting in new belief: w (cid:48) k
= w j ( αw j +(1 − α )(1 /K )) .
When α > 0 , soft resampling produces non-zero gradients for backpropagation.
We take α = 0 .
5 in training and α = 1 . 0 in inference.
When t = T , we decode the node features to get the ﬁnal predictions of the target placement x ∗ i,k = π dec ( h (cid:48)
T i,k ) and the weights w ∗ i,k = w (cid:48) T i,k .
End-to-end Training
We train the framework end-to-end by minimizing two objectives.
One is the negative log-likelihood of dataset ¯ X : J l = (cid:80) ¯ x ∈ ¯ X − log (cid:80) k
w ∗ k N (¯ x ; x ∗ k , Σ) , where ¯ x ∈ ¯ X is the labeled placement and the likelihood is a mixture of Gaussian represented by particles of target object { ( x ∗ k , w ∗ k ) }
Kk =1 .
We also minimize the auxiliary loss for improving soft parsing: J s = (cid:80) (cid:96) ∈
¯ X || (cid:96) − ˜ (cid:96) ||
2 = (cid:80) (cid:96) ∈ ¯ X
(cid:80) l ( ω l − ˜ ω l ) 2 .
As such, the ﬁnal objective is: J =
J l + λ J s , where λ is a hyper-parameter.
Experiments 7.1 Experimental Setup
Tabletop Dataset.
The tabletop dataset, as shown in Fig. 5, consists of 30K visually realistic scenes generated by PyBullet [42] and NVISII [43].
The objects are sampled from 48 objects with various shapes and colors.
The images contain random lighting conditions, light reﬂections, and partial occlusions, reﬂecting the challenging language grounding in the real world.
We use Mask-RCNN [38]
[Figure] [Figure]
=">AAAB9XicbVBNSwMxEM3Wr1q/qh69BIvgqeyKqMdiLwoeKtgPaNeSTbNtaDZZklm1LP0fXjwo4tX/4s1/Y9ruQVsfDDzem2FmXhALbsB1v53c0vLK6lp+vbCxubW9U9zdaxiVaMrqVAmlWwExTHDJ6sBBsFasGYkCwZrBsDrxmw9MG67kHYxi5kekL3nIKQEr3XeAPUFavbmuKQ3jbrHklt0p8CLxMlJCGWrd4lenp2gSMQlUEGPanhuDnxINnAo2LnQSw2JCh6TP2pZKEjHjp9Orx/jIKj0cKm1LAp6qvydSEhkzigLbGREYmHlvIv7ntRMIL/yUyzgBJulsUZgIDApPIsA9rhkFMbKEUM3trZgOiCYUbFAFG4I3//IiaZyUvbOyd3taqlxmceTRATpEx8hD56iCrlAN1RFFGj2jV/TmPDovzrvzMWvNOdnMPvoD5/MHrm2Sog==</latexit> C L
I P
o r t
Seen spatial relations
[Figure]
for object detection.
We also select three tasks in CLIPort’s benchmark: placing objects inside a bowl or a box to test object reference grounding without positional ambiguity involved.
Due to the lack of bounding box labels, we use the unsupervised object detector SPACE [39] for CLIPort’s tasks.
Other tasks of CLIPort’s dataset focus on assembly or deformable object manipulation.
They are not in the scope of our research.
We prepare human-labeled instructions to test the model with natural requests using humanprovided natural language.
Human-labeled instructions are from bit.ly/TableSetData .
However, collecting enough human-labeled data is expensive, while synthesized data is easily generated.
Hence, we use synthesized structured language instructions to train the models and ﬁne-tune the model using human-labeled data.
The training dataset of structured language instructions contains 20K instructions with single spatial relations and 20K with compositions of multiple spatial relations.
Our testing dataset contains 15K instructions, including instructions with unseen compositions of seen spatial relations.
We also have instructions containing ambiguous reference expressions, i.e., multiple objects are semantically identical to a reference expression.
We prepare 9K human-labeled language instructions, where 7K instructions are for training, and the remaining 2K instructions are for testing.
The human-labeled instructions are pre-collected from Mechanical Turk.
We use image pairs to show the scene before and after an object is moved and let humans provide language requests for such object placement.
The details of the dataset can be viewed at section A in the appendix.
Evaluation Metric.
We evaluate the success rate of object placement, repeated 5 times.
The successful placements should satisfy all the spatial constraints given in the language instructions.
The placement should not be too far away from the reference objects with a threshold of 0.4 meters, which is approximated from the human-labeled dataset.
The evaluated models are trained for 300K steps with a batch size of 1.
7.2 Results
We assess the performance of object placement by synthesized structured language instructions as well as human-labeled instructions, and compare our performance with CLIPort [8].
CLIPort has a fully convolutional design with a two-stream architecture that handles spatial and semantic information.
We ﬁrst use three tasks from CLIPort’s benchmark to test the models in grounding referred objects with referential ambiguity.
We then use the Tabletop dataset to test language grounding with the presence of referential ambiguity, positional ambiguity, compositionality, as well as humanlabeled noisy instructions.
[Table]
Object Reference Grounding.
In
CLIPort’s benchmark, spatial relations are simple as the placements are always inside a speciﬁc object.
As such, learning placement generation equals learning object grounding.
In these tasks, packing-shapes requires object grounding without referential ambiguity, packing-google-objects contains referential ambiguity for target objects, and put-blocks-in-bowls involves referential ambiguity for context objects.
Table 1 reveals that both
CLIPort and P ARA G ON have good performance in object grounding even with referential ambiguity.
[Figure]
in a multimodal distribution of object placement.
We use particle- based GNN to capture the multimodality to adapt to referential ambiguity.
[Table]
Object Placing with Ambiguity.
The Tabletop dataset involves placing the object next to a context object, causing positional ambiguity.
As the placement is not inside a speciﬁc object, object grounding no longer equals object placement.
Table 2 suggests that P ARA G ON has a better performance with positional ambiguity.
CLIPort uses convolutional architecture, which can capture strong local correlations.
This inductive bias is useful if the placement is inside an object.
However, positional ambiguity is only in weak agreement with the inductive bias as pixels far away can also be closely related, compromising the performance of CLIPort.
P ARA G ON has an encoded graphical model for the spatial relations between objects to generate placement.
It learns a distribution of the usual distance for placement from data and naturally adapts to positional ambiguity.
We also test the performance of object placement with referential ambiguity ( With Ref Ambiguity in Table 2), in which the scenes contain a few semantically identical objects.
Referential ambiguity makes the correct placement non-unique, resulting in a multimodal distribution of correct placement.
As shown in Table 2, P ARA G ON has a good performance with referential ambiguity.
The core idea is to represent a distribution as a set of particles to capture multimodality and employ this idea in GNN for placement generation.
Fig.
7 demonstrates that the GNN outputs a multimodal distribution when there is referential ambiguity. [Table]
Compositionality.
Table 2 also reports the results on instructions with seen and unseen compositions of seen spatial relations ( Comp and Comp* in Table 2, respectively), as well as the performance of the models trained with 2% and 10% of the training data.
The composition substantially increases the complexity of language comprehension and the data required for training.
Table 2 shows that P ARA G ON is data-efﬁcient in learning structured compositional instructions and can generalize to instructions with unseen compositions.
It converts a complex language into a set of simple, structured relations represented as triplets to reduce complexity.
Our approach operates on the grammatical structure of natural language that is generalizable to different semantic meanings.
CLIPort uses single embeddings to “memorize” seen compositions.
It has poorer generalization in the presence of composition when training data is limited and does not generalize well to unseen compositions.
Human-labeled Instructions.
We use human-labeled instructions to test natural human requests in noisy natural language.
The instructions in this testing dataset may not be grammatically correct and contain unseen, noisy expressions for objects and spatial relations.
The results are shown in Table 3, where Fine-Tuned means the models are ﬁne-tuned by the human-labeled dataset, and Not Fine-Tuned models are trained only on structured instructions.
The results show that P ARA G ON performs better on human-labeled instructions.
P ARA G ON is data-driven and optimizes all the modules to adapt to imperfect, noisy linguistic data and extract useful relational information.
P ARA G ON extracts relations from grammatical structures of instructions, which is highly generalizable and helps tackle unseen expressions.
CLIPort is also data-driven but represents instructions as single embeddings that do not generalize well to unseen, noisy language expressions
[Table] [Table]
Ablation Study.
We conduct an ablation study to assess each module’s contribution in P ARA G ON .
We design a variation of P ARA G ON without soft parsing, using single embeddings of language instructions to ground objects and represent relations.
We also assess a version of P ARA G ON that uses a mean-ﬁeld message passing neural network [17] without the use of particles, and test P ARA G ON without resampling.
P ARA G ON uses pretrained CLIP to encode text and visual inputs for object reference grounding.
To test the impact of using pre-trained models that are trained separately for vision and language, we replace CLIP in P ARA G ON with Visual Transformer [44] and BERT [45].
We report ablation study results in Table 4.
The results for No Soft Parsing demonstrate that soft parsing is essential in learning compositional instructions.
It converts complex language sentences into simple phrases for objects and relations.
The embeddings of those phrases have more straightforward semantic meanings than the entire sentence and are much easier for them to be grounded in the visual scene.
As shown in the row of No Particle , using GNN without particles cannot capture multimodal distribution when referential ambiguity occurs and compromise the performance.
The results in No Resampling indicate that resampling is helpful because particle degeneracy can occur without resampling, compromising the performance of capturing multimodal distributions with referential ambiguity.
The last 4 rows in Table 4 show that P ARA G ON requires fewer data to obtain good results using pre-trained CLIP than pre-trained ViT+Bert.
CLIP is pre-trained by a large dataset of image-caption pairs and is better for aligning language with visual features.
Conclusion and Limitation
P ARA G ON leverages object-centric representation for visual grounding of natural language. It integrates parsing into a probabilistic, data-driven framework for language-conditioned object placement.
It tries to break the lines between parsing-based and embedding-based methods for language grounding, as well as combine the strength of rule-based and data-driven approaches.
P ARA G ON reduces the complexity of embedding-based grounding by parsing complex sentences into simple structures, and learns generalizable parsing rules from data for robustness.
Those combinations are beneﬁcial and would facilitate future research.
Our experiments reﬂect the difﬁculties of language grounding in real situations to show P ARA G ON ’s potential for real-world application.
P ARA G ON is limited by 2D object representations to ground 3D spatial relations situated by the 3D shape of objects, such as “lean”.
It also limits us to ground rotation instructions conditioned on the shape of objects, such as “rotate the fork to make it point to an apple”.
Leveraging 3D object representations would empower stronger language grounding skills and facilitate future research. Acknowledgments
This research is supported in part by the National Research Foundation (NRF), Singapore and DSO
National Laboratories under the AI Singapore Program (AISG Award
No.
AISG2-RP-2020-016) and the Agency of Science, Technology and Research, Singapore, under the National Robotics Program (Grant No. 192 25 00054)
A Tabletop Dataset
We extend the dataset from bit.ly/TableSetData with randomly sampled objects and generate placement from several sampled spatial relations.
The dataset will be released in our website at bit.ly/ParaGonProj .
A.1 Objects
We sample 50 dining objects with different classes, colors, and shapes, including forks, knives, spoons, plates, wine cups, water cups [Table] and napkin cloths.
The visualization of objects is in Fig.(a).
A.2 Language Expressions
Our dataset draws samples from commonly used spatial relations with various language expressions.
The spatial relations includes left, right, front, behind, on top of, and their compositions.
We also have ternary spatial relation “between”.
The language expressions for each spatial relations are illustrated in Table
[Table]
[Table]
A.3 Data Generation A.3.1 Instruction Generation
We generate the instructions according to language templates.
The template for tasks with single spatial relation is illustrated below:
[Equation]
object phrase]
[Equation]
If the language instruction contains compositional spatial relations, the template is:
[Equation]
The spatial relations are sampled from Table 5, and the subject phrases and object phrases are sampled according to the object class, object colors, and shapes, such as “yellow square plate”.
We sample the verb phrases uniformly as random from “put”, “place”, and “move”.
[Figure]
[Figure]
A.3.2 Image Generation
• Camera Setup.
We use four cameras to capture the visual observation.
Similar to the Raven simulator, there are three RGB-D cameras with resolutions of 640 × 480 that are placed on the rectangular table’s top, left, and right, pointing towards the center (see (b), (c), and (d) in Figure 9).
We also have an RGB camera with a resolution of 640 ×placed at the top of the table (see (a) in Figure 9 for the example picture).
• Rendering.
We use NVISII as the rendering interface to get the photorealistic pictures of the visual scene.
We randomly sample the position of the lights and their colors to add difﬁculties.
In addition, due to the random lighting conditions, the visual scenes contain various light reﬂections, hampering the visual grounding of natural language.
A.3.3 Expert Demonstration.
We provide a handcrafted policy for expert demonstration in placement generation.
According to the ground truth position of objects in the simulator, we sample from a handcrafted Gaussian distribution with a speciﬁc direction as the placement generation policy.
Speciﬁcally, the demonstration is sampled from a truncated Gaussian distribution.
There are two examples shown in Fig. 8.
In this ﬁgure, case (a) illustrates a distribution of “left-hand side”, where the highlighted area indicates a high probability of placement.
We truncate areas that derive from the left side and are too close to the reference object.
Case (b) illustrates the composition of spatial relations “left-hand side” and “upper side”, where the conjunction of both directions truncates the distribution. The mean of the Gaussian distribution is located at the intersection of the central axes of the spatial relations. We also use axis-aligned bounding boxes (AABB) of the target object and context objects to detect intersections and avoid object collision after generating the placement.
A.4 Evaluation Critera
We evaluate the success rate of the object placing by examining whether the placement satisﬁes the spatial relational constraints in the natural language instructions.
Speciﬁcally, we assess whether the generated particle is located at the truncated regions.
For CLIPort, we take the argmax to get the generated placement.
As ParaGon generates a set of particles as the samples of a placement distribution, we randomly sample from the particles ten times to evaluate the success rate [Table]
B Implementation and Training Details
[Table]
[Equation]
B.1 Soft Parsing
We illustrate the implementation details of the soft parsing module.
The algorithm is shown in
Algorithm 3.
B.1.1 Structural Prior from Dependency Parsing
We use spaCy dependency parser with transformer backend to get the dependency tree of a given sentence.
We use embeddings v ∈ R D embed to represent the part-of-speech (POS) tag and dependency (DEP) tag in the dependency tree.
The message passing neural network uses MLP as the functions f m : R D embed × R D embed × R D embed → R D embed and f u : R D embed × R D embed → R D embed .
The POS tags are used as the initial node features and the DEP tags are the edge attributes.
During the message passing, the node features and edge attributes are concatenated as the input of f m and f u .
Thus, the structural prior from the dependency tree is encoded in the output structural features of the messagepassing neural network.
In addition, the structure lacks the order of position of the tokens in the sentence.
We use cosine positional encoding added in the output structural embeddings to encode the position of tokens in the sentence.
B.1.2 Relation Extraction
The output structural features (cid:126) l from the message-passing neural network are used for relation extraction.
We use attention to evaluate the normalized score of each token in the sentence from message passing for its contribution to the components in relational triplets.
We use a single-layer
MLP with softmax to compute the normalized score of each token contributing to the subject phrase.
To discover potential object phrases, we use slot attention [36] from visual object discovery for unsupervised object phrase discovery.
The slot attention algorithm is illustrated in Algorithm 2.
After discovering the object phrases, we use a multi-head attention module to evaluate the relational phrases conditioned on the corresponding object phrases.
The object structure embeddings from slot attention are used as the query, and the structural features from message passing as the key and value.
After we compute the weights of each token in the sentence for the subject phrase, object phrase, and relational phrase, we use the expected embedding to represent the word embeddings for the phrases in triplets.
We use CLIP as the encoder to compute the word embeddings of tokens in the sentence.
B.1.3 Hyper-parameters
The dimension of embedding is D embed = 64 , and the layer number of message passing neural network is 4.
We use 3 iteration for slot attention, with slot number N slot = 4 , hidden dimension
D slots = 128 and (cid:15)
= 1 × 10 − 8 .
As for the multi-head attention module for relational phrase discovery, the number of heads is eight, and the embedding dimension is 64.
Language encoder φ ( · ) is trained CLIP (ViT-32/B), in which the word embedding dimension is D word = 512 .
[Table]
Layer params: k, q, v : linear projections for attention; GRU; MLP; LayerNorm ( × 3) 1: inputs ← LayerNorm ( inputs ) 2: for t = 0 , . . .
, T do 3: slots prev ← slots 4: slots ← LayerNorm ( slots ) 5: attn ← SoftMax ( [Table] D k ( inputs ) q ( slots ) (cid:62) , axis = slots ) 6:
Updates ← WeightedMean ( weights = attn
+ (cid:15), values = v ( inputs )) 7: slots ← GRU ( state = slots prev , inputs = updates ) 8: slots ← slots + MLP ( LayerNorm ( slots )) 9: end for
Output: slots
B.2 Particle Mean-ﬁeld GNN
The pseudo-code of particle mean-ﬁeld GNN is illustrated in Algorithm 3.
B.2.1 Functions in Particle Mean-ﬁeld GNN
For the observation, we use masks of bounding boxes to represent the object positions and shapes in the observed image.
We use a simple CNN model f pos : R W pos
× H pos → R D pmpnn to encode the positional feature of the bounding boxes detected in the scene.
We then use an MLP to project the positional feature into embeddings.
The particle proposal network f ψ : R D pmpnn × R D pmpnn → R D pmpnn and aggregation network f u : R D pmpnn → R D pmpnn are all single-layer MLP.
The reweighting function of observation model g φ : R D pmpnn × R D pmpnn → R is also a MLP.
B.2.2 Hyper-parameters
The layer number of particle-based mean-ﬁeld GNN is T = 8 .
In message passing, we use embeddings with dimension D pmpnn =
In soft resampling, we use α = 0 .
5 for training and α = 1 .for inference.
For visual grounding, the visual encoder is pre-trained CLIP (ViT-32/B).
B.3 Training
We conduct supervised learning to train the model end-to-end.
There are two components to the training objective.
One is the negative log-likelihood of dataset ¯ X : J l = (cid:80) ¯ x ∈ ¯ X − log (cid:80) k
w ∗ k N (¯ x ; x ∗ k , Σ) , where ¯ x ∈ ¯ X is the labeled placement and the likelihood is a mixture of Gaussian represented by each output particle of target object { ( x ∗ k , w ∗ k ) }
Kk =1 .
We also minimize the auxiliary loss for improving soft parsing: J s = (cid:80) (cid:96) ∈
¯ X || (cid:96) − ˜ (cid:96) ||
2 = (cid:80) (cid:96) ∈ ¯ X
(cid:80) l ( ω l − ˜ ω
l ) 2 .
As such, the ﬁnal objective is:
J =
J l + λ J s , where λ is a hyper-parameter.
The hyper-parameter λ = 10 . 0 .
For target object grounding, we use cross-entropy loss, where the labels for the loss function are the closest bounding box to the ground truth object position.
For training, we use Adam optimizer, and the learning rate is 1 × 10 − 4 .
We train the model for 200K steps when the data is more signiﬁcant than 1000 and 50K when trained by 300.
Input: Number of layer T , number of particles K , detected object bounding boxes B = { ( z m , x m ) }
Mm =1 , relational graph G =
( V , E ) , edge attributes (spatial relation embeddings) r ij 1: h (0) i,k ∼ N (0 , 1) , w (0) i,k ← 1 /K, t ←{ Initialization } 2: for t < T do 3: for ( i, j ) ∈
E do { Message Computing } 4: m tji,k = f
ψ ( h t − 1 j,k , r ij ) 5: m t obs ,j = f φ ( (cid:80)
Mm =
b jm f pos ( y m )) 6: w tji,k = ηw t − 1 j,k log g φ ( h t − 1 j,k , m t obs ,j ) 7: end for 8: for i ∈ V do { Message Aggregation } 9: h ti,k = f u
( (cid:80) j ∈ N
( i ) m tji,k + m t obs ,i ) 10: log w ti,
k = c
+ log g φ ( h ti,k , m t obs ,i )
+ (cid:80) j ∈ N ( i )
w tji,k 11: { ( h (cid:48) ti,k , w (cid:48)
ti,k ) }
Kk =1 ← SoftResam ( { ( h ti,k , w ti,k ) }
Kk =1 ) 12: end for 13: t ← t + 1 14: end for 15: x ∗ i,k
= π dec ( h (cid:48)
Ti,k ) , ∀ k ≤ K { Decode Placement } Output: { ( x ∗ i,k , w (cid:48)
Ti,k ) }
Kk =
B.4 Baseline Implementation B.4.1 No Soft Parsing
This baseline model directly uses the embedding of the entire instruction sentence instead of the embeddings from triplets for object grounding and message passing in step 4 of Algorithm
. B.4.2 No Particle
This baseline model does not have particles in the message passing, and the initially hidden features in step 1 of Algorithm 3 are vectors with zero values.
It also does not have step 11 for soft parsing.
B.4.3 No Resampling
The third baseline is built without soft resampling in step 11 of Algorithm 3.
B.4. P ARA G ON (ViT + Bert)
This baseline model uses the ImageNet-pretrained ViT 32/B model for visual encoding and the pretrained Bert model for text encoding in visual grounding, representing the edge attribute r ij .