A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Natural language provides an accessible and expressive interface to specify longterm tasks for robotic agents.
However, non-experts are likely to specify such taskswith high-level instructions, which abstract over speciﬁc robot actions throughseveral layers of abstraction.
We propose that key to bridging this gap betweenlanguage and robot actions over long execution horizons are persistent representations.
We propose a persistent spatial semantic representation method, and showhow it enables building an agent that performs hierarchical reasoning to effectivelyexecute long-term tasks.
We evaluate our approach on the ALFRED benchmarkand achieve state-of-the-art results, despite completely avoiding the commonlyused step-by-step instructions.
https://hlsm-alfred.github.io/
Introduction
Mobile manipulation in a home environment requires addressing multiple challenges, includingexploration and making long-term inference about actions to perform.
In addition to reasoning,robots require an accessible, yet sufﬁciently expressive interface to specify their tasks.
Natural Language provides an intuitive mechanism for task speciﬁcation, and coupled with advances inautomated language understanding, is increasingly applied to embodied agents [e.g., 1–11].
In this paper, we study the problem of learning to map high-level natural language instructionsto low-level mobile manipulation actions in an interactive 3D environment [12].
Existing worklargely studies language tightly aligned to the robot actions, either using single-sentence instructions [e.g., 1, 2, 5, 9] or sequences of instructions [13–18].
In contrast, we focus on high-levelinstructions, which provide more efﬁcient human-robot communication, but require long-horizonreasoning across layers of abstraction to generate actions not explicitly speciﬁed in the instruction.
Robust reasoning about manipulation goals from unrestricted high-level natural language instructions has a variety of open challenges.
Consider the instruction secure two discs in a bedroom safe(Figure 1).
The robot must ﬁrst locate the safe in the bedroom .
It then needs to distribute the actions entailed by secure to two objects ( two discs ), each requiring a distinct sequence of actions, buttargeting the same safe .
It is also required to map the verb secure to its action space.
In parallel,the robot must address core mobile manipulation challenges, and often can only identify requiredactions as it observes and manipulates the world (e.g., if the safe needs to be opened).
We propose to construct and continually update a spatial semantic representation of the world fromrobot observations (Figure 2). Similar to widely used map representations [19–22], we retain the spatial properties of the environment, allowing the robot to navigate and reason about relations betweenobjects, as required to accomplish its task. We propose the Hierarchical Language-conditioned
Spatial Model (HLSM), a hierarchical approach that uses our spatial representation as a long-termmemory to solve long-horizon tasks. HLSM consists of a high-level controller that generates subgoals, and a low-level controller that generates sequences of actions to accomplish them.
In ourexample (Figure 1), the sequence of subgoals is (cid:104) pick up a CD, open the safe, put the CD in the safe,... (cid:105) , each requiring a sequence of actions.
The spatial representation allows selecting subgoals thatuse previously observed objects outside of the agent’s view, or to decide about needed exploration. Observations
[Figure] [Figure] [Figure] [Figure]
OTATE
[Figure]
IGHT
P ICKUP
[Figure]
OTATE L EFT
L OOK
[Figure] BJECT ......
[Figure]
P ICKUP [Figure] Time
[Figure] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] do
[Table] [Figure] [Table] then
[Table] [Table] [Figure] [Table]
s t , (cid:104)
g i (cid:105) i<k ) 9: if g k = g STOP then 10:
End episode 11: a t ∼ π L ( g k , ˆ s t ) 12: if a t = a PASS then 13: k ← k +14: if a t = a FAIL then 15: g k ← null 16: while a t ∈ { a FAIL , a PASS } 17: Perform a t , observe o t + 18: End episode We evaluate our approach on the ALFRED [12] benchmark and achieve state-of-the-art resultswithout using the low-level instructions used by previous work [16–18], neither during trainingnor at test-time.
This paper makes three key contributions: (a) a modular representation learningapproach for the problem of mapping high-level natural language task descriptions to actions in a3D environment; (b) a method for utilizing a spatial semantic representation within a hierarchicalmodel for solving mobile manipulation tasks; and (c) state-of-the-art performance on the ALFREDbenchmark, even outperforming all approaches that use detailed sequential instructions.
Related Work
Natural language has been extensively studied in robotics research, including with focus on instruction [1, 23], reference resolution [24], question generation [25–27], and dialogue [4, 28, 29].
Mostwork in this area has considered either synthetic instructions of relatively simple goals [7, 30–32], ornatural language instructions where all intermediate steps are explained in detail [5, 12–14, 33–37].
In contrast, we focus on high-level instructions, which are more likely in home environments [38].
Representation of world state, action history, and language semantics plays a central role in robotsystems and their algorithm design.
Symbolic representations have been extensively studied for instruction following agents [1–4, 19, 20, 38–44].
While they simplify the symbol grounding problemand enable robustness, the ontologies on which they rely on are laborious to scale to new, unstructured environments and language.
Representation learning presents an alternative by learning tomap observations and language directly to actions [5, 8, 9, 11, 13, 33].
World state and languagesemantics are represented with vectors [13] or by memorizing past observations [8, 17].
Modellingimprovements have enabled these approaches to achieve good performance on complex navigationtasks [7, 9, 11, 13, 14, 36], a success that has not yet translated to mobile manipulation [12, 45, 46].
We propose integrating a semantic voxel map state representation within a hierarchical representation learning system.
Similar semantic 2D maps have been successfully used in instruction- following tasks [7, 8, 47–49].
We extend these maps to 3D and show state-of-the-art results ona challenging mobile manipulation benchmark. Our map design is related to sparse metric, topological and semantic maps [10, 19–21, 50] that have enabled grounding symbolic instruction representations. Our map does not impose a topological structure or require reasoning about objectinstances, instead modelling a distribution over semantic classes for every voxel.
Problem Deﬁnition
Let A be the set of agent actions, and S the set of world states.
Given a natural languageinstruction L and an initial state s 0 ∈ S , the agent’s goal is to generate an execution Ξ =(cid:104) s 0 , a 0 , s 1 , a 1 , . . .
, s T , a T (cid:105) , where a t ∈ A is an action taken by the agent at time t , s t ∈ S isthe state before taking a t , and s t
+1 =
T ( s t , a t ) under environment dynamics T : S ×
A → S .
The state s t is deﬁned by the environment layout and the poses and states of all objects and theagent.
The agent does not have access to the state s t , but only to an observation o t .
An observationo t =
( I t , P t , v St , L ) includes a ﬁrst-person RGB camera image I t , the agent’s pose P t , a one-hotencoding of the object class the agent is holding v St , and the instruction L [-. 
 The-].The task is consideredsuccessful if all goal-conditions corresponding to the task L are true at the ﬁnal state s T .
Partialsuccess is measured as the fraction of goal-conditions that have been achieved.
The ALFRED dataset includes sets of seen and unseen environments.
The setof actions A =
A nav ∪ A int includes parameter-free navigation actions A nav = { M OVE A HEAD , R OTATE L EFT , R OTATE R IGHT } and interaction actions A int ={ P ICKUP , P UT , T OGGLE O N , T OGGLE O FF , O PEN , C LOSE , S LICE } parameterized by a binarymask that identiﬁes the argument of the interaction in the agent’s current ﬁrst-person view.
Wecompute P t and v St using dead-reckoning from RGB observations and actions.
Hierarchical Model with a Persistent Spatial Semantic Representation
We model the agent behavior with a policy π that maps an instruction L and the observation o t attime t to an action a t .
The policy π is made of an observation model F and two controllers: ahigh-level controller π H and a low-level controller π L .
The observation model builds a spatial staterepresentation ˆ s t that captures the cumulative agent knowledge of the world at time t .
ˆ s t is used byboth π H for high-level long-horizon task planning, and π L for near-term reasoning, such as objectsearch, navigation, collision avoidance, and manipulation.
Figure 2 illustrates the policy.
The high-level controller π H computes a probability over subgoals .
A subgoal g is a tuple( type , arg C , arg M ) , where type ∈ A int is an interaction type (e.g., O PEN , P ICKUP ), arg C isthe semantic class of the interaction argument (e.g., S AFE , CD), and arg M is a 3D mask identifyingthe location of the argument instance. In ALFRED, each interaction action in the set A int corresponds to a subgoal type.
When predicting the k -th subgoal at time t , π H considers the instruction L , the current state representation ˆ s t , and the sequence of past subgoals (cid:104) g i , (cid:105) i<k .
During inference,we sample from π H .
Unlike arg max , sampling allows the agent to re-try the same or differentsubgoal incase of a potentially random failure (e.g., if a M UG was not found, pick up a C UP ).
The low-level controller π L is given the subgoal g k as its goal speciﬁcation at time t .
At everytimestep j > t , π L maps the state representation
ˆ s j and subgoal g k to an action a j , until it outputsone of the stop actions: a PASS or a FAIL to indicate successful or failed subgoal completion.
Algorithm 1 describes the execution ﬂow.
At time t = {+
 the initial observation o+} 0 is received.
Ateach timesep, we update the state representation ˆ s t (Line 5). If needed, we sample a new subgoalg k from π H (Line 8), and then sample an action a t from π L . If a t is a PASS , we increment subgoalcounter k (Line 13).
If it is a FAIL , we discard the current subgoal k (Line 15). We repeat Lines 8–15until an executable action a t is sampled.
We execute a t , receive the next observation (Line 17), andproceed to the next timestep. The episode ends when the subgoal g STOP is sampled (Line 10) or thehorizon H is exceeded (Line 18).
4.1 State Representation
The state representation ˆ s t at time t captures the agent’s current understanding of the state of theworld, including the locations of objects observed and the agent’s relation to them.
The state representation is a tuple ( V St , V Ot , v St , P t ) .
The semantic map V St ∈ [0 , 1] W × L × H × C is a 3D voxel map that for every position indicates which of the c ∈ [1 , C ] object classes are present in the voxel.
Theobservability map V Ot ∈ { 0 , 1 } W × L × H is a 3D voxel map that indicates whether the correspondingposition has been observed.
The inventory vector v St ∈ { 0 , 1 } C indicates which of the C objectclasses the agent is currently holding.
The agent pose P t =
( x, y, ω p , ω y ) is speciﬁed by the 2Dposition ( x, y ) , pitch angle ω p , and yaw angle ω y .
We also compute 2D state affordance features A FFORD (ˆ s t ) ∈ [0 , 1] 7 × W × L in a top-down view thatrepresent each position with one or more of seven affordance classes { pickable , receptacle ,togglable , openable , ground , obstacle , observed }.
Each [ A FFORD (ˆ s t )] ( τ,x,y ) = 1 .
if atleast one of the voxels at position ( x, y ) has affordance class τ , otherwise it is zero.
A FFORD (ˆ s t ) issuited for object class agnostic reasoning, for example predicting a pose to pick up an object.
4.2 Observation Model
The observation model F (ˆ s t − 1 , o t , g k ) updates the state representation with new observations.
Itconsiders the current subgoal g k to actively acquire information relevant to g k .
The computation of
F consists of three steps: perception, projection, accumulation.
Perception Step We predict semantic segmentation I St and depth map
I
Dt from the RGB observation
I t .
We use neural networks pre-trained in the ALFRED environment.
The semantic segmentation[ I St ] ( u,v ) is a distribution over C object classes at pixel ( u, v ) .
The depth map [ I Dt ] ( u,v ) is a binneddistribution over B bins.
2 We also heuristically compute a binary mask M Dt that indicates whichpixels have conﬁdent depth readings.
We allow more conﬁdence slack in pixels that correspondto the current subgoal argument arg Ct
according to I St .
Appendix ?? provides further details.
Weuse perception models based on the U-Net [51] architecture, but our framework supports other,potentially more powerful models as well (e.g. [52, 53]).
Projection Step We use a pinhole camera model to convert depth I
Dt and segmentation I St to apoint cloud that represents each image pixel ( u, v ) with a 3D position ( x, y, z ) ∈ R W × L × H anda semantic distribution [ I St ] ( u,v ) .
We use arg max B ( I Dt ) to compute the 3D positions, and discardpoints at pixels ( u, v ) when the binary mask value is [ M Dt ] ( u,v ) = 0 .
We construct a discretesemantic voxel map ˆ V St ∈ [0 , 1] W × L × H × C , where W , L , and H are the width, height, and length.
The value at each voxel
[ ˆ V St ] ( x,y,z ) is the element-wise maximum of the segmentation distributions[ I St ] ( u,v ) across all points ( u, v ) within the voxel.
We additionally compute a binary observabilitymap ˆ V Ot ∈ { 0 , 1 } W × L × H that indicates the voxels observed at time t .
A voxel is observed if itcontains points, or if a ray cast from the camera through the voxel centroid has expected depthgreater than the distance from the camera to the centroid.
Accumulation Step We integrate ˆ V St and ˆ V Ot into a persistent state representation:
[Equation]
This operation updates each voxel with the most recent semantic distribution, while retaining thevalues of all voxels not visible at time t .
The output of the observation model is the spatial staterepresentation
ˆ s t =
( V St , V Ot , v St , P t ) .
The inventory v St and pose P t are taken directly from o t .
4.3 High-level Controller ( π H )
At timestep t , when invoked for the k -th time, the input to π H is the instruction L , the sequenceof past subgoals (cid:104)
g
i (cid:105) i<k , and the current state representation ˆ s t .
The output is the next subgoalg k = ( type k , arg Ck , arg Mk ) .
Figure 3 illustrates the high-level controller architecture.
Input Encoding We encode the text L using a pre-trained BERT [54] model that we ﬁne-tuneduring training.
We use the CLS token embedding as the task embedding φ L .
We encode thestate representation ˆ s t to account for classes of all observed objects, and the object that the agent isholding: φ s (ˆ s t ) =
[ v St ; max ( x,y,z ) ( V St )] , where max ( x,y,z ) is a max-pooling operation over spatialdimensions and [ · ; · ] denotes concatenation. We compute the representations of previous subgoals as
[Figure]
𝑡𝑠
[Figure]
𝑡𝑠 [Figure]
State [Figure] Subgoal history tensor arg [Figure]
E GOCENTRIC
[Figure]
RANSFORM
R EFINER
A FFORD (
[Figure]
𝑡 ) [Figure] arg [Figure]
type
[Figure] (cid:104) R EPR ( g i ) (cid:105) k − 1 i =0 , where R EPR ( g i ) is the sum of a sinusoidal positional encoding [55] of index i andlearned embeddings for type i and arg Ci . We process this sequence with a two-layer Transformerautoregressive encoder [55] to compute (cid:104) φ gi (cid:105) k − 1 i =0 .
We take φ gk − 1 as the subgoal history embeddingvector.
We additionally encode the argument mask information arg Mi from the subgoal history inan integer-valued subgoal history tensor H k [- 
-]− 1 ∈ N K × W × L where [ H k − 1 ] ( τ,x,y ) is the number oftimes an interaction action type τ was performed at 2D position ( x, y ) in the birds-eye view:
[Equation]
2) 168 Subgoal Prediction We concatenate the three representations
[Equation] ; φ gk − 1 ] .
We use adensely connected multi-layer perceptron [56] to predict two distributions P ( type k | h ( t,k ) ) and
P ( arg Ck | type k , h ( t,k ) ) , from which we sample a subgoal type type k and argument class arg Ck .
The remaining component of the subgoal is the action argument mask arg Mk .
Let [ V St ] ( arg Ck ) be avoxel map that only retains the object information for objects of class arg Ck in the semantic map V St .
We reﬁne it to identify a single object instance.
We compute a birds-eye view representation:
[Equation]
where A FFORD (ˆ s t ) is a birds-eye view state affordance feature map (Section 4.1) and 1 type k is aone-hot encoding of type k . {+
+}
Finally, we compute the 3D argument mask arg Mk ∈ [0 , 1] W × L × H :
[Equation]
177 where E GO T RANSFORM ( x , P t ) transforms the map x to the agent egocentric pose P t , R EFINERis a neural network based on the LingUNet architecture [35], and φ L is the language embedding.
The reﬁned arg Mk is a [0 , 1] -valued 3D mask that identiﬁes the instance of the interaction argumentobject.
If the object is believed to be unobserved, then arg Mk contains all zeroes.
The controlleroutput is the subgoal g k = ( type k , arg Ck , arg Mk ) .
4.4 Low-level Controller ( π L )
The low-level controller π L is conditioned on the most recent subgoal g k =
( type k , arg Ck , arg Mk ) .
At time t , it maps the state representation ˆ s t to an action a t .
It combines engineered andlearned components.
Appendix ?? provides the implementation details.
The controller π L invokesa set of procedures: NavigateTo , SampleExplorationPosition , SampleInteractionPose ,and InteractMask .
Their invokation follows a pre-speciﬁed execution ﬂow across multipletimesteps.
First, we perform a 360° rotation to observe the nearby environment. If no objects of type arg Ck are observed, we explore the environment by sampling a position ( x, y ) =
SampleExplorationPosition (ˆ s t ) , navigating there using the procedure NavigateTo ( x, y, ˆ s t ) ,and performing a 360° rotation.
We repeat exploration until a voxel in V St contains the class arg C kwith > 50% probability.
To interact with an object, we sample an interaction pose ( x, y, ω y , ω p ) =
SampleInteractionPose (ˆ s t , g k ) , invoke NavigateTo ( x, y, ˆ s t ) to reach the position (x,y), and then rotate according to yaw and pitch angles ( ω y , ω p ) .
Finally, we generate the egocentric interaction mask mask t =
InteractMask
(ˆ s t , arg Mk ) , and output the interaction action ( type k , mask t ) .
All procedures use the spatial representation ˆ s t .
NavigateTo navigates to a goal position using a value iteration network (VIN) [57] that reasons over obstacle and observability maps fromˆ s t .
SampleExplorationPosition samples positions on the boundary of observed space in ˆ s t .
SampleInteractionPose uses a learned neural network N AV M ODEL to predict a distributon ofposes from which the interaction g k will likely succeed.
InteractMask uses the segmentationimage I St and the 3D argument mask arg Mt to compute the ﬁrst-person mask of the target object.
Learning
The policy contains four learned models: the segmentation and depth networks, π H , and the navigation model N AV M ODEL used by π L .
We train all networks using supervised learning.
We assumeaccess to a training dataset D = { ( L ( j ) , Ξ ( j ) ) } N D j =1 of high-level natural language instructions L ( j )paired with demonstration execution Ξ ( j ) in a set of seen environments.
Each execution Ξ ( j ) is asequence of states and actions (cid:104) s ( j ) 0 , a ( j ) 0 , . .
, s ( j ) T , a ( j ) T (cid:105) .
We denote N P the total number of statesin D , and N G the total number of subgoals.
We process D into three datasets.
The perception dataset D P = { ([ I ] ( i ) , [ I D ] ( i ) , [ I S ]
( i ) } N P i =1 includes RGB images [ I ] ( i ) with ground truth depth [ I D ]
( i ) and segmentation [ I S ]
( i ) .
The subgoaldataset D g = { ( L ( i ) , ˆ s ( i ) t , (cid:104) g ( i ) j (cid:105)
kj =0 )
} N G i =1 contains natural language instructions
L ( i ) , state representations ˆ s ( i ) t at the start of k -th subgoal execution, and sequences of the ﬁrst k subgoals (cid:104) g
( i ) j (cid:105)
kj =0extracted from Ξ ( j ) .
The navigation dataset D N = { (ˆ s ( i ) , g ( i ) , P ( i ) ) } N P
i =1 consists of state representations
ˆ s ( i ) , subgoals g ( i ) , and agent poses P ( i ) at the time of taking the interaction actioncorresponding to subgoal g ( i ) .
The state representations ˆ s ( · ) in datasets D g and D N are constructedusing the observation model (Section 4.2), but using ground-truth depth and segmentation images.
We train the perception models on D P and the π H on D g to predict the k -th subgoal by optimizingcross-entropy losses.
We use D N to train the navigation model N AV M ODEL by optimizing a crossentropy loss for positions and yaw angles, and an L2 loss for the pitch angle.
Experimental Setup
Environment, Data, and Evaluation We evaluate our approach on the ALFRED [12] benchmark.
It contains 108 training scenes, 88/4 validation seen/unseen scenes, and 107/8 test seen/unseenscenes.
There are 21,023 training tasks, 820/821 validation seen/unseen tasks, and 1533/1529 testseen/unseen tasks.
Each task is speciﬁed with a high-level natural language instruction.
The goalof the agent is to map raw RGB observations to actions to complete the task.
ALFRED also provides detailed low-level step-by-step instructions, which simplify the reasoning process.
We donot use these instructions for training or evaluation.
We collect a training dataset of languagedemonstration pairs for learning (Section 5).
To extract subgoal sequences, we label each interaction action a t = ( type t , mask t ) and any preceding navigation actions with a single subgoal oftype = type t .
We compute the subgoal argument class arg C and 3D mask arg M labels from theﬁrst-person mask mask t , and ground truth segmentation and depth.
Completing a task requires satisfying several goal conditions.
Following the common evaluation [58, 59], we report two metrics.
Success rate (SR) is the fraction of tasks for which all goal conditions were satisﬁed.
Goal conditionrate (GC) is the fraction of goal-conditions satisﬁed across all tasks.
Systems We compare our approach, the Hierarchical Language-conditioned Spatial Model (HLSM)to others on the ALFRED leaderboard that only use the high-level instructions.
At the time ofwriting, the only such published approach is HiTUT [46], an approach that uses a ﬂat BERT [54]architecture to model a hierarchical task structure without using a spatial representation. See Appendix ??
for a detailed comparison. We also compare to approaches that use the step-by-stepinstructions, which puts our method at a disadvantage.
Additionally, we perform ablations and study sensory oracles.
To study the observation model, wecompare to using sensory oracles for ground truth depth, ground truth segmentation, and both.
Wereport high-level controller ablations that remove the subgoal encoder, language encoder, and state
[Figure]
✔ ✔ ✔
Success ✔ ✔ ✔ ✔ ✔
[Figure]
✔
[Figure] representation encoder as used for predicting subgoal type type k and argument class arg Ck , whilestill using the state representation ˆ s t to predict the subgoal argument mask arg Mk . We also study alow-level controller ablation that removes the exploration procedure.
Results
Table 1 shows test and validation results.
Our approach achieves state-of-the-art performance acrossboth seen and unseen environments in the setting with only high-level instructions.
We achieve6.06% absolute (59.2% relative) improvement in SR on the test unseen split, and 6.7% absolute(36.4% relative) improvement in SR on the test seen split compared to HiTUT G-only.
Our approach performs competitively even when compared to approaches that also use the low-levelstep-by-step instructions.
We achieve 0.86% absolute (5.6% relative) improvement in SR on the testunseen split compared to ABP [16].
On the test seen split, our approach performs reasonably well,however ABP [16], E.T. [17], and LWIT [18] perform better, reﬂecting stronger scene overﬁtting.
Tables 2 and 3 show development results.
Ground truth depth (+ gt depth) provides 3.2%/1%absolute improvement in seen/unseen scenes.
Ground truth segmentation (+ gt seg) provides16.3%/19.3% absolute improvement.
Using both (+ gt depth, gt seg) provides 19.1%/30.2% absolute improvement and narrows the seen/unseen gap from 13.2% to 2.1%.
We report high-level controller π H input encoder ablations.
The poor performance without thelanguage encoder reﬂects task difﬁculty.
Zeroing the input to the subgoal history encoder (butkeeping position encodings) reduces performance by only 1.7%/1.8%, showing that knowing theindex of the current subgoal in addition to the state representation is often sufﬁcient. Not usingthe state representation for predicting subgoal type and argument class reduces SR by 1.7% in seenscenes, but increases SR by 1.1% in unseen scenes, achieving state-of-the-art validation-unseen SR
[Table]
[Table] [Table]
[Table]
of 12.9% on the ALFRED benchmark Therefore, predicting the sequence of subgoal types andargument classes (i.e., what to do) is at times possible without spatial reasoning, while groundingthe subgoal (i.e., where to do it) requires spatial information.
Removing random exploration fromπ L reduces SR by 0.1%/0.5% on seen/unseen scenes, showing it is not very effective.
Figure 4 illustrates the model behavior, showing both successes and common failures.
The mainfailures in valid unseen scenes are due to (1) perception errors that result in missing or extraneousobstacles or picking up wrong objects; (2) insufﬁciency of random exploration (e.g., not searchinginside cabinets); (3) navigation model errors (e.g., blocking objects from opening); (4) subgoalprediction errors (e.g., picking up wrong objects); and (5) lack of state-aware multi-step planningand backtracking.
More qualitative results are available in Appendix ?? .
Discussion and Limitations
We showed that a persistent spatial semantic representation enables a hierarchical model to achievestate-of-the-art performance on a challenging instruction-following mobile manipulation task.
Themain performance bottlenecks include long-horizon exploration, perception generalization to unseen environments, and low-level motion planning for continuous collision avoidance.
In terms oflearning, incorporating reinforcement learning to train π H , π L , and observation model F jointlycould improve robustness. We deﬁned the interface to π L to be faithful to skills available on physical robots, but the exact implementation of π L is not the focus of our work.
Physical deploymentwould require changes to π L , and study on robustness to errors in continuous environments, such aslocalization or motion uncertainty.