H IERARCHICAL P OLICY L EARNING FOR L ONG H ORIZON I NTERACTIVE I NSTRUCTION F OLLOWING
Anonymous authors
Paper under double-blind review
A BSTRACT
Robotic agents performing domestic chores using natural language directives require to learn a complex task of navigating an environment and interacting with objects in it.
To address such composite tasks, we propose a hierarchical modular approach to learn agents that navigate and manipulate objects in a divide-andconquer manner for the diverse nature of the entailing tasks.
Speciﬁcally, our policy operates at three levels of hierarchy.
We ﬁrst infer a sequence of subgoals to be executed based on language instructions by high-level policy composition controller (PCC).
We then discriminatively control the agent’s navigation by a master policy by alternating between navigation policy and various independent interaction policies.
Finally, we infer manipulation actions with the corresponding object masks using the appropriate interaction policy .
Our hierarchical agent, named HACR (Hierarchical Approach for Compositional Reasoning) , generates a human interpretable and short sequence of sub-objectives, leading to efﬁcient interaction with an environment, and achieves the state-of-the-art performance on the challenging ALFRED benchmark.
[Figure]
our hierarchical policy decomposes a long-horizon task into multiple subtasks and leverages the high-level abstract planning, which enables an agent to better address long-horizon planning.
I NTRODUCTION
For the long-awaited dream of building a robot to assist humans in daily life, we now witness rapid advances in various embodied AI tasks such as visual navigation (Anderson et al., 2018; Chen et al.,
2019; Chaplot et al., 2017; Krantz et al., 2020), object interaction (Zhu et al., 2017; Misra et al., 2017) and interactive reasoning (Das et al.,
; Gordon et al., 2018).
Towards building such realistic robotic assistant, the agent should be capable of all of these tasks to address more complex problems.
A typical approach for combining these abilities is to build a uniﬁed model (Shridhar et al., 2020; Singh et al., 2021) to jointly perform different sub-tasks.
For example, the type of reasoning necessary for navigation differs signiﬁcantly from the reasoning required for object interaction; the former focuses on inferring the free space and a search strategy to reach a target location while the latter requires detection of objects and inferring their distances and states (Singh et al., 2021).
Meanwhile, human cognition process learns to divide a task into sub-objectives such as navigation or interaction, which enables humans to facilitate complex reasoning in a variety of circumstances (Hayes-Roth & Hayes-Roth, 1979).
Inspired by those, we propose a hierarchical agent that learns and infers low-level policies for different sub-tasks.
To evaluate our proposal in challenging scenarios, we particularly focus on the task of interactive instruction following, where the goal is to execute a set of navigation and object interaction tasks speciﬁed by natural language.
This task poses numerous challenges including long-term planning, partial observability, and irreversible state changes.
To complete a task successfully, an agent needs to navigate through hundreds of action sequences and interact with various objects in novel environments ( e.g ., dining room, and kitchen etc .) which necessitates thorough comprehension of both visual scenes and natural language instructions.
Speciﬁcally, we propose to disentangle the task into high-level subgoals which are executed by the respective subgoal policies.
Our agent is composed of (1) a policy composition controller (PCC) that speciﬁes sub-policy execution sequence, (2) a master policy (MP) that specialises in navigation and provides a signal to conduct interaction tasks, and (3) several interaction policies (IP), which execute autonomous low-level interactions for each subgoal type.
This disentanglement results in easier to analyze subtasks with shorter horizons.
Inspired by visual navigation literature (Yang et al., 2018; Wortsman et al., 2019) that encodes target object information for navigation, we propose an object encoding module (OEM) that provides such target object information and serves as a navigation subgoal monitor.
During navigation to locate an object, the agent faces uncertain obstacles that forces it to traverse in a loop (deadlock state).
We additionally propose a loop escape module (LEM) that assist the agent in circumventing such obstacles, based on a stored history of prior visual observations, thus resulting in better navigation.
We summarize our contributions as follows;
• We propose a hierarchical framework that decomposes the instruction following task into semantic subgoals and effectively addresses them with corresponding submodules.
• We propose object encoding module (OEM) that encodes object information from natural language instructions for effective navigation and a loop escape module (LEM) to escape a deadlock state with a stochastic policy.
• The proposed agent achieves state-of-the-art performance on the ALFRED benchmark.
R ELATED W ORK
Prior works propose numerous task setups and benchmarks for developing an agent to complete complicated tasks given natural language directives, such as agents trained to navigate (Li et al., 2020; Xu et al., 2021) or solve household tasks (Shridhar et al., 2020).
However, the vast majority of approaches for these tasks employ a ﬂat reasoning (Singh et al., 2021; Nguyen et al., 2021), in which the agent decides on the low-level actions accessible while moving through the environment (Gupta et al., 2017; Zhu et al., 2020).
When the prior arts seek to deﬁne subtasks, they are frequently deﬁned in a shallow manner with only two layers of hierarchy (Zhang & Chai, 2021;
Corona et al., 2020; Blukis et al., 2021;
Andreas et al., 2017; Gordon et al., 2018; Yu et al., 2019;
Das et al., 2019).
Furthermore, these strategies are inefﬁcient in terms of data requirements.
Natural Language is subjective, and even a seemingly simple command can contain several unstated meanings.
Because of this semantic gap, most present solutions involve either a large amount of labelled data or trial-and-error learning to map language to low-level actions.
Compounding errors also impede performance for activities with a long and complex horizon (Xu et al., 2019).
We introduce a system for representing and using hierarchical knowledge in order to better the control of embodied agents doing complex tasks stated in natural language directives.
Furthermore, because of the modular structure, our agent reasons and accomplishes tasks along longer paths than previous studies, spanning numerous subgoals.
The task of interactive instruction following necessitates not only navigation but also the interaction with surrounding objects by following natural language directives.
(Shridhar et al., 2020) proposed a CNN-LSTM-based baseline agent with progress tracking (Ma et al., 2019).
(Singh et al., 2021) offer a modular strategy that factorises perception for mask prediction and policy for action prediction.
(Nottingham et al., 2021) offer a modular system in which language encoding, visual state extraction, and action prediction are all trained independently in different modules.
(Nguyen et al., 2021) suggested a system for predicting actions and masks corresponding to different instructions independently.
They combine numerous visual inputs from navigable directions to provide a larger ﬁeld of view for improved navigation.
(Zhang & Chai, 2021) propose a Transformer-based hierarchical agent whereas (Suglia et al., 2021) present a Transformer-based agent that uses object landmarks for navigation.
(Pashevich et al., 2021) present a Transformer-based agent that uses a multimodal Transformer for exploiting the multiple input modalities.
(Blukis et al., 2021) propose a technique for successful high-level reasoning by reconstructing a 3D map for encode spatial semantic representation.
Finally, a modular policy with a two levels of hierarchy for instruction following have previously been proposed by (Corona et al., 2020) which results in poor performance on longhorizon task.
In contrast, our policy operates at three hierarchical levels exploiting the fact that navigation and interaction are semantically diverse activities that requires independent processing.
A PPROACH
We observe that the visual information for navigation varies considerably over time while for object interaction it is largely stationary.
As a consequence, the agent beneﬁts from learning different policy modules for the two tasks, because the former needs to reason about temporal history and global environment information, whilst the latter requires to focus on local visual cues for mask prediction.
Furthermore, the class imbalance between navigation and interaction atomic actions ( i.e. , navigation actions are far more frequent than the interaction actions) would bias a model towards more frequent navigation actions.
Inspired by these motivations, we design our modular architecture with three levels of hierarchy; (1) A high-level policy composition controller (PCC) that uses language instructions to generate a sequence of sub-objectives, (2) a master policy that specialises in navigation and determines when and where the agent is required to perform the interaction task, and (3) an interaction policy that is a collection of subgoal policies that specialises in precise interaction tasks.
We ﬁrst analyze each language instruction, and use the information to determine the basic interaction policies required to perform the task.
Depending on the underlying composition, the control of the agent is alternated between (1) master policy and (2) various specialised policies for object interaction.
Importantly, all interaction policies are compositional and independent, which allows formulating an instance speciﬁc policy execution sequence.
In particular, we learn seven interaction policies, each of which specialises in a different sub-objective and can be integrated in a precise order to accomplish long-horizon planning tasks.
We illustrate the overview of our proposed agent in Figure 2.
3. P OLICY C OMPOSITION C ONTROLLER
For the task of interactive instruction following, the trajectories can be divided into several meaningful subgoals and detailed instructions are provided for each subgoal.
Due to the requirement of long horizon planning for the task, we believe that it should be beneﬁcial to ﬁrst generate a highlevel subgoal layout for the given task and then tackle each subgoal individually.
To this end, we propose a policy composition controller (PCC), shown in Figure 2 which generates a sequence of subgoals S
= { s i } (where s i belongs to the set of subgaols deﬁned by benchmark) for each ‘stepby-step’ instructions corresponding to the given task.
The PCC’s predictions correlate to semantic subgoals, subjecting the agent’s logic to observation (‘What is the agent attempting to accomplish right now?’), which enables us to monitor the progress of task completion by the agent.
Speciﬁcally, we ﬁrst encode the language instructions with a Bi-LSTM, followed by a self-attention module.
The encoded step-by-step language instructions ˆ x i are used as input for the PCC to generate the subgoal
Surrounding Perception 3.
Turn left and walk to the microwave.
Heat the potato in the microwave.
Ego-Perception
Language Instruction
Policy Composition Controller
Interaction Policy k
Master Policy <M ANIPULATE >
Predicted Navigation Action equity Stochastic
Deterministic
False
L S T M m VLGround c on ca t
L S T M v VLGround c on ca t
M ask G e n e r a t
o r
Predicted Object Mask
P UT
Predicted Interaction Action
L S T M a VLGround c on ca t
Interaction Policies
Interaction Policy Interaction Policyk
Interaction PolicyK
Object Encoding
L S T M F C
R OTATE R IGHT
M OVE A HEAD P ICK U P
G OTO L OCATION H EAT O BJECT
G OTO L OCATION
Loop Escape
F C
Navigation Policy
Interactive Perception Module
Action Policy Module
G O T O L O C A
T I ON ?
N o
R e s N e t
Y e s ...
[Table]
sequences.
The agent must complete these subgoals in the speciﬁed order to accomplish the goal task.
Formally, for each language instruction ˆ x i , the PCC deﬁnes a distribution over the subgoal action space as:
[Equation]
where FC 1 denotes a single layer perceptron, and N subgoals denotes the number of subgoals i.e .
We train the PCC module in a supervised manner with the associated subgoal labels.
On the
ALFRED validation fold, the controller achieves 98.5% accuracy.
Kindly refer Sec.
A.3 for more details about these subgoals.
3. M ASTER P OLICY
We observed that the agent does not require information about the interactions it must perform for the given task when navigating in an environment.
We require the agent to learn how to move from one interaction location to another with the help of language instruction.
Inspired by these observations, we contend to use an independent module that can perform navigation while simultaneously marking the locations for object interaction along the way.
For the task, we propose a master policy, which is illustrated by the upper-right block in Figure 2, for generating the navigational action sequence based on the multi-modal inputs.
Let A n denote the set of primitive navigation actions { MoveAhead, RotateRight, RotateLeft, LookUp, LookDown } .
The master policy learns to navigate in the environment by learning a distribution over A n ∪ <MANIPULATE> , where <MANIPULATE> is the abstract token we introduced for the agent to signify when to move control to the next level of hierarchy, i.e ., the interaction policies, for completing manipulation sub-tasks.
The master policy comprises of three basic modules: (1) an object encoding module that provides information about the object the agent needs to locate for interaction.
(2) a navigation policy that outputs the navigation action sequence based on the multi-modal input for traversing the environment.
(3) a loop escape module that helps the agent in escaping deadlock states.
3.2. S UBTASK L ANGUAGE C OMBINATION
The language instructions for the interactive instruction following task can be divided into two types: a) Navigation and b) Interaction.
The agent needs to navigate to the necessary location and then perform interaction tasks, because of which the two types of instructions occur in pairs.
To describe the complete task, the combination is repeated several times with varying locations and interaction subgoals.
For the better utilisation of the language input, we propose a new way of encoding stepby-step instructions for the navigation task.
Speciﬁcally, we deﬁne a subtask instruction as a combination of ‘navigation to discover the relevant object’ and the ‘corresponding interactions’.
For instance, in the subtask Turn around and walk forward to the garbage bin on the ﬂoor by the TV.
Pick up the blue credit card on the TV stand.
, the agent needs to interact with the credit card, which is a crucial information for the agent and also serves as a navigational criterion, i.e ., the agent should stop if it encounters the credit card in close vicinity.
We observed that this information is missing in the ﬁrst navigation command, but is present in the second interaction instruction which works as a motivation for the proposed combination.
The navigation instruction provides the low-level information about the path to be traversed and the interaction instruction provides information about the object the agent needs to locate for appropriate interaction.
We encode these subtasks using an instruction encoder comprised of a Bi-LSTM followed by a self attention module.
ˆ l T : T +1 ,n refers to the encoded feature for the combined subtask instruction corresponding to the navigation subgoal T .
3.2. O BJECT E NCODING M ODULE
The task of instruction following can be broken down into two components, according to our observations: 1) locating the required object, and 2) interacting with the object.
We propose a module that takes as input the subtask language instruction l T : T +1 ,n and outputs an encoding for the object the agent must locate for interaction.
This guides the agent’s navigation, and we recommend using it as a subgoal monitor to signify the completion of the current navigation subgoal.
Navigation Subgoal Monitor.
The master policy learns to output an abstract token <MANIPULATE> along its navigation route, which passes control to a pretrained object detector (Mask RCNN) that validates whether the object provided by the OEM is present in the current frame or not.
If the agent discovers the item, it switches to the appropriate interaction policy; otherwise, it continues with the navigation.
The object encoder is composed of a Bi-LSTM, followed by a two layer perceptron which outputs the object class.
For this object, the master policy utilises a learnable 100-dimensional embedding as input, as demonstrated in Eq. 2.
3.2. N AVIGATION P OLICY
The navigation policy (cid:81) n receives the processed multi-modal data as input and produces a sequence of actions as output.
The architecture is based on the action prediction module of (Singh et al., 2021).
It uses visual features, subtask instruction features (Sec. 3.2.1), and the embedding of the preceding time step action as inputs.
Moreover, the object encoding supplied by the object encoding module (Sec. 3.2.2) is provided as input in order to further enhance the navigation capabilities of the agent.
The subtask instruction encoding l T : T +1 provides low-level information relevant for navigation as well as the information about the object that the agent needs to interact with.
This aids the agent in arriving at the correct location.
To capture the relationship between the visual observation and language instructions, we dynamically generate ﬁlters based on the attended language features and convolve visual features with the ﬁlters, denoted by ”VL-Ground” in Figure 2.
To summarise, the
LSTM hidden state h t,n of the master policy decoder, LSTM n , is updated with four different features concatenated together as:
[Equation]
k
[Equation]
[1 , N objects ]
[Equation]
where ˆ v pant denotes the attended visual features for the surrounding view as explained in Sec.
A.2 in appendix at time step t , ˆ l T : T +1 ,n denotes the attended subtask language features, a t − 1 ,n represents the action given by (cid:81) n in the previous time step, and o t denotes the object encoding given by the object encoding module .
In addition, we use the subgoal progress and overall progress monitors similar to (Shridhar et al., 2020) for training the navigation policy.
Details about the surrounding views is given in Sec.
A.2 in the Appendix.
[Figure]
3.2. L OOP E SCAPE M ODULE
While navigating through an environment, the agent is prone to be caught around unanticipated obstructions such as tables or chairs, which might cause navigation failure.
To circumvent such obstructions during navigation, (Singh et al., 2021) introduced obstruction evasion (OE).
However, in the scenario where the agent predicts a repeated series of activities (i.e., a deadlock state), the agent may fail to navigate even without such barriers, resulting in limited exploration.
The OE mechanism is unable to handle such challenges since repeated sequences of visual observations might not be recognised as obstacles based on visual similarity in consecutive frames.
To address the problem, we propose a loop escape module, which allows for improved exploration at inference time by avoiding deadlock conditions.
We use an external memory to memorise the history of visual observations, inspired by (Du et al., 2020).
The agent identiﬁes a deadlock state if any t (cid:48) exists that satisﬁes Equation 3, given a history of visual observations, v 0:
t , up to the present time step, t .
[Equation]
[Equation]
where W is a hyperparameter that indicates the length of the repeated actions.
In the event of a deadlock, the stochastic policy executes a random navigation action to assist the agent in escaping.
Figure 3 depicts an example of a deadlock state and the behaviour of loop escape module in the case when the agent is unable to ﬁnd a target object (a garbage can) due to occlusion (by the countertop).
As shown in Figure 3(a), the agent in a deadlock state tries to revisit previously explored areas, which may limit exploration.
Loop escape module assists the agent in escaping the impasse condition, thus ﬁnding the garbage can, as depicted in Figure 3(b).
3. I NTERACTION P OLICIES
To abstract a visual observation to a consequent action, the agent requires a global scene-level comprehension of the visual observation whereas for localisation task, the agent needs to focus on both global as well local object speciﬁc information.
Inspired by (Singh et al., 2021), we exploit separate streams for action prediction and object localization due to the contrasting nature of the two tasks, illustrated as ‘Interaction Policy k ’ in Figure 2.
The task require execution of varied subgoals with different levels of complexity.
For instance, Heat
Subgoal might require interaction with either a stove or a microwave whereas for Pickup Subgoal there is a variety of recepticles but the action sequence is simpler.
To focus on individual subobjectives, we train a module, (cid:81) m,k , for each subgoal where k ∈ [1 , N subgoals ] .
We observed that each interaction has its own properties and the navigation information history is irrelevant to
[Table]
the task that makes it necessary to use an isolated hidden state for each interaction subgoal.
More details about the architecture and training can be found in Sec.
A.1 in the Appendix.
R ESULTS AND D ISCUSSION
First we carry out a quantitative analysis of task success (Success Rate or SR) and goal condition success rates (Goal Condition or GC), and summarize the ﬁndings with prior approaches in Table 1.
We compare our method with following prior-arts on ALFRED benchmark, Seq2Seq (Shridhar et al., 2020), LAV (Nottingham et al., 2021), FPP (Singh et al., 2021), EmBERT (Suglia et al., 2021), E.T. (Pashevich et al., 2021), LWIT (Nguyen et al., 2021), HiTUT (Zhang & Chai, 2021), and HLSM (Blukis et al., 2021).
HACR outperforms the preceding approaches on all metrics, as indicated in the table.
Its ability to perform in unknown situations is evidenced by its increased success rate in unseen environments.
For the validation fold, HACR outperforms the state-of-the-art (Blukis et al., 2021) with an improvement of 83.84 % in seen task SR, and 69.15 % in unseen task SR.
For the test fold, HACR shows an improvement of 61.92 % in seen SR, and 2.5 % in in unseen SR.
It indicates that our approach enhances the agent’s overall grasp of all sub-tasks and their connection needed to complete a task successfully.
We also discuss performance over individual subgoals in Table 6 in Appendix.
4. C OMPARISON BETWEEN H IERARCHICAL AND F LAT P OLICY A GENTS
We present the analysis of the learning process for the hierarchical and ﬂat policy agents for seen and unseen environments.
The performance of our hierarchical agent and the ﬂat agent are compared quantitatively in relation to the number of iterations (expressed in terms of epochs), and the results are presented in Figure 4.
As shown, hierarchical policy gives a major improvement over the ﬂat policy.
Its ability to perform in novel environments is evidenced by its increased success rate in unseen scenes.
As depicted in Table 2 (#b), for seen and unseen task SR, the hierarchical agent outperforms the ﬂat policy by 9.5% and 49.5%, respectively.
In both seen and unseen ‘Goal-Condition’
SR, the hierarchical approach outperforms, with improvements of 8.3% and 29.1%, respectively.
The greater performance of the hierarchical approach on both overall task success rate and goal condition suggests its comprehension of both short-term subtasks and long-horizon whole tasks.
Also, the hierarchical agent approach convergences signiﬁcantly sooner than the monolithic agent (25th epoch vs 37th epoch), as shown in Figure 4a, demonstrating the computation-efﬁciency of our approach.
Our policies go through two stages of training.
We start by training the seven interaction
[Figure]
P i ck &
P l a c e
C oo l & P l a c e
S t a ck &
P l a c e
H ea t & P l a c e
C l ean & P l a c e E x a
m
i ne
P i ck 2 &
P l a c e
Hierarchical
Flat
[Figure]
policies, which collectively take two epochs to converge.
We included them in the computation and began the hierarchical curve from the third epoch, which is the ﬁrst epoch of our master policy.
Figure 4b represents the average length of a successful trajectory traversed by the hierarchical or ﬂat policy agent, contrasting the efﬁciency of each agent.
The hierarchical agent consist of the master policy that is dedicated solely to navigation, giving it a signiﬁcant advantage over the ﬂat policy agent that learns everything using the same network parameters.
It was observed that due to the wide action space, the ﬂat policy agent occasionally executes irrelevant interactions along the trajectory, which is not the case with HACR.
The dedicated action sets for the master policy and interaction policies improve HACR by allowing the agent to avoid any unnecessary interactions while traversing to discover the desired object.
The interaction policy modules also perform signiﬁcantly better because they only master certain short-horizon tasks, which speeds up and simpliﬁes the learning process.
Table 6 in Appendix provides information on the subgoal performances for each module.
4. I NTERPRETABLE S UBGOAL M ODULES
One of the beneﬁts of the proposed approach is that it generates semantically meaningful subgoals.
The generated subgoals are far more interpretable than a low-level action sequence.
For instance, the low-level put action might be associated to any of the subgoals such as Heat, Cool, or Put.
The highlevel semantics are reasoned about by the hierarchical agent, and the agent’s intent is considerably clearer.
If the agent attempts to place the object in a refrigerator, it is most likely cooling it; if the receptacle is a microwave, it is attempting to heat it; and if the receptacle is a table, it is simply a pickup subgoal.
In contrast, the ﬂat policy agent considers it as a single atomic action regardless of the object and receptable.
We present an example for more detail in Figure 7 in appendix.
4. C OMPARISON BETWEEN M ODULAR AND N ON -M ODULAR P OLICY A GENTS
In modular networks, the decision-making process is separated into numerous modules.
Each module is designed to perform a certain function and is put together in a structure that is unique to each trajectory instance.
Because of their compositional nature, such networks with the help of specialised modules often perform better in new environments than their ﬂat counterparts (Hu et al., 2019;
Blukis et al., 2019)
We present a quantitative comparison of interaction policies’ modular structure (Table 2 #c).
For this experiment, we train a single policy module to learn all interaction tasks.
The decoupled pipeline for action and mask prediction, as well as the rest of the settings, are preserved.
The modular agent outperforms the non-modular agent by 2.30% and 7.08% in seen and unseen task SR, respectively.
It also performs signiﬁcantly well in both seen and unseen ’Goal-Condition’ criteria, with gains of
[Table]
2.19% and 9.17%, respectively.
The greater performance of the modular policy in both task and goal-condition metrics highlights the beneﬁts of modular structure in long-horizon planning tasks.
4. A BLATION S TUDY
We conduct a series of ablation analysis on the components of HACR and report the results in Table 2 to evaluate the signiﬁcance of each module with empirical studies.
(a) Object encoding module (OEM).
First, we investigate the ablation of Object Encoding Module.
We remove the navigation subgoal monitor and train the navigation policy without object information.
The agent can complete some objectives, but it lacks object information, which functions as a stopping criterion, preventing it from navigating properly.
Therefore, it is unable to completely comprehend the relationship between the step-by-step instruction and the visual trajectory, limiting the agent’s capacity to explore and connect various interaction policies required for the task’s completion, thus resulting in a huge performance drop.
(b) Navigation interaction hierarchy (NIH).
Next, we demonstrate the importance of hierarchy between navigation and interaction policies.
For this ablation, we utilize the same network for learning the navigation and interaction action prediction.
For target class prediction, we preserve the IPM.
To eliminate the beneﬁt of the language distribution, we use the concatenation of all stepby-step instructions as the language input and conduct action and mask prediction while leaving the other modules unaltered.
The ablated model’s task success rates drop dramatically, showing that it is unable to effectively utilise the available inputs.
(d) and (e) Individual Components Next, we provide the individual performance of the two major components of our framework which brings the most empirical gain, OEM and NIH in row (d) and (e) respectively in the absence of other components.
Even in the lack of hierarchy and modular structure, the agent (d) has signiﬁcantly higher performance than (Singh et al., 2021), highlighting the relevance of target object information for navigation and the effectiveness of our OEM.
The lower performance of agent (e) denotes that the hierarchical architecture requires the inclusion of other components for optimal performance.
The agent’s performance increases when these components are combined (#c vs (#d and #e)) indicating that the components are complimentary.
C ONCLUSION
We address the problem of interactive instruction following.
To effectively tackle the long horizon task, we propose Hierarchical Approach for Compositional Reasoning (HACR), that decomposes the task into high-level subgoals and accomplishes each subgoal with a specialized submodule.
We propose a policy composition controller (PCC) to disentangle the task into high-level subgoals.
For improving the navigation performance, we propose an object encoding module (OEM) to explicitly encode target object information during internal state update.
We also propose a loop escape module to circumvent the deadlock states during navigation, for better exploration.
Our agent outperforms prior arts by signiﬁcant margins and achieves a new state-of-the-art.
E THICS S TATEMENT
The embodied AI research will expedite the deployment of AI systems to help humans by its versatility of adapting to a new environment out of the factory or research labs.
As all embodied AI, however, would suffer from adversarially annotated data as well as data bias in a form of language instructions, which may cause ethnic, gender or biased gender issues, the proposed method would not be an exception.
Although the proposed method has no intention to allow such problematic cases, the method may be exposed to such threats.
Relentless efforts should be made to develop mechanisms to prevent such usage cases in order to make the robotics AI safer and enjoyable to be used by humans.
R EPRODUCIBILITY S TATEMENT
We take the reproducibility of research very seriously and describe the algorithm in detail in Sec.and Sec.
A.1.
We also solemnly promise to release all codes, environment information, learned models and complete task conﬁgurations in a public repository
A A PPENDIX A. D ETAILS FOR I NTERACTION P OLICY
Let A m denote the set of primitive interaction actions { Open, Close, ToggleOn, ToggleOff, Pickup,
Put } .
The APM architecture consist of an LSTM decoder that takes as input the attended language encoding ˆ l T,m corresponding to the current manipulation subgoal, the egocentric visual features ˆ v egot,m and the previous time step action a t,m and outputs a distribution over the manipulation action space
A m ∪ <STOP> .
The manipulation policy uses <
STOP> token as an indicator for completion of the interaction task and shifts the control to the master policy for further task completion.
Formally:
[Equation]
k
[Equation] [Equation]
| A m
[Equation] [Equation]
The IPM uses the same inputs as APM and outputs the object class that the agent needs to interact with at the current time step.
Similar to (Singh et al., 2021), we use a pretrained Mask-RCNN to obtain the interaction mask corresponding to the object class.
For more details about the architecture, kindly refer to (Singh et al., 2021).
[Equation]
We train our interaction policies in a supervised manner using imitation learning from expert groundtruth trajectories.
Speciﬁcally, each interaction policy is trained to minimize the cross entropy loss as:
[Equation]
where p m =
FC m ([ˆ v egot ; ˆ l T,m ; a t − 1 ,m ; h apmt,m ]) and p v =
FC v ([ˆ v egot ; ˆ l T,v ; a t − 1 ,v ; h vpmt,v ]) denotes the probability distribution of action and class prediction, respectively, and a ∗ t,m and c ∗ t,v denotes the corresponding ground truth actions and classes obtained from given expert trajectories, respectively.
A. S URROUNDING P ERCEPTION
HACR investigates and perceives surrounding views to improve the agent’s perception and gain a better comprehension of the environment.
Let v dt be a visual feature from each navigable direction, d ∈ [0 , D ] , at the time step, t , where D represents the number of navigable directions and d =the agent’s egocentric direction.
After gathering additional observations, v 0: D t , we acquire attended visual features, ˆ v pant , using the dynamic ﬁlters and concatenate all attended visual features as:
[Equation]
Using the attended visual features, we update the master policy decoder and predict masks and actions as described in Equation 2.
A. E XPERIMENTAL D ETAILS
Dataset and Metrics.
For training and evaluation, we used the ALFRED benchmark (Shridhar et al., 2020).
This benchmark provides expert trajectories for the agents performing household tasks (e.g. Boil an egg) in a simulated environment on AI2-THOR (Kolve et al., 2017).
The dataset is divided into three parts; ’train’, ’validation’, and ’test’ set.
To evaluate the generalisation ability of an embodied agent to novel environments, the benchmark further divides ’validation’ and ’test’ trajectories into seen and unseen splits.
Unseen comprises a set of rooms which are held out during training and scenes that are exposed to the agent during the training are termed as seen .
The benchmark provides a high-level Goal Statement describing the ﬁnal task and several low-level
Step-by-Step Instruction describing each subgoal that needs to be accomplished in the given order for successful completion of goal task.
The benchmark has 7 types of subgoals; G OTO L OCATION , P ICKUP O BJECT , P UT O BJECT , C OOL O BJECT , H EAT O BJECT , C LEAN O BJECT , S LICE O BJECT , and T OGGLE O BJECT .
G OTO L OCATION denotes navigation subgoal where the agent requires to locate a target object for interaction.
P ICKUP O BJECT requires the agent to pick up an object.
P UT O BJECT requires the agent to put an object onto a receptacle.
C OOL O BJECT , H EAT O BJECT , and C LEAN O BJECT requires the agent to cool, heat, and clean an object while interacting with the required receptacle ( e.g ., Fridge, Microwave, SinkBasin, etc.).
S LICE O BJECT requires the agent to slice an object using a knife, resulting in sliced objects.
T OGGLE O BJECT requires the agent to toggle on or off an object such as a ﬂoor lamp.
Following the common evaluation protocols (Blukis et al., 2021), we evaluate our agent on two metrics.
Success rate (SR) provides the percentage of tasks that had all successful goal conditions.
Goal condition rate (GC) is the percentage of satisﬁed goal-conditions across all tasks.
Training and Evaluation.
We begin by assessing the performance of each subgoal policy in relation to its speciﬁc task.
On the ALFRED validation set, we present the success rate for each subgoal in Table 6.
To accomplish this, we employ the expert trajectory to guide the agent through the episode until it reaches the subgoal task.
The agent then uses the task speciﬁc policy to infer based on the language directive and current visual frame.
Following that, we train and evaluate the master policy’s performance during independent behaviour cloning on expert trajectories, i.e. assuming oracle subgoal-policies.
With ground truth subgoal sequences, the master policy is able to generalise effectively and achieves 29.96% success rate (SR) on unseen validation data.
Then we train the Object Encoding Module and the Policy Composition Controller in isolation and combine all the modules together as described in Sec. 3.
Implementation Details.
Visual features are encoded with a pretrained ResNet-
(He et al., 2016) after resizing input images to 224 × 224 .
To perceive surrounding views, we additionally gather visual observations from four navigable directions
(i
[Table]
e., R OTATE L EFT , R OTATE R IGHT ,
L OOK U P
[Table]
and L OOK D OWN ).
We train each module of our approach independently using the Adam
Optimizer with the initial learning rate of 10 − 3 .
We augment visual features by shufﬂing the channel order of each image (Singh et al., 2021) and applying predeﬁned image augmentations (Cubuk et al., 2019).
We set the coefﬁcients for the subgoal and overall progress monitor objectives as 0.2, and 0.3 respectively.
The value of W for the loop escape module is set as 10 using a grid search on validation set.
A. F URTHER A BLATION S TUDY A.4.
[Table]
a) Data augmentation (DA).
We start by ablating the data augmentations (described
in Sec.
A.
which lowers the performance across the board, highlighting its usefulness in training a smarter agent by reducing imitation learning sample complexity.
This results in a performance drop of 12.6% and 25.0% in the seen and unseen task success rate respectively.
(b) Factorised interaction policies (FIP).
Then, under the interaction policies, we replaced the decoupled pipeline with a uniﬁed pipeline (Singh et al., 2021) for both action sequence prediction and interaction mask generation.
Due to shared processing for mask and action modules, the ablated model’s performance drops substantially, showing its inability to effectively utilize the language information.
(c) Object-centric mask prediction (OCMP).
Finally, we train the interaction policies without the use of object based mask prediction which uses a Mask-RCNN as described in Sec.
A.3.
Similar to (Shridhar et al., 2020), we directly upsample the joint embedding using deconvolution layers to generate the interaction mask.
We see a signiﬁcant decline in performance on both seen and unseen folds due to poor mask generating ability, emphasising the need of object information for consistently accurate mask formation
[Table]
A.4. I NPUT A BLATION
To further investigate our agent’s vision and language bias, we ablate the inputs to our model in
Table 4.
Egocentric View only.
This ablation speciﬁes the condition under which the agent is deprived of its surrounding views (Sec A.2) while navigating without affecting any other module.
The agent continues to perform admirably and outperforms the HLSM (Blukis et al.
on the validation fold by a large margin in seen trajectories, while also providing a signiﬁcant improvement in task success rate in unseen environment.
This demonstrates the signiﬁcance of peripheral perceptions in navigating a novel environment.
No Language.
We discovered that when the agent receives only visual inputs without the language instructions, it is still capable of doing some tasks by memorising common sequences and target classes in the ‘seen’ fold.
but does not adapt to the unseen environment.
No Vision.
This ablation demonstrates that the agent can still perform some subgoals, mostly navigation, by following the instructions to move along a few short trajectories, but not able to complete goal tasks because it involves mask generation (object selection), which is entirely dependent on visual observation.
Goal-Only.
We deﬁne the setting where both the navigation and interaction policies receive only the goal task information.
It gives a general idea of the task but doesn’t contain any information about the tedious action combinations the agent needs to perform to get the task done.
This leads to a signiﬁcant depletion in the task success rate.
Single Navigation Instruction.
This is when the master policy only receives the navigation instruction corresponding to the current navigational subgoal.
The agent is able to somewhat navigate but doesn’t have the object information that acts as the stopping criterion which hinders the agent’s ability to perform interaction at the correct place in the environment.
This demonstrates the beneﬁt of our object-centric navigational approach.
A. A DDITIONAL R ELATED W ORKS
In this section we extend our realated works, to provide a detailed technical comparison with the prior arts involving hierarchical frameworks (Blukis et al., 2021; Zhang & Chai, 2021).
Comparison with HLSM (Blukis et al., 2021).
Here we compare the technical aspects of HACR and HLSM.
Both the works uses action hierarchy to decompose trajectory into high-level subgoal actions and subgoals into low-level action sequences.
There are differences between the subgoals anticipated by HACR and HLSM.
HLSM only employs interaction subgoals and considers navigation to be part of the same subgoal in order to reach the place of interaction, whereas HACR includes an extra navigation subgoal ‘GoTo(object)’ in which the agent navigates the environment to discover the object of interest.
HACR also contains the compound subgoals for Clean and Heat , whereas
HLSM would forecast the complete sequence of primary subgoals like Put, Pick, Toggle etc.
In terms of model architecture, HLSM deﬁnes the task layout by using a hierarchical model with high-level and low-level controllers.
The navigation and interaction components of a particular subgoal are collaboratively worked on by the low-level controller.
HACR, on the other hand, adopt a three-level hierarchy with independent modules for each level, namely, PCC, which provides the high-level structure of the trajectory, MP, which is responsible for navigation, and IP, which is composed of modules specialised for each interaction job.
We believe that HLSM and HACR are complementary because HLSM uses depth supervision to estimate the 3D layout of the environment for detailed visual understanding while our HACR exploits high-level language instructions for action planning. The depth supervision could be complementary to low-level instructions because the supervision of HLSM provides rich visual information by depth, which contains visual details, while the supervision of our HACR provides abstract language information for high level guidance for actions.
We argue that it is non trivial to claim which supervision is less or more.
Using the Object Encoding Module, HACR also adds a navigational subgoal monitor that checks the availability of object in the scene before executing the interaction policy.
A possible reason that HACR shows strong performance in seen but less in unseen is insufﬁciency of data.
Due to the limited data size for imitation learning of the given benchmark (note that despite its size being largest in the literature, it is an ofﬂine dataset, i.e. , the training data are obtained from the simulator by a single expert trajectory not using the simulator to learn various possible scenarios) and large language domain gap in seen/unseen environments, generalization is challenging.
As the detailed supervision about the environment, e.g., depth information, may help generalization (HLSM), using both supervision for environments and detailed language would reduce the generalization gap further.
Comparison with HiTuT (Zhang & Chai, 2021).
We also compare the technical aspects of HACR and HiTuT. HiTut also employs action hierarchy to decompose trajectory into high-level subgoal actions and subgoals into low-level action sequences like HACR and HLSM.
Even though HiTuT divides the task into a hierarchy of high-level subgoals and low-level actions, they utilize a uniﬁed transformer model with multiple heads to predict the subgoals and actions.
In contrast, HACR consists of independent modules for subgoal planning, navigation and varying interaction tasks.
The modular design allows for separate processing via specialised modules, making the learning process simpler due to short-term reasoning and enables easy inclusion of additional interaction subgoals.
A. A DDITIONAL E XPERIMENTAL R ESULTS
There are seven high-level task categories in the ALFRED benchmark (Shridhar et al., 2020).
Table 5 presents the quantitative comparison of our approach with a ﬂat policy, a seq2seq model (Shridhar et al., 2020), and prior hierarchical baselines (Blukis et al., 2021; Zhang & Chai, 2021) for each task type.
Blukis et al.
(2021) outperforms our agent on short-horizon tasks like ‘Pick & Place’ and ‘Examine.’
The two most complex and longest-horizon task categories in ALFRED are ‘Stack &
Place’ and ‘Pick Two & Place.’
On these task types HACR gives an unseen success rate of 13.8% and 22.2%, respectively, compared to HLSM’s 1.8% and 12.3%, and outperforms all baselines inof the 7 task types in unseen SR.
Overall, HACR outperforms all the baselines in terms of average seen and unseen success rates.
This is inline with the detailed analysis in Sec.
4.3 and 4.1 which describes the effectiveness of our approach to long-horizon trajectories. [Table]
[Table] Table 6 presents the performance of individual subgoal policies on corresponding subgoals.
Our hierarchical agent relatively outperforms the ﬂat policy by 50% and 21% average on unseen and seen validation set respectively.
The modular nature of our interaction policies allows them to learn speciﬁc short horizon subgoal tasks as illustrated in Figure 8-14, which reduces the complexity and makes the learning faster and easier
[Table]
A.6. A BLATION STUDY ON VISUAL INPUT FOR POLICY COMPOSITION CONTROLLER
The impact of visual features on PCC is presented in the Table 7.
The agent does not gain from accessing visual information, as evidenced by the drop in performance for PCC w/ vision. We believe that this happens due to the ‘causal misidentiﬁcation’ phenomenon (de Haan et al., 2019); when receiving more information ( e.g. , RGB observation), the agent could learn to exploit irrelevant information ( e.g. , a chair in the room) to a target task ( e
[Table]
g. , Heat a potato) rather than necessary information ( e.g. , Microwave and potato), leading to performance drop.
[Table]
A.6. A BLATION STUDY ON VISUAL INPUT FOR MASTER POLICY
The impact of visual features on MP is presented in the Table 8.
The MPP w/o vision demonstrates that the agent can still complete a few goal conditions, mostly the ﬁrst navigation instruction, by following the instructions to move along a few short trajectories, but not able to complete goal tasks because it involves interaction with objects in the environment i.e. localising objects, which is entirely dependent on visual observation.
[Table]
A.6. S ENSITIVITY ANALYSIS FOR LOOP ESCAPE MODULE (LEM) WINDOW ( W )
The impact of the value of W for Loop Escape Module (LEM) is examined in Figure 5 by conducting a grid search.
We observe that our agent achieves the best performance with W = 10 , which indicates the efﬁcacy of LEM.
Note that the performance does not drastically decrease by other values of W , which indicates that it prevents the agent from performance degradation by other values of W .
A. W HAT TO LEARN ? VS . W HAT TO BAKE IN AS AN INDUCTIVE BIAS ?
We agree that our method is a more or less ‘classical’ pipelined approach and has many carefully designed components as a set of inductive biases.
We attribute our design to (1) lack of sufﬁcient training data and (2) requirement of heavy computational cost.
(1) Given our framework is the behavior cloning (BC) (Bain & Sammut, 1995), though the benchmark we used is the largest in the literature, the rigid nature of the BC approach ( i.e. , any deviation from the expert trajectory over time is not allowed to learn a behavior) requires vast amount of training examples ( i.e. , may be not sufﬁcient to be the largest dataset in the literature) to learn a satisfactory model.
(2) To overcome the rigidity of learning a BC model, there are many approaches proposed in the literature (Ho & Ermon, 2016; Fu et al., 2017).
But they are mostly computationally expensive (Kostrikov et al., 2018).
We aim the niche between both problems by designing well crafted components for better encoding the given dataset even in the rigid BC framework.
Similar to our answer to the previous question, our inductive bias could be useful for this task not just for this benchmark by the analogy of convolutional neural networks for images.
By the universal approximation theorem, the 2D information, e.g., images, could be encoded by a multi-layer perceptron (MLP) with sufﬁcient amount of data.
Given that it is difﬁcult to fathom the sufﬁcient amount of data to train the MLP, we have a very successful design choice for encoding 2D information; the patch wise encoding and convolutional operations for its processing.
Now, with 1M images, e.g. , the ImageNet-1K, we can have a decently performing image classiﬁer.
We hope that our designed components could be useful to inspire the future research in designing a data driven model to address the Embodied AI task successfully.
In sum, we pose our work as a stepping stone for designing and learning a successful data driven model in the future.
A. D OES SEPARATE NETWORKS INCREASE HARDWARE / TRAINING / COMPUTATIONAL COST ?
In this section, we provide a discussion about the hardware, training, and computational cost when learning and exploiting separate networks for each subgoal.
A.8. H ARDWARE COST
Our model would increase the memory cost linearly by the number of subgoals (0.23GB per subgoal) because models for each policy should be loaded in the memory proportional to the number of subgoals.
Considering the large sized memory in modern computer systems ( e.g. , usually larger than 8GB for PC and 6GB for high-end mobile phones), the additional memory cost may not be considered large, but it is certainly a cost.
A.8. T RAINING COST
The training cost for each policy may marginally or may not increase because we divide a task into multiple subgoals and learning the specialized model for each subgoal takes a few epochs as the training set for each policy is easier ( i.e. , sharing similar semantics) for training than the data for the ﬂat model .
In Figure 4 (a) in the revision, we show that learning the specialized networks requires less training epochs compared to learning a uniﬁed network.
Therefore, even though there are multiple subgoals to be learned, the total training cost would increase marginally (and even embarrassingly parallelizable) or even decrease thanks to the reduced training cost per each subgoal added to an overhead of predicting subgoals.
A.8. C OMPUTATIONAL COST
The computational cost may marginally increase or even decrease because specialized networks for subgoals could infer each subgoal efﬁciently by reducing unnecessary exploration.
Figure 4 (b) in the revision shows that the specialized networks require shorter mean episode lengths to accomplish tasks.
With the efﬁcient subgoal achievement, the overall computational cost at inference would be similar or even decrease, compared to the model with the ﬂat policy learning.
[Figure]
[Figure]
Pickup
Put
Cool
Heat
Clean
Slice
Toggle
F l a t
P o li cy
H i e r a r c h
i ca l P
o li cy
Policy CompositionController
MASTERPOLICY
M A N
I P
M A N
I P
M A N
I P t= t=4 t= t= t= t= t= t= t= t= t= t=
HEAT Pickup
Cool t= t= t= t= t= t= t= t=
A HEAD
A HEAD
R IGHT R IGHT
A HEAD
L EFT
R IGHT
L EFT t= t= t= t= t= t= t= t= t= t= t= t=
[Figure]
potato
[Figure]
the microwave on the left.
Cook the potato in the microwave then remove it.
[Figure]
Open
Put
Close
On
Off
Open
Pick
Close
Open
Put
Close
Pick
>
On completion of the interaction task, the control is shifted back to the navigation policy.
hidden state of the navigation policy is keeps a continuous encoding of navigation history without any interference of the whereas each interaction uses an independent hidden state.
The ﬂat shown at the on the hand shares the same parameters for both navigation and interaction
This
[Figure]
[Figure]