L EARNING W HEN AND W HAT TO A SK : A H IERARCHICAL R EINFORCEMENT L EARNING F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Reliable AI agents should be mindful of the limits of their knowledge and consult humans when sensing that they do not have sufﬁcient knowledge to make sound decisions.
We formulate a hierarchical reinforcement learning framework for learning to decide when to request additional information from humans and what type of information would be helpful to request.
Our framework extends partially-observed Markov decision processes (POMDPs) by allowing an agent to interact with an assistant to leverage their knowledge in accomplishing tasks.
Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework: aided with an interaction policy learned by our method, a navigation policy achieves up to a 7 × improvement in task success rate compared to performing tasks only by itself.
We ﬁnd that the ability to request subgoals enables the agent to generalize effectively to tasks in unseen environments.
We analyze beneﬁts and challenges of learning with a hierarchical policy structure and suggest directions for future work.
I NTRODUCTION
Human-agent communication at deployment time has been under-explored in machine learning, where the traditional focus has been on building agents that can accomplish tasks on their own (full autonomy). Nevertheless, enabling an agent to exchange information with humans during its operation can potentially enhance its helpfulness and trustworthiness. The ability to request and interpret human advice would help the agent accomplish tasks beyond its built-in knowledge, while the ability to accurately convey when and why it is about to fail would make the agent safer to use.
In his classical work, Grice (1975) outlines the desired characteristics of cooperative communication, commonly known as the Gricean maxims of cooperation .
Among these characteristics are informativeness (the maxim of quantity) and faithfulness (the maxim of quality).
Human-agent communication in current work has fallen short in these two aspects. Traditional frameworks like imitation learning and reinforcement learning employ limited communication protocols where the agent and the human can only exchange simple intentions (requesting low-level actions or rewards Torrey & Taylor (2013); Knox & Stone (2009)). More powerful frameworks like (Nguyen & Daum´e III, 2019; Nguyen et al., 2019;
Kim et al., 2019) allow the agent to process high-level instructions from humans, but the agent still only requests generic help.
Recent work in natural language processing endows the agent with the ability to generate rich natural language utterances (Camburu et al., 2018; Rao & Daum´e III, 2018; De Vries et al., 2017; Das et al., 2017), but the communication is not faithful in the sense that the agent only mirrors human-generated utterances without grounding its communication in self-perception of its (in)capabilities and (un)certainties. Essentially it learns to convey what a human may be concerned about, not what it is concerned about.
This paper presents a hierarchical reinforcement learning framework named H ARI ( H uman- A ssisted R einforced I nteraction), which supports richer and more faithful human-agent communication. Our framework allows the agent to learn to convey intrinstic needs for speciﬁc information and to incorporate diverse types of information from humans to make better decisions.
Speciﬁcally, the agent in H ARI is equipped with three information-seeking intentions: in every step, it can choose to request more information about (i) its current state, (ii) the goal state, or (iii) a subgoal state which, if reached, helps it make progress on the current task.
Upon receiving a request, the human can transfer new information to the agent by giving new descriptions of the requested state.
These de-
[Figure]
scriptions will be incorporated as new inputs to the agent’s decision-making policy. The human thus can transfer any form of information that can be interpreted by the policy (e.g., asking the agent to execute skills that it has learned, giving side information that connects the agent to a situation it is more familiar with).
Because the agent’s policy can implement a variety of model architectures and learning algorithms, our framework opens up many possibilities for human-agent communication. To enable faithful communication, we teach the agent to understand its intrinsic needs by interacting with the human and the environment (rather than imitating human behaviors). By requesting different types of information and observing how much each type of information enhances its decisions, the agent gradually learns to determine which information is most useful to obtain in a given situation.
With this capability, at deployment time, it can choose when and what information to ask from the human to improve its task performance. To demonstrate the effectiveness of H ARI , we simulate a human-assisted navigation problem where an agent has access to only sparse information about its current state and the goal, and can request additional information about these states.
On tasks that take place in previously unseen environments, the ability to ask for help improves the agent’s success rate by 7 × higher compared to performing tasks only by itself.
This human-assisted agent even outperforms an agent that always has access to dense information in unseen environments, thanks to the ability to request subgoals.
We show that performance of the agent can be further improved by recursively asking for subgoals of subgoals.
We discuss limitations of the policy’s model and feature representation, which suggest room for future improvements.
M OTIVATION : L IMITATIONS OF THE S TANDARD POMDP F RAMEWORK
We consider an environment deﬁned by a partially-observed Markov decision process (POMDP)
E = ( S , A , T, c, D , ρ ) with state space S , action space A , state-transition function T : S ×
A → ∆( S ) , cost function c : S ×
A → R , description space D , and description function ρ : S → ∆( D ) .
Here, ∆( Y ) denotes the set of all probability distributions over a set Y .
We refer to this environment as the operation environment because it is where the agent operates to accomplish tasks.
Each task in the environment is deﬁned as a tuple ( s 1 , g 1 , d g 1 ) where s 1 is the start state, g 1 is the goal state, and d g 1 is a limited description of g 1 .
Initially, a task ( s 1 , g 1 , d g 1 ) is sampled from a task distribution T .
An agent starts in s 1 and is only given the goal description d g 1 .
It has to reach the goal state g 1 within H time steps.
Let g t and d gt be the goal state and goal description being executed at time step t , respectively.
In a standard POMDP, g t = g 1 and d gt = d g 1 for 1 ≤ t ≤ H .
But later, we will enable the agent to set new goals via communication with humans.
At any time step t , the agent does not know its true state s t but only receives a description d st ∼ ρ ( s t ) of the state.
Generally, the description can include any information coming from any knowledge source (e.g., an RGB image and/or a verbal description describing the current view).
Given d st and d gt , the agent then makes a decision a t ∈ A , transitions to the next state s t +1 ∼ T ( s t , a t ) , and receives a cost c t (cid:44)
c ( s t , a t ) .
A special action a done ∈
A is taken when the agent decides to terminate its execution.
The goal of the agent is to reach g 1 with minimum cumulative cost
C ( τ ) =
(cid:80)
Ht =1 c t , where τ = ( s 1 , d s 1 , a 1 , s 2 , d s 2 , a 2 , . . .
, s H , d sH ) is an execution of the task.
As the agent does not have access to its true state, it can only make decisions based on the (observable) partial execution τ 1: t =
( d s 1 , a 1 , . . .
, d st ) .
Kaelbling et al.
(1998) introduce the notion of a belief state b ∈ ∆( S ) , which sufﬁciently summarizes a partial execution as a distribution over the state space S .
In practice, when S is continuous or high-dimensional, representing and updating a full belief state (whose dimension is |S| ) is intractable.
We follow Hausknecht & Stone (2015), using recurrent neural networks to learn compact representation of partial executions.
We denote by b st a representation of the partial execution τ 1: t and by B the set of all possible representations.
The agent maintains an operation policy ˆ π : B × D → ∆( A ) that maps a belief state b s and a goal description d g to a distribution over A .
The learning objective for solving a standard POMDP is to estimate an operation policy that minimizes the expected cumulative cost of performing tasks:
[Equation]
where P π ( · | s 1 , d g 1 ) is the distribution over executions generated by a policy π given start state s 1 and goal description d g 1 .
In a standard POMDP, an agent performs tasks by executing its own operation policy without asking for any external assistance.
Moreover, the description function ρ and the goal description d g 1 are assumed to be ﬁxed during a task execution.
As seen from Equation 1, given a ﬁxed environment and task distribution, the expected performance of the agent is solely determined by the operation policy ˆ π .
Thus, the standard POMDP framework does not provide any mechanism for improving the agent’s performance other than enhancing the operation policy.
L EVERAGING H UMAN K NOWLEDGE TO B ETTER A CCOMPLISH T ASKS
We introduce an assistant into the operation environment, who can provide information about the environment’s states.
We assume the agent possesses a pre-learned operation policy ˆ π .
This policy serves as the common ground between the agent and the assistant, which is a prerequisite for communication between them to occur. For example, this policy represents a set of basic skills that agent has mastered and the assistant can refer to. In general, the more knowledge encoded in this policy, the more effectively the agent can communicate with and leverage help from the assistant.
Our goal is to learn an interaction policy ψ θ (parametrized by θ ) that controls how the agent communicates with the assistant to gather additional information.
The operation policy ˆ π will be invoked by the interaction policy if the latter decides that the agent does not need new information and wants to take an operating action.
The assistant aids the agent by giving new (current or goal) state descriptions, connecting the agent to situations on which it can make better decisions.
Consider an object-ﬁnding navigation task where a robot has been trained to reliably navigate to the kitchen from the living room of a house.
Suppose the robot is then asked to “ ﬁnd a mug ”, an object that it has never heard of. The assistant can help the robot accomplish this task by giving a more informative goal description “ ﬁnd a mug in the kitchen ”, relating the current task to the kitchen-ﬁnding task that the robot has been familiar with.
The robot may also have problems with localization: it knows how to get the kitchen from the living room but it may not realize that it is currently the living room. In this case, giving a current-state description that speciﬁes this information provides the robot with a useful hint on what actions to take next.
Our framework allows the assistant to convey any form of information that the agent can incorporate into its input. As discussed in §2, the notion of “state description” in our framework is general that captures various types of information, including but not limited to visual perception and verbal description. Communication between the agent and the assistant can be ﬂexibly enriched by designing the agent’s operation policy to be able to consume the forms of information of interest (e.g., a policy that takes natural language as input).
Communication with the Assistant.
The assistant is present all the time and knows the agent’s current state s t and the goal state g t .
It is speciﬁed by two functions: a description function ρ A :
S × D → ∆( D ) and a subgoal function ω A : S × S → ∆( S ) .
ρ A ( d (cid:48) | s, d ) speciﬁes the probability of giving d (cid:48) as the new description of state s given a current description d .
ω
A ( g (cid:48) | s, g ) indicates the probability of proposing g (cid:48) as a subgoal given a current state s and a goal state g .
At time step t , the assistant accepts three types of request from the agent:
(a) C UR : requests a new description of s t and receives d st +1 ∼ ρ A ( · | s t , d st ) ; (b) G OAL : requests a new description of g t and receives d gt +1 ∼
ρ A ( · | g t , d gt ) ; (c) S UB : requests a description of a subgoal g t +1 and receives d gt +1 ∼
ρ A ( · | g t +1 , ∅ ) where g t +
∼ ω
A ( · | s t , g t ) and ∅ is an empty description.
Interaction Policy.
The action space of the interaction policy ψ θ consists of ﬁve actions: { C UR , G OAL , S UB , D O , D ONE } .
The ﬁrst three actions correspond to making the three types of request that the assistants accepts.
The remaining two actions are used to traverse in the operation environment:
(d) D O : executes the action a do t (cid:44) arg max a ∈A ˆ π ( a | b st , d gt ) .
The agent transitions to a new operation state s t +1 ∼ T ( s t , a do t ) ; (e) D ONE : determines that the current goal g t has been reached.
2 If g t is a main goal ( g t = g 1 ), the task episode ends.
If g t is a subgoal ( g t (cid:54) = g 1 ), the agent may choose a new goal to follow.
Our problem formulation leaves it open on what goal should be selected next.
By selecting among these actions, the interaction policy essentially decides when to ask the assistant for additional information, and what types of information to ask for.
Our formulation does not specify the input space of the interaction policy, as this space depends on how the agent implements its goal memory (i.e. how it stores and retrieves the subgoals).
In the next section, we introduce an instantiation where the agent uses a stack data structure to manage (sub)goals.
H IERARCHICAL R EINFORCEMENT L EARNING F RAMEWORK
In this section, we describe the H ARI framework. We ﬁrst formulate the POMDP environment that the interaction policy acts in, referred to as the interaction environment (§4.1).
Our construction employs a goal stack to manage multiple levels of (sub)goals (§4.2).
A goal stack stores all the tasks the agent has been assigned but has not yet decided to terminate (by choosing the D ONE action).
It is updated in every step depending on the taken action.
We design a cost function (§4.4) that speciﬁes a trade-off between the cost of taking actions (acting cost) and the cost of not completing a task (task error).
4. I NTERACTION E NVIRONMENT
Given an operation environment E =
( S , A , T, c, D , ρ ) , the interaction environment constructed on top of E is a POMDP ¯ E
= ( ¯ S , ¯ A , ¯ T , ¯ c, ¯ D , ¯ ρ ) with:
• State space ¯ S
= S
× D × G L where G L is the set of all goal stacks containing at most L elements ( L is a hyperparameter).
Each state ¯ s =
( s, d s , G ) ∈
¯ S is a tuple of an operation state s ∈ S , its description d s ∈ D , and a goal stack G ∈ G L .
Each element in the goal stack G is a tuple ( g, d g ) of a goal state g ∈ S and its description d g ∈ D ; • Action space ¯
A = { C UR , G OAL , S UB , D O , D ONE } ; • State-transition function ¯
T =
T s · T G where T s : S × D
× ¯ A → ∆( S × D ) and T G : G L × ¯ A → ∆( G L ) ; • Cost function ¯ c : ( S×G L ) × ¯
A → R (deﬁned in §4.4 to trade off acting cost and task error); • Description space ¯ D
= D
× G dL where G dL is the set of all goal-description stacks of size L .
At any time, the agent cannot access the environment’s goal stack G , which contains true goal states.
Instead, it can only observe the descriptions in G .
We call this partial stack a goal-description stack , denoted by G d ; • Description function ¯ ρ : ¯ S → ¯ D , where ¯ ρ (¯ s ) =
¯ ρ ( s, d s , G ) =
( d s , G d ) .
Unlike in the standard POMDP formulation, this description function is deterministic.
A belief state ¯ b t summarizes a partial execution (¯ s 1 , ¯ a 1 , · · · , ¯ s t ) .
We formally deﬁne the interaction policy as ψ θ : ¯ B → ∆( A ) , where ¯ B is the set of all interaction belief states.
4. G OAL S TACK
A goal stack is an ordered set of tasks that the agent has not declared completion (by calling D ONE ).
The initial stack G 1 = { ( g 1 , d g 1 ) } contains the main goal g 1 , and its description d g 1 .
Let G t be the goal stack at time step t .
Only the G OAL , S UB , D ONE actions alter the stack.
The G OAL action replaces the top goal description with d gt +1 , the new description given by the assistant.
The S UB action pushes a new subtask ( g t +1 , d gt +1 ) to the stack.
The D ONE action pops the top (sub)task from the stack.
[Equation]
otherwise (2)
The S UB action is not available to the agent when the current stack contains L elements, guaranteeing that goal stack always has at most L elements.
The goal-stack transition function T G is deﬁned as T G ( G t +1 | G t , ¯ a t )
= 1 { G t +1 =
G t .
update (¯ a t ) } where 1 { . } is an indicator function.
4. T RANSITION OF THE C URRENT O PERATION S TATE AND I TS D ESCRIPTION
To complete the deﬁnition of the state-transition function ¯ T , we deﬁne the transition function T s .
This function is factored into two terms by the chain rule:
[Equation]
Only taking the D O action may change the current operation state
[Equation]
The description d st may vary when the agent moves to a new operation state (by taking the D O action) or requests a new description of s t (by taking the C UR action)
[Equation]
4. C OST F UNCTION
The interaction policy needs to balance between two types of cost: the cost of taking actions (acting cost) and the cost of not completing a task (task error).
The acting cost also subsumes the cost of communicating with the assistant because, in reality, such interactions consume time, human effort, and possibly trust.
Assuming that the assistant is helpful, acting cost and task error usually conﬂict with each other; for example, the agent may lower its task error if it is willing to suffer a larger acting cost by increasing the number of requests to the assistant.
We employ a simpliﬁed model where all types of cost are non-negative real numbers of the same unit.
Making a request of type a is assigned a constant cost γ a .
The cost of taking the D O action is c ( s t , a do t ) , the cost of executing the a do t action in the operation environment.
Calling D ONE to terminate execution of the main goal g 1 incurs a task error c ( s t , a done ) .
We exclude the task errors of executing subgoals because the interaction policy is only evaluated on reaching the main goal.
The magnitudes of the costs naturally specify a trade-off between acting cost and task error.
For example, setting the task errors much larger than the other costs indicates that completing tasks is prioritized over taking few actions.
L EARNING W HEN AND W HAT TO A SK IN H UMAN -A SSISTED N AVIGATION
Problem.
We apply H ARI to modeling a human-assisted navigation (H AN ) problem.
In H AN , a human requests an agent to ﬁnd an object in an indoor environment.
Each task request asks the agent to go to a room of type r and ﬁnd an object of type o (e.g., ﬁnd a mug in a kitchen).
The agent is equipped with a camera and shares its camera view with the human.
We assume that the human is sufﬁciently familiar with the environment that they can recognize the agent’s location by looking at its current view.
Before issuing a task request, the human imagines a goal location (but do not reveal it to the agent).
We are primarily interested in evaluating success in goal-ﬁnding , i.e. whether the agent can arrive at the human’s intended goal location.
Even though there could be multiple locations that match a request, the agent only succeeds if it arrives exactly at the chosen goal location.
We also determine success in request-fulﬁlling , where the agent successfully fulﬁlls a request if it navigates to any node that is within two meters of an object that matches the request. While an agent is performing a task, it may request the human to provide additional information via telecommunication (e.g., a chat app).
Speciﬁcally, it can ask for a description of its current location (C UR ), the goal location (G OAL ), or a subgoal location that is on the path from its current location to the goal location (S UB ).
Detail about how the subgoals are determined is in the Appendix.
Operation Environment.
We construct the operation environments using the environment graphs provided by the Matterport3D simulator (Anderson et al., 2018).
Each environment graph is generated from a 3D model of a house where each node is a location in the house and each edge connects two nearby unobstructed locations.
Each operation state s corresponds to a node in the graph.
At any time, the agent’s operation action space A consists of traversing to any of the nodes that are adjacent to its current node.
We employ a discrete bag-of-features representation for state descriptions.
3 A bag of features represents the information that the agent extracts from the raw input that the agent perceives (e.g., an image, a language sentence).
Working with this intermediate input allows us to easily vary the type and amount of information given to the agent.
Speciﬁcally, we simulate two settings of descriptions: dense and sparse .
At evaluation time, the agent perceives sparse descriptions and request the assistant for dense descriptions.
A dense description of a current location contains the room name at the location, and the features of M objects restricted to be within δ meters of the location.
The features of each object consists of (i) its name, (ii) horizontal and vertical angles (relative to the current viewpoint), and (iii) distance (in meters) from the object to the current location.
A dense description of a goal follows the same representation scheme.
In the sparse setting, the current-location description does not include the room name.
Moreover, we remove the features of objects that are not in the top 100 most frequent objects, emulating an imperfect object detector module.
The sparse goal description (the task request) has only features of the target object and the room name where the object is located at.
Especially, if a subgoal location is adjacent or coincides with the agent’s current location, instead of describing room and object features, the human speciﬁes the ground-truth action to go to the subgoal (an action is speciﬁed by its horizontal and vertical angles, and travel distance).
Experimental Procedure.
We conduct our experiments in three phases.
In the pre-training phase, we learn an operation policy ˆ π with dense descriptions of the current location and the goal.
In the training phase, the agent perceives a sparse description of its current location and is given a sparse initial goal description.
We use advantage actor-critic (Mnih et al., 2016) to learn an interaction policy ψ θ that controls how the agent communicates with the human and navigates in an environment.
The human always returns dense descriptions. The interaction policy is trained in environments
[Table]
[Table]
[Table] [Table] [Table]
that are previously seen as well as unseen during pre-training.
Finally, in the evaluation phase, the interaction policy is tested on three conditions: seen environment and target object type but starting from a new room (U NSEEN S TR ), seen environment but new target object type (U NSEEN O BJ ), and new environment (U NSEEN E NV ).
The pre-trained operation policy ˆ π is ﬁxed during the training and evaluation phases.
We create 82,104 examples for pre-training, 65,133 for training, and approximately 2,000 for each validation or test set.
Details about the training procedure and the dataset are included in the Appendix.
R ESULTS AND A NALYSES
Settings.
In our main experiments, we set: the cost of taking a C UR , G OAL , S UB , or D O action to be 0.01 (we will consider other settings subsequently), the cost of calling D ONE to terminate the main goal (i.e. task error) equal the (unweighted) length of the shortest-path from the agent’s location to the goal, and the goal stack’s size ( L ) to be 2.
We compare our RL-learned interaction policy with a rule-based baseline that ﬁrst takes the G OAL action and then randomly selects actions.
In each episode, we enforce that the rule-based policy can take at most (cid:98) X a (cid:99) + y actions of type a , where y ∼ Bernoulli ( X a − (cid:98)
X a (cid:99) ) and X a is a constant.
We tune each X a on the validation sets so that the rule-based policy has the same average count of each action as the RL-learned policy.
To prevent early termination, we enforce that the rule-based policy cannot take more D ONE actions than
S UB actions unless when its S UB action’s budget is exhausted.
We also construct a skyline where the interaction policy is also learned by RL but with an operation policy that executes subgoals perfectly.
As discussed in §5, we are primarily interested in goal-ﬁnding success rate and will refer to this metric brieﬂy as success rate.
Main Results (Table 1).
To inspect the potential beneﬁts of asking for additional information, we compute how much the operation policy ˆ π improves when it is supplied with dense information about the current and/or goal states.
As seen, success rate of the operation policy is lifted dramatically when both the current-state and goal descriptions are dense ( ∼ 2 × increase on U NSEEN S TR , ∼ 5 × on U NSEEN O BJ , and ∼ 3 × on U NSEEN E NV ).
We ﬁnd that dense information about the current state is more helpful on U NSEEN S TR , while dense information about the goal is more valuable on U NSEEN O BJ and U NSEEN E NV .
This is reasonable because on U NSEEN S TR , the agent has been trained to ﬁnd similar goals.
In contrast, the initial goal descriptions in U NSEEN O BJ and U NSEEN E NV are completely new to the agent, thus gathering more information about them is necessary.
Aided by our RL-learned interaction policy, the agent observes a substantial ∼ 2 × increase in success rate on U NSEEN S TR , ∼ 5 × on U NSEEN O BJ , and ∼ 7 × on U NSEEN E NV , compared to when performing tasks using only the operation policy.
In unseen environments, with its capability of requesting subgoals, the agent impressively doubles the success rate of the operation policy that has access to dense descriptions.
The RL-learned interaction policy is signiﬁcantly more effective than the rule-based baseline (+7.1% on U NSEEN E NV ).
Compared to a policy that calls G OAL at the
[Figure]
[Figure]
beginning and calls C UR at every step (which is equivalent to the dense- d s -and-
d g baseline)
, our policy achieves higher success rate on U NSEEN E NV while making two times fewer requests. This is due to the ability to request subgoals.
On U NSEEN S TR and U NSEEN O BJ , the RL-learned policy has not closed the gap with the operation policy that performs tasks with dense descriptions.
Our investigation ﬁnds that limited information often causes the policy to not request information about the current state (i.e. taking C UR ) and terminate prematurely or go to a wrong place.
Encoding uncertainty in the current-state description (e.g., Finkel et al.
(2006); Nguyen & O’Connor (2015)) is a plausible future direction for tackling this issue.
Finally, results obtained by replacing the learned operation policy with one that behaves optimally on subgoals shows that further improving performance of the operation policy on shortdistance goals would effectively enhance the agent’s performance on long-distance goals.
Behavior of the RL-Learned Interaction Policy.
Figure 2a characterizes behaviors of the RLlearned interaction policy in three evaluation conditions.
We expect that tasks in U NSEEN S TR are the easiest and those in U NSEEN E NV are the hardest.
As the difﬁculty of the evaluation condition increases, the interaction policy issues more C UR , S UB , and D O actions.
The average number of
G OAL actions does not vary, showing that the interaction policy has correctly learned that making more than one goal-clarifying request is unnecessary.
Figure 2b illustrates the distribution of each action along the length of an episode in the validation U NSEEN E NV dataset.
The G OAL action, if taken, is always taken only once and immediately in the ﬁrst step.
The number of C UR actions gradually decreases over time.
The agent makes most S UB requests in the middle of an episode, after its has attempted but failed to accomplish the main goals.
We observe similar patterns on the other two validation sets.
Effects of Varying Action Cost.
As mentioned, we assign the same cost to each C UR , G OAL , S UB , or D O action.
Figure 3a demonstrates the effects of changing this cost on the success rate of the agent.
Setting the cost equal to 0.5 makes it too costly to take any action, inducing a policy that always calls D ONE in the ﬁrst step and thus fails on all tasks.
Overall, the success rate of the agent rises as we reduce the action cost.
The increase in success rate is most visible in U NSEEN E NV and least visible in U NSEEN S TR .
Figure 3b provides more insights.
As the action cost decreases, we observe a growth in the number of S UB and D O actions taken by the interaction policy.
Meanwhile, the numbers of C UR and G OAL actions are mostly static.
Since requesting subgoals is more helpful in unseen environments than in seen environments, the increase in the number of S UB actions leads the more visible boost in success rate on U NSEEN E NV tasks.
Performing Tasks with Deeper Goal Stacks.
In Table 2, we test the functionality of our framework with a stack size 3, allowing the agent to request subgoals of subgoals.
As expected, success rate on U NSEEN E NV is boosted signiﬁcantly (+11.9% compared to using a stack of size 2).
Success rate on U NSEEN O BJ is largely unchanged; we ﬁnd that the agent makes more S UB requests (averagely 4.5 requests per episode compared to 1.0 request made when the stack size is 2), but doing so does not further enhance performance.
The agent makes less C UR requests, possibly in order to [Table] offset the cost of making more S UB requests.
Due to this behavior, success rate on U NSEEN S TR declines with larger stack sizes, as information about the current state is more valuable for these tasks than subgoals.
These results show that the critic model overestimates the V values in states where S UB actions are taken, leading to the agent learning to request subgoals more than needed.
R ELATED W ORK AND C ONCLUSION
Transfer Learning in Reinforcement Learning.
Various frameworks have been proposed to model knowledge transfer from a more capable agent to a novice one (Da Silva & Costa, 2019).
Torrey & Taylor (2013) introduce the action-advising framework where a learner requests reference actions from a teacher.
Following this work, Da Silva et al.
(2020) investigate uncertainty-based strategies for deciding when to request. In an agent-to-agent setting, (Da Silva et al., 2017; Zimmer et al., 2014; Omidshaﬁei et al., 2019) focuses on learning a teaching policy in addition to an advicerequesting policy. An important assumption in these papers is that the teacher must share a common action space with the learner. More recent frameworks (Kim et al., 2019; Nguyen et al., 2019;
Nguyen & Daum ´ e III, 2019) relax this assumption by allowing the teacher to specify high-level subgoals instead of low-level actions. H ARI can viewed as a strict extension of these frameworks. It allows the human to specify not only subgoals, but also any additional information about the current state and the goal that agent can interpret. Moreover, H ARI equips the agent with multiple communication intentions and teaches it to select the most useful intention to convey in a given situation.
Another line of work employs standard RL communication protocol, where the human can only transfer knowledge through numerical scores or categorical feedback (Knox & Stone, 2009; Judah et al., 2010;
Peng et al., 2016; Grifﬁth et al., 2013).
(Maclin & Shavlik, 1996) propose a framework where the human advise the agent via a domain-speciﬁc language, specifying rules that can be incorporated into the agent’s model.
In contrast, H ARI operates with a black-box agent model.
Sumers et al. (2020) extract features from various types of language feedback to construct a reward function for reinforcement learning. We instead focus on deployment-time communication and directly incorporate the human feedback as input to agent’s operation policy.
H ARI essentially models a task-oriented dialog problem. Many variants of this problem requires the agent to compose speciﬁc questions (De Vries et al., 2017; Das et al., 2017; Thomason et al., 2020). The dominant approach in these problems is to mimic pre-collected human utterances.
As discussed previously, naively mirroring human external behavior cannot enable agents to understand the limits of their knowledge.
We teach the agent to understand its intrinsic needs through interaction with the human and the environment rather than through imitation of human behaviors. Another related line of work concerns generating natural language explanations of model decisions (Camburu et al., 2018;
Hendricks et al., 2016; Rajani et al., 2019).
In summary, this paper presents a general POMDP framework for modeling human-agent communication. While we only demonstrate this framework on a simpliﬁed navigation problem, our framework can theoretical capture rich types of human-agent communication. Hence, an important empirical question is how well our formulation generalizes to richer environments with more complex interactions and state spaces (Shridhar et al., 2020).
Enhancing the sample efﬁciency of the learning policy by exploiting the hierarchical policy structure is an exciting future direction.
Furthermore, techniques for generating faithful explanations (Kumar & Talukdar, 2020; Madsen et al., 2021) can be applied to enhance the speciﬁcity of the generated questions. A T RAINING P ROCEDURE Cost function.
The cost function in our framework is given as follows
[Equation]
Subgoals.
Let p t be the shortest path from the agent’s current state s t to the current goal g t , and p t,i be the i -th node on the path ( 0 ≤ i < | p t | ).
The subgoal location is chosen as p t,k where k = min( (cid:98)| p | / 2 (cid:99) , l max ) , where l max is a pre-deﬁned constant.
Training Algorithms.
We pre-train the operation policy
ˆ π with DAgger (Ross et al., 2011), minimizing the cross entropy between its action distribution with that of a shortest-path oracle (which is a one-hot distribution with all probability concentrated on the optimal action).
We use advantage actor-critic (Mnih et al., 2016) to train the interaction policy ψ θ .
This method simultaneously estimates an actor policy ψ θ : ¯ B → ∆( ¯ A ) and a critic function V φ : ¯ B → R .
Given an execution ¯ τ =
(¯ s 1 , ¯ a 1 , ¯ c 1 · · · , ¯ s H ) , the gradients with respect to the actor and critic are
[Equation]
(8) where C t = (cid:80) Hj =
t c j , ¯ b at is a belief state that summarizes the partial execution ¯ τ 1: t for the actor, and ¯ b vt is a belief state for the critic.
Cost function.
The cost function introduced in §4.4 is not effective for learning the interaction policy because the task error is given only at the end of an episode.
We extend the reward-shaping method proposed by Ng et al.
(1999) to goal-conditioned policies, augmenting the original cost function with a shaping function Φ( s, g ) with s, g ∈ S .
We set Φ( s, g ) to be the (unweighted) shortest-path distance from s to g .
The cost received by the agent at time step t is ˜ c t (cid:44) ¯ c t + Φ( s t +1 , g t +1 ) − Φ( s t , g t ) .
We assume that the agent transitions to a special terminal state s term ∈ S and remains there after it terminates execution of the main goal.
We set Φ( s term , None ) = 0 , where g t =
None signals that the episode has ended.
Hence, the cumulative cost of an execution under the new cost function is
[Equation]
Since Φ( s 1 , g 1 ) does not depend on the action taken in s 1 , minimizing the new cumulative cost does not change the optimal policy for the task ( s 1 , g 1 ) .
Model Architecture.
We adapt the V&L BERT architecture (Hong et al., 2020) for modeling the operation policy ˆ π .
Our model has two components: an encoder and a decoder; both are implemented as Transformer models (Vaswani et al., 2017).
The encoder takes as input a description d st or d gt and generates a sequence of hidden vectors.
In every step, the decoder takes as input the previous hidden vector b st − 1 , the sequence of vectors representing d st , and the sequence of vectors representing d gt .
It then performs self-attention on these vectors to compute the current hidden vector b st and a probability distribution over navigation actions p t .
The interaction policy ψ θ (the actor) is an LSTM-based recurrent neural network.
The input of this model is the operation policy’s model outputs, b st and p t , and the embedding of the previously taken action ¯ a t − 1 .
The critic model also has a similar architecture but outputs a real number (the
V value) rather than an action distribution.
When training the interaction policy, we always ﬁx the parameters of the operation policy.
We ﬁnd it necessary to pre-train the critic before training it jointly with the actor.
[Table]
Representation of State Descriptions.
The representation of each object, room, or action is computed as follows.
Let f name , f horz , f vert , f dist , and f type are the features of an object f , consisting of its name, horizontal angle, vertical angle, distance, and type (a type is either Object , Room , or
Action ; in this case, the type is Object ).
For simplicity, we discretize real-valued features, resulting in 12 horizontal angles (corresponding to π/ 6 · k, 0 ≤ k < 12 ), 3 vertical angles (corresponding to π/ 6 · k, − 1 ≤ k ≤ 1 ), and 5 distance values (we round down a real-valued distance to the nearest integer).
We then lookup the embedding of each feature from an embedding table and sum all the embeddings into a single vector that represents the corresponding object.
For a room, f horz , f vert f dist are zeroes.
For an action, f name is either ActionStop for the stop action a done or ActionGo otherwise.
During pre-training, we randomly drop features in d st and d gt so that the operation policy is familiar with making decisions under sparse information.
Concretely, we refer to all features of an object, room or action as a feature set .
For d st , let M be the number objects in a description.
We uniformly randomly keep m feature sets among the M +1 feature sets of d st (the plus one is the room’s feature set), where m ∼ Uniform (min(5 , M + 1) , M + 1) .
For d st , we have two cases.
If g 1 is not adjacent or equals to s 1 , we uniformly randomly alternate between giving a dense and a sparse description.
In this case, the sparse description contains the features of the target object and the goal room’s name.
Otherwise, with a probability of 1 ⁄ 3 , we give either (a) a dense description (b) a (sparse) description that contains the target object’s features and the goal room’s name, or (c) a (sparse) description that describes the next ground-truth action.
We pre-train the operation policy on various path lengths (ranging from 1 to 10 graph nodes) so that it learns to accomplish both long-distance main goals and short-distance subgoals.
Data.
Table 3 summarizes the data splits.
From a total of 72 environments provided by the Matterport3D dataset, we use 36 environments for pre-training, 18 as unseen environments for training,for validation U NSEEN E NV , and 11 for test U NSEEN E NV .
We use a vocabulary of size 1738, which includes object and room names, and special tokens representing the distance and direction values.
The length of a navigation path ranges from 5 to 10 graph nodes.
Hyperparameters.
See Table 4.
[Table]