Original: P OSSIBILITY B EFORE U TILITY : L EARNING A ND U SING H IERARCHICAL A FFORDANCES
Revised: P OSSIBILITY B EFORE U TILITY : L EARNING A ND U SING H IERARCHICAL A FFORDANCES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures.
Revised: Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures.

Original: Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but insteadconsider only  ones they deem possible in the ﬁrst place.
Revised: Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the ﬁrst place.

Original: By focusing only on what is feasible, or “afforded”, at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters .
Revised: By focusing only on what is feasible, or “afforded”, at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters .

Original: To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning.
Revised: To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning.

Original: Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our deﬁnition of hierarchical affordances in the present state , our approach is more ﬂexible than the multitude of approaches that ground their subtask dependencies in a symbolic history .
Revised: Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our deﬁnition of hierarchical affordances in the present state , our approach is more ﬂexible than the multitude of approaches that ground their subtask dependencies in a symbolic history .

Original: While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic speciﬁcations.
Revised: While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic speciﬁcations.

Original: Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efﬁciently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision—all of which are hallmarks of human learning.
Revised: Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efﬁciently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision—all of which are hallmarks of human learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) methods have recently achieved success in a variety of historically difﬁcult domains (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), but they continue to struggle on complex hierarchical tasks.
Revised: Reinforcement learning (RL) methods have recently achieved success in a variety of historically difﬁcult domains (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), but they continue to struggle on complex hierarchical tasks.

Original: Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do.
Revised: Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do.

Original: In other words, they do not waste time attempting the impossible.
Revised: In other words, they do not waste time attempting the impossible.

Original: Gibson (1977) coins the term “affordances“ to articulate the observation that humans and other animals largely interpret the world around them by which behaviors the environment affords them.
Revised: Gibson (1977) coins the term “affordances” to articulate the observation that humans and other animals largely interpret the world around them in terms of which behaviors the environment affords them.

Original: While some previous works apply the concept of affordances to the RL setting, none of these methods easily translate to environments with hierarchical tasks.
Revised: While some previous works apply the concept of affordances to the RL setting, none of these methods easily translate to environments with hierarchical tasks.

Original: In this work, we introduce Hierarchical Affordance Learning (HAL), a method that addresses the challenges inherent to learning affordances over high-level subtasks, enabling more efﬁcient learning in environments with complex subtask dependency structures.
Revised: In this work, we introduce Hierarchical Affordance Learning (HAL), a method that addresses the challenges inherent to learning affordances over high-level subtasks, enabling more efﬁcient learning in environments with complex subtask dependency structures.

Original: Many real-world environments have an underlying hierarchical dependency structure (Fig. 1a), and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them.
Revised: Many real-world environments have an underlying hierarchical dependency structure (Fig. 1a), and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them.

Original: Consider the task of preparing a simple pasta dish.
Revised: Consider the task of preparing a simple pasta dish.

Original: Some sets of subtasks, like chopping vegetables or ﬁlling a pot with water, can be successfully performed in any order.
Revised: Some sets of subtasks, like chopping vegetables or ﬁlling a pot with water, can be successfully performed in any order.

Original: However, there are many cases in which the dependencies between subtasks must be obeyed.
Revised: However, there are many cases in which the dependencies between subtasks must be obeyed.

Original: For instance, it is inadvisable chop vegetables after having mixed them with the sauce, or to boil a pot of water before the pot is ﬁlled with water in the ﬁrst place.
Revised: For instance, it is inadvisable chop vegetables after having mixed them with the sauce, or to boil a pot of water before the pot is ﬁlled with water in the ﬁrst place.

Original: Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks, hierarchical reinforcement learning (HRL) methods are well-suited for tasks with complex high-level dependencies.
Revised: Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks, hierarchical reinforcement learning (HRL) methods are well-suited for tasks with complex high-level dependencies.

Original: Existing HRL methods fall along a spectrum ranging from ﬂexible approaches that discover useful subtasks automatically, to the structured approaches that provide someamount of prior information about subtasks and their interdependencies.
Revised: Existing HRL methods fall along a spectrum ranging from ﬂexible approaches that discover useful subtasks automatically, to the structured approaches that provide some prior information about subtasks and their interdependencies.

Original: The former approaches (e.g. Vezhnevets et al., 2017; 
Revised: The former approaches (e.g. Vezhnevets et al., 2017; Eysenbach

Original: [Figure]
Revised: [Figure]

Original: Eysenbach et al., 2018) have seen limited success, as the automatic identiﬁcation of hierarchical abstractions is an open problem in deep learning (Hinton, 2021).
Revised: et al., 2018) have seen limited success, as the automatic identiﬁcation of hierarchical abstractions is an open problem in deep learning (Hinton, 2021).

Original: But approaches that endow the agent with more structure, in hopes of making certain tasks feasible, do so at the cost of rigid assumptions.
Revised: But approaches that endow the agent with more structure, to make complex tasks feasible, do so at the cost of rigid assumptions.

Original: Methods that use a form of ﬁnite automata (Fig. 1b) to express subtask dependencies (e.g. Icarte et al., 2020) require the set of symbols, or atomic propositions , provided to the agent to be complete, in that the history of symbols maps deterministically to the current context (i.e. how much progress has been made; which subtasks are available).
Revised: Methods that use a form of ﬁnite automata (Fig. 1b) to express subtask dependencies (e.g. Icarte et al., 2020) require the set of symbols, or atomic propositions , provided to the agent to be complete, in that the history of symbols maps deterministically to the current context (i.e. how much progress has been made; which subtasks are available).

Original: Importantly, these methods and many others (e.g. Andreas et al., 2016; Sohn et al., 2020) consider subtasks to be dependent merely on the completion of others.
Revised: Importantly, these methods and many others (e.g. Andreas et al., 2017; Sohn et al., 2020) consider subtasks to be dependent merely on the completion of others.

Original: Unfortunately, these assumptions do not hold in the real world (Fig. 1c).
Revised: Unfortunately, these assumptions do not hold in the real world (Fig. 1c).

Original: For instance, if one completes the subtask cook noodles , but they clumsily spill them all over the ﬂoor, are they now ready for the next subtask, mix noodles and sauce ?
Revised: For instance, if one completes the subtask cook noodles , but they clumsily spill them all over the ﬂoor, are they now ready for the next subtask, mix noodles and sauce ?

Original: While the subtask cook noodles is somehow necessary for this further subtask, it is not sufﬁcient to have completed it in the past.
Revised: While the subtask cook noodles is somehow necessary for this further subtask, it is not sufﬁcient to have completed it in the past.

Original: The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone.
Revised: The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone.

Original: This is possible, but extraordinarily restrictive, since, unless the set of symbols is complete, none of the subtask completion information can be used to reliably learn and utilize subtask dependencies.
Revised: This is possible, but extraordinarily restrictive, since, unless the set of symbols is complete, none of the subtask completion information can be used to reliably learn and utilize subtask dependencies.

Revised: Modeling probabilistic transitions allows the symbolic signal to be incomplete, but still requires a complete set of symbols, in addition to predeﬁned contexts.

Original: Incontrast, in order to make use ofthis incomplete symbolic information, our approach  learns a representation of context grounded in the present state to determine which subtasks are possible, rather than solely relying on symbols (Fig. 1d).
Revised: In order to make use of incomplete symbolic information, our approach instead learns a representation of context grounded in the present state to determine which subtasks are possible (Fig. 1d), rather than solely relying on symbols .

Original: The contributions of this paper are as follows.
Revised: The contributions of this paper are as follows.

Original: First we deﬁne the milestone signal (§4), a ﬂexible alternative to atomic propositions used in automata-based approaches.
Revised: First we deﬁne the milestone signal (§4), a ﬂexible alternative to atomic propositions used in automata-based approaches.

Original: Deﬁning a set of milestones is signiﬁcantly easier, as the assumptions are not as rigid.
Revised: Specifying a set of milestones is signiﬁcantly easier, as the assumptions are not as rigid.

Original: Unlike a dense reward function, the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives.
Revised: Unlike a dense reward function, the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives.

Original: Next, we introduce hierarchical affordances , which can be deﬁned over any arbitrary set of milestones, and describe HAL (§5), a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks.
Revised: Next, we introduce hierarchical affordances , which can be deﬁned over any arbitrary set of milestones, and describe HAL (§5), a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks.

Original: Finally, we demonstrate HAL’s superior performance on two complex hierarchical tasks in terms of learning speed, robustness, generalizability, and ability to explore complex subtask hierarchies without extrinsic supervision, relative to baselines provided with the same information (§6.3).
Revised: Finally, we demonstrate HAL’s superior performance on two complex hierarchical tasks in terms of learning speed, robustness, generalizability, and ability to explore complex subtask hierarchies without extrinsic supervision, relative to baselines provided with the same information (§6.3).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our approach uses the options framework (Sutton et al., 1999), which allows an agent to take temporally extended actions, enabling more efﬁcient planning in certain tasks.
Revised: Our approach uses the options framework (Sutton et al., 1999), which allows an agent to take temporally extended actions, enabling more efﬁcient planning in certain tasks.

Original: Among the various deep-RL incarnations of the options framework (e.g. Kulkarni et al., 2016;
Revised: Among the various deep-RL incarnations of the options framework (e.g. Kulkarni et al., 2016;

Original: Vezhnevets et al., 2017;
Revised: Vezhnevets et al., 2017;

Original: Nachum et al., 2018), Hierarchical DQN (h-DQN) (Kulkarni et al., 2016) is the most relevant to our approach, and we discuss it in more detail in Section 3.
Revised: Nachum et al., 2018), Hierarchical DQN (h-DQN) (Kulkarni et al., 2016) is the most relevant to our approach, and we discuss it in more detail in Section 3.

Original: Our approach most notably deviates from h-DQN by learning a model of affordances and using it to reduce the initiation set for each option.
Revised: Our approach most notably deviates from h-DQN by learning a model of affordances and using it to reduce the initiation set for each option.

Original: Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution (Andreas et al., 2016;
Revised: Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution (Andreas et al., 2017;

Original: Shiarlis et al., 2018; Devin et al., 2019; Sohn et al., 2020;
Revised: Shiarlis et al., 2018; Devin et al., 2019; Sohn et al., 2020;

Original: Lu et al., 2020).
Revised: Lu et al., 2021).

Original: Sohn et al.
Revised: Sohn et al.

Original: (2020) learn subtask preconditions, but use symbol-based contexts and do not learn and use their model of preconditions concurrently.
Revised: (2020) learn subtask preconditions, but use symbol-based contexts and do not learn and use their model of preconditions concurrently.

Original: Instead they assume a naive policy can sufﬁciently reach all subtasks.
Revised: Instead they assume a naive policy can sufﬁciently reach all subtasks.

Original: Furthermore, they assume ground-truth affordances are provided at each step.
Revised: Furthermore, they assume ground-truth affordances are provided at each step.

Original: Some works provide the agent with high-level task sketches (Andreas et al., 2016; Shiarlis et al., 2018) describing the order in which subtasks must be completed.
Revised: Some works provide the agent with high-level task sketches (Andreas et al., 2017; Shiarlis et al., 2018) describing the order in which subtasks must be completed.

Original: While these sketches are advertised as “ungrounded” Andreas et al.
Revised: While these sketches are advertised as “ungrounded” Andreas et al.

Original: , they are in fact grounded by the inclusion of short sketches, which are the ﬁrst to be introduced to the agent in a curriculum learning scheme (Bengio et al., 2009).
Revised: , they are in fact grounded by the inclusion of short sketches, which are the ﬁrst to be introduced to the agent in a curriculum learning scheme (Bengio et al., 2009).

Original: Our approach instead uses a direct signal, which  need notalone determine task progress, and can learn without exposure to other tasks with shared structure.
Revised: Our approach instead uses a direct signal, which alone need not determine task progress, and can learn without exposure to other tasks with shared structure.

Original: In using a set of discrete symbols to indicate subtask completion, our work is similar to the variety of approaches that apply temporal logic (TL) to the RL setting (Yuan et al., 2019; Hasanbeig et al., 2018; 2020; Li et al., 2017; 2018).
Revised: In using a set of discrete symbols to indicate subtask completion, our work is similar to the variety of approaches that apply temporal logic (TL) to the RL setting (Yuan et al., 2019; Hasanbeig et al., 2018; 2020; Li et al., 2017; 2018).

Original: These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step.
Revised: These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step.

Original: Some works use reward shaping to encourage satisfaction of the formula (Li et al., 2017; 2018), whereas others convert the TL formula to some ﬁnite state machine, which provides the agent with a structure that roughly expresses subtask dependencies (Hasanbeig et al., 2018; 2020; Yuan et al., 2019).
Revised: Some works use reward shaping to encourage satisfaction of the formula (Li et al., 2017; 2018), whereas others convert the TL formula to some ﬁnite state machine, which provides the agent with a structure that roughly expresses subtask dependencies (Hasanbeig et al., 2018; 2020; Yuan et al., 2019).

Original: Icarte et al.
Revised: Icarte et al.

Original: (2020) bypass this formula-to-automata conversion, and instead directly provide the automata to the agent in the form of a reward machine (RM).
Revised: (2020) bypass this formula-to-automata conversion, and instead directly provide the automata to the agent in the form of a reward machine (RM).

Original: While RMs are more expressive than LTL formulas, they are less ﬂexible than HAL, which can deal with incomplete sets of symbols, as well as context stochasticity.
Revised: While RMs are more expressive than LTL formulas, they are less ﬂexible than HAL, which can deal with incomplete sets of symbols, as well as context stochasticity.

Original: Gibson (1977) introduces a theory of affordances , deﬁned roughly as properties of the environment which must be measured relative to the agent.
Revised: Gibson (1977) introduces a theory of affordances , deﬁned roughly as properties of the environment which must be measured relative to the agent.

Original: Heft (1989) and Chemero (2003) clarify affordances as relations between the agent and the environment.
Revised: Heft (1989) and Chemero (2003) clarify affordances as relations between the agent and its environment.

Original: Khetarpal et al.
Revised: Khetarpal et al.

Original: (2020) formalize this relational deﬁnition of affordances in the context of RL, and model which low-level actions, giventheir corresponding intents  are afforded in the present state.
Revised: (2020) formalize this relational deﬁnition of affordances in the context of RL, and model which low-level actions, given corresponding intents , are afforded in each state.

Revised: In this work we implicitly deﬁne a set of high-level intents by designating an intended milestone for each subtask.

Original: They  demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions, and allows for the learning of more accurate and generalizable partial world models.
Revised: They also demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions, and allows for the learning of more accurate and generalizable partial world models.

Original: This approach does not directly translate to the hierarchical setting because subtasks, unlike actions, may fail for reasons other than affordances, meaning we do not have access to ground-truth affordance label with which to train our model.
Revised: This approach does not directly translate to the hierarchical setting because subtasks, unlike actions, may fail for reasons other than affordances, meaning we do not have access to ground-truth affordance label with which to train our model.

Original: Manoury et al.
Revised: Manoury et al.

Original: (2020) and Khazatsky et al.
Revised: (2019) and Khazatsky et al.

Original: (2021) present approaches that can discover and use affordances to learn new skills, but their deﬁnition of affordances (i.e. a behavior is either afforded or not, with no notion of preconditions) does not translate to the hierarchical setting 1 .
Revised: (2021) present approaches that can discover and use affordances to learn new skills, but their deﬁnition of affordances (i.e. a behavior is either afforded or not, with no notion of preconditions) does not translate to the hierarchical setting 1 .

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Our setting is concerned with learning behavior policies in Markov Decision Processes (MDP).
Revised: Our setting is concerned with learning behavior policies in Markov Decision Processes (MDP).

Original: An
Revised: An

Original: MDP is deﬁned by the state space S , action space A , reward function R : S×A → R , and state tran- sition distribution P :
Revised: MDP is deﬁned by the state space S , action space A , reward function R : S×A → R , and state tran- sition distribution P :

Original: S × A → (cid:52) ( S ) .
Revised: S × A → (cid:52) ( S ) .

Original: The objective is to learn a policy, π : S → (cid:52) ( A ) , which se- lects actions that maximize expected future returns: G ( π ) =
Revised: The objective is to learn a policy, π : S → (cid:52) ( A ) , which se- lects actions that maximize expected future returns: G ( π ) =

Original: E [ (cid:80) ∞ t =0 γ t r t | a t ∼ π, s t ∼ P ] , where γ is the discount factor.
Revised: E [ (cid:80) ∞ t =0 γ t r t | a t ∼ π, s t ∼ P ] , where γ is the discount factor.

Original: We speciﬁcally follow the paradigm of hierarchical RL where we learn lowlevel controllers each trained to attain some discrete goal, and a high-level meta-controller trained to select goals in a manner that maximizes rewards.
Revised: We speciﬁcally follow the paradigm of hierarchical RL where we learn lowlevel controllers each trained to attain some discrete goal, and a high-level meta-controller trained to select goals in a manner that maximizes rewards.

Original: A bi-level controller of this form allows an agent to learn intermediate skills, in the form of competent sub-policies, enabling the agent to compose these intermediate behaviors in order to achieve rewards in long horizon tasks.
Revised: A bi-level controller of this form allows an agent to learn intermediate skills, in the form of competent sub-policies, enabling the agent to compose these intermediate behaviors in order to achieve rewards in long horizon tasks.

Original: Our controllers are learned via hierarchical deep Q-learning (Kulkarni et al., 2016).
Revised: Our controllers are learned via hierarchical deep Q-learning (Kulkarni et al., 2016).

Original: Goal-directed controllers We aim to learn a controller, π : S ×G → (cid:52) ( A ) , that selects low-level actions, a ∈ A , given a state, s ∈ S , and goal, g ∈ G  that maximizes expected future goal- attainment rewards, b g .
Revised: Goal-directed controllers We aim to learn a controller, π : S ×G → (cid:52) ( A ) , that selects low-level actions, a ∈ A , given a state, s ∈ S , and goal, g ∈ G , in some speciﬁed goal space, that maximizes expected future goal-attainment rewards, b g (our milestones discussed in Section 4).

Original: In the discrete goal case, we can treat this as learning a separate controller π g for each goal.
Revised: In the discrete goal case, we can treat this as learning a separate controller π g for each goal.

Original: Q-Learning (Watkins, 1989) trains these controllers by learning an estimate of the optimal Q-function: Q ∗ c ( s, a ; g )
Revised: Q-Learning (Watkins, 1989) trains these controllers by learning an estimate of the optimal Q-function: Q ∗ c ( s, a ; g )

Original: = max π E [ (cid:80) ∞ t
Revised: = max π E [ (cid:80) ∞ t

Original: =0 γ t b g,t | s 0 =
Revised: =0 γ t b g,t | s 0 =

Original: s, a 

Revised: s, a

Original: = a, a t ∼ π, s t ∼ P ] and deriving a policy from the Q-function as such: π g ( a | s, g ) = 1 ( a = arg max a (cid:48)
Revised: = a, a t ∼ π, s t ∼ P ] and deriving a policy from the Q-function as such: π g ( a | s, g ) = 1 ( a = arg max a (cid:48)

Original: Q c ( s, a (cid:48) ; g )) .
Revised: Q c ( s, a (cid:48) ; g )) .

Original: Deep Q-Learning (Mnih et al., 2015) aims to estimate Q ∗ via deep neural network function approximation.
Revised: Deep Q-Learning (Mnih et al., 2015) estimates Q ∗ using deep neural networks.

Original: This Q-function is parameterized by θ = { θ base , ..., θ g , ... } , where θ base is a set of shared base parameters and θ g is a goal-speciﬁc head.
Revised: This Q-function is parameterized by θ = { θ base , ..., θ g , ... } , where θ base is a set of shared base parameters and θ g is a goal-speciﬁc head.

Original: It is updated via gradient descent on the following loss function, derived from the original Q-learning update:
Revised: It is updated via gradient descent on the following loss function, derived from the original Q-learning update:

Original: [Equation]
Revised: [Equation]

Original: where D c is a replay buffer that stores previously collected transitions, and ¯ θ are the parameters of a periodically updated target network.
Revised: where D c is a replay buffer that stores previously collected transitions, and ¯ θ are the parameters of a periodically updated target network.

Original: Both of these components are included to avoid the instability associated with using function approximation in Q-Learning.
Revised: Both of these components are included to avoid the instability associated with using function approximation in Q-Learning.

Original: Goal-selecting meta-controller
Revised: Goal-selecting meta-controller

Original: The objective of our meta-controller Π : S → (cid:52) ( G ) is to select goals in a manner that maximizes extrinsic rewards received by the environment when executing the low-level controllers corresponding to the selected goals.
Revised: The objective of our meta-controller Π : S → [Equation] ( G ) is to select goals in a manner that maximizes extrinsic rewards received by the environment when executing the low-level controllers corresponding to the selected goals.

Original: Again, we estimate a Q-function, this time over a dilated time scale (i.e. we allow the lowlevel controllers to run for multiple steps before choosing new goals):Q ∗ m ( s, g ) = E
Revised: Again, we estimate a Q-function, this time over a dilated time scale (i.e. we allow the lowlevel controllers to run for multiple steps before choosing new goals):

Original: [Equation]
Revised: [Equation]

Original: + max [Equation] Q ∗
Revised: , where N is the (variable) number of steps the goal-directed controller runs.

Original: [Equation] s N , g [Equation] ) | s
Revised: When collecting data in the environment, we add transitions ( s t , g t , (cid:80) t + N

Original: [Equation]
Revised: [Equation]

Original: s, g 0 = g, a t ∼ π g , s t ∼ P (cid:105) , where N is the number of steps the goal-directed controller runs.

Original: When collecting data in the environment, we add transitions ( s t , g t , (cid:80) t + N t r t , s t + N ) to a separate meta-replay buffer, D m , used to train our meta-Q-function (parameterized by Θ ) with a loss similar to that in Eq. 1, but without any goal-conditioning.
Revised: r t , s t + N ) to a separate meta-replay buffer, D m , used to train our meta-Qfunction (parameterized by Θ ) with a loss similar to that in Eq. 1, but without any goal-conditioning.

Original:  M ILESTONES A ND H IERARCHICAL A FFORDANCES
Revised:  M ILESTONES A ND H IERARCHICAL A FFORDANCES

Original: We consider tasks that can be decomposed into subtasks represented by milestone symbols g ∈ G , where G is the set of symbols relevant to the task, and |G| =
Revised: We consider tasks that can be decomposed into subtasks represented by milestone symbols g ∈ G , where G is the set of symbols relevant to the task, and |G| =

Original: K .
Revised: K .

Original: At each time-step, in addition to the extrinsic reward signal provided by the environment to indicate successfulbehavior on the overall task, we have access to a milestone signal , which is a vector b t where each element b tg ∈ { 0 , 1 } indicates the completion of subtask g ∈ G at time t .
Revised: At each time-step, in addition to the extrinsic reward signal provided by the environment to indicate successful on the overall task, we have access to a milestone signal , which is a vector b t where each element b tg ∈ { 0 , 1 } indicates the whether g ∈ G was completed on time-step t .

Original: In our pasta example, we might receive a milestone each time we cut a vegetable, make the sauce, cook the noodles, etc.
Revised: In our PASTA example, we might receive a milestone each time we cut a vegetable, make the sauce, cook the noodles, etc.

Original: Milestones give us indication that we have achieved something notable  without having to  assign it utility.
Revised: Milestones indicate that we have achieved something notable (e.g. a subtask or subgoal) without having to quantify or assign utility to it.

Original: We deﬁne hierarchical affordances over a set of milestones in the following way.
Revised: We deﬁne hierarchical affordances over a set of milestones in the following way.

Original: The vector f s = f ∗ ( s ) of size K represents which milestones are achievable from the present state s , without requiring the collection of any intermediate milestones, where f is a hierarchical affordance classiﬁer , and f ∗ ( s ) is the optimal one.
Revised: The vector f s = f ∗ ( s ) of size K represents which milestones are achievable from the present state s , without requiring the collection of any intermediate milestones, where f is a hierarchical affordance classiﬁer , and f ∗ ( s ) is the optimal one.

Original: f
Revised: f

Original: sg
Revised: sg

Original: = 1 if at time t 0 it is possible for  b
Revised: = 1 if at time t 0 it is possible for future b

Original: Tg = 1 without any b tj = 1 , j (cid:54) =
Revised: Tg = 1 without any b tj = 1 , j (cid:54) =

Original: g , where t 0 < t < T .
Revised: g , where t 0 < t < T .

Original: In our simple pasta example, the milestone mix cooked noodles and sauce is not afforded at the beginning since it requires cook noodles .
Revised: In PASTA , for example, the milestone mix cooked noodles and sauce is not afforded at the beginning since it requires cook noodles .

Original: Many logic-based RL approaches learn a similar notion of context as affordances by performing temporal logic (Gotzhein, 1992) over a set of atomic propositions , which are sequentially fed to the agent in order to keep track of context changes.
Revised: Many logic-based RL approaches learn a similar notion of context as affordances by performing temporal logic (Gotzhein, 1992) over a set of atomic propositions , which are sequentially fed to the agent in order to keep track of context changes.

Original: Crucially, unlike atomic propositions, the history of milestones alone need not deﬁne which subtasks are currently possible.
Revised: Crucially, unlike atomic propositions, the history of milestones alone need not deﬁne which subtasks are currently possible.

Original: In fact, for G to be useful to an affordance-aware agent, only the following requirements must be met: G must contain a ﬁnal milestone that indicates task completion, and each additional milestone must correspond to a  behavior that is necessary for achieving the ﬁnal milestone.
Revised: In fact, for G to be useful to an affordance-aware agent, only the following requirements must be met: G must contain a ﬁnal milestone that indicates task completion, and each additional milestone should correspond to a (unique) behavior that is necessary for achieving the ﬁnal milestone.

Original: Figure 5 empirically supports this ﬂexible deﬁnition by demonstrating that a careful choice of G is unnecessary to observe beneﬁts from an affordance classiﬁer, meaning future work might more easily be able to combine HAL with orthogonal approaches that automatically discover meaningful subgoals.
Revised: Figure 5 empirically supports this ﬂexible deﬁnition by demonstrating that a careful choice of G is unnecessary to observe beneﬁts from an affordance classiﬁer, meaning future work might more easily be able to combine HAL with orthogonal approaches that automatically discover meaningful subgoals.

Original: If milestones are reliably given to the agent upon subtask completion, hierarchical affordances for each subtask are determined simply by the states in which their immediate attainment is possible.
Revised: If milestones are reliably given to the agent upon subtask completion, hierarchical affordances for each subtask are determined simply by the states in which their immediate attainment is possible.

Original: There are two forms of stochasticity that hierarchical affordances, by virtue of being grounded in the present state, can more naturally address than symbolically-deﬁned dependencies.
Revised: There are two forms of stochasticity that hierarchical affordances, by virtue of being grounded in the present state, can more naturally address than symbolically-deﬁned dependencies.

Original: We can conceptualize potential trajectories as graphs where nodes represent the attainment of milestones, and edges are the segments between them.
Revised: We can conceptualize potential trajectories as graphs where nodes represent the attainment of milestones, and edges are the segments between them.

Original: The ﬁrst type of stochasticity we call node stochasticity , or affordance-affecting randomness that occurs either when milestones are collected (e.g. receiving varying quantity of  item), or at the beginning of the episode (i.e. starting in different contexts).
Revised: The ﬁrst type of stochasticity we call node stochasticity , or affordance-affecting randomness that occurs either when milestones are collected (e.g. receiving varying quantity of an item), or at the beginning of the episode (i.e. starting in different contexts).

Original: The second form of stochasticity is edge stochasticity , or when affordances may change within segments where no milestones are collected.
Revised: The second form of stochasticity is edge stochasticity , or when affordances may change within segments where no milestones are collected.

Original: Spilling cooked noodles on the ﬂoor is one example of this.
Revised: Spilling cooked noodles on the ﬂoor is one example of this.

Original: [Figure]

Original: A more unlikely example would be receiving fully cooked noodles from a neighbor while in the midst of cooking.
Revised: A more unlikely example would be receiving fully cooked noodles from a neighbor while in the midst of cooking.

Original: Edge stochasticity requires us to clarify our working deﬁnition of hierarchical affordances.
Revised: Edge stochasticity requires us to clarify our working deﬁnition of hierarchical affordances.

Original: If cooking noodles for PASTA generally requires intermediate milestones, but there is some chance a neighbor will save the day, should an agent rely on this possibility?
Revised: If cooking noodles for PASTA generally requires intermediate milestones, but there is some chance a neighbor will save the day, should an agent rely on this possibility?

Original: So long as edge  stochasticity events can be viewed as exceptions to a general set of dependency rules, the deﬁnition still stands.
Revised: So long as edge [Figure] stochasticity events can be viewed as exceptions to a general set of dependency rules, the deﬁnition still stands.

Original: By grounding these rules in the current state, an affordance-aware agent can detect and adapt to edge anomalies.
Revised: By grounding these rules in the current state, an affordance-aware agent can detect and adapt to edge anomalies.

Original: In the next section, we describe in detail how hierarchical affordances allow us to learn and use these rules from milestones in stochastic environments where symbolic history would fail to reliably determine the current context.
Revised: In the next section, we describe in detail how hierarchical affordances allow us to learn and use these rules from milestones in stochastic environments where symbolic history would fail to reliably determine the current context.

Original:  H IERARCHICAL A FFORDANCE L EARNING
Revised:  H IERARCHICAL A FFORDANCE L EARNING

Original: In typical HRL methods, if the meta-controller is yet to receive extrinsic reward from the environment, there will be no preference for selecting any subtask  the others.
Revised: In typical HRL methods, if the meta-controller is yet to receive extrinsic reward from the environment, there will be no preference for selecting any subtask over the others.

Original: However, by restricting the selection of subtasks to ones that have proven merely to be possible , an agent can avoid wasting time attempting the impossible, and reach more fruitful subtasks faster.
Revised: However, by restricting the selection of subtasks to ones that have proven merely to be possible , an agent can avoid wasting time attempting the impossible, and reach more fruitful subtasks faster.

Original: Suppose from experience, gained through random exploration, the agent achieves milestone g (where achievement means b g = 1 ) very often from the initial state set I , but never j ∈ G , despite being able to achieve j in later states.
Revised: Suppose from experience, gained through random exploration, the agent achieves milestone g (where achievement means b g = 1 ) very often from the initial state set I , but never j ∈ G , despite being able to achieve j in later states.

Original: With enough experience, the agent should becomerelatively conﬁdent that j is not achievable without completing other milestones ﬁrst, and should not bother selecting j from any s ∈ I , while g , and any others that are achievable from those states, should instead be considered.
Revised: With enough experience, the agent should become conﬁdent that j is not achievable without completing other milestones ﬁrst, and should not bother selecting j from any s ∈ I , while g , and any others that are achievable from those states, should instead be considered.

Original: If we had access to an oracle function, f ∗ ( s ) , that accurately computes hierarchical affordances for our task, we could prune impossible subtasks by masking the otherwise uninformed policy with the affordance oracle output: p ( g | s ) 
Revised: If we had access to an oracle function, f ∗ ( s ) , that accurately computes hierarchical affordances for our task, we could prune impossible subtasks by masking the otherwise uninformed policy with the affordance oracle output: p ( g | s ) ∝ f s g ∗ Π Θ

Original: ∝ f sg ∗ Π Θ ( g | s ) .
Revised: ( g | s ) .

Original: In the following sections, we describe a method that can learn an approximate f ( s )
Revised: In the following sections, we describe a method that can learn an approximate f ( s )

Original: ≈
Revised: ≈

Original: f ∗ ( s ) from experience and leverage it in real-time for more effective learning.
Revised: f ∗ ( s ) from experience and leverage it in real-time for more effective learning.

Original: The false negative problem Recall that the purpose of f ( s ) is to output a vector f s , where each f sg indicates the possibility of collecting milestone g from state s without requiring intermediate milestones.
Revised: The false negative problem Recall that the purpose of f ( s ) is to output a vector f s , where each f sg indicates the possibility of collecting milestone g from state s without requiring intermediate milestones.

Original: To train each binary classiﬁcation head, f g ( s ) , we must somehow generate labeled data for each milestone.
Revised: To train each binary classiﬁcation head, f g ( s ) , we must somehow generate labeled data for each milestone.

Original: Suppose an option was initialized at time-step t o , and in time-step T milestone g is received.
Revised: Suppose an option was initialized at time-step t o , and in time-step T milestone g is received.

Original: If no others were received since the start of the option, we may assume that for any t where t o ≤ t ≤ T , the collection of milestone g was afforded, so we can use the set of states { s t o , s t o +1 , . . .
Revised: If no others were received since the start of the option, we may assume that for any t where t o ≤ t ≤ T , the collection of milestone g was afforded, so we can use the set of states { s t o , s t o +1 , . . .

Original: , s T } as positive (i.e. f ∗ g ( s ) = 1 ) training examples for the affordance classiﬁer.
Revised: , s T } as positive (i.e. f ∗ g ( s ) = 1 ) training examples for the affordance classiﬁer.

Original: Even if g was not the intended milestone, we can still generate positive training examples for f g in this way.
Revised: Even if g was not the intended milestone, we can still generate positive training examples for f g in this way.

Original: If an option has failed to collected intended milestone g , either through timing out or collecting an unintended milestone, we might be tempted to use the states encountered during that option as negative examples (i.e. f ∗ g ( s ) = 0 ).
Revised: If an option has failed to collect intended milestone g , either through timing out or collecting an unintended milestone, we might be tempted to use the states encountered during that option as negative examples (i.e. f ∗ g ( s ) = 0 ).

Original: However, this occurrence can either be indicative of the states not affording g , or that the subpolicy corresponding to g is sub-optimal and has failed despite g being afforded.
Revised: However, this occurrence can either be indicative of the states not affording g , or that the subpolicy corresponding to g is sub-optimal and has failed despite g being afforded.

Original: These false negatives are a problem for any approach requiring function approximation via neural networks, which are generally not robust to label noise (Song et al., 2020).
Revised: These false negatives are a problem for any approach requiring function approximation via neural networks, which are generally not robust to label noise (Song et al., 2020).

Original: It is particularly troublesome in our case since the noise is greater than the true signal when the subpolicies are under-trained.
Revised: It is particularly troublesome in our case since the noise is greater than the true signal when the subpolicies are under-trained.

Original: Context learning Suppose we had access to an abstract state representation
Revised: Context learning Suppose we had access to an abstract state representation

Original: z s aﬀ = z ∗ aﬀ ( s ) , where any states s i and s j are mapped to the same value only when f ∗ ( s i ) =
Revised: z s aﬀ = z ∗ aﬀ ( s ) , where any states s i and s j are mapped to the same value only when f ∗ ( s i ) =

Original: f ∗ ( s j ) .
Revised: f ∗ ( s j ) .

Original: With a representation that could cluster states in this way, we could trivially determine the falsity of a collected negative s ∈ D − g by checking if z ∗ aﬀ ( s ) =
Revised: With a representation that could cluster states in this way, we could trivially determine the falsity of a collected negative s ∈ D − g by checking if z ∗ aﬀ ( s ) =

Original: z ∗ aﬀ ( s j ) for any s j ∈ D + g , that is, if we have encountered a true positive with the same representation.
Revised: z ∗ aﬀ ( s j ) for any s j ∈ D + g , that is, if we have encountered a true positive with the same representation.

Original: The classiﬁcation procedure could be interpreted as “labeling” these contexts with affordance values.
Revised: The classiﬁcation procedure could be interpreted as “labeling” these contexts with affordance values.

Original: This is somewhat of a “chicken and egg” problem, since to learn affordances, we require a representation that maps states to contexts with the same affordance values, which clearly requires some prior knowledge about affordances.
Revised: This is somewhat of a “chicken and egg” problem, since to learn affordances, we require a representation that maps states to contexts with the same affordance values, which clearly requires some prior knowledge about affordances.

Original: Fortunately, from Section 4, we know that affordances will only change when either (1) a milestone is collected and (2) when edge stochasticity occurs.
Revised: Fortunately, from Section 4, we know that affordances will only change when either (1) a milestone is collected and (2) when edge stochasticity occurs.

Original: Since (2) is by deﬁnition a rare occurrence, states s t and s t +1 are more likely than not to satisfy f ∗ ( s i ) =
Revised: Since (2) is by deﬁnition a rare occurrence, states s t and s t +1 are more likely than not to satisfy f ∗ ( s t ) =

Original: f ∗ ( s j ) , so long as they exist in the same segment between milestones.
Revised: f ∗ ( s t +1 ) , so long as they exist in the same segment between milestones.

Original: In this case, we can say that s t and s t +1 share the same achievement context , z s =
Revised: In this case, we can say that s t and s t +1 share the same achievement context , z s =

Original: z ( s ) .
Revised: z ( s ) .

Original: Let z ψ ( s ) be an achievement context embedding represented by a differentiable function parameterized by ψ .
Revised: Let z ψ ( s ) be an achievement context embedding represented by a differentiable function parameterized by ψ .

Original: We can train z ψ ( s ) from experience using the following contrastive loss:
Revised: We can train z ψ ( s ) from experience using the following contrastive loss:

Original: [Equation]
Revised: [Equation]

Original: [Table] where each s aj is a randomly chosen anchor , each s pj is a positive 2 example chosen  according to a segment-truncated normal distribution, N T ( a, σ 2 ) (centered around the anchor point), s nj is chosen randomly among other segments and is treated as a negative example, and α is an arbitrary margin value.
Revised: where each s aj is a randomly chosen anchor , each s pj is a positive 2 example chosen within the segment according to a (truncated) normal distribution, N T (0 , σ 2 ) , centered around (and excluding) s aj , s nj is chosen randomly among other segments and is treated as a negative example, and α is an arbitrary margin value.

Original: This loss will have the effect of pushing representations of states from the same achievement context together, and pulling representations of states from different contexts apart.
Revised: This loss pushes representations of states from the same achievement context together, and pulls representations of states from different contexts apart.

Original: Depending on the level of  stochasticity expected in an environment, σ can be adjusted to balance the prevention of sampling across edge stochasticity occurrences with the generalizing capabilities of the representation. We show in Figure 5 that this parameter is relatively easy to set in that the same value can be used across different frequencies of edge stochasticity.
Revised: We show in Appendix B that a wide range of σ produce useful representations. For our edge stochasticity experiments (Figure 5) we use a low σ = 2 . 0 to reduce the risk of sampling across affordance changes.

Original: False negative ﬁltering In the learned representation space, we expect false negative points to be closer to positive points than true negatives.
Revised: False negative ﬁltering In the learned representation space, we expect false negative points to be closer to positive points than true negatives.

Original: Given an unveriﬁed negatively-labeled point s q for classiﬁer head f g , we compute  the mean distance from z ψ ( s q ) to the representations of the k closest positive points3 in  D + g , denotedas d kq .
Revised: Given a negatively-labeled state s q for classiﬁer head f g , we compute 3 the mean distance from z ψ ( s q ) to the representations of the k closest positive points in a population uniformly sampled 4 from D + g , denoted d kq .

Original: This procedure is akin to the particle entropy approach used in (Hao & Pieter, 2021).

Original: We expect d kq to be larger for true negatives and smaller for false ones, and we deﬁne a separating margin in the following way.
Revised: We expect d kq to be large for true negatives and small for false ones, and we can determine an effective separating margin in the following way.

Original: Since weknow all positives are true positives, we can compute distance scores for a set of positive points { d kp }  that are disjoint from the set used in the KNN procedure.
Revised: First we compute distance scores for a random sample of positive points, denoted { d kp } , to use as reference. We ensure these points come from segments that are disjoint from the population points’ segments to avoid trivially low scores that might skew the distribution.

Original: Wecan then ﬁt a Gaussian distribution to { d k p } andthen compute an upper conﬁdence bound ρ for a given percentile value and conﬁdence level.
Revised: We then ﬁt a Gaussian distribution to { d kp } and compute an upper conﬁdence bound ρ for a given percentile value and conﬁdence level.

Original: Any d kq < ρ is very similar to positive points in the representation space, so we count s q as a false negative and exclude it from our training set.
Revised: Any d kq < ρ is very similar to positive points in the representation space, so we count s q as a false negative and exclude it from our training set.

Original: This process presupposes access to positive examples for each classiﬁer head f g , but we do not traina head  until we have access to both positives and negatives for g .
Revised: Note, we do not train head f g (and therefore do not reliably prune) until we have access to both positives and negatives for g .

Original: Method overview The HAL architecture consists of a bi-level policy like h-DQN, a context embedding network, and an affordance classiﬁer (see Figure 2), which are all learned concurrently (full algorithm in Appendix D).
Revised: Method overview The HAL architecture consists of a bi-level policy like h-DQN, a context embedding network, and an affordance classiﬁer (see Figure 2), which are all learned concurrently (full algorithm in Appendix E).

Revised: Intuitively, the affordance classiﬁer is able to generalize to a novel state, s , by ﬁrst identifying the abstract achievement context, z ψ ( s ) , associated with the state, and then outputting an affordance value based on previous experience in that context.

Revised: If z ψ ( s ) has also not been encountered, that context will not be strongly “labeled” either way, so we will not be invariably pruning it.

Original: The meta-controller selects new subtasks at the start of each episode, after collecting any milestone, and whenever an option times out after a predeﬁned number of steps.
Revised: During each episode, the meta-controller selects new subtasks at the start of the episode, after collecting any milestone, and whenever an option times out after a predeﬁned number of steps.

Original: At each step, the current state is fed to the controller, which outputs an action conditioned on the most recently selected subtask.
Revised: At each step, the current state is fed to the controller, which outputs an action conditioned on the most recently selected subtask.

Original: After discretizing the classiﬁer’s output to a binary mask, we perform an affordance aware version of (cid:15) -greedy as follows.
Revised: After discretizing the classiﬁer’s output to a binary mask (over threshold 0 . 5 ), we perform an affordance aware version of (cid:15) -greedy as follows.

Original: Given parameters (cid:15) aff and (cid:15) mc , we select a random subtask within the mask with probability (cid:15) aff , randomly across all subtasks with probability (cid:15) mc , and otherwise select greedily with respect to Q within the mask.
Revised: Given parameters (cid:15) aff and (cid:15) mc , we select a random subtask within the mask with probability (cid:15) aff , randomly across all subtasks with probability (cid:15) mc , and otherwise select greedily with respect to meta-Q within the mask.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we aim to answer the following questions: (1) Does HAL improve learning in tasks with complex dependencies?
Revised: In our experiments we aim to answer the following questions: (1) Does HAL improve learning in tasks with complex dependencies?

Original: (2) Is HAL robust to milestone selection and context stochasticity?
Revised: (2) Is HAL robust to milestone selection and context stochasticity?

Original: (3) Can HAL more effectively learn a diverse set of skills when training task-agnostically?
Revised: (3) Can HAL more effectively learn a diverse set of skills when trained task-agnostically?

Original: 6. E NVIRONMENTS
Revised: 6. E NVIRONMENTS

Original: We evaluate our method, along with several baselines, on two complex environments with intricate subtask dependency structures: CRAFTING and TREASURE .
Revised: We evaluate our method, along with several baselines, on two complex environments with intricate subtask dependency structures: CRAFTING and TREASURE .

Original: Both environments (visualized in Figure 3) are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).
Revised: Both environments (visualized in Figure 3) are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).

Original: Agents receive an egocentric image of the environment, as well as a vector describing their inventory (items picked up from the environment and currently in their possession) as observations.
Revised: Agents receive an egocentric image of the environment, as well as a vector describing their inventory (items picked up from the environment and currently in their possession) as observations.

Original: The action spaces are discrete and include actions for turning left/right, and moving forward/backward, as well as environment speciﬁc actions detailed below.
Revised: The action spaces are discrete and include actions for turning left/right, and moving forward/backward, as well as environment speciﬁc actions detailed below.

Original: Walk-throughs of successful trajectories, as well as more information about each environment, are included in Appendix A.
Revised: Task hierarchies, walk-throughs of successful trajectories, and additional information about both environments are included in Appendix A.

Original: CRAFTING is based on MineCraft, a popular open-ended video game in which players collect resources from their environment and use them to craft objects which can be used to then obtain more resources.
Revised: CRAFTING is based on Minecraft, a popular open-ended video game in which players collect resources from their environment and use them to craft objects which can be used to then obtain more resources.

Original: As such, the hierarchy of possible subtasks is immensely complex and presents a signiﬁcant challenge for AI agents to reach subtasks deeper in the hierarchy.
Revised: As such, the hierarchy of possible subtasks is immensely complex and presents a signiﬁcant challenge for AI agents to reach subtasks deeper in the hierarchy.

Original: We develop an environment that replicates this hierarchical complexity without the commensurate visuomotor complexity which our method does not aim to address.
Revised: We develop an environment that replicates this hierarchical complexity without the commensurate visuomotor complexity which our method does not aim to address.

Original: In addition to movement actions, CRAFTING includes actions to mine the object immediately in front of the agent (which requires an appropriate pickaxe), as well as to craft and smelt the various objects (pickaxes, iron ingots, etc.).
Revised: In addition to movement actions, CRAFTING includes actions to mine the object immediately in front of the agent (which requires an appropriate pickaxe), as well as to craft and smelt the various objects (pickaxes, iron ingots, etc.).

Original: The full set of milestones contains items that are either craftable or collectable.
Revised: The full set of milestones contains items that are either craftable or collectable.

Original: CRAFTING naturally contains node stochasticity since the collection of certain items, due to the random procedural generation, requires slightly different milestone trajectories across episodes (e.g. mining variable amount of stone to encounter diamond).
Revised: CRAFTING naturally contains node stochasticity since the collection of certain items, due to the random procedural generation, requires slightly different milestone trajectories across episodes (e.g. mining variable amount of stone to encounter diamond).

Original: TREASURE is a navigation task that requires the agent to collect various items and use them to unlock rooms to reach further items.
Revised: TREASURE is a navigation task that requires the agent to collect various items and use them to unlock rooms to reach further items.

Original: The ultimate goal is to unlock a treasure chest, which requires a sequence of collecting several keys, as well as placing an object on a weight-based sensor, in order to open the requisite doors.
Revised: The ultimate goal is to unlock a treasure chest, which requires a sequence of collecting several keys, as well as placing an object on a weight-based sensor, in order to open the requisite doors.

Original: Agents can only carry one object at a time, so they must reason about which object to pick up based on what it will afford them (e.g. if the weight-based sensor room is locked, the weight object is not currently useful).
Revised: Agents can only carry one object at a time, so they must reason about which object to pick up based on what it will afford them (e.g. if the weight-based sensor room is locked, the weight object is not currently useful).

Original: Like CRAFTING , TREASURE contains node stochasticity due to the procedural generation.
Revised: Like CRAFTING , TREASURE contains node stochasticity due to the procedural generation.

Original: For example, the central room that the agent is spawned in can contain either of the red or yellow key individually, or both together.
Revised: For example, the central room that the agent is spawned in can contain either of the red or yellow key individually, or both together.

Original: Unlike CRAFTING , which has a large action space to accommodate the various crafting recipes, this environment only contains actions to move and a single “interaction” action that is used to pick up keys, open doors, etc.
Revised: Unlike CRAFTING , which has a large action space to accommodate the various crafting recipes, this environment only contains actions to move and a single “interaction” action that is used to pick up keys, open doors, etc.

Original: While
Revised: While

Original: CRAFTING has a more complex hierarchy and greater diversity in the potential ordering of subtasks,
Revised: CRAFTING has a more complex hierarchy and greater diversity in the potential ordering of subtasks,

Original: TREASURE has on average more difﬁcult subtasks.
Revised: TREASURE has on average more difﬁcult subtasks.

Original: The  set of milestones contains each object the agent can successfully interact with in the environment.
Revised: The full set of milestones contains each object the agent can successfully interact with in the environment (e.g. opening door, collecting key).

Original: 6. B ASELINES
Revised: 6. B ASELINES

Original:  Our set of baselines is summarized in Table 1.
Revised: [Table] Our set of baselines is summarized in Table 1.

Original: All methods are based on the Rainbow (Hessel et al., 2018)
Revised: All methods are based on the Rainbow (Hessel et al., 2018)

Original: Deep Q-Learning algorithm, which combines several improvements to vanilla DQNs (Mnih et al., 2015).
Revised: Deep Q-Learning algorithm, which combines several improvements to vanilla DQNs (Mnih et al., 2015).

Original: To compensate for the lack of milestone signals, non-hierarchical methods use a dense reward function that incorporates milestone signals for the ﬁrst time each mile-
Revised: To compensate for the lack of milestone signals, non-hierarchical methods use a dense reward function that incorporates milestone signals for the ﬁrst time each milestone is obtained in the episode. To evaluate the efﬁcacy of our affordance classiﬁer

Original: [Figure]
Revised: [Figure]

Original: called timing stone is obtained in the episode.

Original: We compare our method to an oracle version that uses groundtruth affordances in order to evaluate the efﬁcacy of our affordance classiﬁer learning procedure
Revised: online learning procedure, we compare our method to an “oracle” that differs only by using groundtruth affordances for masking subtasks.

Original: [Figure] [Figure] [Table] The node stochasticity inherent to both environments, as well as the edge stochasticity later explored, preclude the use of any methods that reason solely over symbols (i.e. automata-, sketch-, or subtask dependency-based approaches).
Revised: The node stochasticity inherent to both environments, as well as the edge stochasticity later explored, preclude the use of any methods that reason solely over symbols (i.e. automata-, sketch-, or subtask dependency-based approaches).

Original: We also incorporate a version of Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) adapted for the discrete milestone setting, which involves re-using “failed” trajectories that result in the collection of an unintended milestone (given the selected subpolicy) as successful data for the relevant subpolicy.
Revised: We also incorporate a version of Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) adapted for the discrete milestone setting, which involves re-using “failed” trajectories that result in the collection of an unintended milestone (given the selected subpolicy) as successful data for the relevant subpolicy.

Original: For instance, if an agent accidentally collects iron while executing its wooden pickaxe sub-policy, it can use this trajectory to train its iron sub-policy.
Revised: For instance, if an agent accidentally collects iron while executing its wooden pickaxe sub-policy, it can use this trajectory to train its iron sub-policy.

Original: 6. R ESULTS
Revised: 6. R ESULTS

Original: Learning efﬁcacy First, we evaluate the ability of HAL and our baselines to learn successful policies for complex tasks in each environment.
Revised: Learning efﬁcacy First, we evaluate the ability of HAL and our baselines to learn successful policies for complex tasks in each environment.

Original: Learning curves are shown in Figure 4.
Revised: Learning curves are shown in Figure 4.

Original: In both environments, HAL signiﬁcantly outperforms the strongest baseline, H-Rainbow(+HER) (HR+H), despite both methods receiving the same information, and performs only slightly worse than the oracle, which has access to ground truth.
Revised: In both environments, HAL signiﬁcantly outperforms the strongest baseline, H-Rainbow(+HER) (HR+H), despite both methods receiving the same information, and performs only slightly worse than the oracle, which has access to ground truth.

Original: Incorporating HER into H-Rainbow appears tobe a signiﬁcant improvement.
Revised: Incorporating HER into H-Rainbow leads to a signiﬁcant improvement.

Original: False negative ﬁltering also appears crucial for successful learning in the TREASURE environment, but not so much for CRAFTING .
Revised: False negative ﬁltering (and all other HAL components; see Appendix B) appears crucial for successful learning in the TREASURE environment, but not as much for CRAFTING , though in both cases ﬁltering improves mask accuracy.

Original: Removing false negative ﬁltering causes HAL(-FNF) to be pessimistic (i.e. over-pruning subtasks, see Figures 9a and 9c), ultimately leading to its unstable learning.
Revised: Removing false negative ﬁltering causes HAL(-FNF) to be pessimistic (i.e. over-pruning subtasks, see Figures 15a and 15c in the Appendix D), ultimately leading to its unstable learning.

Original: Since masking impossible subpolicies would have no impact on an optimal meta-controller, HAL’s success must stem from its ability to learn a useful mask before TD errors are able to propagate through the meta-controller’s Q function.
Revised: Since masking impossible subpolicies would have no impact on an optimal meta-controller, HAL’s success must stem from its ability to learn a useful mask before TD errors are able to propagate through the meta-controller’s Q function.

Original: HAL utilizes a more easily learnable function (the affordance classiﬁer) to reduce the amount of unnecessary expensive learning (TD error propagation) required.
Revised: HAL utilizes a more easily learnable function (the affordance classiﬁer) to reduce the amount of unnecessary expensive learning (TD error propagation) required.

Original: We see from Figures 9a and 9c that, throughout training, HAL’s mask has an impact on greedy subtask selection ∼ 60% of the time, which is evidence that HAL avoids wasting time learning Q-values that the mask is able to prune.
Revised: We see from Figure 15 that, throughout training, HAL’s mask has an impact on greedy subtask selection ∼ 60% of the time, which is evidence that HAL avoids wasting time learning Q-values that the mask is able to prune.

Original: Lastly, because affordance-aware methods are more likely to initiate subtasks in an appropriate context, we see they achieve a signiﬁcantly higher average subpolicy success rate (Figure 4, right).
Revised: Lastly, because affordance-aware methods are more likely to initiate subtasks in an appropriate context, we see they achieve a signiﬁcantly higher average subpolicy success rate (Figure 4, right).

Original: Robustness to milestone selection In this section we evaluate HAL’s robustness to the selection of milestones used for training.
Revised: Robustness to milestone selection In this section we evaluate HAL’s robustness to the selection of milestones.

Original: When a milestone is removed, the environment’s deﬁnition of affordances changes, since the milestone no longer acts as an intermediate link between others.
Revised: Affordances change when a milestone is removed since that milestone no longer acts as an intermediate link between others.

Original: One downside of some approaches that use symbol-based contexts is that an entirely different automata or subtask dependency graph must be deﬁned over the new set of symbols.
Revised: One downside of some approaches that use symbol-based contexts is that an entirely different automata or subtask dependency graph must be deﬁned over the new set of symbols.

Original: HAL does not use prior information of this kind, so the learning process is the same across all sets.
Revised: HAL does not use prior information of this kind, so the learning process is the same across all sets.

Original: Figure 5 shows HAL’s success on the CRAFTING environment’s iron task when using “incomplete” milestone[Figure] sets, relative to the full human-designed set.
Revised: Figure 5 shows HAL’s success on the CRAFTING environment’s iron task when using “incomplete” milestone sets, relative to the full human-designed set.

Original: We see that randomly removing 1 milestone makes no signiﬁcant difference for HAL, and even after removing 4 milestones, HAL still achieves better performance than HR+H using the full set.
Revised: We see that randomly removing 1 milestone makes no signiﬁcant difference for HAL, and even after removingmilestones, HAL still achieves better performance than HR+H using the full set.

Original: We imagine that a slight increase in  training time would lead HAL  to the same convergence as observed for the full set.
Revised: HAL’s performance drops when 5 milestones are removed likely due to the increased sparsity of the signal (i.e. greater subtask length) and variance in milestone set quality. However, when we double the training time for these same sets, we ﬁnd that HAL is able to converge to a 97% success rate on at least one set, while HR+H fails to converge on any set and ends with a maximum success rate of around 70% . [Figure]

Original: Robustness to stochasticity We modifythe CRAFTINGenvironment so that at each environment step, there is a certain probability that an item in the inventory will disappear.
Revised: Robustness to stochasticity We modify CRAFTING so that at each environment step, there is a certain probability that an item in the inventory will disappear.

Original:  This procedure produces edge stochasticity , since the disappearance of items may change what is afforded, and this can occur at anypoint in time.In order to make the task feasible, we design it so that rare items are less likely to disappear than common ones.
Revised: In order to make the task feasible, rare items are less likely to disappear than common ones. This procedure produces edge stochasticity , since the disappearance of items may alter affordances, and this can occur at any time.

Original: We test three different levels of stochasticity, and display the learning curves in Figure 5.
Revised: We test three different levels of stochasticity, and display the learning curves in Figure 5.

Original: With a disappearance frequency of 1 / 100 that cuts HR+H’s success rate in half, HAL is still able to reach its non-stochastic success rate.
Revised: With a disappearance frequency of 1 / 100 that cuts HR+H’s success rate in half, HAL is still able to reach its non-stochastic success rate.

Original: With a frequency of 1 / 50 , HAL performs comparably to HR+H with no stochasticity.
Revised: With a frequency of 1 / 50 , HAL performs comparably to HR+H with no stochasticity.

Original: To put these stochasticity rates into context, the algorithm’s average episode length about halfway through training is still oversteps, meaning dozens of items are removed from the agent’s inventory over the course of an episode.
Revised: To put these stochasticity rates into context, the algorithm’s average episode length about halfway through training is still over 1000 steps (see Figure 14 in Appendix C), meaning dozens of items are removed from the agent’s inventory over the course of an episode.

Original: By learning  affordances grounded in the  state, HAL is able to  adaptwell to these stochastic events.
Revised: By learning a model of affordances grounded in the present state, HAL is able to detect and adapt to these stochastic events.

Original: L og W oo d S t i c k C r a f t i n g B e n c h W oo d P i c
Revised: [Figure]

Original: k a x e S t o n e F u r n a ce
Revised: ce

Original: S t o n e P
Revised: S t o n e P

Original: i c k a x e C
Revised: i c k a x e C

Original: o a l
Revised: o a l

Original: I r o n O r
Revised: I r o n O r

Original: e
Revised: e

Original: I r o n I r o n P
Revised: I r o n I r o n P

Original: i
Revised: i

Original: c
Revised: c

Original: k a x e
Revised: k a x e

Original: D i a m o nd
Revised: D i a m o nd

Original: [Figure]
Revised: Subtask 0 .0 .0 .0 .1 . S u cce ss R a t e HAL (ours) H-Rainbow( + HER) H i e r a r c h y D

Original: [Figure]
Revised: e p t h

Original: Task-agnostic learning We next test the ability of HAL to learn skills when no task-speciﬁc extrinsic rewards (only milestones) are provided by the environment.
Revised: Task-agnostic learning We next test the ability of HAL to learn skills when no task-speciﬁc extrinsic rewards (only milestones) are provided by the environment.

Original: Since we cannot learn a meta-controller in the absence of rewards, we instead randomly select subtasks with some probability (cid:15) mc  and random afforded subtasks otherwise.
Revised: Since we cannot learn a meta-controller in the absence of rewards, we instead randomly select subtasks with some probability (cid:15) mc , and random afforded subtasks otherwise (only for HAL).

Original: We test both HAL andour strongest baseline – HR+H. 
Revised: We evaluate both HAL and HR+H. In

Original: In Figure 6 we see that by the end of 10 6 steps, HAL is able to more reliably complete the milestones deeper in the hierarchy in the CRAFTING environment.
Revised: Figure 6 we see that by the end of 10 6 steps, HAL is able to more reliably complete the milestones deeper in the hierarchy in the CRAFTING environment.

Original: We note that HR+H is able to marginally outperform HAL in tasks shallower in the hierarchy (e.g. wood pickaxe, stone, furnace), potentially as a result of failing to reach deeper tasks and getting more practice on shallower ones.
Revised: We note that HR+H is able to marginally outperform HAL in tasks shallower in the hierarchy (e.g. wood pickaxe, stone, furnace), potentially as a result of failing to reach deeper tasks and getting more practice on shallower ones.

Original: This result is an indication of the general utility of HAL in environments with complex task hierarchies.
Revised: This result is an indication of the general utility of HAL in environments with complex task hierarchies.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND F UTURE W ORK

Original: The present work can be viewed as a ﬁrst step towards bridging the substantial gap between ﬂexible hierarchical approaches that are presently intractable, and methods that impose useful structures but are too rigid to be of practical use.
Revised: The present work can be viewed as a ﬁrst step towards bridging the substantial gap between ﬂexible hierarchical approaches that are presently intractable, and methods that impose useful structures but are too rigid to be of practical use.

Original: We introduce HAL, a method that is able to utilize incomplete symbolic information in order to learn a more general form of subtask dependency.
Revised: We introduce HAL, a method that is able to utilize incomplete symbolic information in order to learn a more general form of subtask dependency.

Original: By grounding subtask dependencies in the present state through the learning  of hierarchical affordances, HAL is able to navigate stochastic environments that approaches relying solely on symbolic history are unable to.
Revised: By grounding subtask dependencies in the present state by learning a model of hierarchical affordances, HAL is able to navigate stochastic environments that approaches relying solely on symbolic history are unable to.

Original: We demonstrate that HAL learns more efﬁciently than baselines provided with the same information, is more robust to milestone selection and affordance stochasticity, and can more thoroughly explore the environment’s subtask hierarchy.
Revised: We demonstrate that HAL learns more effectively than baselines provided with the same information, is more robust to milestone selection and affordance stochasticity, and can more thoroughly explore the environment’s subtask hierarchy.

Original:  HAL’s  success in the face of incomplete and stochastic symbolic information bodes well for its ability to mesh with approaches that automatically detect salient subgoals in complex environments.
Revised: Given HAL’s ﬂexible formulation and success in the face of incomplete and stochastic symbolic information, we foresee future work integrating HAL with option (or subgoal) discovery methods (e.g. Bacon et al., 2017; Machado et al., 2017; Bagaria & Konidaris, 2019) to obtain performance gains in complex tasks without requiring prespeciﬁed milestones. Additionally, future work might be able to extend HAL to continuous goalspaces, but this would require revising the deﬁnition of hierarchical affordances provided here, as it currently requires a notion of intermediate subgoal completion.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: All code for environments, algorithms, and baselines is provided at the following link: https://github.com/jamesdoewhat/hal-anonymous.
Revised: All code for environments, algorithms, and baselines is provided at the following link: https://github.com/jamesdoewhat/hal-anonymous.

Original: We enumerate the list of commands to reproduce all experiments in the paper, as well as a ﬁle specifying the python packages and versions required which can be installed via the Anaconda python package manager.
Revised: We enumerate the list of commands to reproduce all experiments in the paper, as well as a ﬁle specifying the python packages and versions required which can be installed via the Anaconda python package manager.

Original: All hyperparameters and additional implementation details are listed in the Appendix.
Revised: All hyperparameters and additional implementation details are listed in the Appendix.

Original: A  E NVIRONMENT D ETAILS
Revised: A A DDITIONAL E NVIRONMENT D ETAILS

Original: In both environments, extrinsic rewards are a sparse binary signal provided upon the completion of a ﬁnal goal.
Revised: In both environments, extrinsic rewards are a sparse binary signal provided upon the completion of a ﬁnal goal.

Original: Milestone signals are similarly formulated for each of the possible subtasks.
Revised: Milestone signals are similarly formulated for each of the possible subtasks.

Original: For both types of rewards (extrinsic and milestone) we introduce a per step penalty that encourages agents to achieve their goal as quickly as possible.
Revised: For both types of rewards (extrinsic and milestone) we introduce a per step penalty that encourages agents to achieve their goal as quickly as possible.

Revised: TREASURE and CRAFTING environments are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).

Original: A. CRAFTING
Revised: A. CRAFTING

Original: As discussed in Section 6.3, our CRAFTING environment is designed to maximize hierarchical complexity while minimizing visuomotor complexity, which we do not aim to address with our method.
Revised: As discussed in Section 6.3, our CRAFTING environment is designed to maximize hierarchical complexity while minimizing visuomotor complexity, which we do not aim to address with our method.

Original: Similar MineCraft-based environments have been used in the literature; however they do not meet these requirements.
Revised: Similar Minecraft-based environments have been used in the literature; however they do not meet these requirements.

Original: The MineRL environment (Guss et al., 2019) provides an interface into the full game of MineCraft; however, effective behavior in this environment requires learning complex visuomotor policies in addition to understanding the hierarchical relationships between subtasks.
Revised: The MineRL environment (Guss et al., 2019) provides an interface into the full game of Minecraft; however, effective behavior in this environment requires learning complex visuomotor policies in addition to understanding the hierarchical relationships between subtasks.

Original: In order to evaluate our method effectively, we only aim to test the former.
Revised: In order to evaluate our method effectively, we only aim to test the latter.

Original: Other work has used
Revised: Other work has used

Original: MineCraft-inspired environments (Andreas et al., 2015; Sohn et al., 2020); however, these versions involve simpliﬁed subtask hierarchies.
Revised: Minecraft-inspired environments (Andreas et al., 2016; Sohn et al., 2020); however, these versions involve simpliﬁed subtask hierarchies.

Original: Our environmentfully replicates the complexity of the MineRL subtask hierarchy while remaining perceptually simple.
Revised: Our environment replicates the complexity of the MineRL subtask hierarchy while remaining perceptually simple.

Original: A concurrently developed environment contains similar hierarchical complexity while minimizing perceptual complexity (Hafner, 2021).
Revised: A concurrently developed environment contains similar hierarchical complexity while minimizing perceptual complexity (Hafner, 2021).

Revised: Figure 7 displays an abstract representation of the CRAFTING environment subtask hierarchy.

Original: A. W ALK - THROUGHS
Revised: Each arrow points from one subtask to another, where the latter subtask requires the former in some way. The CRAFTING environment contains a variety of dependency types. Many items must be built or crafted from other resources. Some of these require the agent to be within the vicinity of a crafting bench.

Revised: Other items require speciﬁc pickaxes to be mined, and yet others must be smelted in a furnace using a raw material and a fuel source.

Revised: Another complexity missing from Figure 7 is that crafting recipes require speciﬁc amounts of items.

Revised: For instance, in our environment a stone pickaxe requires three stone and two sticks.

Revised: Each stone must be mined individually, resulting in three separate milestones, while just two wood are required to craft four sticks.

Revised: Lastly, there are

Revised: [Figure]

Revised: [Figure]

Revised: other implicit dependencies than the ones shown, depending on the deﬁnition of the milestones set.

Revised: For example, although the only ofﬁcial prerequisite for obtaining diamond is having an iron pickaxe, in practice, an agent will need to mine stone and other blocks to reach the diamond, and the successful mining of each of these blocks may be considered a milestone.

Revised: CRAFTING is procedurally generated in the following way.

Revised: For the lower half of the environment, each cell is randomly assigned stone, coal, iron, or dirt to each cell with varying probabilities.

Revised: Diamond is randomly assigned to a cell in the bottom-most layer.

Revised: Trees (from which logs are obtained) are abundantly scattered in the upper half, and a few irrelevant dirt blocks are placed in this region as well.

Revised: If the generated environment does not have enough resources, it is regenerated with the next random seed.

Revised: A. TREASURE

Revised: In Figure 8 one possible TREASURE subtask hierarchy is displayed.

Revised: Unlike CRAFTING , there are multiple different abstract hierarchies depending on the initial state of the environment.

Revised: At the beginning of each episode, either both keys are available, only the yellow key, or only the red key (the instance shown in Figure 8).

Revised: While the environment dynamics remain the same across all episodes, the agent must infer which hierarchy is appropriate based on the initial conﬁguration of the environment.

Revised: TREASURE is procedurally generated by randomly assigning each colored door to the room entrances, randomly determining which keys are accessible from the starting room, and lastly placing all objects behind their appropriate doors at random positions within the room.

Revised: A. W ALK - THROUGHS

Original: We provide walk-throughs of successful episodes in TREASURE and CRAFTING  in Figures 7 andrespectively.
Revised: Successful human walk-throughs for both TREASURE and CRAFTING environments are described in Figures 9 and 10 respectively. [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: B A BLATION AND R OBUSTNESS R ESULTS B. C OMPONENT A BLATIONS

Revised: In Figure 11 we plot success on TREASURE after removing various integral components of the full

Revised: HAL method.

Revised: We ﬁnd that all components introduced in this work are necessary for achieving the best performance on this task.

Revised: The only variation that also reliably converges to 100% success rate is when the affordance classiﬁer is provided with the raw state as input rather than using the learned representation, but this variation is undesirable since more learnable parameters are introduced.

Revised: [Figure]

Revised: B. H YPERPARAMETER R OBUSTNESS

Revised: In Figure 12 we demonstrate HAL’s robustness to modiﬁcations of the most signiﬁcant newly introduced hyperparameters on the TREASURE task.

Revised: In only two cases over the wide range of values we tested did all runs not converge.

Revised: The ﬁrst is when the upper conﬁdence bound percentile is set too low, which results in more false negatives being left unﬁltered.

Revised: The second is when the affordance classiﬁer threshold is set too low, which results in less pruning.

Revised: We see that a wide range of more aggressive settings for both of these hyperparameters are reliable.

Revised: We ﬁnd that the standard deviation hyperparameter is not sensitive across the values we test in the non-edge-stochastic TREASURE environment, and that lower values (e.g. σ = 2 . 0 ), which we initially hypothesized might fare better with edge stochasticity but could lead to worse representations, are in actuality still effective.

Revised: [Figure]

Revised: B. V ARYING σ UNDER S TOCHASTICITY

Original: [Figure]
Revised: In Appendix B.2, we demonstrated that in TREASURE , various values of σ centered around the value we used in that setting ( σ = 7 . 0 ) are all conducive to good performance on that task. In Figure 13, we plot the results of using different values in a stochastic version of the CRAFTING environment in the iron task. Although all values lead to signiﬁcantly better performance than HR+H, the value we happened to used in this setting ( σ = 2 . 0 ) appears to strike the best balance. Very low values (e.g. σ = 1 . 0 ) likely do not learn as general a context representation, while higher values (e.g. σ = 8 . 0 , 16 . 0 ) are more likely to sample positive points across occurrences of edge stochasticity.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: It is intriguing that the ﬁnal loss for σ = 2 . 0 is lower than for σ = 1 . 0 .

Original: [Figure]
Revised: We speculate that the context learned by sampling points too close to the anchor could be a less natural representation to learn than a more general one, which considers further points.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: In Figure 14 we display the episode length plots corresponding to the success rate results shown in

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Figures 4 and 5.

Original: [Figure]
Revised: [Figure]

Original: B A FFORDANCE P LOTS
Revised: D A FFORDANCE P LOTS

Original: We provide metrics tracked over the course of training for affordance masking as well as false negative ﬁltering in Figure 9.
Revised: We provide metrics tracked over the course of training for affordance masking as well as false negative ﬁltering in Figure 15.

Original: The mask metrics (Figure 9a for CRAFTING iron and Figure 9c for TREASURE ) consist of (from left to right):
Revised: The mask metrics (Figure 15a for CRAFTING iron and Figure 15c for TREASURE ) consist of (from left to right):

Original: • Affordance Classiﬁer Accuracy: The accuracy of the affordance classiﬁer w.r.t truth.
Revised: • Affordance Classiﬁer Accuracy: The accuracy of the affordance classiﬁer w.r.t truth.

Original: • Mask Impact:
Revised: • Mask Impact:

Original: The percentage of times that the affordance mask prevents selecting an subtask that would have otherwise had the highest Q-value.
Revised: The percentage of times that the affordance mask prevents selecting an subtask that would have otherwise had the highest Q-value.

Original: • Pruning Percentage: Percentage of subtasks pruned • Overpruning Percentage: Percentage of subtasks that are afforded but are pruned.
Revised: • Pruning Percentage: Percentage of subtasks pruned. • Overpruning Percentage: Percentage of subtasks that are afforded but are pruned.

Original: • Underpruning Percentage: Percentage of subtasks that are not afforded but are not pruned.
Revised: • Underpruning Percentage: Percentage of subtasks that are not afforded but are not pruned.

Original: The false negative ﬁltering metrics (Figure 9b for CRAFTING iron and Figure 9d for TREASURE ) consist of:
Revised: The false negative ﬁltering metrics (Figure 15b for CRAFTING iron and Figure 15d for TREASURE ) consist of:

Original: • Filtering Margin: L2 distance at which we consider negatives to be true negatives.
Revised: • Filtering Margin: L2 distance at which we consider negatives to be true negatives.

Original: • True Negative Accuracy: Percentage of true negatives falling above ﬁltering margin.
Revised: • True Negative Accuracy: Percentage of true negatives falling above ﬁltering margin.

Original: • False Negative Accuracy: Percentage of false negatives falling below ﬁltering margin.
Revised: • False Negative Accuracy: Percentage of false negatives falling below ﬁltering margin.

Original: • Percentage False Negatives: Percentage of negatives that are false negatives.
Revised: • Percentage False Negatives: Percentage of negatives that are false negatives.

Original: • False Negatives Flagged: Percentage of negatives that are ﬂagged as false by our margin.
Revised: • False Negatives Flagged: Percentage of negatives that are ﬂagged as false by our margin.

Original: [Figure]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: Step

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: r d a n

Original: [Table]
Revised: [Table]

Original:  C l a ssi ﬁ e r A
Revised: A ﬀ o r d a n ce C l a ss

Revised: i ﬁ e r A cc u r a c y

Revised: Method

Original: [Table]
Revised: [Table]

Original: u r a c y
Revised: HAL (ours

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: ×

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: M a s k

Original: [Table] [Figure] [Table] [Figure]
Revised: I m p a c t

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]{+× 
+}
Revised: {+× 
+}

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table] [Figure]
Revised: [Table] [Table]

Original:  n g P e r ce n t a g e 0 .0 .1 .1 .
Revised: P r un i n g P e r ce n t a g e 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: O v e r p r un i n g
Revised: O v e r p r un i n g

Original: P e r ce n t a g e 0 .0 .1 .1 .
Revised: P e r ce n t a g e 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: U nd e r p r un i n g
Revised: U nd e r p r un i n g

Original: P e r ce n t a g e (
Revised: P e r ce n t a g e (

Original: 5 Step × 

Revised: 5 Step ×

Original: K NN F il t e r
Revised: K NN F il t e r

Original: i n g
Revised: i n g

Original: M a r g i n
Revised: M a r g i n

Original: Method HAL (ours) 0 .0 .1 .1 .
Revised: Method HAL (ours) 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: T r u e N e g a t
Revised: T r u e N e g a t

Original: i v e
Revised: i v e

Original: A cc u r a c y 0 .0 .1 .1 .
Revised: A cc u r a c y 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e
Revised: F a l s e N e g a t i v e

Original: A cc u r a c y 0 .0 .1 .1 .
Revised: A cc u r a c y 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: P e r ce n t a g e F a l s e N e g a t
Revised: P e r ce n t a g e F a l s e N e g a t

Original: i v e s 0 .0 .1 .1 .
Revised: i v e s 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e s F l a gg e d
Revised: F a l s e N e g a t i v e s F l a gg e d

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: ﬀ
Revised: ﬀ

Original: [Table]
Revised: [Table]

Original: r d a n ce C l a ss i ﬁe r A
Revised: l a ss i ﬁ

Original: [Table]
Revised: [Table]

Original: u r a c y

Original: [Table]

Original: [Table]
Revised: [Table]

Original: a s k I m p a c t [Table] [Table] K NN F il t e r 
Revised: il t e r i n g

Original: i n g

Original: M a r g i n
Revised: M a r g i n

Original: [Table]
Revised: [Table]

Original: F a l s e N e g a t i v e
Revised: F a l s e N e g a t i v e

Original: A cc u r a c y 0 . 

Revised: A cc u r a c y 0 .

Original: Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: P e r ce n t a g e F a l s e N e g a t
Revised: P e r ce n t a g e F a l s e N e g a t

Original: i v e s 0 . 0 0 .
Revised: i v e s 0 . 0 0 .

Original: Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e s F l a gg e dC
Revised: F a l s e N e g a t i v e s F l a gg e d

Original: E PISODE L ENGTH P LOTS

Original: [Figure]
Revised: [Figure]

Original: i s o d e L e n

Original: [Figure]
Revised: Figure

Original: D
Revised: E

Original: A LGORITHM
Revised: A LGORITHM

Original: 1: Initialize
Revised: 1: Initialize

Original: 2: steps ← 0 3: while steps < max steps do 4: env steps , option steps ←5:
Revised: 2: steps ← 0 3: while steps < max steps do 4: env steps , option steps ←5:

Original: Receive initial state, s , from the environment 6: while s is not terminal or env steps < max env steps do 7: if env steps = 0 or option steps > max option steps or (cid:80) j b j > 0 then 8: option steps ← 0 9: if env steps > 0 then 10:
Revised: Receive initial state, s , from the environment 6: while s is not terminal or env steps < max env steps do 7: if env steps = 0 or option steps > max option steps or (cid:80) j b j > 0 then 8: option steps ← 0 9: if env steps > 0 then 10:

Original: Store transition ( s init , g, b g , s ) in D m 11:
Revised: Store transition ( s init , g, b g , s ) in D m 11:

Original: Store examples s t
Revised: Store examples s t

Original: | t ∈ [ init , env steps ] in positive buffers D + j 12: where b j = 1 and negative buffers D − k where b k =13: end if 14: init ← env steps 15: f s ← f φ ( z ψ ( s ))
Revised: | t ∈ [ init , env steps ] in positive buffers D + j 12: where b j = 1 and negative buffers D − k where b k =13: end if 14: init ← env steps 15: f s ← f φ ( z ψ ( s ))

Original: (cid:46) Get predicted affordances 16: q ← Q m ( s, · ; Θ) (cid:46) Compute Q-values for all goals 17: q j ← −∞ ∀ f sj =(cid:46)
Revised: (cid:46) Get predicted affordances 16: q ← Q m ( s, · ; Θ) (cid:46) Compute Q-values for all goals 17: q j ← −∞ ∀ f sj =(cid:46)

Original: Mask non-afforded goals 18: g max ← arg max q (cid:46)
Revised: Mask non-afforded goals 18: g max ← arg max q (cid:46)

Original: Get best afforded goal 19: g aff ← Rand ( j | f sj = 1) (cid:46) Get random afforded goal 20: g rand ← Rand ( G ) (cid:46)
Revised: Get best afforded goal 19: g aff ← Rand ( j | f sj = 1) (cid:46) Get random afforded goal 20: g rand ← Rand ( G ) (cid:46)

Original: Get random goal 21: g ← g aff w.p (cid:15) aff , g rand w.p (cid:15) mc , else g max (cid:46) (cid:15) 2 -greedy goal selection 22: end if 23: a max ← arg max Q c ( s, · ; g, Θ) (cid:46) Get best action – conditioned on goal 24:
Revised: Get random goal 21: g ← g aff w.p (cid:15) aff , g rand w.p (cid:15) mc , else g max (cid:46) (cid:15) 2 -greedy goal selection 22: end if 23: a max ← arg max Q c ( s, · ; g, Θ) (cid:46) Get best action – conditioned on goal 24:

Original: a rand ← Rand ( A ) (cid:46) Get random action 25: a ← a rand w.p (cid:15) c , else a max (cid:46) (cid:15) -greedy action selection 26: steps + + , option steps + + , env steps + + 27:
Revised: a rand ← Rand ( A ) (cid:46) Get random action 25: a ← a rand w.p (cid:15) c , else a max (cid:46) (cid:15) -greedy action selection 26: steps + + , option steps + + , env steps + + 27:

Original: Send action a to the environment and receive next state s (cid:48) , rewards r , and milestones b 28:
Revised: Send action a to the environment and receive next state s (cid:48) , rewards r , and milestones b 28:

Original: Store transition ( s, a, r, s (cid:48) , g ) in D c 29: s ← s (cid:48) 30: if steps
Revised: Store transition ( s, a, r, s (cid:48) , g ) in D c 29: s ← s (cid:48) 30: if steps

Original: % update freq = 0 then 31:
Revised: % update freq = 0 then 31:

Original: Sample batch of transitions from D c and use it to update θ via Q-learning loss 32: end if 33: if steps % meta update freq = 

Revised: Sample batch of transitions from D c and use it to update θ via Q-learning loss 32: end if 33: if steps % meta update freq =

Original: then 34:
Revised: then 34:

Original: Sample batch of transitions from D m and use it to update Θ via Q-learning loss 35: end if 36: if steps % margin update freq = 0 then 37:
Revised: Sample batch of transitions from D m and use it to update Θ via Q-learning loss 35: end if 36: if steps % margin update freq = 0 then 37:

Original: Sample disjoint sets of positive examples from D + g and update negative ﬁltering 38: margin 39: end if 40: if steps % aff update freq = 

Revised: Sample disjoint sets of positive examples from D + g and update negative ﬁltering 38: margin 39: end if 40: if steps % aff update freq =

Original: then 41:
Revised: then 41:

Original: Sample positive and negative examples from D + g and D − g respectively to update φ 42: with binary cross entropy loss  43:  end if 44: if steps % rep update freq = 0 then 45:{+
+}
Revised: Sample positive and negative examples from D + g and D − g respectively to update φ 42: with binary cross entropy loss (ignoring any negatives with computed distance 43: scores less than false negative ﬁltering margin) 44: end if 45: if steps % rep update freq = {+
+}

Original:  Sample anchor, positive, and negative states from D c to update ψ via contrastive loss46: end if 47: end while 48: end while 49: 
Revised: then 46: Sample anchor, positive, and negative states from D c to update ψ via contrastive loss 47: end if 48: end while 49: end while 50:

Original: E I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original:  Wecollect use 4 parallel environments for data collection with all methods.
Revised: All experiments are run with 5 random seeds each. We use 4 parallel environments for data collection with all methods.

Original: We do not include Noisy Nets or C51 in our implementation of Rainbow, as we do not ﬁnd them to improve performance in our domains and signiﬁcantly increase training time.
Revised: We do not include Noisy Nets or C51 in our implementation of Rainbow, as we do not ﬁnd them to improve performance in our domains and signiﬁcantly increase training time.

Original: Hyperparameters are shown in Table 2.
Revised: Hyperparameters are shown in Table 2.

Original: All  frequency parameters are  with respect to total environment steps (across parallel environments).
Revised: All update frequency parameters are computed with respect to total environment steps (across parallel environments).

Revised: Relative to the controller’s Q-learning updates, all other algorithmic mechanisms are updated less frequently for the sake of efﬁciency, at rates which do not appear to affect the performance of our method.

Original: All experiments are run with 5 random seeds each.
Revised: The meta-controller, affordance classiﬁer, and the context representation are all trained every 10 controller updates. Since the optimal separation margin between false negatives and true negatives changes slowly over time [Table] and computing a new margin is expensive, each classiﬁer head’s margin is updated every 150 controller updates.

Original: [Table]
Revised: [Table]