Original: P OSSIBILITY B EFORE U TILITY : L EARNING A ND U SING H IERARCHICAL A FFORDANCES
Revised: P OSSIBILITY B EFORE U TILITY : L EARNING A ND U SING H IERARCHICAL A FFORDANCES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures.
Revised: Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures.

Original: Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the ﬁrst place.
Revised: Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the ﬁrst place.

Original: By focusing only on what is feasible, or “afforded”, at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters .
Revised: By focusing only on what is feasible, or “afforded”, at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters.

Original: To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning.
Revised: To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning.

Original: Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our deﬁnition of hierarchical affordances in the present state , our approach is more ﬂexible than the multitude of approaches that ground their subtask dependencies in a symbolic history .
Revised: Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our deﬁnition of hierarchical affordances in the present state , our approach is more ﬂexible than the multitude of approaches that ground their subtask dependencies in a symbolic history .

Original: While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic speciﬁcations.
Revised: While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic speciﬁcations.

Original: Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efﬁciently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision—all of which are hallmarks of human learning.
Revised: Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efﬁciently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision—all of which are hallmarks of human learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) methods have recently achieved success in a variety of historically difﬁcult domains (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), but they continue to struggle on complex hierarchical tasks.
Revised: Reinforcement learning (RL) methods have recently achieved success in a variety of historically difﬁcult domains (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), but they continue to struggle on complex hierarchical tasks.

Original: Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do.
Revised: Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do.

Original: In other words, they do not waste time attempting the impossible.
Revised: In other words, they do not waste time attempting the impossible.

Original: Gibson (1977) coins the term “affordances” to articulate the observation that humans and other animals largely interpret the world around them in terms of which behaviors the environment affords them.
Revised: Gibson (1977) coins the term “affordances” to articulate the observation that humans and other animals largely interpret the world around them in terms of which behaviors the environment affords them.

Original: While some previous works apply the concept of affordances to the RL setting, none of these methods easily translate to environments with hierarchical tasks.
Revised: While some previous works apply the concept of affordances to the RL setting, none of these methods easily translate to environments with hierarchical tasks.

Original: In this work, we introduce Hierarchical Affordance Learning (HAL), a method that addresses the challenges inherent to learning affordances over high-level subtasks, enabling more efﬁcient learning in environments with complex subtask dependency structures.
Revised: In this work, we introduce Hierarchical Affordance Learning (HAL), a method that addresses the challenges inherent to learning affordances over high-level subtasks, enabling more efﬁcient learning in environments with complex subtask dependency structures.

Original: Many real-world environments have an underlying hierarchical dependency structure (Fig. 1a), and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them.
Revised: Many real-world environments have an underlying hierarchical dependency structure (Fig. 1a), and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them.

Original: Consider the task of preparing a simple pasta dish.
Revised: Consider the task of preparing a simple pasta dish.

Original: Some sets of subtasks, like chopping vegetables or ﬁlling a pot with water, can be successfully performed in any order.
Revised: Some sets of subtasks, like chopping vegetables or ﬁlling a pot with water, can be successfully performed in any order.

Original: However, there are many cases in which the dependencies between subtasks must be obeyed.
Revised: However, there are many cases in which the dependencies between subtasks must be obeyed.

Original: For instance, it is inadvisable chop vegetables after having mixed them with the sauce, or to boil a pot of water before the pot is ﬁlled with water in the ﬁrst place.
Revised: For instance, it is inadvisable chop vegetables after having mixed them with the sauce, or to boil a pot of water before the pot is ﬁlled with water in the ﬁrst place.

Original: Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks, hierarchical reinforcement learning (HRL) methods are well-suited for tasks with complex high-level dependencies.
Revised: Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks, hierarchical reinforcement learning (HRL) methods are well-suited for tasks with complex high-level dependencies.

Revised: [Figure]

Original: Existing HRL methods fall along a spectrum ranging from ﬂexible approaches that discover useful subtasks automatically, to the structured approaches that provide some prior information about subtasks and their interdependencies.
Revised: Existing HRL methods fall along a spectrum ranging from ﬂexible approaches that discover useful subtasks automatically, to the structured approaches that provide some prior information about subtasks and their interdependencies.

Original: The former  approaches (e.g. Vezhnevets et al., 2017;Eysenbach
Revised: The former set of approaches (e.g. Vezhnevets et al., 2017;

Original: [Figure] et al., 2018) have seen limited success, as the automatic identiﬁcation of hierarchical abstractions is an open problem in deep learning (Hinton, 2021).
Revised: Eysenbach et al., 2018) have seen limited success, as the automatic identiﬁcation of hierarchical abstractions is an open problem in deep learning (Hinton, 2021).

Original: But approaches that endow the agent with more structure, to make complex tasks feasible, do so at the cost of rigid assumptions.
Revised: But approaches that endow the agent with more structure, to make complex tasks feasible, do so at the cost of rigid assumptions.

Original: Methods that usea form of ﬁnite automata (Fig. 1b) to express subtask dependencies (e.g. Icarte et al., 2020) require the set of symbols, or atomic propositions , provided to the agent to be complete, in that the history of symbols maps deterministically to the current context (i.e. how much progress has been made; which subtasks are available).
Revised: Methods that use ﬁnite automatas (Fig. 1b) to express subtask dependencies (e.g. Icarte et al., 2020) require the set of symbols, or atomic propositions , provided to the agent to be complete, in that the history of symbols maps deterministically to the current context (i.e. how much progress has been made; which subtasks are available).

Original: Importantly, these methods and many others (e.g. Andreas et al., 2017; Sohn et al., 2020) consider subtasks to be dependent merely on the completion of others.
Revised: Importantly, these methods and many others (e.g. Andreas et al., 2017; Sohn et al., 2020) consider subtasks to be dependent merely on the completion of others.

Original: Unfortunately, these assumptions do not hold in the real world (Fig. 1c).
Revised: Unfortunately, these assumptions do not hold in the real world (Fig. 1c).

Original: For instance, if one completes the subtask cook noodles , but they clumsily spill them all over the ﬂoor, are they now ready for the next subtask, mix noodles and sauce ?
Revised: For instance, if one completes the subtask cook noodles , but they clumsily spill them all over the ﬂoor, are they now ready for the next subtask, mix noodles and sauce ?

Original: While the subtask cook noodles is somehow necessary for this further subtask, it is not sufﬁcient to have completed it in the past.
Revised: While the subtask cook noodles is somehow necessary for this further subtask, it is not sufﬁcient to have completed it in the past.

Original: The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone.
Revised: The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone.

Original: This is possible, but extraordinarily restrictive, since, unless the set of symbols is complete, none of the subtask completion information can be used to reliably learn and utilize subtask dependencies.
Revised: This is possible, but extraordinarily restrictive, since, unless the set of symbols is complete, none of the subtask completion information can be used to reliably learn and utilize subtask dependencies.

Original: Modeling probabilistic transitions allows the symbolic signal to be incomplete, but still requires a complete set of symbols, in addition to predeﬁned contexts.
Revised: Modeling probabilistic transitions allows the symbolic signal to be incomplete, but still requires a complete set of symbols, in addition to predeﬁned contexts.

Original: In order to make use of incomplete symbolic information, our approach instead learns a representation of context grounded in the present state to determine which subtasks are possible (Fig. 1d), rather than solely relying on symbols .
Revised: In order to make use of incomplete symbolic information, our approach instead learns a representation of context grounded in the present state to determine which subtasks are possible (Fig. 1d), rather than solely relying on symbols .

Original: The contributions of this paper are as follows.
Revised: The contributions of this paper are as follows.

Original: First we introduce milestones (§4), which serve the dual purpose of subgoals for training options (Sutton et al., 1999) and as high-level intents (Kulkarni et al., 2016) for training our affordance model.
Revised: First we introduce milestones (§4), which serve the dual purpose of subgoals for training options (Sutton et al., 1999) and as high-level intents (Kulkarni et al., 2016) for training our affordance model.

Original: Milestones are a ﬂexible alternative to atomic propositions used in automata-based approaches, and they are easier to specify due to less rigid assumptions.
Revised: Milestones are a ﬂexible alternative to atomic propositions used in automata-based approaches, and they are easier to specify due to less rigid assumptions.

Original: Unlike a dense reward function, the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives.
Revised: Unlike a dense reward function, the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives.

Original: Next, we introduce hierarchical affordances , which can be deﬁned over any arbitrary set of milestones, and describe HAL (§6), a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks.
Revised: Next, we introduce hierarchical affordances , which can be deﬁned over any arbitrary set of milestones, and describe HAL (§6), a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks.

Original: Finally, we demonstrate HAL’s superior performance on two complex hierarchical tasks in terms of learning speed, robustness, generalizability, and ability to explore complex subtask hierarchies without extrinsic supervision, relative to baselines provided with the same information (§7.3).
Revised: Finally, we demonstrate HAL’s superior performance on two complex hierarchical tasks in terms of learning speed, robustness, generalizability, and ability to explore complex subtask hierarchies without extrinsic supervision, relative to baselines provided with the same information (§7.3).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution (Andreas et al., 2017;
Revised: Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution (Andreas et al., 2017;

Original: Shiarlis et al., 2018; Devin et al., 2019; Sohn et al., 2020;
Revised: Shiarlis et al., 2018; Devin et al., 2019; Sohn et al., 2020;

Original: Lu et al., 2021).
Revised: Lu et al., 2021).

Original: Sohn et al.
Revised: Sohn et al.

Original: (2020) learn subtask preconditions, but use symbol-based contexts and do not learn and use their model of preconditions concurrently.
Revised: (2020) learn subtask preconditions, but use symbol-based contexts and do not learn and use their model of preconditions concurrently.

Original: Instead they assume a naive policy can sufﬁciently reach all subtasks.
Revised: Instead they assume a naive policy can sufﬁciently reach all subtasks.

Original: Furthermore, they assume ground-truth affordances are provided at each step.
Revised: Furthermore, they assume ground-truth affordances are provided at each step.

Original: Some works provide the agent with high-level task sketches (Andreas et al., 2017; Shiarlis et al., 2018) describing the order in which subtasks must be completed.
Revised: Some works provide the agent with high-level task sketches (Andreas et al., 2017; Shiarlis et al., 2018) describing the order in which subtasks must be completed.

Original: While these sketches are advertised as “ungrounded” Andreas et al.
Revised: While these sketches are advertised as “ungrounded” Andreas et al.

Original: , they are in fact grounded by the inclusion of short sketches, which are the ﬁrst to be introduced to the agent in a curriculum learning scheme (Bengio et al., 2009).
Revised: , they are in fact grounded by the inclusion of short sketches, which are the ﬁrst to be introduced to the agent in a curriculum learning scheme (Bengio et al., 2009).

Original: Our approach instead uses a direct signal, which alone need not determine task progress, and can learn without exposure to other tasks with shared structure.
Revised: Our approach instead uses a direct signal, which alone need not determine task progress, and can learn without exposure to other tasks with shared structure.

Original: In using a set of discrete symbols to indicate subtask completion, our work is similar to the variety of approaches that apply temporal logic (TL) to the RL setting (Yuan et al., 2019; Hasanbeig et al., 2018; 2020; Li et al., 2017; 2018).
Revised: In using a set of discrete symbols to indicate subtask completion, our work is similar to the variety of approaches that apply temporal logic (TL) to the RL setting (Yuan et al., 2019; Hasanbeig et al., 2018; 2020; Li et al., 2017; 2018).

Original: These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step.
Revised: These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step.

Original: Some works use reward shaping to encourage satisfaction of the formula (Li et al., 2017; 2018), whereas others convert the TL formula to some ﬁnite state machine, which provides the agent with a structure that roughly expresses subtask dependencies (Hasanbeig et al., 2018; 2020; Yuan et al., 2019).
Revised: Some works use reward shaping to encourage satisfaction of the formula (Li et al., 2017; 2018), whereas others convert the TL formula to some ﬁnite state machine, which provides the agent with a structure that roughly expresses subtask dependencies (Hasanbeig et al., 2018; 2020; Yuan et al., 2019).

Original: Icarte et al.
Revised: Icarte et al.

Original: (2020) bypass this formula-to-automata conversion, and instead directly provide the automata to the agent in the form of a reward machine (RM).
Revised: (2020) bypass this formula-to-automata conversion, and instead directly provide the automata to the agent in the form of a reward machine (RM).

Original: While RMs are more expressive than LTL formulas, they are less ﬂexible than HAL, which can deal with incomplete sets of symbols, as well as context stochasticity.
Revised: While RMs are more expressive than LTL formulas, they are less ﬂexible than HAL, which can deal with incomplete sets of symbols, as well as context stochasticity.

Original: Gibson (1977) introduces a theory of affordances , deﬁned roughly as properties of the environment which must be measured relative to the agent.
Revised: Gibson (1977) introduces a theory of affordances , deﬁned roughly as properties of the environment which must be measured relative to the agent.

Original: Heft (1989) and Chemero (2003) clarify affordances as relations between the agent and its environment.
Revised: Heft (1989) and Chemero (2003) clarify affordances as relations between the agent and its environment.

Original: Khetarpal et al.
Revised: Khetarpal et al.

Original: (2020) formalize this relational deﬁnition of affordances in the context of RL, and model which low-level actions, given corresponding intents , are afforded in each state.
Revised: (2020) formalize this relational deﬁnition of affordances in the context of RL, and model which low-level actions, given corresponding intents , are afforded in each state.

Original: In this work, milestones represent high-level intents corresponding to each subtask.
Revised: In this work, milestones represent high-level intents corresponding to each subtask.

Original: They also demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions, and allows for the learning of more accurate and generalizable partial world models.
Revised: They also demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions, and allows for the learning of more accurate and generalizable partial world models.

Original: This approach does not directly translate to the hierarchical setting because subtasks, unlike actions, may fail for reasons other than affordances, meaning we do not have access to ground-truth affordance labels with which to train our model.
Revised: This approach does not directly translate to the hierarchical setting because subtasks, unlike actions, may fail for reasons other than affordances, meaning we do not have access to ground-truth affordance labels with which to train our model.

Original: Manoury et al.
Revised: Manoury et al.

Original: (2019) and Khazatsky et al.
Revised: (2019) and Khazatsky et al.

Original: (2021) present approaches that can discover and use affordances to learn new skills, but their deﬁnition of affordances (i.e. a behavior is either afforded or not, with no notion of preconditions) does not translate to the hierarchical setting.
Revised: (2021) present approaches that can discover and use affordances to learn new skills, but their deﬁnition of affordances (i.e. a behavior is either afforded or not, with no notion of preconditions) does not translate to the hierarchical setting.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Our setting involves learning behavior policies in Markov Decision Processes (MDP) using RL.
Revised: Our setting involves learning behavior policies in Markov Decision Processes (MDP) using RL.

Original: An
Revised: An

Original: MDP is deﬁned by the state space S , action space A , reward function R : S ×
Revised: MDP is deﬁned by the state space S , action space A , reward function R : S ×

Original: A → R , and state transition distribution P :
Revised: A → R , and state transition distribution P :

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: → (cid:52) ( S ) .
Revised: (cid:52) ( S ) .

Original: The objective is to learn a policy, π : S → (cid:52) ( A ) , which selects actions that maximize expected future returns: G ( π
Revised: The objective is to learn a policy, π : S → (cid:52) ( A ) , which selects actions that maximize expected future returns: G ( π

Original: [Equation]
Revised: [Equation]

Original: ∼ P ] , where γ is the discount factor.
Revised: ∼ P ] , where γ is the discount factor.

Original: Sutton et al.
Revised: Sutton et al.

Original: (1999) introduce the options framework , which ﬂexibly models hierarchical abstractions with minimal modiﬁcation to the RL paradigm.
Revised: (1999) introduce the options framework , which ﬂexibly models hierarchical abstractions with minimal modiﬁcation to the RL paradigm.

Original: Each option , o :=
Revised: Each option , o :=

Original: (cid:104)I o , π o , β o (cid:105) , is deﬁned by an initiation set, I o ⊆ S , indicating where the option can be selected, the corresponding option policy, π o , and the termination condition, β o : S + → [0 , 1] , indicating the probability of termination in each state.
Revised: (cid:104)I o , π o , β o (cid:105) , is deﬁned by an initiation set, I o ⊆ S , indicating where the option can be selected, the corresponding option policy, π o , and the termination condition, β o : S + → [0 , 1] , indicating the probability of termination in each state.

Original: Options turn our typical MDP into a semi-Markov decision process (SMDP) since the state transition distribution is, in general, no longer dependent on the current state and action, but also the present option, which was decided in a previous time-step.
Revised: Options turn our typical MDP into a semi-Markov decision process (SMDP) since the state transition distribution is, in general, no longer dependent on the current state and action, but also the present option, which was decided in a previous time-step.

Original: The design of this framework allows options to be treated similarly to actions, except they may be executed across multiple time-steps, interrupted, composed, and learned as separate subpolicies.
Revised: The design of this framework allows options to be treated similarly to actions, except they may be executed across multiple time-steps, interrupted, composed, and learned as separate subpolicies.

Original:  M ILESTONES A ND H IERARCHICAL A FFORDANCES
Revised:  M ILESTONES A ND H IERARCHICAL A FFORDANCES

Original: We consider tasks that can be decomposed into subtasks , each represented by a milestone symbol, g ∈ G , where G is the set of symbols relevant to the task, and |G| =
Revised: We consider tasks that can be decomposed into subtasks , each represented by a milestone symbol, g ∈ G , where G is the set of symbols relevant to the task, and |G| =

Original: K .
Revised: K .

Original: For each subtask, we introduce a separate option, (cid:104)I g , π g , β g (cid:105) , and we call π g a subpolicy .
Revised: For each subtask, we introduce a separate option, (cid:104)I g , π g , β g (cid:105) , and we call π g a subpolicy .

Original: At each time-step, in addition to the extrinsic reward signal provided by the environment to indicate success on the overall task, we have access to a milestone signal , which is a vector b t where each element b t g ∈ { 0 , 1 } indicates whether g ∈ G was completed on time-step t .
Revised: At each time-step, in addition to the extrinsic reward signal provided by the environment to indicate success on the overall task, we have access to a milestone signal , which is a vector b t where each element b t g ∈ { 0 , 1 } indicates whether g ∈ G was completed on time-step t .

Original: In our PASTA example, we might receive a milestone each time we cut a vegetable, make the sauce, cook the noodles, etc.
Revised: In our PASTA example, we might receive a milestone each time we cut a vegetable, make the sauce, cook the noodles, etc.

Original: Milestones serve two main purposes.
Revised: Milestones serve two main purposes.

Original: Firstly, milestones function as option subgoals (Sutton et al., 1999) that are in this work used to train each subpolicy (discussed in Section 5).
Revised: Firstly, milestones function as option subgoals (Sutton et al., 1999) that are in this work used to train each subpolicy (discussed in Section 5).

Original: Secondly, each milestone represents the intent of its corresponding subtask—similar to the action intents introduced to learn action-level affordances in the work of Khetarpal et al.
Revised: Secondly, each milestone represents the intent of its corresponding subtask—similar to the action intents introduced to learn action-level affordances in the work of Khetarpal et al.

Original: (2020)—which we use to learn hierarchical affordances (discussed in
Revised: (2020)—which we use to learn hierarchical affordances (discussed in

Original: Section 6).
Revised: Section 6).

Original: In contrast to the standard options framework, primitive actions can only be executed as part of an option’s subpolicy in our method.
Revised: In contrast to the standard options framework, primitive actions can only be executed as part of an option’s subpolicy in our method.

Original: Generally, policies trained solely over options have no guarantee of optimality (Sutton et al., 1999), but we ensure the existence of an optimal solution by requiring g K ∈ G , where g K is the task’s ﬁnal milestone (indicating task success).
Revised: Generally, policies trained solely over options have no guarantee of optimality (Sutton et al., 1999), but we ensure the existence of an optimal solution by requiring g K ∈ G , where g K is the task’s ﬁnal milestone (indicating task success).

Original: When G = { g k } , our setting is standard, ﬂat RL.
Revised: When G = { g k } , our setting is standard, ﬂat RL.

Original: Each additional milestone g (cid:48) added to
Revised: Each additional milestone g (cid:48) added to

Original: G is useful as an intermediate signal so long as g (cid:48) corresponds to a unique behavior necessary for achieving g K .
Revised: G is useful as an intermediate signal so long as g (cid:48) corresponds to a unique behavior necessary for achieving g K .

Original: Hierarchical affordances are deﬁned over G in the following way.
Revised: Hierarchical affordances are deﬁned over G in the following way.

Original: The vector f s = f ∗ ( s ) of size K represents which milestones are immediately achievable from the present state s , without requiring the collection of any intermediate milestones, where f is a hierarchical affordance classiﬁer , and f ∗ is the optimal one 1 .
Revised: The vector f s = f ∗ ( s ) of size K represents which milestones are immediately achievable from the present state s , without requiring the collection of any intermediate milestones, where f is a hierarchical affordance classiﬁer , and f ∗ is the optimal one 2 .

Original: Formally, f sg = 1 if at time t 0 it is possible for future b
Revised: Formally, f sg = 1 if at time t 0 it is possible for future b

Original: Tg = 1 without any b tj = 1 , j (cid:54) =
Revised: Tg = 1 without any b tj = 1 , j (cid:54) =

Original: g , where t 0 < t < T .
Revised: g , where t 0 < t < T .

Original: In PASTA , the milestone mix cooked noodles and sauce is not afforded at the beginning since cook noodles is required ﬁrst.
Revised: In PASTA , the milestone mix cooked noodles and sauce is not afforded at the beginning since cook noodles is required ﬁrst.

Original: A successful policy trained within the vanilla options framework will eventually learn to execute options in contexts where they are most useful, regardless of each option’s predeﬁned initiation set.
Revised: A successful policy trained within the vanilla options framework will eventually learn to execute options in contexts where they are most useful, regardless of each option’s predeﬁned initiation set.

Original: However, hierarchical affordances give us a principled way to directly adjust this set: for subtask g , we can set I g
Revised: However, hierarchical affordances give us a principled way to directly adjust this set: for subtask g , we can set I g

Original: = { s | s ∈ S , f
Revised: = { s | s ∈ S , f

Original: sg = 1 } .
Revised: sg = 1 } .

Original: One can think of hierarchical affordances as using milestones to impose a state-grounded subtask dependency structure on top of the options framework, which we can use to prune impossible subtasks.
Revised: One can think of hierarchical affordances as using milestones to impose a state-grounded subtask dependency structure on top of the options framework, which we can use to prune impossible subtasks.

Original: If G = { g (cid:48) , g K } , an affordance-aware agent with access to optimal f ∗ ( s ) will never initiate subtask g K from the beginning if g (cid:48) is a necessary intermediate behavior.
Revised: If G = { g (cid:48) , g K } , an affordance-aware agent with access to optimal f ∗ ( s ) will never initiate subtask g K from the beginning if g (cid:48) is a necessary intermediate behavior.

Original: Some logic-based RL approaches (e.g. Yuan et al., 2019; Icarte et al., 2020) use atomic propositions as markers of subtask achievement to transition between contexts in a ﬁnite state machine.
Revised: Some logic-based RL approaches (e.g. Yuan et al., 2019; Icarte et al., 2020) use atomic propositions as markers of subtask achievement to transition between contexts in a ﬁnite state machine.

Original: These approaches, and many other HRL works (e.g. Andreas et al., 2017; Sohn et al., 2020) deﬁne subtask preconditions in terms of other subtasks .
Revised: These approaches, and many other HRL works (e.g. Andreas et al., 2017; Sohn et al., 2020) deﬁne subtask preconditions in terms of other subtasks .

Original: There are two forms of stochasticity that hierarchical affordances, by virtue of being grounded in the present state, can more naturally address than symbolically-deﬁned dependencies.
Revised: There are two forms of stochasticity that hierarchical affordances, by virtue of being grounded in the present state, can more naturally address than symbolically-deﬁned dependencies.

Original: We can conceptualize potential agent trajectories as graphs where nodes represent the attainment of milestones, and edges are the segments between them.
Revised: We can conceptualize potential agent trajectories as graphs where nodes represent the attainment of milestones, and edges are the segments between them.

Original: Node stochasticity is affordance-affecting randomness that occurs either when milestones are attained (e.g. receiving varying quantity of an item), or at the beginning of the episode (i.e. starting in different contexts).
Revised: Node stochasticity is affordance-affecting randomness that occurs either when milestones are attained (e.g. receiving varying quantity of an item), or at the beginning of the episode (i.e. starting in different contexts).

Original: Edge stochasticity is when affordances change at any time within a segment.
Revised: Edge stochasticity is when affordances change at any time within a segment.

Original: We treat edge stochasticity events as infrequent exceptions to the typical subtask dependency rules.
Revised: We treat edge stochasticity events as infrequent exceptions to the typical subtask dependency rules.

Original: For example, after cook noodles is complete, mix cooked noodles and sauce is afforded, even if the agent may eventually spill the noodles on the ﬂoor.
Revised: For example, after cook noodles is complete, mix cooked noodles and sauce is afforded, even if the agent may eventually spill the noodles on the ﬂoor.

Original: By grounding these rules in the current state, an affordance-aware agent can detect and adapt to edge anomalies.
Revised: By grounding these rules in the current state, an affordance-aware agent can detect and adapt to edge anomalies.

Original: In Section 6, we describe in detail how hierarchical affordances are learned and used in stochastic environments where symbols alone would fail to reliably determine the current context.
Revised: In Section 6, we describe in detail how hierarchical affordances are learned and used in stochastic environments where symbols alone would fail to reliably determine the current context.

Original:  L EARNING C ONTROLLERS
Revised:  L EARNING C ONTROLLERS

Original: Like h-DQN (Kulkarni et al., 2016), we use a meta-controller that selects the current subtask to attempt and a low-level controller which executes the subpolicy relevant to that subtask.
Revised: Like h-DQN (Kulkarni et al., 2016), we use a meta-controller that selects the current subtask to attempt and a low-level controller which executes the subpolicy relevant to that subtask.

Original: The controller, π : S × G → (cid:52) ( A ) , selects low-level actions, a ∈ A , given a state, s ∈ S , and milestone, g ∈ G , and aims to maximize the expected milestone signal rewards, b g .
Revised: The controller, π : S × G → (cid:52) ( A ) , selects low-level actions, a ∈ A , given a state, s ∈ S , and milestone, g ∈ G , and aims to maximize the expected milestone signal rewards, b g .

Original: Q-Learning (Watkins, 1989) trains these controllers by learning an estimate of the optimal Q-function: Q ∗ c ( s, a ; g )
Revised: Q-Learning (Watkins, 1989) trains these controllers by learning an estimate of the optimal Q-function: Q ∗ c ( s, a ; g )

Original: =
Revised: =

Original: max π E (cid:2)(cid:80) ∞ t =

Revised: max π E (cid:2)(cid:80) ∞ t =

Original: γ t b tg | s 

Revised: γ t b tg | s

Original: =
Revised: =

Original: s, a 

Revised: s, a

Original: = a, a t ∼ π, s t ∼ P (cid:3) and deriving a policy from the Q-function as such: π g ( a | s, g ) =
Revised: = a, a t ∼ π, s t ∼ P (cid:3) and deriving a policy from the Q-function as such: π g ( a | s, g ) =

Original: 1 ( a = arg max a (cid:48)
Revised: 1 ( a = arg max a (cid:48)

Original: Q c ( s, a (cid:48) ; g )) .
Revised: Q c ( s, a (cid:48) ; g )) .

Original: Deep Q-Learning (Mnih et al., 2015) estimates
Revised: Deep Q-Learning (Mnih et al., 2015) estimates

Original: Q ∗ using deep neural networks.
Revised: Q ∗ using deep neural networks.

Original: This Q-function is parameterized by θ = { θ base , ..., θ g , ... } , where θ base is a set of shared base parameters and θ g is a goal-speciﬁc head.
Revised: This Q-function is parameterized by θ = { θ base , ..., θ g , ... } , where θ base is a set of shared base parameters and θ g is a goal-speciﬁc head.

Original: It is updated via gradient descent on the following loss function, derived from the original Q-learning update:
Revised: It is updated via gradient descent on the following loss function, derived from the original Q-learning update:

Original: [Equation]
Revised: [Equation]

Original: where D c is a replay buffer that stores previously collected transitions, and ¯ θ are the parameters of a periodically updated target network.
Revised: where D c is a replay buffer that stores previously collected transitions, and ¯ θ are the parameters of a periodically updated target network.

Original: Both of these components are included to avoid the instability associated with using function approximation in Q-Learning.
Revised: Both of these components are included to avoid the instability associated with using function approximation in Q-Learning.

Original: The meta-controller Π : S → (cid:52) ( G ) aims to execute subtasks to maximize extrinsic rewards
Revised: The meta-controller Π : S → (cid:52) ( G ) aims to execute subtasks to maximize extrinsic rewards

Original: re-
Revised: re-

Original: ceived by the environment.
Revised: ceived by the environment.

Original: Again, we estimate a Q-function, this time over a dilated time scale
Revised: Again, we estimate a Q-function, this time over a dilated time scale

Original: [Figure]
Revised: [Figure]

Original: [Equation] [Equation] g,
Revised: (i.e. we allow the low-level controllers to run for multiple steps before choosing new goals):

Original: [Equation]
Revised: [Equation]

Original: g , s t ∼ P (cid:105) , where N is the (variable) number of steps the option runs for.
Revised: , where N is the (variable) number of steps the option runs for.

Original: When collecting data in the environment,  transitions ( s t , g t , (cid:80) t + N t r t , s t + N ) to a separate meta-replay buffer, D mc , used to train our meta-Qfunction (parameterized by Θ ) with a loss similar to Eq. 1, but without any goal-conditioning.
Revised: When collecting data in the environment, we add transitions ( s t , g t , (cid:80) t + N t r t , s t + N ) to a separate meta-replay buffer, D mc , used to train our meta-Qfunction (parameterized by Θ ) with a loss similar to Eq. 1, but without any goal-conditioning.

Original: In
Revised: In

Original: Section 6 we describe how hierarchical affordances are integrated into this training procedure.
Revised: Section 6 we describe how hierarchical affordances are integrated into this training procedure.

Original:  H IERARCHICAL A FFORDANCE L EARNING
Revised:  H IERARCHICAL A FFORDANCE L EARNING

Original: In typical HRL methods, if the meta-controller is yet to receive extrinsic reward from the environment, there will be no preference for selecting any subtask over the others.
Revised: In typical HRL methods, if the meta-controller is yet to receive extrinsic reward from the environment, there will be no preference for selecting any subtask over the others.

Original: However, by restricting the selection of subtasks to ones that have proven merely to be possible , an agent can avoid wasting time attempting the impossible, and reach more fruitful subtasks faster.
Revised: However, by restricting the selection of subtasks to ones that have proven merely to be possible , an agent can avoid wasting time attempting the impossible, and reach more fruitful subtasks faster.

Original: Suppose from experience, gained through random exploration, the agent achieves milestone g (where achievement means b g = 1 ) very often from the initial state set I , but never j ∈ G , despite being able to achieve j in later states.
Revised: Suppose from experience, gained through random exploration, the agent achieves milestone g (where achievement means b g = 1 ) very often from the initial state set I , but never j ∈ G , despite being able to achieve j in later states.

Original: With enough experience, the agent should become conﬁdent that j is not achievable without completing other milestones ﬁrst, and should not bother selecting j from any s ∈ I , while g , and any others that are achievable from those states, should instead be considered.
Revised: With enough experience, the agent should become conﬁdent that j is not achievable without completing other milestones ﬁrst, and should not bother selecting j from any s ∈ I , while g , and any others that are achievable from those states, should instead be considered.

Original: If we had access to an oracle function, f ∗ ( s ) , that accurately computes hierarchical affordances for our task, we could prune impossible subtasks by masking the otherwise uninformed policy with the affordance oracle output: p ( g | s )
Revised: If we had access to an oracle function, f ∗ ( s ) , that accurately computes hierarchical affordances for our task, we could prune impossible subtasks by masking the otherwise uninformed policy with the affordance oracle output: p ( g | s )

Original: ∝ f sg ∗ Π Θ ( g | s ) .
Revised: ∝ f sg ∗ Π Θ ( g | s ) .

Original: In the following sections, we describe a method that can learn an approximate f ( s )
Revised: In the following sections, we describe a method that can learn an approximate f ( s )

Original: ≈
Revised: ≈

Original: f ∗ ( s ) from experience and leverage it in real-time for more effective learning.
Revised: f ∗ ( s ) from experience and leverage it in real-time for more effective learning.

Original: The false negative problem Recall that f ( s ) outputs a vector f s , where each f sg indicates the possibility of collecting milestone g from state s without requiring intermediate milestones.
Revised: The false negative problem Recall that f ( s ) outputs a vector f s , where each f sg indicates the possibility of collecting milestone g from state s without requiring intermediate milestones.

Original: To train each binary classiﬁcation head, f g ( s ) , we must somehow generate labeled data for each milestone.
Revised: To train each binary classiﬁcation head, f g ( s ) , we must somehow generate labeled data for each milestone.

Original: Suppose an option was initialized at time t o , and in time T milestone g is received.
Revised: Suppose an option was initialized at time t o , and in time T milestone g is received.

Original: If no others were received since the start of the option, we may assume that for any t where t o ≤ t ≤ T , the collection of milestone g was afforded, so we can use the set of states { s t o , s t o +1 , . . .
Revised: If no others were received since the start of the option, we may assume that for any t where t o ≤ t ≤ T , the collection of milestone g was afforded, so we can use the set of states { s t o , s t o +1 , . . .

Original: , s T } as positive (i.e. f ∗ g ( s ) = 1 ) training examples for the affordance classiﬁer.
Revised: , s T } as positive (i.e. f ∗ g ( s ) = 1 ) training examples for the affordance classiﬁer.

Original: Even if g was not the intended milestone, we can still generate positive training examples for f g in this way.
Revised: Even if g was not the intended milestone, we can still generate positive training examples for f g in this way.

Original: If an option has failed to collect intended milestone g , either through timing out or collecting an unintended milestone, we might be tempted to use the states encountered during that option as negative examples (i.e. f ∗ g ( s ) = 0 ).
Revised: If an option has failed to collect intended milestone g , either through timing out or collecting an unintended milestone, we might be tempted to use the states encountered during that option as negative examples (i.e. f ∗ g ( s ) = 0 ).

Original: However, this occurrence can either be indicative of the states not affording g , or that the subpolicy corresponding to g is sub-optimal and has failed despite g being afforded.
Revised: However, this occurrence can either be indicative of the states not affording g , or that the subpolicy corresponding to g is sub-optimal and has failed despite g being afforded.

Original: These false negatives are a problem for any approach requiring function approximation via neural networks, which are generally not robust to label noise (Song et al., 2020).
Revised: These false negatives are a problem for any approach requiring function approximation via neural networks, which are generally not robust to label noise (Song et al., 2020).

Original: It is particularly troublesome in our case since the noise is greater than the true signal when the subpolicies are under-trained.
Revised: It is particularly troublesome in our case since the noise is greater than the true signal when the subpolicies are under-trained.

Original: Context learning Suppose we had access to an abstract state representation
Revised: Context learning Suppose we had access to an abstract state representation

Original: z s aﬀ = z ∗ aﬀ ( s ) , where any states s i and s j are mapped to the same value only when f ∗ ( s i ) =
Revised: z s aﬀ = z ∗ aﬀ ( s ) , where any states s i and s j are mapped to the same value only when f ∗ ( s i ) =

Original: f ∗ ( s j ) .
Revised: f ∗ ( s j ) .

Original: With a representation that could cluster states in this way, we could trivially determine the falsity of a collected negative s ∈ D − g by checking if z ∗ aﬀ ( s ) =
Revised: With a representation that could cluster states in this way, we could trivially determine the falsity of a collected negative s ∈ D − g by checking if z ∗ aﬀ ( s ) =

Original: z ∗ aﬀ ( s j ) for any s j ∈ D + g , that is, if we have encountered a true positive with the same representation.
Revised: z ∗ aﬀ ( s j ) for any s j ∈ D + g , that is, if we have encountered a true positive with the same representation.

Original: The classiﬁcation procedure could be interpreted as “labeling” these contexts with affordance values.
Revised: The classiﬁcation procedure could be interpreted as “labeling” these contexts with affordance values.

Original: This is somewhat of a “chicken and egg” problem, since to learn affordances, we require a representation that maps states to contexts with the same affordance values, which clearly requires some prior knowledge about affordances.
Revised: This is somewhat of a “chicken and egg” problem, since to learn affordances, we require a representation that maps states to contexts with the same affordance values, which clearly requires some prior knowledge about affordances.

Original: Fortunately, from Section 4, we know that affordances will only change when either (1) a milestone is collected and (2) when edge stochasticity occurs.
Revised: Fortunately, from Section 4, we know that affordances will only change when either (1) a milestone is collected and (2) when edge stochasticity occurs.

Original: Since (2) is by deﬁnition a rare occurrence, states s t and s t +1 are more likely than not to satisfy f ∗ ( s t ) =
Revised: Since (2) is by deﬁnition a rare occurrence, states s t and s t +1 are more likely than not to satisfy f ∗ ( s t ) =

Original: f ∗ ( s t +1 ) , so long as they exist in the same segment between milestones.
Revised: f ∗ ( s t +1 ) , so long as they exist in the same segment between milestones.

Original: In this case, we can say that s t and s t +1 share the same achievement context , z s =
Revised: In this case, we can say that s t and s t +1 share the same achievement context , z s =

Original: z ( s ) .
Revised: z ( s ) .

Original: Let z ψ ( s ) be an achievement context embedding represented by a differentiable function parameterized by ψ .
Revised: Let z ψ ( s ) be an achievement context embedding represented by a differentiable function parameterized by ψ .

Original: We can train z ψ ( s ) from experience using the following contrastive loss:
Revised: We can train z ψ ( s ) from experience using the following contrastive loss:

Original: [Equation]
Revised: [Equation]

Original: where each s aj is a randomly chosen anchor , each s pj is a positive 2 example chosen within the segment according to a (truncated) normal distribution, N T ( [Equation] 2 ) , centered around (and excluding) s aj , s nj is chosen randomly among other segments and is treated as a negative example, and α is an arbitrary margin value.
Revised: where each s aj is a randomly chosen anchor , each s pj is a positive 3 example chosen within the segment according to a (truncated) normal distribution, N T (0 , σ 2 ) , centered around (and excluding) s aj , s nj is chosen randomly among other segments and is treated as a negative example, and α is an arbitrary margin value.

Original: This loss pushes representations of states from the same achievement context together, and pulls representations of states from different contexts apart.
Revised: This loss pushes representations of states from the same achievement context together, and pulls representations of states from different contexts apart.

Original: We show in Appendix B that a wide range of σ produce useful representations.
Revised: We show in Appendix B that a wide range of σ produce useful representations.

Original: For our edge stochasticity experiments (Figure 5) we use a low σ = 2 . 0 to reduce the risk of sampling across affordance changes.
Revised: For our edge stochasticity experiments (Figure 5) we use a low σ = 2 . 0 to reduce the risk of sampling across affordance changes.

Original: False negative ﬁltering In the learned representation space, we expect false negative points to be closer to positive points than true negatives.
Revised: False negative ﬁltering In the learned representation space, we expect false negative points to be closer to positive points than true negatives.

Original: Given a negatively-labeled state s q for classiﬁer head f g , we compute 3 the mean distance from z ψ ( s q ) to the representations of the k closest positive points in a population uniformly sampled 4 from D + g , denoted d kq .
Revised: Given a negatively-labeled state s q for classiﬁer head f g , we compute 4 the mean distance from z ψ ( s q ) to the representations of the k closest positive points in a population uniformly sampled 5 from D + g , denoted d kq .

Original: We expect d kq to be large for true negatives and small for false ones, and we can determine an effective separating margin in the following way.
Revised: We expect d kq to be large for true negatives and small for false ones, and we can determine an effective separating margin in the following way.

Original: First we compute distance scores for a random sample of positive points, denoted { d kp } , to use as reference.
Revised: First we compute distance scores for a random sample of positive points, denoted { d kp } , to use as reference.

Original: We ensure these points come from segments that are disjoint from the population points’ segments to avoid trivially low scores that might skew the distribution.
Revised: We ensure these points come from segments that are disjoint from the population points’ segments to avoid trivially low scores that might skew the distribution.

Original: We then ﬁt a Gaussian distribution to { d kp } and compute an upper conﬁdence bound ρ for a given percentile value and conﬁdence level.
Revised: We then ﬁt a Gaussian distribution to { d kp } and compute an upper conﬁdence bound ρ for a given percentile value and conﬁdence level.

Original: Any d kq < ρ is very similar to positive points in the representation space, so we count s q as a false negative and exclude it from our training set.
Revised: Any d kq < ρ is very similar to positive points in the representation space, so we count s q as a false negative and exclude it from our training set.

Original: Note, we do not train head f g (and therefore do not reliably prune) until we have access to both positives and negatives for g .
Revised: Note, we do not train head f g (and therefore do not reliably prune) until we have access to both positives and negatives for g .

Original: Method overview The HAL architecture consists of a bi-level policy like h-DQN, a context embedding network, and an affordance classiﬁer (see Figure 2), which are all learned concurrently (full algorithm in Appendix E).
Revised: Method overview The HAL architecture consists of a bi-level policy like h-DQN, a context embedding network, and an affordance classiﬁer (see Figure 2), which are all learned concurrently (full algorithm in Appendix E).

Original: Intuitively, the affordance classiﬁer is able to generalize to a novel state, s , by ﬁrst identifying the abstract achievement context, z ψ ( s ) , associated with the state, and then outputting an affordance value based on previous experience in that context.
Revised: Intuitively, the affordance classiﬁer is able to generalize to a novel state, s , by ﬁrst identifying the abstract achievement context, z ψ ( s ) , associated with the state, and then outputting an affordance value based on previous experience in that context.

Original: If z ψ ( s ) has also not been encountered, that context will not be strongly “labeled” either way, so we will not be invariably pruning it.
Revised: If z ψ ( s ) has also not been encountered, that context will not be strongly “labeled” either way, so we will not be invariably pruning it.

Original: The meta-controller selects a subtask g at the beginning of the episode, and selects a new subtask g (cid:48) after collecting any milestone or whenever an option times out after a predeﬁned number of steps (our β g ).
Revised: The meta-controller selects a subtask g at the beginning of the episode, and selects a new subtask g (cid:48) after collecting any milestone or whenever an option times out after a predeﬁned number of steps (our β g ).

Original: At each step, the current state is fed to the controller, which outputs an action conditioned on the most recently selected subtask.
Revised: At each step, the current state is fed to the controller, which outputs an action conditioned on the most recently selected subtask.

Original: After discretizing the classiﬁer’s output to a binary mask, we perform an affordance aware version of (cid:15) -greedy as follows.
Revised: After discretizing the classiﬁer’s output to a binary mask, we perform an affordance aware version of (cid:15) -greedy as follows.

Original: Given parameters (cid:15) aff and (cid:15) mc , we select a random subtask within the mask with probability (cid:15) aff , randomly across all subtasks with probability (cid:15) mc , and otherwise select greedily with respect to meta-Q within the mask.
Revised: Given parameters (cid:15) aff and (cid:15) mc , we select a random subtask within the mask with probability (cid:15) aff , randomly across all subtasks with probability (cid:15) mc , and otherwise select greedily with respect to meta-Q within the mask.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we aim to answer the following questions: (1) Does HAL improve learning in tasks with complex dependencies?
Revised: In our experiments we aim to answer the following questions: (1) Does HAL improve learning in tasks with complex dependencies?

Original: (2) Is HAL robust to milestone selection and context stochasticity?
Revised: (2) Is HAL robust to milestone selection and context stochasticity?

Original: (3) Can HAL more effectively learn a diverse set of skills when trained task-agnostically?
Revised: (3) Can HAL more effectively learn a diverse set of skills when trained task-agnostically?

Original: 7. E NVIRONMENTS
Revised: 7. E NVIRONMENTS

Original: We evaluate our method, along with several baselines, on two complex environments with intricate subtask dependency structures: CRAFTING and TREASURE .
Revised: We evaluate our method, along with several baselines, on two complex environments with intricate subtask dependency structures: CRAFTING and TREASURE .

Original: Both environments (visualized in Figure 3) are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).
Revised: Both environments (visualized in Figure 3) are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).

Original: Agents receive an egocentric image of the environment, as well as a vector describing their inventory (items picked up from the environment and currently in their possession) as observations.
Revised: Agents receive an egocentric image of the environment, as well as a vector describing their inventory (items picked up from the environment and currently in their possession) as observations.

Original: The action spaces are discrete and include actions for turning left/right, and moving forward/backward, as well as environment speciﬁc actions detailed below.
Revised: The action spaces are discrete and include actions for turning left/right, and moving forward/backward, as well as environment speciﬁc actions detailed below.

Original: Task hierarchies, walk-throughs of successful trajectories, and additional information about both environments are included in Appendix A.
Revised: Task hierarchies, walk-throughs of successful trajectories, and additional information about both environments are included in Appendix A.

Original: CRAFTING is based on Minecraft, a popular open-ended video game in which players collect resources from their environment and use them to craft objects which can be used to then obtain more resources.
Revised: CRAFTING is based on Minecraft, a popular open-ended video game in which players collect resources from their environment and use them to craft objects which can be used to then obtain more resources.

Original: As such, the hierarchy of possible subtasks is immensely complex and presents a signiﬁcant challenge for AI agents to reach subtasks deeper in the hierarchy.
Revised: As such, the hierarchy of possible subtasks is immensely complex and presents a signiﬁcant challenge for AI agents to reach subtasks deeper in the hierarchy.

Original: We develop an environment that replicates this hierarchical complexity without the commensurate visuomotor complexity which our method does not aim to address.
Revised: We develop an environment that replicates this hierarchical complexity without the commensurate visuomotor complexity which our method does not aim to address.

Original: In addition to movement actions, CRAFTING includes actions to mine the object immediately in front of the agent (which requires an appropriate pickaxe), as well as to craft and smelt the various objects (pickaxes, iron ingots, etc.).
Revised: In addition to movement actions, CRAFTING includes actions to mine the object immediately in front of the agent (which requires an appropriate pickaxe), as well as to craft and smelt the various objects (pickaxes, iron ingots, etc.).

Original: The full set of milestones contains items that are either craftable or collectable.
Revised: The full set of milestones contains items that are either craftable or collectable.

Original: CRAFTING naturally contains node stochasticity since the collection of certain items, due to the random procedural generation, requires slightly different milestone trajectories across episodes (e.g. mining variable amount of stone to encounter diamond).
Revised: CRAFTING naturally contains node stochasticity since the collection of certain items, due to the random procedural generation, requires slightly different milestone trajectories across episodes (e.g. mining variable amount of stone to encounter diamond).

Original: TREASURE is a navigation task that requires the agent to collect various items and use them to unlock rooms to reach further items.
Revised: TREASURE is a navigation task that requires the agent to collect various items and use them to unlock rooms to reach further items.

Original: The ultimate goal is to unlock a treasure chest, which requires a sequence of collecting several keys, as well as placing an object on a weight-based sensor, in order to open the requisite doors.
Revised: The ultimate goal is to unlock a treasure chest, which requires a sequence of collecting several keys, as well as placing an object on a weight-based sensor, in order to open the requisite doors.

Original: Agents can only carry one object at a time, so they must reason about which object to pick up based on what it will afford them (e.g. if the weight-based sensor room is locked, the weight object is not currently useful).
Revised: Agents can only carry one object at a time, so they must reason about which object to pick up based on what it will afford them (e.g. if the weight-based sensor room is locked, the weight object is not currently useful).

Original: Like CRAFTING , TREASURE contains node stochasticity due to the procedural generation.
Revised: Like CRAFTING , TREASURE contains node stochasticity due to the procedural generation.

Original: For example, the central room that the agent is spawned in can contain either of the red or yellow key individually, or both together.
Revised: For example, the central room that the agent is spawned in can contain either of the red or yellow key individually, or both together.

Original: Unlike CRAFTING , which has a large action space to accommodate the various crafting recipes, this environment only contains actions to move and a single “interaction” action that is used to pick up keys, open doors, etc.
Revised: Unlike CRAFTING , which has a large action space to accommodate the various crafting recipes, this environment only contains actions to move and a single “interaction” action that is used to pick up keys, open doors, etc.

Original: While
Revised: While

Original: CRAFTING has a more complex hierarchy and greater diversity in the potential ordering of subtasks,
Revised: CRAFTING has a more complex hierarchy and greater diversity in the potential ordering of subtasks,

Original: TREASURE has on average more difﬁcult subtasks.
Revised: TREASURE has on average more difﬁcult subtasks.

Original: The full set of milestones contains each object the agent can successfully interact with in the environment (e.g. opening door, collecting key).
Revised: The full set of milestones contains each object the agent can successfully interact with in the environment (e.g. opening door, collecting key).

Original: 7. B ASELINES
Revised: 7. B ASELINES

Original: [Table]
Revised: [Table]

Original: Our set of baselines is summarized in Table 1.
Revised: Our set of baselines is summarized in Table 1.

Original: All methods are based on the Rainbow (Hessel et al., 2018)
Revised: All methods are based on the Rainbow (Hessel et al., 2018)

Original: Deep Q-Learning algorithm, which combines several improvements to vanilla DQNs (Mnih et al., 2015).
Revised: Deep Q-Learning algorithm, which combines several improvements to vanilla DQNs (Mnih et al., 2015).

Original: To compensate for the lack of milestone signals, non-hierarchical methods use a dense reward function that incorporates milestone signals for the ﬁrst time each milestone is obtained in the episode.
Revised: To compensate for the lack of milestone signals, non-hierarchical methods use a dense reward function that incorporates milestone signals for the ﬁrst time each milestone is obtained in the episode.

Original: To evaluate the efﬁcacy of our affordance classiﬁer
Revised: To evaluate the efﬁcacy of our affordance classiﬁer

Original: [Figure]
Revised: [Figure]

Original: online learning procedure, we compare our method to an “oracle” that differs only by using groundtruth affordances for masking subtasks.
Revised: online learning procedure, we compare our method to an “oracle” that differs only by using groundtruth affordances for masking subtasks.

Original: The node stochasticity inherent to both environments, as well as the edge stochasticity later explored, preclude the use of any methods that reason solely over symbols (i.e. automata-, sketch-, or subtask dependency-based approaches).
Revised: The node stochasticity inherent to both environments, as well as the edge stochasticity later explored, preclude the use of any methods that reason solely over symbols (i.e. automata-, sketch-, or subtask dependency-based approaches).

Original: We also incorporate a version of Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) adapted for the discrete milestone setting, which involves re-using “failed” trajectories that result in the collection of an unintended milestone (given the selected subpolicy) as successful data for the relevant subpolicy.
Revised: We also incorporate a version of Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) adapted for the discrete milestone setting, which involves re-using “failed” trajectories that result in the collection of an unintended milestone (given the selected subpolicy) as successful data for the relevant subpolicy.

Original: For instance, if an agent accidentally collects iron while executing its wooden pickaxe sub-policy, it can use this trajectory to train its iron sub-policy.
Revised: For instance, if an agent accidentally collects iron while executing its wooden pickaxe sub-policy, it can use this trajectory to train its iron sub-policy.

Original: 7. R ESULTS
Revised: 7. R ESULTS

Original: Learning efﬁcacy First, we evaluate the ability of HAL and our baselines to learn successful policies for complex tasks in each environment.
Revised: Learning efﬁcacy First, we evaluate the ability of HAL and our baselines to learn successful policies for complex tasks in each environment.

Original: Learning curves are shown in Figure 4.
Revised: Learning curves are shown in Figure 4 (plots depict mean and 95% conﬁdence interval over 5 seeds).

Original: In both environments, HAL signiﬁcantly outperforms the strongest baseline, H-Rainbow(+HER) (HR+H), despite both methods receiving the same information, and performs only slightly worse than the oracle, which has access to ground truth.
Revised: In both environments, HAL signiﬁcantly outperforms the strongest baseline, H-Rainbow(+HER) (HR+H), despite both methods receiving the same information, and performs only slightly worse than the oracle, which has access to ground truth.

Original: Incorporating HER into H-Rainbow leads to a signiﬁcant improvement.
Revised: Incorporating HER into H-Rainbow leads to a signiﬁcant improvement.

Original: False negative ﬁltering (and all other HAL components; see Appendix B) appears crucial forsuccessful learning in the TREASURE environment, but not as much for CRAFTING , though in both cases ﬁltering improves mask accuracy.
Revised: False negative ﬁltering (and all other HAL components; see Appendix B) appears crucial for learning in the TREASURE environment, but not as much for CRAFTING , though in both cases ﬁltering improves mask accuracy.

Original: Removing false negative ﬁltering causes HAL(-FNF) to be pessimistic (i.e. over-pruning subtasks, see Figures 15a and 15c inthe Appendix D), ultimately leading to its unstable learning.
Revised: Removing false negative ﬁltering causes HAL(-FNF) to be pessimistic (i.e. over-pruning subtasks, see Figure 15 in Appendix D), ultimately leading to its unstable learning.

Original: Since masking impossible subpolicies would have no impact on an optimal meta-controller, HAL’s success must stem from its ability to learn a useful mask before TD errors are able to propagate through the meta-controller’s Q function.
Revised: Since masking impossible subpolicies would have no impact on an optimal meta-controller, HAL’s success must stem from its ability to learn a useful mask before TD errors are able to propagate through the meta-controller’s Q function.

Original: HAL utilizes a more easily learnable function (the affordance classiﬁer) to reduce the amount of unnecessary expensive learning (TD error propagation) required.
Revised: HAL utilizes a more easily learnable function (affordance classiﬁer) to reduce the amount of unnecessary expensive learning (TD error propagation) required.

Original: We see from Figure 15 that, throughout training, HAL’s mask has an impact on greedy subtask selection ∼ 60% of the time, which is evidence that HAL avoids wasting time learning Q-values that the mask is able to prune.
Revised: We see from Figure 15 that, throughout training, HAL’s mask has an impact on greedy subtask selection ∼ 60% of the time, which is evidence that HAL avoids wasting time learning Q-values that the mask is able to prune.

Original: Lastly, because affordance-aware methods are more likely to initiate subtasks in an appropriate context, we see they achieve a signiﬁcantly higher average subpolicy success rate (Figure 4, right).
Revised: Lastly, because affordance-aware methods are more likely to initiate subtasks in an appropriate context, we see they achieve a signiﬁcantly higher average subpolicy success rate (Figure 4, right).

Original: Robustness to milestone selection In this section we evaluate HAL’s robustness to the selection of milestones.
Revised: Robustness to milestone selection In this section we evaluate HAL’s robustness to the selection of milestones.

Original: Affordances change when a milestone is removed since that milestone no longer acts as an intermediate link between others.
Revised: Affordances change when a milestone is removed since that milestone no longer acts as an intermediate link between others.

Original: One downside of some approaches that use symbol-based contexts is that an entirely different automata or subtask dependency graph must be deﬁned over the new set of symbols.
Revised: One downside of some approaches that use symbol-based contexts is that an entirely different automata or subtask dependency graph must be deﬁned over the new set of symbols.

Original: HAL does not use prior information of this kind, so the learning process is the same across all sets.
Revised: HAL does not use prior information of this kind, so the learning process is the same across all sets.

Original: Figure 5 shows HAL’s success on the CRAFTING environment’s iron task when using “incomplete” milestone sets, relative to the full human-designed set.
Revised: Figure 5 shows HAL’s success on the CRAFTING environment’s iron task when using “incomplete” milestone sets, relative to the full human-designed set.

Original: We see that randomly removing 1 milestone makes no signiﬁcant difference for HAL, and even after removingmilestones, HAL still achieves better performance than HR+H using the full set.
Revised: We see that randomly removing 1 milestone makes no signiﬁcant difference for HAL, and even after removingmilestones, HAL still achieves better performance than HR+H using the full set.

Original: HAL’s performance drops when 5 milestones are removed likely due to the increased sparsity of the signal (i.e. greater subtask length) and variance in milestone set quality.
Revised: HAL’s performance drops when 5 milestones are removed likely due to the increased sparsity of the signal (i.e. greater subtask length) and variance in milestone set quality.

Original: However, when we double the training time for these same sets, we ﬁnd that HAL is able to converge to a 97% success rate on at least one set, while HR+H fails to converge on any set and ends with a maximum success rate of around 70% .
Revised: However, when we double the training time for these same sets, we ﬁnd that HAL is able to converge to a 97% success rate on at least one set, while HR+H fails to converge on any set and ends with a maximum success rate of around 70% .

Original: [Figure]
Revised: [Figure]

Original: Robustness to stochasticity We modify CRAFTING so that at each environment step, there is a certain probability that an item in the inventory will disappear.
Revised: Robustness to stochasticity We modify CRAFTING so that at each environment step, there is a certain probability that an item in the inventory will disappear.

Original: In order to make the task feasible, rare items are less likely to disappear than common ones.
Revised: In order to make the task feasible, rare items are less likely to disappear than common ones.

Original: This procedure produces edge stochasticity , since the disappearance of items may alter affordances, and this can occur at any time.
Revised: This procedure produces edge stochasticity , since the disappearance of items may alter affordances, and this can occur at any time.

Original: We test three different levels of stochasticity, and display the learning curves in Figure 5.
Revised: We test three different levels of stochasticity [Figure] and display the learning curves in Figure 5.

Original: With a disappearance frequency of 1 / 100 that cuts HR+H’s success rate in half, HAL is still able to reach its non-stochastic success rate.
Revised: With a disappearance frequency of 1 / 100 that cuts HR+H’s success rate in half, HAL is still able to reach its non-stochastic success rate.

Original: With a frequency of 1 / 50 , HAL performs comparably to HR+H with no stochasticity.
Revised: With a frequency of 1 / 50 , HAL performs comparably to HR+H with no stochasticity.

Revised: To put these stochasticity rates into context

Original: To put these stochasticity rates into context, the algorithm’s average episode length about halfway through training is still over 1000 steps (see Figure 14 in Appendix C), meaning dozens of items are removed from the agent’s inventory over the course of an episode.
Revised: [Figure] the algorithm’s average episode length about halfway through training is still over 1000 steps (see Figure 14 in Appendix C) [Figure] meaning dozens of items are removed from the agent’s inventory over the course of an episode.

Original: By learning a model of affordances grounded in the present state, HAL is able to detect and adapt to these stochastic events.
Revised: By learning a model of affordances grounded in the present state, HAL is able to detect and adapt to these stochastic events.

Original: [Figure]
Revised: [Figure]

Original: ce

Original: S t o n e P
Revised: S t o n e P

Original: i c k a x e C
Revised: i c k a x e C

Original: o a l
Revised: o a l

Original: I r o n O r
Revised: I r o n O r

Original: e
Revised: e

Original: I r o n I r o n P
Revised: I r o n I r o n P

Original: i
Revised: i

Original: c
Revised: c

Original: k a x e
Revised: k a x e

Original: D i a m o nd
Revised: D i a m o nd

Original: Subtask 0 .0 .0 .0 .1 .
Revised: Subtask 0 .0 .0 .0 .1 .

Original: S u cce ss R a t e
Revised: S u cce ss R a t e

Original: HAL (ours) H-Rainbow( + HER)
Revised: HAL (ours) H-Rainbow( + HER)

Original: H i e r a r c h y D
Revised: H i e r a r c h y D

Original: e
Revised: e

Original: p
Revised: p

Original: t h
Revised: t h

Original: Task-agnostic learning We next test the ability of HAL to learn skills when no task-speciﬁc extrinsic rewards (only milestones) are provided by the environment.
Revised: Task-agnostic learning We next test the ability of HAL to learn skills when no task-speciﬁc extrinsic rewards (only milestones) are provided by the environment.

Original: Since we cannot learn a meta-controller in the absence of rewards, we instead randomly select subtasks with some probability (cid:15) mc , and random afforded subtasks otherwise (only for HAL).
Revised: Since we cannot learn a meta-controller in the absence of rewards, we instead randomly select subtasks with some probability (cid:15) mc , and random afforded subtasks otherwise (only for HAL).

Original: We evaluate both HAL and HR+H. In
Revised: We evaluate both HAL and HR+H. In

Original: Figure 6 we see that by the end of 10 6 steps, HAL is able to more reliably complete the milestones deeper in the hierarchy in the CRAFTING environment.
Revised: Figure 6 we see that by the end of 10 6 steps, HAL is able to more reliably complete the milestones deeper in the hierarchy in the CRAFTING environment.

Original: We note that HR+H is able to marginally outperform HAL in tasks shallower in the hierarchy (e.g. wood pickaxe, stone, furnace), potentially as a result of failing to reach deeper tasks and getting more practice on shallower ones.
Revised: We note that HR+H is able to marginally outperform HAL in tasks shallower in the hierarchy (e.g. wood pickaxe, stone, furnace), potentially as a result of failing to reach deeper tasks and getting more practice on shallower ones.

Original: This result is an indication of the general utility of HAL in environments with complex task hierarchies.
Revised: This result is an indication of the general utility of HAL in environments with complex task hierarchies.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: The present work can be viewed as a ﬁrst step towards bridging the substantial gap between ﬂexible hierarchical approaches that are presently intractable, and methods that impose useful structures but are too rigid to be of practical use.
Revised: The present work can be viewed as a ﬁrst step towards bridging the substantial gap between ﬂexible hierarchical approaches that are currently intractable, and methods that impose useful structures but are too rigid to be of practical use.

Original: We introduce HAL, a method that is able to utilize incomplete symbolic information in order to learn a more general form of subtask dependency.
Revised: We introduce HAL, a method that is able to utilize incomplete symbolic information in order to learn a more general form of subtask dependency.

Original: By grounding subtask dependencies in the present state by learning a model of hierarchical affordances, HAL is able to navigate stochastic environments that approaches relying solely on symbolic history are unable to.
Revised: By grounding subtask dependencies in the present state by learning a model of hierarchical affordances, HAL is able to navigate stochastic environments that approaches relying solely on symbolic history are unable to.

Original: We demonstrate that HAL learns more effectively than baselines provided with the same information, is more robust to milestone selection and affordance stochasticity, and can more thoroughly explore the environment’s subtask hierarchy.
Revised: We demonstrate that HAL learns more effectively than baselines provided with the same information, is more robust to milestone selection and affordance stochasticity, and can more thoroughly explore the environment’s subtask hierarchy.

Original: Given HAL’s ﬂexible formulation and success in the face of incomplete and stochastic symbolic information, we foresee future work integrating
Revised: Given HAL’s ﬂexible formulation and success in the face of incomplete and stochastic symbolic information, we foresee future work integrating

Original: HAL with option (or subgoal) discovery methods (e.g. Bacon et al., 2017; Machado et al., 2017;
Revised: HAL with option (or subgoal) discovery methods (e.g. Bacon et al., 2017; Machado et al., 2017;

Original: Bagaria & Konidaris, 2019) to obtain performance gains in complex tasks without requiring prespeciﬁed milestones.
Revised: Bagaria & Konidaris, 2019) to obtain performance gains in complex tasks without requiring prespeciﬁed milestones.

Original: Additionally, future work might be able to extend HAL to continuous goalspaces, but this would require revising the deﬁnition of hierarchical affordances provided here, as it currently requires a notion of intermediate subgoal completion.  
Revised: Additionally, future work might be able to extend HAL to continuous goalspaces, but this would require revising the deﬁnition of hierarchical affordances provided here, as it currently requires a notion of intermediate subgoal completion.  A CKNOWLEDGEMENTS

Revised: We thank Natasha Jaques, S´ebastien Arnold and the anonymous reviewers for their feedback on mature drafts of this manuscript.

Revised: This work is partially supported by NSF Awards IIS1513966/ 1632803/1833137, CCF-1139148, DARPA Award#: FA8750-18-2-0117, FA8750-19-10504, DARPAD3M - Award UCB-00009528, Google Research Awards, gifts from Facebook and

Revised: Netﬂix and ARO# W911NF-12- 1-0241 and W911NF-15-1-0484.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: All code for environments, algorithms, and baselines is provided at the following link: https://github.com/jamesdoewhat/hal-anonymous.
Revised: All code for environments, HAL, and other relevant baseline algorithms is provided at the following link: https://github.com/robbycostales/HAL.

Original: We enumerate the list of commands  to reproduce all experiments in the paper, as well as a ﬁle specifying the python packages and versions required which can be installed via the Anaconda python package manager.
Revised: We provide instructions for installing the necessary dependencies, and enumerate commands that allow researchers to replicate results in this paper.

Original: All hyperparameters and additional implementation details are  listed in the Appendix.
Revised: Most relevant hyperparameters and additional implementation details are also listed in the Appendix.

Original: A A DDITIONAL E NVIRONMENT D ETAILS
Revised: A A DDITIONAL E NVIRONMENT D ETAILS

Original: In both environments, extrinsic rewards are a sparse binary signal provided upon the completion of a ﬁnal goal.
Revised: In both environments, extrinsic rewards are a sparse binary signal provided upon the completion of a ﬁnal goal.

Original: Milestone signals are similarly formulated for each of the possible subtasks.
Revised: Milestone signals are similarly formulated for each of the possible subtasks.

Original: For both types of rewards (extrinsic and milestone) we introduce a per step penalty that encourages agents to achieve their goal as quickly as possible.
Revised: For both types of rewards (extrinsic and milestone) we introduce a per step penalty that encourages agents to achieve their goal as quickly as possible.

Original: TREASURE and CRAFTING environments are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).
Revised: TREASURE and CRAFTING environments are extensions of the minigrid framework (Chevalier-Boisvert et al., 2018).

Original: A. CRAFTING
Revised: A. CRAFTING

Original: As discussed in Section 7.3, our CRAFTING environment is designed to maximize hierarchical complexity while minimizing visuomotor complexity, which we do not aim to address with our method.
Revised: As discussed in Section 7.3, our CRAFTING environment is designed to maximize hierarchical complexity while minimizing visuomotor complexity, which we do not aim to address with our method.

Original: Similar Minecraft-based environments have been used in the literature; however they do not meet these requirements.
Revised: Similar Minecraft-based environments have been used in the literature; however they do not meet these requirements.

Original: The MineRL environment (Guss et al., 2019) provides an interface into the full game of Minecraft; however, effective behavior in this environment requires learning complex visuomotor policies in addition to understanding the hierarchical relationships between subtasks.
Revised: The MineRL environment (Guss et al., 2019) provides an interface into the full game of Minecraft; however, effective behavior in this environment requires learning complex visuomotor policies in addition to understanding the hierarchical relationships between subtasks.

Original: In order to evaluate our method effectively, we only aim to test the latter.
Revised: In order to evaluate our method effectively, we only aim to test the latter.

Original: Other work has used
Revised: Other work has used

Original: Minecraft-inspired environments (Andreas et al., 2016; Sohn et al., 2020); however, these versions involve simpliﬁed subtask hierarchies.
Revised: Minecraft-inspired environments (Andreas et al., 2016; Sohn et al., 2020); however, these versions involve simpliﬁed subtask hierarchies.

Original: Our environment replicates the complexity of the MineRL subtask hierarchy while remaining perceptually simple.
Revised: Our environment replicates the complexity of the MineRL subtask hierarchy while remaining perceptually simple.

Original: A concurrently developed environment contains similar hierarchical complexity while minimizing perceptual complexity (Hafner, 2021).
Revised: A concurrently developed environment contains similar hierarchical complexity while minimizing perceptual complexity (Hafner, 2021).

Original: Figure 7 displays an abstract representation of the CRAFTING environment subtask hierarchy.
Revised: Figure 7 displays an abstract representation of the CRAFTING environment subtask hierarchy.

Original: Each arrow points from one subtask to another, where the latter subtask requires the former in some way.
Revised: Each arrow points from one subtask to another, where the latter subtask requires the former in some way.

Original: The CRAFTING environment contains a variety of dependency types.
Revised: The CRAFTING environment contains a variety of dependency types.

Original: Many items must be built or crafted from other resources.
Revised: Many items must be built or crafted from other resources.

Original: Some of these require the agent to be within the vicinity of a crafting bench.
Revised: Some of these require the agent to be within the vicinity of a crafting bench.

Original: Other items require speciﬁc pickaxes to be mined, and yet others must be smelted in a furnace using a raw material and a fuel source.
Revised: Other items require speciﬁc pickaxes to be mined, and yet others must be smelted in a furnace using a raw material and a fuel source.

Original: Another complexity missing from Figure 7 is that crafting recipes require speciﬁc amounts of items.
Revised: Another complexity missing from Figure 7 is that crafting recipes require speciﬁc amounts of items.

Original: For instance, in our environment a stone pickaxe requires three stone and two sticks.
Revised: For instance, in our environment a stone pickaxe requires three stone and two sticks.

Original: Each stone must be mined individually, resulting in three separate milestones, while just two wood are required to craft four sticks.
Revised: Each stone must be mined individually, resulting in three separate milestones, while just two wood are required to craft four sticks.

Original: Lastly, there are
Revised: Lastly, there are

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: other implicit dependencies than the ones shown, depending on the deﬁnition of the milestones set.
Revised: other implicit dependencies than the ones shown, depending on the deﬁnition of the milestones set.

Original: For example, although the only ofﬁcial prerequisite for obtaining diamond is having an iron pickaxe, in practice, an agent will need to mine stone and other blocks to reach the diamond, and the successful mining of each of these blocks may be considered a milestone.
Revised: For example, although the only ofﬁcial prerequisite for obtaining diamond is having an iron pickaxe, in practice, an agent will need to mine stone and other blocks to reach the diamond, and the successful mining of each of these blocks may be considered a milestone.

Original: CRAFTING is procedurally generated in the following way.
Revised: CRAFTING is procedurally generated in the following way.

Original: For the lower half of the environment, each cell is randomly assigned stone, coal, iron, or dirt to each cell with varying probabilities.
Revised: For the lower half of the environment, each cell is randomly assigned stone, coal, iron, or dirt to each cell with varying probabilities.

Original: Diamond is randomly assigned to a cell in the bottom-most layer.
Revised: Diamond is randomly assigned to a cell in the bottom-most layer.

Original: Trees (from which logs are obtained) are abundantly scattered in the upper half, and a few irrelevant dirt blocks are placed in this region as well.
Revised: Trees (from which logs are obtained) are abundantly scattered in the upper half, and a few irrelevant dirt blocks are placed in this region as well.

Original: If the generated environment does not have enough resources, it is regenerated with the next random seed.
Revised: If the generated environment does not have enough resources, it is regenerated with the next random seed.

Original: A. TREASURE
Revised: A. TREASURE

Original: In Figure 8 one possible TREASURE subtask hierarchy is displayed.
Revised: In Figure 8 one possible TREASURE subtask hierarchy is displayed.

Original: Unlike CRAFTING , there are multiple different abstract hierarchies depending on the initial state of the environment.
Revised: Unlike CRAFTING , there are multiple different abstract hierarchies depending on the initial state of the environment.

Original: At the beginning of each episode, either both keys are available, only the yellow key, or only the red key (the instance shown in Figure 8).
Revised: At the beginning of each episode, either both keys are available, only the yellow key, or only the red key (the instance shown in Figure 8).

Original: While the environment dynamics remain the same across all episodes, the agent must infer which hierarchy is appropriate based on the initial conﬁguration of the environment.
Revised: While the environment dynamics remain the same across all episodes, the agent must infer which hierarchy is appropriate based on the initial conﬁguration of the environment.

Original: TREASURE is procedurally generated by randomly assigning each colored door to the room entrances, randomly determining which keys are accessible from the starting room, and lastly placing all objects behind their appropriate doors at random positions within the room.
Revised: TREASURE is procedurally generated by randomly assigning each colored door to the room entrances, randomly determining which keys are accessible from the starting room, and lastly placing all objects behind their appropriate doors at random positions within the room.

Original: A. W ALK - THROUGHS
Revised: A. W ALK - THROUGHS

Original: Successful human walk-throughs for both TREASURE and CRAFTING environments are described in
Revised: Successful human walk-throughs for both TREASURE and CRAFTING environments are described in

Original: Figures 9 and 10 respectively.
Revised: Figures 9 and 10 respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B A BLATION AND R OBUSTNESS R ESULTS B. C OMPONENT A BLATIONS
Revised: B A BLATION AND R OBUSTNESS R ESULTS B. C OMPONENT A BLATIONS

Original: In Figure 11 we plot success on TREASURE after removing various integral components of the full
Revised: In Figure 11 we plot success on TREASURE after removing various integral components of the full

Original: HAL method.
Revised: HAL method.

Original: We ﬁnd that all components introduced in this work are necessary for achieving the best performance on this task.
Revised: We ﬁnd that all components introduced in this work are necessary for achieving the best performance on this task.

Original: The only variation that also reliably converges to 100% success rate is when the affordance classiﬁer is provided with the raw state as input rather than using the learned representation, but this variation is undesirable since more learnable parameters are introduced.
Revised: The only variation that also reliably converges to 100% success rate is when the affordance classiﬁer is provided with the raw state as input rather than using the learned representation, but this variation is undesirable since more learnable parameters are introduced.

Original: [Figure]
Revised: [Figure]

Original: B. H YPERPARAMETER R OBUSTNESS
Revised: B. H YPERPARAMETER R OBUSTNESS

Original: In Figure 12 we demonstrate HAL’s robustness to modiﬁcations of the most signiﬁcant newly introduced hyperparameters on the TREASURE task.
Revised: In Figure 12 we demonstrate HAL’s robustness to modiﬁcations of the most signiﬁcant newly introduced hyperparameters on the TREASURE task.

Original: In only two cases over the wide range of values we tested did all runs not converge.
Revised: In only two cases over the wide range of values we tested did all runs not converge.

Original: The ﬁrst is when the upper conﬁdence bound percentile is set too low, which results in more false negatives being left unﬁltered.
Revised: The ﬁrst is when the upper conﬁdence bound percentile is set too low, which results in more false negatives being left unﬁltered.

Original: The second is when the affordance classiﬁer threshold is set too low, which results in less pruning.
Revised: The second is when the affordance classiﬁer threshold is set too low, which results in less pruning.

Original: We see that a wide range of more aggressive settings for both of these hyperparameters are reliable.
Revised: We see that a wide range of more aggressive settings for both of these hyperparameters are reliable.

Original: We ﬁnd that the standard deviation hyperparameter is not sensitive across the values we test in the non-edge-stochastic TREASURE environment, and that lower values (e.g. σ = 2 . 0 ), which we initially hypothesized might fare better with edge stochasticity but could lead to worse representations, are in actuality still effective.
Revised: We ﬁnd that the standard deviation hyperparameter is not sensitive across the values we test in the non-edge-stochastic TREASURE environment, and that lower values (e.g. σ = 2 . 0 ), which we initially hypothesized might fare better with edge stochasticity but could lead to worse representations, are in actuality still effective.

Original: [Figure]
Revised: [Figure]

Original: B. V ARYING σ UNDER S TOCHASTICITY
Revised: B. V ARYING σ UNDER S TOCHASTICITY

Original: In Appendix B.2, we demonstrated that in TREASURE , various values of σ centered around the value we used in that setting ( σ = 7 . 0 ) are all conducive to good performance on that task.
Revised: In Appendix B.2, we demonstrated that in TREASURE , various values of σ centered around the value we used in that setting ( σ = 7 . 0 ) are all conducive to good performance on that task.

Original: In Figure 13, we plot the results of using different values in a stochastic version of the CRAFTING environment in the iron task.
Revised: In Figure 13, we plot the results of using different values in a stochastic version of the CRAFTING environment in the iron task.

Original: Although all values lead to signiﬁcantly better performance than HR+H, the value we happened to used in this setting ( σ = 2 . 0 ) appears to strike the best balance.
Revised: Although all values lead to signiﬁcantly better performance than HR+H, the value we happened to used in this setting ( σ = 2 . 0 ) appears to strike the best balance.

Original: Very low values (e.g. σ = 1 . 0 ) likely do not learn as general a context representation, while higher values (e.g. σ = 8 . 0 , 16 . 0 ) are more likely to sample positive points across occurrences of edge stochasticity.
Revised: Very low values (e.g. σ = 1 . 0 ) likely do not learn as general a context representation, while higher values (e.g. σ = 8 . 0 , 16 . 0 ) are more likely to sample positive points across occurrences of edge stochasticity.

Original: It is intriguing that the ﬁnal loss for σ = 2 . 0 is lower than for σ = 1 . 0 .
Revised: It is intriguing that the ﬁnal loss for σ = 2 . 0 is lower than for σ = 1 . 0 .

Original: We speculate that the context learned by sampling points too close to the anchor could be a less natural representation to learn than a more general one, which considers further points.
Revised: We speculate that the context learned by sampling points too close to the anchor could be a less natural representation to learn than a more general one, which considers further points.

Original: [Figure]
Revised: [Figure]

Original: C E PISODE L ENGTH P LOTS
Revised: C E PISODE L ENGTH P LOTS

Original: In Figure 14 we display the episode length plots corresponding to the success rate results shown in
Revised: In Figure 14 we display the episode length plots corresponding to the success rate results shown in

Original: Figures 4 and 5.
Revised: Figures 4 and 5.

Original: [Figure]
Revised: [Figure]

Original: D A FFORDANCE P LOTS
Revised: D A FFORDANCE P LOTS

Original: We provide metrics tracked over the course of training for affordance masking as well as false negative ﬁltering in Figure 15.
Revised: We provide metrics tracked over the course of training for affordance masking as well as false negative ﬁltering in Figure 15.

Original: The mask metrics (Figure 15a for CRAFTING iron and Figure 15c for TREASURE ) consist of (from left to right):
Revised: The mask metrics (Figure 15a for CRAFTING iron and Figure 15c for TREASURE ) consist of (from left to right):

Original: • Affordance Classiﬁer Accuracy: The accuracy of the affordance classiﬁer w.r.t truth.
Revised: • Affordance Classiﬁer Accuracy: The accuracy of the affordance classiﬁer w.r.t truth.

Original: • Mask Impact:
Revised: • Mask Impact:

Original: The percentage of times that the affordance mask prevents selecting an subtask that would have otherwise had the highest Q-value.
Revised: The percentage of times that the affordance mask prevents selecting an subtask that would have otherwise had the highest Q-value.

Original: • Pruning Percentage: Percentage of subtasks pruned.
Revised: • Pruning Percentage: Percentage of subtasks pruned.

Original: • Overpruning Percentage: Percentage of subtasks that are afforded but are pruned.
Revised: • Overpruning Percentage: Percentage of subtasks that are afforded but are pruned.

Original: • Underpruning Percentage: Percentage of subtasks that are not afforded but are not pruned.
Revised: • Underpruning Percentage: Percentage of subtasks that are not afforded but are not pruned.

Original: The false negative ﬁltering metrics (Figure 15b for CRAFTING iron and Figure 15d for TREASURE ) consist of:
Revised: The false negative ﬁltering metrics (Figure 15b for CRAFTING iron and Figure 15d for TREASURE ) consist of:

Original: • Filtering Margin: L2 distance at which we consider negatives to be true negatives.
Revised: • Filtering Margin: L2 distance at which we consider negatives to be true negatives.

Original: • True Negative Accuracy: Percentage of true negatives falling above ﬁltering margin.
Revised: • True Negative Accuracy: Percentage of true negatives falling above ﬁltering margin.

Original: • False Negative Accuracy: Percentage of false negatives falling below ﬁltering margin.
Revised: • False Negative Accuracy: Percentage of false negatives falling below ﬁltering margin.

Original: • Percentage False Negatives: Percentage of negatives that are false negatives.
Revised: • Percentage False Negatives: Percentage of negatives that are false negatives.

Original: • False Negatives Flagged: Percentage of negatives that are ﬂagged as false by our margin. 
Revised: • False Negatives Flagged: Percentage of negatives that are ﬂagged as false by our margin. 0 .0 .1 .1 .

Original: [Table] [Table] [Table] Step 
Revised: 5 Step × 10 6 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A ﬀ o r d a n ce C l a ss i ﬁ e r A cc u r a c y
Revised: ﬀ o r d a n ce C l a ss i ﬁ e r A cc u r a c y

Original: Method [Table] HAL(ours [Table]
Revised: Method Oracle HAL (ours) HAL

Original: [Table]

Original: [Table]
Revised: [Table]

Original: ×
Revised: ×

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: M

Original: [Table]
Revised: [Table]

Original: M a s k I m pa c t [Table] [Table]
Revised: m p

Original: [Table]

Original: [Table]
Revised: [Table]

Original: [-× 
 [Table] [Table] [Table]-]
Revised: [-× 
 [Table] [Table] [Table]-]c t

Original: [Table]

Original: [Table]
Revised: [Table]

Original: P r un i n g P e
Revised: P r un i n g P e

Original: r
Revised: r

Original: ce n t a g e 0 .0 .1 .1 .
Revised: ce n t a g e 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: O v e r p r un i n g
Revised: O v e r p r un i n g

Original: P e r ce n t a g e 0 .0 .1 .1 .
Revised: P e r ce n t a g e 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: U nd e r p r un i n g
Revised: U nd e r p r un i n g

Original: P e r ce n t a g e (
Revised: P e r ce n t a g e (

Original: 5 Step × 

Revised: 5 Step ×

Original: K NN F il t e r
Revised: K NN F il t e r

Original: i n g
Revised: i n g

Original: M a r g i n
Revised: M a r g i n

Original: Method HAL (ours) 0 .0 .1 .1 .
Revised: Method HAL (ours) 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: T r u e N e g a t
Revised: T r u e N e g a t

Original: i v e
Revised: i v e

Original: A cc u r a c y 0 .0 .1 .1 .
Revised: A cc u r a c y 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e
Revised: F a l s e N e g a t i v e

Original: A cc u r a c y 0 .0 .1 .1 .
Revised: A cc u r a c y 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: P e r ce n t a g e F a l s e N e g a t
Revised: P e r ce n t a g e F a l s e N e g a t

Original: i v e s 0 .0 .1 .1 .
Revised: i v e s 0 .0 .1 .1 .

Original: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: 5 Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e s F l a gg e d
Revised: F a l s e N e g a t i v e s F l a gg e d

Original: [Table]
Revised: [Table]

Original:  ﬀ 
Revised: A ﬀ o

Original: [Table]
Revised: [Table]

Original:  l a ss i ﬁ 
Revised: d a n ce C l a ss i ﬁ e r A cc u r a c y [Table] [Table] I

Original: [Table]
Revised: [Table]

Revised: p a

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  il t e r i n g
Revised: K NN F il t e r i n g

Original: M a r g i n
Revised: M a r g i n

Original: [Table]
Revised: [Table]

Original: F a l s e N e g a t i v e
Revised: F a l s e N e g a t i v e

Original: A cc u r a c y 0 . 

Revised: A cc u r a c y 0 .

Original: Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: P e r ce n t a g e F a l s e N e g a t
Revised: P e r ce n t a g e F a l s e N e g a t

Original: i v e s 0 . 0 0 .
Revised: i v e s 0 . 0 0 .

Original: Step × 10 6 0 .0 .0 .0 .0 .1 .
Revised: Step × 10 6 0 .0 .0 .0 .0 .1 .

Original: F a l s e N e g a t i v e s F l a gg e d
Revised: F a l s e N e g a t i v e s F l a gg e d

Original: [Figure]
Revised: [Figure]

Original: Figure

Original: E
Revised: E

Original: A LGORITHM
Revised: A LGORITHM

Original: 1: Initialize 2: steps ← 0 3: while steps < max steps do 4: env steps , option steps ←5:
Revised: 2: steps ← 0 3: while steps < max steps do 4: env steps , option steps ←5:

Original: Receive initial state, s , from the environment 6: while s is not terminal or env steps < max env steps do 7: if env steps = 0 or option steps > max option steps or (cid:80) j b j > 0 then 8: option steps ← 0 9: if env steps > 0 then 10:
Revised: Receive initial state, s , from the environment 6: while s is not terminal or env steps < max env steps do 7: if env steps = 0 or option steps > max option steps or (cid:80) j b j > 0 then 8: option steps ← 0 9: if env steps > 0 then 10:

Original: Store transition ( s init , g, b g , s ) in D mc 11:
Revised: Store transition ( s init , g, b g , s ) in D mc 11:

Original: Store examples s t
Revised: Store examples s t

Original: | t ∈ [ init , env steps ] in positive buffers D + j 12: where b j = 1 and negative buffers D − k where b k =13: end if 14: init ← env steps 15: f s ← f φ ( z ψ ( s ))
Revised: | t ∈ [ init , env steps ] in positive buffers D + j 12: where b j = 1 and negative buffers D − k where b k =13: end if 14: init ← env steps 15: f s ← f φ ( z ψ ( s ))

Original: (cid:46) Get predicted affordances 16: q ← Q mc ( s, · ; Θ) (cid:46) Compute Q-values for all goals 17: q j ← −∞ ∀ f sj =(cid:46)
Revised: (cid:46) Get predicted affordances 16: q ← Q mc ( s, · ; Θ) (cid:46) Compute Q-values for all goals 17: q j ← −∞ ∀ f sj =(cid:46)

Original: Mask non-afforded goals 18: g max ← arg max q (cid:46)
Revised: Mask non-afforded goals 18: g max ← arg max q (cid:46)

Original: Get best afforded goal 19: g aff ← Rand ( j | f sj = 1) (cid:46) Get random afforded goal 20: g rand ← Rand ( G ) (cid:46)
Revised: Get best afforded goal 19: g aff ← Rand ( j | f sj = 1) (cid:46) Get random afforded goal 20: g rand ← Rand ( G ) (cid:46)

Original: Get random goal 21: g ← g aff w.p (cid:15) aff , g rand w.p (cid:15) mc , else g max (cid:46) (cid:15) 2 -greedy goal selection 22: end if 23: a max ← arg max Q c ( s, · ; g, Θ) (cid:46) Get best action – conditioned on goal 24:
Revised: Get random goal 21: g ← g aff w.p (cid:15) aff , g rand w.p (cid:15) mc , else g max (cid:46) (cid:15) 2 -greedy goal selection 22: end if 23: a max ← arg max Q c ( s, · ; g, Θ) (cid:46) Get best action – conditioned on goal 24:

Original: a rand ← Rand ( A ) (cid:46) Get random action 25: a ← a rand w.p (cid:15) c , else a max (cid:46) (cid:15) -greedy action selection 26: steps + + , option steps + + , env steps + + 27:
Revised: a rand ← Rand ( A ) (cid:46) Get random action 25: a ← a rand w.p (cid:15) c , else a max (cid:46) (cid:15) -greedy action selection 26: steps + + , option steps + + , env steps + + 27:

Original: Send action a to the environment and receive next state s (cid:48) , rewards r , and milestones b 28:
Revised: Send action a to the environment and receive next state s (cid:48) , rewards r , and milestones b 28:

Original: Store transition ( s, a, r, s (cid:48) , g ) in D c 29: s ← s (cid:48) 30: if steps
Revised: Store transition ( s, a, r, s (cid:48) , g ) in D c 29: s ← s (cid:48) 30: if steps

Original: % update freq = 0 then 31:
Revised: % update freq = 0 then 31:

Original: Sample batch of transitions from D c and use it to update θ via Q-learning loss 32: end if 33: if steps % meta update freq = 

Revised: Sample batch of transitions from D c and use it to update θ via Q-learning loss 32: end if 33: if steps % meta update freq =

Original: then 34:
Revised: then 34:

Original: Sample batch of transitions from D mc and use it to update Θ via Q-learning loss 35: end if 36: if steps % margin update freq = 0 then 37:
Revised: Sample batch of transitions from D mc and use it to update Θ via Q-learning loss 35: end if 36: if steps % margin update freq = 0 then 37:

Original: Sample disjoint sets of positive examples from D + g and update negative ﬁltering 38: margin 39: end if 40: if steps % aff update freq = 

Revised: Sample disjoint sets of positive examples from D + g and update negative ﬁltering 38: margin 39: end if 40: if steps % aff update freq =

Original: then 41:
Revised: then 41:

Original: Sample positive and negative examples from D + g and D − g respectively to update φ 42: with binary cross entropy loss (ignoring any negatives with computed distance 43: scores less than false negative ﬁltering margin) 44: end if 45: if steps % rep update freq = 

Revised: Sample positive and negative examples from D + g and D − g respectively to update φ 42: with binary cross entropy loss (ignoring any negatives with computed distance 43: scores less than false negative ﬁltering margin) 44: end if 45: if steps % rep update freq =

Original: then 46:
Revised: then 46:

Original: Sample anchor, positive, and negative states from D c to update ψ via contrastive loss 47: end if 48: end while 49: end while 50:
Revised: Sample anchor, positive, and negative states from D c to update ψ via contrastive loss 47: end if 48: end while 49: end while 50:

Original: F I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original: All experiments are run with 5 random seeds each.
Revised: All experiments are run with 5 random seeds each.

Original: We use 4 parallel environments for data collection with all methods.
Revised: We use 4 parallel environments for data collection with all methods.

Original: We do not include Noisy Nets or C51 in our implementation of Rainbow, as we do not ﬁnd them to improve performance in our domains and signiﬁcantly increase training time.
Revised: We do not include Noisy Nets or C51 in our implementation of Rainbow, as we do not ﬁnd them to improve performance in our domains and only increase training time.

Original: Hyperparameters are shown in Table 2.
Revised: Hyperparameters are shown in Table 2.

Original: All update frequency parameters are computed with respect to total environment steps (across parallel environments).
Revised: All update frequency parameters are computed with respect to total environment steps (across parallel environments).

Original: Relative to the controller’s Q-learning updates, all other algorithmic mechanisms are updated less frequently for the sake of efﬁciency, at rates which do not appear to affect the performance of our method.
Revised: Relative to the controller’s Q-learning updates, all other algorithmic mechanisms are updated less frequently for the sake of efﬁciency, at rates which do not appear to affect the performance of our method.

Original: The meta-controller, affordance classiﬁer, and the context representation are all trained every 10 controller updates.
Revised: The meta-controller, affordance classiﬁer [Table] and the context representation are all trained every 10 controller updates.

Original: Since the optimal separation margin between false negatives and true negatives changes slowly over time
Revised: Since the optimal separation margin between false negatives and true negatives changes slowly over time, and each computation is expensive, each classiﬁer head’s margin is updated every 150 controller updates

Original: [Table]
Revised: [Table]

Original: and computing a new margin is expensive, each classiﬁer head’s margin is updated every 150 controller updates.
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: negative ﬁltering margins

Revised: [Figure]

Revised: [Table]