Original: E XPLAINABLE
Revised: E XPLAINABLE

Original: R EINFORCEMENT L EARNING
Revised: R EINFORCEMENT L EARNING

Original: T HROUGH G OAL -B ASED E XPLANATIONS
Revised: T HROUGH G OAL -B ASED E XPLANATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many algorithms in Reinforcement Learning rely on neural networks to achieve state-of-the-art performance, but this has the cost of making the agents blackboxes, hard to interpret and understand, making their use difﬁcult in trusted applications, such as robotics or industrial applications.
Revised: Many algorithms in Reinforcement Learning rely on neural networks to achieve state-of-the-art performance, but this has the cost of making the agents blackboxes, hard to interpret and understand, making their use difﬁcult in trusted applications, such as robotics or industrial applications.

Original: To address this lack of explainability, we introduce goal-based explanations, a new explanation mechanism where the agent produces goals and attempts to reach those goals one-by-one while maximizing the collected reward.
Revised: Our key contribution to improve explainability is introducing goal-based explanations , a new explanation mechanism where the agent produces goals and attempts to reach those goals one-by-one while maximizing the collected reward.

Original: These goals form the agent’s plan to solve the task, explaining the purpose of its current actions (reach the current goal) and predicting its future behavior.
Revised: These goals form the agent’s plan to solve the task, explaining the purpose of its current actions (reach the current goal) and predicting its future behavior.

Original: To obtain the agent’s goals without domain knowledge, we use 2-layer hierarchical agents where the top layer produces goals and the bottom layer attempts to reach those goals.
Revised: To obtain the agent’s goals without domain knowledge, we use 2-layer hierarchical agents where the top layer produces goals and the bottom layer attempts to reach those goals.

Original: To train those hierarchical agents, we develop HAC-General with Teacher, a generalization of the Hindsight ActorCritic (HAC) algorithm that maximizes the rewards collected from the environment and does not require the environment to provide an end-goal. The algorithm leverages a black-box agent to vastly improve the training process.
Revised: The goals produced by trained hierarchical agents form clear and reliable explanations that can be visualized to make them easier to understand for non-experts. Hierarchical agents are more explainable but are difﬁcult to train: Hindsight Actor-Critic (HAC), a state-of-the-art algorithm, fails to train the agent in many environments.

Original: A goal is composed of an end state s to be reached and an amount of reward r to be collected.We demonstrate that leveraging a black-box agent allows HAC-General with
Revised: As an additional contribution, we generalize it and create HACGeneral with Teacher , which maximizes the rewards collected from the environment, does not require the environment to provide an end-goal, and vastly improves training by leveraging a black-box agent and using more complex goals composed of a state s to be reached and a reward r to be collected.

Original:  Teacher to solve environments that are not solvable using HAC (the current stateof-the-art hierarchical algorithm) even if it is helped by knowing the desired endgoal. The trained hierarchical agent can produce clear and reliable explanations which can be visualized to make them easier to understand for non-experts.
Revised: Our experiments show HAC-General with Teacher can train agents successfully in environments where HAC fails (even if it is helped by knowing the desired end-goal), making it possible to create explainable agents in more settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has had a huge impact on Reinforcement Learning, making it possible to solve certain problems for the ﬁrst time, vastly improving performance in many old problems and often exceeding human performance in difﬁcult tasks (Schrittwieser et al., 2019; Badia et al., 2020).
Revised: Deep learning has had a huge impact on Reinforcement Learning, making it possible to solve certain problems for the ﬁrst time, vastly improving performance in many old problems and often exceeding human performance in difﬁcult tasks (Schrittwieser et al., 2019; Badia et al., 2020).

Original: These improvements come at a price though: deep agents are black-boxes which are difﬁcult to understand and their decisions are hard to explain due to the complexity and non-obvious behavior of neural networks.
Revised: These improvements come at a price though: deep agents are black-boxes which are difﬁcult to understand and their decisions are hard to explain due to the complexity and non-obvious behavior of neural networks.

Original: In safety-critical applications, it is often fundamental to check that certain properties are respected or to understand what the behavior of the agent will be (Garc´ıa & Fern´andez, 2015; Bragg & Habli, 2018).
Revised: In safety-critical applications, it is often fundamental to check that certain properties are respected or to understand what the behavior of the agent will be (Garc´ıa & Fern´andez, 2015; Bragg & Habli, 2018).

Original: Simply observing the behavior of the agent is often not enough, since it might take its actions for the wrong reasons or it might have surprising behavior when faced with an unexpected state.
Revised: Simply observing the behavior of the agent is often not enough, since it might take its actions for the wrong reasons or it might have surprising behavior when faced with an unexpected state.

Original: Ideally, the agent would explain its behavior, which would allow for auditing, accountability, and safety-checking (Puiutta & Veith, 2020), unlocking the use of Reinforcement Learning systems in critical areas such as robotics, semi-autonomous driving, or industrial applications.
Revised: Ideally, the agent would explain its behavior, which would allow for auditing, accountability, and safety-checking (Puiutta & Veith, 2020), unlocking the use of Reinforcement Learning systems in critical areas such as robotics, semi-autonomous driving, or industrial applications.

Original: We provide three contributions to better understand deep agents.
Revised: We provide three contributions to better understand deep agents.

Original: First, we develop a new type of explanation for the agent’s behavior.
Revised: First, we develop a new type of explanation for the agent’s behavior.

Original: Imagine the following scenario: a robotic agent has to traverse a difﬁcult terrain until it reaches a speciﬁc building, where it collects a reward.
Revised: Imagine the following scenario: a robotic agent has to traverse a difﬁcult terrain until it reaches a speciﬁc building, where it collects a reward.

Original: The agent decomposes its task into a series of goals (for example, positions it has to reach) and tries to reach these goals successively until it reaches the reward zone.
Revised: The agent decomposes its task into a series of goals (for example, positions it has to reach) and tries to reach these goals successively until it reaches the reward zone.

Original: Knowing the agent’s goals would add clarity to its decision-making process: the current goal explains its short-term behavior (the joint movements are done to reach the current goal position) and the remaining goals help us understand the agent’s overall plan to solve the task and predict its future behavior.
Revised: Knowing the agent’s goals would add clarity to its decision-making process: the current goal explains its short-term behavior (the joint movements are done to reach the current goal position) and the remaining goals help us understand the agent’s overall plan to solve the task and predict its future behavior.

Original: In a way, the agent is explaining its own behavior by explicitly producing the successive goals it is trying to accomplish.
Revised: In a way, the agent is explaining its own behavior by explicitly producing the successive goals it is trying to accomplish.

Original: We call this plan composed by a series of goals a goal-based explanation .
Revised: We call this plan composed by a series of goals a goal-based explanation .

Original: Both model-based reinforcement learning (Moerland et al., 2020) and planning techniques (Fox et al., 2017) appear similar to goal-based explanations but there are important differences that make this technique novel.
Revised: Both model-based reinforcement learning (Moerland et al., 2020) and planning techniques (Fox et al., 2017) appear similar to goal-based explanations but there are important differences that make this technique novel.

Original: Goal-based explanations do not require learning a model of the environment (neither the reward function nor the transition function), thus being compatible with both model-free and model-based reinforcement learning.
Revised: Goal-based explanations do not require learning a model of the environment (neither the reward function nor the transition function), thus being compatible with both model-free and model-based reinforcement learning.

Original: Planning can be a useful explainability technique, but it has a few limitations: it typically requires knowing the end goals, they often cannot be applied to complex Markov Decision Problems and they may have difﬁculty handling very large or continuous action spaces or state spaces.
Revised: Planning can be a useful explainability technique, but it has a few limitations: it typically requires knowing the end goals, they often cannot be applied to complex Markov Decision Problems and they may have difﬁculty handling very large or continuous action spaces or state spaces.

Original: Our approach suffers from none of these limitations.
Revised: Our approach suffers from none of these limitations.

Original: Second, we develop a method to create goal-based explanations using hierarchical agents.
Revised: Second, we develop a method to create goal-based explanations using hierarchical agents.

Original: To obtain the goals, the agent is structured as a 2-level hierarchy of policies, with a goal-picking policy that produces goals and a goal-reaching policy that attempts to reach them.
Revised: To obtain the goals, the agent is structured as a 2-level hierarchy of policies, with a goal-picking policy that produces goals and a goal-reaching policy that attempts to reach them.

Original: Goals are (state, minimum desired reward) pairs, meaning the goal-reaching policy has to reach a speciﬁc state in at most H steps and collect a minimum amount of reward along the way.
Revised: Goals are (state, minimum desired reward) pairs, meaning the goal-reaching policy has to reach a speciﬁc state in at most H steps and collect a minimum amount of reward along the way.

Original: To create a goal-based explanation, the goal-picking policy is queried repeatedly: given the agent’s state s , we query for the current goal g 1 =
Revised: To create a goal-based explanation, the goal-picking policy is queried repeatedly: given the agent’s state s , we query for the current goal g 1 =

Original: ( s 1 , r 1 ) ; we then assume the agent reaches the state s 1 and query for the next goal g 

Revised: ( s 1 , r 1 ) ; we then assume the agent reaches the state s 1 and query for the next goal g

Original: = ( s 2 , r 2 ) ; this process repeats until the desired amount of goals has been collected.
Revised: = ( s 2 , r 2 ) ; this process repeats until the desired amount of goals has been collected.

Original: Our third contribution is developing a new algorithm to train hierarchical agents, which we denote
Revised: Our third contribution is developing a new algorithm to train hierarchical agents, which we denote

Original: HAC-General.
Revised: HAC-General.

Original: This algorithm builds upon the Hindsight Actor-Critic (HAC) algorithm (Levy et al., 2019), extending it to environments that do not provide an explicit end-goal.
Revised: This algorithm builds upon the Hindsight Actor-Critic (HAC) algorithm (Levy et al., 2019), extending it to environments that do not provide an explicit end-goal.

Original: Instead of trying to reach the end-goal as fast as possible and ignoring the environment’s rewards, the HAC-General algorithm trains the agent to maximize the collected reward.
Revised: Instead of trying to reach the end-goal as fast as possible and ignoring the environment’s rewards, the HAC-General algorithm trains the agent to maximize the collected reward.

Original: Our extension tries to preserve the key property that makes the Hindsight Actor-Critic algorithm effective: having an effective strategy to deal with non-stationarity by giving the illusion that the policies in sub-levels are optimal.
Revised: Our extension tries to preserve the key property that makes the Hindsight Actor-Critic algorithm effective: having an effective strategy to deal with non-stationarity by giving the illusion that the policies in sub-levels are optimal.

Original: The
Revised: The

Original: HAC-General algorithm is also able to leverage a black-box expert to improve and speed up the training for the hierarchical agent.
Revised: HAC-General algorithm is also able to leverage a black-box expert to improve and speed up the training for the hierarchical agent.

Original:  B ACKGROUND & R ELATED W ORK 2. E XPLAINABLE R EINFORCEMENT L EARNING
Revised:  B ACKGROUND & R ELATED W ORK 2. E XPLAINABLE R EINFORCEMENT L EARNING

Original: The Reinforcement Learning community has recognized the need for interpretable and explainable agents, and researchers have developed several methods to add explainability and interpretability.
Revised: The Reinforcement Learning community has recognized the need for interpretable and explainable agents, and researchers have developed several methods to add explainability and interpretability.

Original: Puiutta & Veith (2020) survey explainability techniques; we brieﬂy describe some key methods.
Revised: Puiutta & Veith (2020) survey explainability techniques; we brieﬂy describe some key methods.

Original: To add interpretability, saliency-map methods determine the importance of each input feature for the policy when it generates its output.
Revised: To add interpretability, saliency-map methods determine the importance of each input feature for the policy when it generates its output.

Original: Perturbation-based methods (Greydanus et al., 2018) measure importance by perturbing different parts of the input and measuring the change in the policy’s output.
Revised: Perturbation-based methods (Greydanus et al., 2018) measure importance by perturbing different parts of the input and measuring the change in the policy’s output.

Original: The larger the change in output, the more important the feature; the magnitude of the change quantiﬁes the relative importance of features, making it possible to build the saliency map.
Revised: The larger the change in output, the more important the feature; the magnitude of the change quantiﬁes the relative importance of features, making it possible to build the saliency map.

Original: In object-based saliency maps (Iyer et al., 2018), in addition to measuring the importance of raw features, they also measure the importance of the whole objects present in the image.
Revised: In object-based saliency maps (Iyer et al., 2018), in addition to measuring the importance of raw features, they also measure the importance of the whole objects present in the image.

Original: The importance of each object is measured by masking it and measuring the change in the policy’s output.
Revised: The importance of each object is measured by masking it and measuring the change in the policy’s output.

Original: Thus, a higher-level object saliency map is created which can be more easily interpreted by non-experts.
Revised: Thus, a higher-level object saliency map is created which can be more easily interpreted by non-experts.

Original: Another approach is to distill the policy of the black-box agent into a simpler, more interpretable model while trying to preserve the behavior and performance of the black-box policy.
Revised: Another approach is to distill the policy of the black-box agent into a simpler, more interpretable model while trying to preserve the behavior and performance of the black-box policy.

Original: Coppens et al.
Revised: Coppens et al.

Original: (2019) distill the black-box policy into a soft decision tree, a type of decision tree where the leaves output a static distribution over the actions and the inner nodes select the sub-branch using a logistic model.
Revised: (2019) distill the black-box policy into a soft decision tree, a type of decision tree where the leaves output a static distribution over the actions and the inner nodes select the sub-branch using a logistic model.

Original: A different approach is taken by Liu et al.
Revised: A different approach is taken by Liu et al.

Original: (2019) which distill the model intoa linear model
Revised: (2019) which distill the model into linear model

Original: U-Trees, a type of decision tree in which leaf nodes use a linear model to produce their output (Qvalues) instead of outputting a constant value.
Revised: U-Trees, a type of decision tree in which leaf nodes use a linear model to produce their output (Qvalues) instead of outputting a constant value.

Original: Both types of decision trees are more interpretable since they follow clear and simpler rules to go down the tree and to pick the output value.
Revised: Both types of decision trees are more interpretable since they follow clear and simpler rules to go down the tree and to pick the output value.

Original: To avoid the problem of having a black-box, another approach is to create inherently interpretable agents, without the need for a post-processing step such as creating saliency maps or distillation.

Original: Among these intrinsic methods are structural causal models (Madumal et al., 2020) which rely on structural equations to explicitly model the causal link between actions and their effect on the environment, and then use those causal links to pick actions which have a favorable effect.

Original: Since this model captures explicit causal relationships, it has inherent explainability.

Original: A different approach is to explicitly specify the degree of complexity that the model can have and then ﬁnd the best model within that class of complexity.

Original: Hein et al.

Original: (2018) pick algebraic equations as their class of models.

Original: The number of operations done in the equation is limited according to the maximum allowed complexity and good policies are found using genetic approaches.

Original: For visual or high-dimensional inputs, models that incorporate attention-like mechanisms focus on some part of the input, adding some explainability since it shows which features the model focused on (Yang et al., 2018).

Original: 2. H IERARCHICAL R EINFORCEMENT L EARNING
Revised: 2. H IERARCHICAL R EINFORCEMENT L EARNING

Original: In Hierarchical Reinforcement Learning (HRL), an agent is composed of a hierarchy of policies.
Revised: In Hierarchical Reinforcement Learning (HRL), an agent is composed of a hierarchy of policies.

Original: The top layer decomposes the task into sub-tasks, the layer below decomposes sub-tasks into sub-subtasks, and so on until the lowest level receives a low-level task and attempts to solve it by interacting with the environment.
Revised: The top layer decomposes the task into sub-tasks, the layer below decomposes sub-tasks into sub-subtasks, and so on until the lowest level receives a low-level task and attempts to solve it by interacting with the environment.

Original: Policies at higher layers learn to act at higher temporal and abstraction levels.
Revised: Policies at higher layers learn to act at higher temporal and abstraction levels.

Original: A subtask φ i can be deﬁned in multiple ways, for example as simpler linearly solvable Markov
Revised: A subtask φ i can be deﬁned in multiple ways, for example as simpler linearly solvable Markov

Original: Decision Problem (Earle et al., 2018) or as a tuple ( P i , C icomp , R i ) were the subtask φ i is eligible to start any time the precondition P
Revised: Decision Problems (Earle et al., 2018) or as a tuple ( P i , C icomp , R i ) where the subtask φ i is eligible to start any time the precondition P

Original: i is satisﬁed and it is completed once the current state is part of the completion set C i comp upon which it receives a reward r t ∼ R
Revised: i is satisﬁed and it is completed once the current state is part of the completion set C icomp upon which it receives a reward r t ∼ R

Original: i (Sohn et al., 2020).
Revised: i (Sohn et al., 2020).

Original: Our approach is based upon goal-oriented hierarchical reinforcement learning, where completing a task means reaching a goal where that goal typically is a state s which the agent must reach.
Revised: Our approach is based upon goal-oriented hierarchical reinforcement learning, where completing a task means reaching a goal where that goal typically is a state s which the agent must reach.

Original: Policies that receive a goal have only H steps to reach it instead of an unlimited time budget.
Revised: Policies that receive a goal have only H steps to reach it instead of an unlimited time budget.

Original: The policy at the bottom of the hierarchy interacts with the environment while the other policies act by picking goals (i.e. their actions are goals for the policy below them).
Revised: The policy at the bottom of the hierarchy interacts with the environment while the other policies act by picking goals (i.e. their actions are goals for the policy below them).

Original: In some problem settings the reward must be maximized.
Revised: In some problem settings, the reward must be maximized.

Original: However, in other settings, the agent receives a goal g env from the environment which must be reached as fast as possible.
Revised: However, in other settings, the agent receives a goal g env from the environment which must be reached as fast as possible.

Original: In that setting, it is important to note that the agent ignores the rewards produced by the environment; it only uses its internal reward scheme which gives the agent a small negative reward at each step, encouraging it to ﬁnd short paths.
Revised: In that setting, it is important to note that the agent ignores the rewards produced by the environment; it only uses its internal reward scheme which gives the agent a small negative reward at each step, encouraging it to ﬁnd short paths.

Original: While goal-oriented hierarchical reinforcement learning has a long history (Dayan & Hinton, 1992), there has been a resurgence in interest in recent years.
Revised: While goal-oriented hierarchical reinforcement learning has a long history (Dayan & Hinton, 1992), there has been a resurgence in interest in recent years.

Original: Hierarchical-DQN (Kulkarni et al., 2016) combines hierarchical learning with deep learning for the ﬁrst time;
Revised: Hierarchical-DQN (Kulkarni et al., 2016) combines hierarchical learning with deep learning for the ﬁrst time;

Original: Hierarchical Actor-Critic (Levy et al., 2017) improves performance by carefully setting up the hierarchy of actor-critic policies;
Revised: Hierarchical Actor-Critic (Levy et al., 2017) improves performance by carefully setting up the hierarchy of actor-critic policies;

Original: Deep
Revised: Deep

Original: Feudal Reinforcement Learning (Vezhnevets et al., 2017) use abstract goals in a latent space instead of an actual state in the real state space S .
Revised: Feudal Reinforcement Learning (Vezhnevets et al., 2017) use abstract goals in a latent space instead of an actual state in the real state space S .

Original: More recently, Hierarchical Learning with Off-Policy
Revised: More recently, Hierarchical Learning with Off-Policy

Original: Correction (Nachum et al., 2018) tries to support off-policy learning even though that all layers are constantly evolving by correcting the goals present in the transitions using a heuristic method.
Revised: Correction (Nachum et al., 2018) tries to support off-policy learning even though that all layers are constantly evolving by correcting the goals present in the transitions using a heuristic method.

Original: Our work leverages the Hindsight Actor-Critic (Levy et al., 2019) algorithm which achieves stateof-the-art performance by addressing the key problem in hierarchical reinforcement learning: the non-stationarity problem .

Original: Since all levels in the hierarchy are training at the same time, a policy at level l must continuously adapt to the changes of the policies below it in the hierarchy.

Original: This nonstationarity makes training difﬁcult; in fact, the policy at level l may train successfully only when all the policies below it in the hierarchy have stabilized.

Original: The key insight of Hindsight Actor-Critic is that optimal policies are static and therefore stable, so if the policy at level l could assume that the policies below it were optimal and therefore stationary, then it would train faster and more efﬁciently.

Original: To create this illusion that the policies below are optimal, the algorithm relies on carefully constructing 3 types of transitions, where a transition is a tuple of the form (state, action, reward, next state, goal, discount) .

Original: Due to space reasons, we detail the 3 types of transitions and their purpose in Appendix A.

Original:  G ENERALIZED H INDSIGHT A CTOR -C RITIC 
Revised:  G ENERALIZED H INDSIGHT A CTOR -C RITIC W ITH T EACHER

Original: While the Hindsight Actor-Critic (HAC) algorithm  achieves good results in speciﬁc settings such as maze environments, it is designed for avery speciﬁc setting: environments that provide an end goal and where the  objective is to reach the goal as fast as possible.
Revised: Our work builds upon the Hindsight Actor-Critic (Levy et al., 2019) or HAC, a state-of-the-art algorithm to train hierarchical agents, which achieves excellent performance in some environments. HAC is designed for a speciﬁc setting: environments that provide an end goal and where the only objective is to reach the goal as fast as possible.

Original: This specialization leads to 2 limitations: (1) HAC requires a goal, making it incompatible with all environments which do not provide a goal for the agent and (2) HAC ignores the rewards given by[Figure] the environment since it uses an internal reward scheme.
Revised: This specialization leads to 2 limitations : (1) HAC requires a goal, making it incompatible with all environments which do not provide a goal for the agent and (2) HAC ignores the rewards given by the environment since it uses an internal reward scheme.

Original: This makes it inapplicable to the majority of environments, in which rewards can be given at any point.
Revised: This makes it inapplicable to most environments, in which rewards can be given anytime.

Original: To address these issues, we generalize HAC, creating the HAC-General  algorithm which doesn’t require a goal and that considers the reward given by the environment.
Revised: To address these issues we generalize HAC, creating the HAC-General with Teacher algorithm which doesn’t require a goal and that considers the reward given by the environment.

Original: To avoid requiring a goal, the policy at the top of the hierarchy(the goal-picking policy) produces its output (a shorter-term goal) using only the state as input (no end goal in the input).
Revised: To avoid requiring a goal, the policy at the top of the hierarchy produces its output (a shorter-term goal) using only the state as input (no end-goal in the input).

Original: To take into account the rewards, the objective of the goal-picking  becomes picking goals such that the maximum amount of reward is collected during the episode.
Revised: To take into account the rewards, the objective of the goal-picking policy becomes picking goals such that the maximum amount of reward is collected during the episode.

Original: The objective of policy at the bottom of the hierarchy (the goal-reaching policy) stays the same: reaching the short-term goal in at most H steps; it ignores environment rewards.
Revised: The objective of policy at the bottom of the hierarchy (the goal-reaching policy) stays the same: reaching the short-term goal in at most H steps, ignoring environment rewards.

Original: 3. P RESERVING THE OPTIMALITY ILLUSION 
Revised: 3. M AINTAINING THE OPTIMALITY ILLUSION TO ADDRESS NON - STATIONARITY

Original: These changes address the 2 limitations of HAC, but they break  the optimality illusion that  makes 
Revised: These changes address the 2 limitations of HAC, but they break HAC’s technique to make training effective: addressing the non-stationarity problem , i.e. the problem that since all policies in the hierarchy train in parallel, each policy needs to continuously adapt to the changes in the policies below it in the hierarchy (whose behavior it relies on), which makes training difﬁcult and unstable.

Original: HAC’s training process effective: giving the goal-picking policy  the illusion that goal-reaching policy below it acts optimally. Figure 2 shows how the HAC algorithm creates theoptimality illusion (assuming environment rewards can be ignored): if thegoal-reaching policy fails to reach the goalpicking policy’s action g and reaches s instead, a  is replaced by the hindsight action s .
Revised: The insight of HAC is that if each policy trained under the illusion that all the policies below it were stationary, then it would train faster and more efﬁciently. Since optimal policies are stationary, HAC attempts to give each policy the illusion that the policy below it is optimal and thus stationary. HAC carefully constructs 3 types of transitions to create this illusion, where a transition is a tuple

Original: [Figure] 
Revised: [Figure] of the form (state, action, reward, next state, goal, discount) While the 3 types of transitions are detailed in Appendix A for space reasons, we summarize how HAC creates the illusion that the policy below is optimal and how HAC-General With Teacher preserves that illusion. We deﬁne some terminology: let π be the policy in the hierarchy for which we create the illusion.

Original: e g o a l - r e a c h i n g p o li c y G o a l p
Revised: We call π the goal-picking policy since it produces goals and call the policy below it in the hierarchy the goal-reaching policy π below , since it attempts to reach the goals it receives from π .

Original: i c
Revised: HAC .

Original: k e d b y t h e g o a l - p i
Revised: In HAC it is simple to give to π the illusion that π below is optimal because rewards are only given when the goal-state is reached.

Original: c
Revised: As shown in Figure 2, if the action of π is to pick the goal g and the goal-reaching policy π below fails to reach it and reaches state s instead, π ’s action is replaced by the hindsight action s .

Original: k i n g p o li c y
Revised: The policy π below now appears optimal since π picked goal s and π below reached it.

Original: [Figure]
Revised: [Figure]

Original: CurrentState Goal
Revised: Problem . This technique breaks down when environment rewards matter and must be maximized, i.e. it breaks down for HAC-General with Teacher.

Original: [Figure] State S [Figure] be is not enough anymore to give the  the  acted optimally while it s the goal-reaching policy might not have collected  maximum amount of reward possible.
Revised: Replacing g by s is not enough anymore to give the illusion that the goal-reaching π below acted optimally: while π below reached state s , it might not have collected the maximum amount of reward possible.

Original: As shown in Figure 3, there might be an alternative path to the same ﬁnal state where more reward would have been collected.
Revised: In other words, there might be an alternative path to the same ﬁnal state where more reward would have been collected (Figure 3).

Original: In most environments, it isunfortunately impractical  to determine if the optimal path was taken (or what the optimal path is).
Revised: Since in most environments, it is impractical or impossible to determine if the optimal path was taken (or what the optimal path is), we cannot guarantee that π below appears optimal.

Original: [Figure]
Revised: [Figure]

Revised: Solution .

Original: To address this issue, HAC-General uses a new deﬁnition of goals.
Revised: To address this issue, HAC-General uses a new deﬁnition of goals.

Original: The new goals have 2 components: a state s which must be reached and a minimum amount of reward r min which must be collected.
Revised: The new goals havecomponents: a state s which must be reached and a minimum amount of reward r min which must be collected.

Original: As shown in Figure 4, if the original action/goal is ( s, r min ) but the goal-reaching policy reaches instead s (cid:48) and collects r (cid:48) reward, then the goal-picking policy’s action is replaced by the hindsight action ( s (cid:48) , r hindsight ) where r hindsight ≤ r , creating again the optimality illusion.
Revised: As shown in Figure 4, if the original action/goal is ( s, r min ) but the goal-reaching policy reaches instead s (cid:48) and collects r (cid:48) reward, then the goal-picking policy’s action is replaced by the hindsight action ( s (cid:48) , r hindsight ) where r hindsight ≤ r , creating again the optimality illusion.

Original: It is important to note HAC-General creates the same 3 types of transitions as HAC; the major change is the way goals are deﬁned.
Revised: It is important to note HAC-General creates the same 3 types of transitions as HAC; the major change is the way goals are deﬁned.

Original: [Figure]
Revised: [Figure]

Original:  optimal  necessarily 
Revised: If we replace the original action ( G, R ) by the hindsight action ( S, R (cid:48) ) , the goal-reaching policy looks optimal again. Optimality is reached because optimality is relaxed: the policy must collect at least the desired reward to appear optimal, but not necessarily the maximum reward possible. Advantages .

Original: The goal-picking policy now has 2 mechanisms to maximize the reward it collects 1 : (1) pick goal states s that lead to high rewards and (2) force the goal-reaching policy to take a high-reward path to s by making the minimum reward threshold r min as high as possible.
Revised: The goal-picking policy now has 2 mechanisms to maximize the reward it collects 1 : (1) pick goal states s that lead to high rewards and (2) force the goal-reaching policy to take a highreward path to s by making the minimum reward threshold r min as high as possible.

Revised: The second point makes it possible to achieve high reward in RL environments where the reward is also tied to the action, not just the state, since the goal-reaching policy will learn to pick actions that lead to the goal-state but also lead to high reward.

Original: 3. L EVERAGING THE B LACK -B OX A GENT
Revised: 3. L EVERAGING THE B LACK -B OX A GENT

Original: We can leverage the black-boxnon-explainable agent to train the explainable agent more quickly and efﬁciently. The black box agent acts as a teacher which provides partial demonstrations to the explainable hierarchical agent, which acts as a student.
Revised: We can leverage the black-box agent to train the explainable agent more quickly and efﬁciently by letting the black box agent act as a teacher which provides partial demonstrations to the explainable hierarchical agent (the student).

Original: Since the goal of this paper is to create an explainable agent and there is no requirement that it should be trained from scratch with no help,  leveraging a black-box agent does not contradict our chosen setting.
Revised: Since the goal of this paper is to create an explainable agent, there is no requirement that it should be trained from scratch with no help, hence leveraging a black-box agent does not contradict our chosen setting.

Original: During an episode, the goal-picking policy produces a series of goals and the goal-reaching policy tries to reach each of those goals.
Revised: During an episode, the goal-picking policy produces a series of goals and the goal-reaching policy tries to reach each of those goals.

Original: To train the goal-producing policy of the hierarchical agent, we need to obtain goals from the black box expert but the expert doesn’t produce any goals explicitly since it is not a hierarchical agent.
Revised: To train the goal-producing policy of the hierarchical agent, we need to obtain goals from the black box expert but the expert doesn’t produce any goals explicitly since it is not a hierarchical agent.

Original: However, it produces goals implicitly: if we let the expert act for roughly H steps, the last states it reaches are good goals.
Revised: However, it produces goals implicitly: if we let the expert act for roughly H steps, the last states it reaches are good goals.

Original: Additionally, the expert reached those ﬁnal states using a series of actions a 1 , ..., a n that can be used to train the goal-reaching policy.
Revised: Additionally, the expert reached those ﬁnal states using a series of actions a 1 , ..., a n that can be used to train the goal-reaching policy.

Original: Therefore, during an episode, each time a new goal needs to be picked, we stochastically decide whether the hierarchical agent or the black-box expert acts.
Revised: Therefore, during an episode, each time a new goal needs to be picked, we stochastically decide whether the hierarchical agent or the black-box expert acts.

Original: If the black-box expert is chosen, it acts for a number n ∈ [0 .
Revised: If the black-box expert is chosen, it acts for a number n ∈ [0 .

Original: 75 H, H ] of steps, reaching the state s n and collecting a total reward r n 
Revised: 75 H, H ] of steps, reaching the state s n and collecting a total reward r n .

Original: Given the last state we can create a goal g hindsight =
Revised: Given the last state we can create a goal g hindsight =

Original: ( s n , r hindsight ) where r hindsight ≤ r n .
Revised: ( s n , r hindsight ) where r hindsight ≤ r n .

Original: The algorithm then proceeds as usual, creating the transitions as if it was the goal-picking policy in the hierarchy that had picked the goal.
Revised: The algorithm then proceeds as usual, creating the transitions as if it was the goal-picking policy in the hierarchy that had picked the goal.

Original: To train the goal-reaching policy, for each action a taken by the expert agent while it was reaching for s n we create a transition as if had been the goal-reaching policy acting with the short-term goal g hindsight .
Revised: To train the goal-reaching policy, for each action a taken by the expert agent while it was reaching for s n we create a transition as if had been the goal-reaching policy acting with the short-term goal g hindsight .

Original: By interleaving the use of the hierarchical agent and the black-box expert during an episode, the behavior cloning problem is avoided (Ross & Bagnell, 2010) which ensures the hierarchical agent can achieve good performance even in states outside the expert-induced state distribution, i.e. even if it is in a state the expert would rarely or never visit when it interacts with the environment by itself (without the intervention of the hierarchical agent).
Revised: By interleaving the use of the hierarchical agent and the black-box expert during an episode, the behavior cloning problem is avoided (Ross & Bagnell, 2010) which ensures the hierarchical agent can achieve good performance even in states outside the expert-induced state distribution, i.e. even if it is in a state the expert would rarely or never visit when it interacts with the environment by itself (without the intervention of the hierarchical agent).

Original: Algorithm 1 shows the pseudo-code to train a speciﬁc level (Appendix B contains the full pseudocode).
Revised: Algorithm 1 shows the pseudo-code to train a speciﬁc level (Appendix B contains the full pseudocode).

Original: The HAC-General With Teacher algorithm uses the generalized goals with a desired state s which must be reached and a minimum reward threshold r min which must be collected.

Original: It interleaves the use of the black-box expert and the hierarchical algorithm during an episode, to leverage the effective behavior of the expert to train the hierarchical agent more efﬁciently.

Original: Note : the 3 types of transitions created in the algorithm are detailed in Appendix A action/
Revised: Note : the 3 types of transitions created in the algorithm are detailed in Appendix A.

Original: i then
Revised: [Figure] [Figure] [Figure] [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The goal of our experiments is to evaluatethe HAC-General With Teacher algorithm in environments that provide no goal and can give a reward at any step.
Revised: The goal of our experiments is to evaluate the HAC-General With Teacher algorithm in environments that provide no goal and can give a reward at any step.

Original: We compare 3 different algorithms: HACGeneral with Teacher, HAC-General without Teacher, and the original HAC algorithm (to which we manually provide the end goal).
Revised: We compare 3 different algorithms: HACGeneral with Teacher, HAC-General without Teacher, and the original HAC algorithm (to which we manually provide the end goal).

Original: We then use the trained hierarchical agent to generate explanations, which are visualized, showing the agent’s plan to collect as much reward as possible.
Revised: We then use the trained hierarchical agent to generate explanations, which are visualized, showing the agent’s plan to collect as much reward as possible.

Original: We evaluate the agent in 2 environments from the Open AI Gym (Brockman et al., 2016) which present different challenges: Mountain Car Continuous and Lunar Lander Continuous.
Revised: We evaluate the agent in 2 environments from the OpenAI Gym (Brockman et al., 2016) which present different challenges: Mountain Car Continuous and Lunar Lander Continuous.

Original: In Mountain
Revised: In Mountain

Original: Car, a car has to reach the top of a mountain to its right but does not have enough power to reach it directly; it has to move left and then leverage momentum to reach the right mountain.
Revised: Car, a car has to reach the top of a mountain to its right but does not have enough power to reach it directly; it has to move left and then leverage momentum to climb the right mountain.

Original: The challenge of Mountain Car is good exploration since it only gets a positive reward at the top of the mountain but random exploration will very rarely lead to it; for other actions it is punished through small negative rewards.
Revised: The challenge of Mountain Car is good exploration since it only gets a positive reward at the top of the mountain but random exploration will very rarely lead to it; for other actions, it is punished through small negative rewards.

Original: In the Lunar Lander environment, a lander has to stabilize and land on the ground softly.
Revised: In the Lunar Lander environment, a lander has to stabilize and land on the ground softly.

Original: This environment has more complex dynamics than Mountain Car, higher dimensional states (which makes picking good goals harder), and more complex actions for controlling the engines.
Revised: This environment has more complex dynamics than Mountain Car, higher dimensional states (which makes picking good goals harder), and more complex actions for controlling the engines.

Original: The main challenge is predicting good states and desired rewards for the short-term goals, and learning how to reach those high dimensional goal states.
Revised: The main challenge is predicting good states and desired rewards for the short-term goals, and learning how to reach those high dimensional goal states.

Original: Figures 5 displays the performance ofthe HAC, HAC-General Without Teacher and HAC-General
Revised: Figures 5 compares the performance of HAC, HAC-General Without Teacher, and HAC-General

Original: With Teacher algorithms.
Revised: With Teacher.

Original: As we can see HAC achieves good performance in the Mountain Car (likely through knowing the end goal in an environment where exploration is themain challenge) but fails to solve the more complex Lunar Lander environment (despite knowing the end goal).
Revised: HAC achieves good performance in the Mountain Car (likely by knowing the end goal in an environment where exploration is the challenge) but fails to solve the more complex Lunar Lander environment (despite knowing the end goal).

Original: HAC-General Without Teacher  solves the Mountain Car environments on some runsand fails on others but always fails to solve Lunar Lander.
Revised: HAC-General Without Teacher only solves the Mountain Car environments on 3 out of 5 runs but always fails on Lunar Lander.

Original: However, HAC-General With Teacher can solve both environments beating the other hierarchical algorithms and demonstrating how effective leveraging a black box expert can be to train the explainable hierarchical agent.
Revised: HACGeneral With Teacher can solve both environments, beating the other hierarchical algorithms and showing how effective leveraging a black-box expert can be to train the explainable hierarchical agent.

Original: [Figure]
Revised: [Figure]

Revised: peting algorithms, showing both its effectiveness despite being in a more general and harder setting than HAC and the usefulness of leveraging a black box expert during training.

Revised: Note: training stops once the success threshold is reached, since it indicates the agent can solve the task.

Original: 4. C REATING G OAL -B ASED E XPLANATIONS
Revised: 4. C REATING G OAL -B ASED E XPLANATIONS

Original: Given a trained hierarchical agent, we can leverage its goal-picking component to explain the agent’s actions and future behavior.
Revised: Given a trained hierarchical agent, we can leverage its goal-picking component to explain the agent’s actions and future behavior.

Original: The current goal explains the short-behavior of the agent since it reveals the state the agent is trying to reach.
Revised: The current goal explains the short-behavior of the agent since it reveals the state the agent is trying to reach.

Original: We can obtain more useful explanations by querying for a series of goals instead of only knowing the current goal.
Revised: We can obtain more useful explanations by querying for a series of goals instead of only knowing the current goal.

Original: These goals form the agent’s plan to solve the environment and collect the maximum amount of reward; the goals help the user understand the agent’s present and future behavior.
Revised: These goals form the agent’s plan to solve the environment and collect the maximum amount of reward; the goals help the user understand the agent’s present and future behavior.

Original: To obtain this series of goals, the goal-picking policy is queried repeatedly: given the agent’s state s , we query for the current goal g 1 =
Revised: To obtain this series of goals, the goal-picking policy is queried repeatedly: given the agent’s state s , we query for the current goal g 1 =

Original: ( s 1 , r 1 ) ; we then assume the agent reaches the state s 1 and query for the next goal g 

Revised: ( s 1 , r 1 ) ; we then assume the agent reaches the state s 1 and query for the next goal g

Original: = ( s 2 , r 2 ) ; this process repeats until the desired amount of goals has been collected.
Revised: = ( s 2 , r 2 ) ; this process repeats until the desired amount of goals has been collected.

Original: Figure 6 displays the goal-based explanations at different time-steps of an episode, for both environments.
Revised: Figure 6 displays the goal-based explanations at different time-steps of an episode, for both environments.

Original: If the goal-reaching policy has good performance and the goal-picking policy is not too sensitive to small changes to the input state, then the explanations have several attractive properties:
Revised: If the goal-reaching policy has good performance and the goal-picking policy is not too sensitive to small changes to the input state, then the explanations have several attractive properties:

Original: • Good predictive abilities : if we generate a plan P at state s and then let the agent act, it will reach each of the goals in P (or come close to them).
Revised: • Good predictive abilities : if we generate a plan P at state s and then let the agent act, it will reach each of the goals in P (or come close to them).

Original: Thus, these goals are an effective tool to predict the future behavior of the agent for the whole episode, given a single state.
Revised: Thus, these goals are an effective tool to predict the future behavior of the agent for the whole episode, given a single state.

Original: Appendix C shows statistics on the percentage of goals that are reached.
Revised: Appendix C shows statistics on the percentage of goals that are reached.

Original: Smoothness : if we generate a plan P 1 at time-step t and another plan P 2 at time-step t +1 , the goals in P 2 should be roughly equal to the goals in P 1 shifted by the effect of one action.
Revised: [Figure]

Original: • Resistance to small errors : if the agent fails the reach the next goal in the plan by a small amount, the remainder of the plan stays relevant because the goal-picking policy isn’t overly sensitive to small changes in input state.
Revised: • Resistance to small errors : if the agent fails to reach the next goal in the plan by a small amount, the remainder of the plan stays relevant because the goal-picking policy isn’t overly sensitive to small changes in input state.

Original: Therefore, the rest of the plan is still useful even though it might be less accurate than if the state was reached.
Revised: Therefore, the rest of the plan is still useful even though it might be less accurate than if the state was reached.

Original: • Dynamic : the plans can be created from any state s , which makes it possible to re-create the plan if the goal-reaching fails to reach the goal or an unexpected stochastic events happens in the environment.
Revised: • Dynamic : the plans can be created from any state s , which makes it possible to re-create the plan if the goal-reaching fails to reach the goal or an unexpected stochastic event happens in the environment.

Original: This acts as a safety measure in case the plan becomes invalid.
Revised: This acts as a safety measure in case the plan becomes invalid.

Original: [Figure]
Revised: These plans make the hierarchical agent more explainable than the black-box agent since the goals form a useful tool to understand the agent’s behavior. Since the 2 policies are neural networks, it is possible to further apply explainability techniques to make the agent’s behavior even clearer. A strength of goal-based explanations is that they are orthogonal to other explainability techniques, and so can be combined with other techniques. Thus, we could further understand how the policies in the hierarchy pick actions by using distillation or saliency maps, for example, as well as future techniques not yet developed.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we tackle the problem of understanding the decisions taken by deep reinforcement learning agents, which tend to be inscrutable black-boxes.
Revised: In this paper, we tackle the problem of understanding the decisions taken by deep RL agents, which tend to be inscrutable black-boxes.

Original: We develop a new technique to understand agents, goal-based explanations , in which the agent explicitly produces a series of goals and attempts to reach those goals one by one, collecting as much reward as possible along the way.
Revised: We develop a new technique to understand agents, goal-based explanations , in which the agent explicitly produces a series of goals and attempts to reach those goals one by one, collecting as much reward as possible along the way.

Original: These goals make the agent more explainable since the agent’s current behavior is clear (reach the next goal) and the user knows the agent’s long term plan for solving the task.
Revised: These goals make the agent more explainable since the agent’s current behavior is clear (reach the next goal) and the user knows the agent’s long-term plan for solving the task.

Original: Given certain assumptions, these goal-based plans are smooth, resistant to small misses to the goal, and predict well the future behavior of the agent.
Revised: Given certain assumptions, these goal-based plans are smooth, resistant to small misses to the goal, and predict well the future behavior of the agent.

Original: To create the goals, we rely on a 2-layer hierarchical agent , where the top layer produces goals and the bottom layer attempts to reach these goals.
Revised: To create the goals, we rely on a 2-layer hierarchical agent , where the top layer produces goals and the bottom layer attempts to reach these goals.

Original: We contribute a new algorithm to train hierarchical agents, which we call HAC-General With Teacher , which generalizes the Hindsight Actor-Critic (HAC) algorithm.
Revised: We contribute a new algorithm to train hierarchical agents, which we call HAC-General With Teacher , which generalizes the Hindsight Actor-Critic (HAC) algorithm.

Original: Contrarily to HAC, HAC-General attempts to maximize the reward collected from the environment and doesn’t require the environment to provide an end goal the agent must reach.
Revised: Contrarily to HAC, HAC-General attempts to maximize the reward collected from the environment and doesn’t require the environment to provide an end goal the agent must reach.

Original: HAC-General is also able to leverage black-box agents to improve the training process of the hierarchical agent.
Revised: HAC-General is also able to leverage black-box agents to improve the training process of the hierarchical agent.

Original: We detail possible improvements in Appendix D and note that the algorithm can be adapted to accommodate different deﬁnitions of goals, allowing for further research.
Revised: We detail possible improvements in Appendix D and note that the algorithm can be adapted to accommodate different deﬁnitions of goals, allowing for further research.

Original: Our experiments show that HAC-General can train hierarchical agents to solve certain environments (Mountain Car and Lunar Lander) where the previous state-of-the-art hierarchical algorithm HAC failed.
Revised: Our experiments show that HAC-General can train hierarchical agents to solve certain environments (Mountain Car and Lunar Lander) where the previous state-of-the-art hierarchical algorithm HAC failed.

Original: The trained hierarchical agent produces reliable and clear explanations which can be visualized, helping users understand the agent’s short-term and long-term behavior.
Revised: The trained hierarchical agent produces reliable and clear explanations which can be visualized, helping users understand the agent’s short-term and long-term behavior.

Original: A T RANSITIONS CREATED IN THE H INDSIGHT -A CTOR -C RITIC ALGORITHM
Revised: A T RANSITIONS CREATED IN THE H INDSIGHT -A CTOR -C RITIC ALGORITHM

Original: Figure 7 summarizes the transitions and we detail them below:
Revised: Figure 7 summarizes the transitions and we detail them below:

Original: Hindsight action transitions : consider the scenario where the policy π l at level l picked as action the goal a l = s and the policy below π l − 1 reached state s (cid:48) .
Revised: Hindsight action transitions : consider the scenario where the policy π l at level l picked as action the goal a l = s and the policy below π l − 1 reached state s (cid:48) .

Original: If s (cid:48) is close enough to s , then the goal was reached and the policy below π l − 1 acted optimally.
Revised: If s (cid:48) is close enough to s , then the goal was reached and the policy below π l − 1 acted optimally.

Original: If the goal wasn’t reached, the action taken by the policy π l at level l is substituted by the hindsight action a hindsight =
Revised: If the goal wasn’t reached, the action taken by the policy π l at level l is substituted by the hindsight action a hindsight =

Original: s (cid:48) .
Revised: s (cid:48) .

Original: Under this scenario, the policy below now acted optimally, since it reached the updated goal.
Revised: Under this scenario, the policy below now acted optimally, since it reached the updated goal.

Original: Hindsight goal transitions : for a policy π l at level l , while hindsight action transitions give the illusion that the policy below π l − 1 is optimal, the reward signal may still be extremely sparse if the policy π l never reaches the goal g l it received from above as input.
Revised: Hindsight goal transitions : for a policy π l at level l , while hindsight action transitions give the illusion that the policy below π l − 1 is optimal, the reward signal may still be extremely sparse if the policy π l never reaches the goal g l it received from above as input.

Original: To ﬁx this issue, new transitions are created by replacing the goal g l of the hindsight action transitions by one of the states reached by policy π l while it was doing actions.
Revised: To ﬁx this issue, new transitions are created by replacing the goal g l of the hindsight action transitions by one of the states reached by policy π l while it was doing actions.

Original: This ensures that during at least one step of the episode the policy gets a positive reward since the hindsight goal is one of the states it visited.
Revised: This ensures that during at least one step of the episode the policy gets a positive reward since the hindsight goal is one of the states it visited.

Original: Subgoal testing transitions : in the two previous types of transitions, the hindsight action is always a state that the policy below reached.
Revised: Subgoal testing transitions : in the two previous types of transitions, the hindsight action is always a state that the policy below reached.

Original: Therefore, there are no transitions in which the action is a goal the policy below didn’t reach in at most H steps, which makes it impossible for the policy π l to learn which states are unreachable in H steps.
Revised: Therefore, there are no transitions in which the action is a goal the policy below didn’t reach in at most H steps, which makes it impossible for the policy π l to learn which states are unreachable in H steps.

Original: These unreachable states should not be picked, because then the policy below π l − 1 can never reach the goal it received, which will considerably slow down its training process and may lead to erratic and unstable behavior.
Revised: These unreachable states should not be picked, because then the policy below π l − 1 can never reach the goal it received, which will considerably slow down its training process and may lead to erratic and unstable behavior.

Original: To avoid picking unreachable goals as actions, we want to assign very low Q-values to those actions.
Revised: To avoid picking unreachable goals as actions, we want to assign very low Q-values to those actions.

Original: However, there is one key uncertainty: if the policy π l produced the goal g l but the policy below π l − 1 reached the state s instead, it might be
Revised: However, there is one key uncertainty: if the policy π l produced the goal g l but the policy below π l − 1 reached the state s instead, it might be

Original: either because the policy π l − 1 hasn’t learned to reach the goal or because g l is unreachable in at most H steps.
Revised: either because the policy π l − 1 hasn’t learned to reach the goal or because g l is unreachable in at most H steps.

Original: Since it is not possible to distinguish between the cases with certainty, subgoal testing transitions are created a random fraction of the time (say 20%).
Revised: Since it is not possible to distinguish between the cases with certainty, subgoal testing transitions are created a random fraction of the time (say 20%).

Original: In these transitions, the raw action is used (instead of the hindsight action) and a very low reward is attributed to that action, to avoid picking that goal again in the future.
Revised: In these transitions, the raw action is used (instead of the hindsight action) and a very low reward is attributed to that action, to avoid picking that goal again in the future.

Original: Note : for the bottom-level policy, we only create hindsight goal transitions (using the original action).
Revised: Note : for the bottom-level policy, we only create hindsight goal transitions (using the original action).

Original: Since the bottom-level policy interacts with the environment instead of producing goals as actions, it is not possible to create hindsight action transitions and subgoal testing transitions.
Revised: Since the bottom-level policy interacts with the environment instead of producing goals as actions, it is not possible to create hindsight action transitions and subgoal testing transitions.

Original: [Figure]
Revised: [Figure]

Original: B P SEUDOCODE FOR THE HAC-G ENERAL WITH T EACHER ALGORITHM
Revised: B P SEUDOCODE FOR THE HAC-G ENERAL WITH T EACHER ALGORITHM

Original: The pseudo-code of the training loop and a more detailed version of the TrainLevel function is shown below.
Revised: The pseudo-code of the training loop and a more detailed version of the TrainLevel function is shown below.

Original: The HAC-General without Teacher algorithm can be obtained by setting C = 0 .
Revised: The HAC-General without Teacher algorithm can be obtained by setting C = 0 .

Original: The full implementation can be consulted in the supplementary material.
Revised: The full implementation can be consulted in the supplementary material.

Original: The supplementary material also contains the implementation of the teacher agents as well as the code to show visualise the goals in the environment.
Revised: The supplementary material also contains the implementation of the teacher agents as well as the code to show visualize the goals in the environment.

Original: For both environment, a GIF ﬁle shows the trained hierarchical agent acting, and both the current goal and future goals are visualised at all steps
Revised: For both environments, a GIF ﬁle shows the trained hierarchical agent acting, and both the current goal and future goals are visualized at all steps.

Original: [ s i a i r =
Revised: if i > 0 and

Original: TBD s (cid:48)

Original: i g = TBD γ = TBD 1 Input : Initialized hierarchy of actor-critics with k levels 2 Input : Expert policy π ∗ 3 Input : Probability C of using the teacher 5 for M episodes do s ← reset environment ()
Revised: 1 Input : Initialized hierarchy of actor-critics with k levels 2 Input : Expert policy π ∗ 3 Input : Probability C of using the teacher 5 for M episodes do s ← reset environment ()

Original: TrainLevel( k − 1 , s, goal =
Revised: TrainLevel( k − 1 , s, goal =

Original: None, π ∗ , C )
Revised: None, π ∗ , C )

Original: Update all the actor-critic networks in the hierarchy 9 end
Revised: Update all the actor-critic networks in the hierarchy 9 end

Original: C P ERCENTAGE OF GOALS THAT ARE REACHED
Revised: C P ERCENTAGE OF GOALS THAT ARE REACHED

Original: [Figure]
Revised: [Figure]

Original: D P OSSIBLE IMPROVEMENTS
Revised: D P OSSIBLE IMPROVEMENTS

Original: While we displayed explanations generated by the good-performance hierarchical agents and saw their effectiveness at gaining insight into the agent’s decision-making mechanism and future behavior, improvements could be made:
Revised: While we displayed explanations generated by the good-performance hierarchical agents and saw their effectiveness at gaining insight into the agent’s decision-making mechanism and future behavior, improvements could be made:

Original: While we generated plans at different time steps to validate them, this does not prove the plans are always effective (this is similar to the principle that tests can show the presence of bugs but cannot prove an absence of bugs).
Revised: While we generated plans at different time steps to validate them, this does not prove the plans are always effective (this is similar to the principle that tests can show the presence of bugs but cannot prove an absence of bugs).

Original: Finding mechanisms to further verify the reliability of the explanations would be interesting future work.
Revised: Finding mechanisms to further verify the reliability of the explanations would be interesting future work.

Original: Work could be done to verify that the assumptions behind the plan-generating routine are met.
Revised: Work could be done to verify that the assumptions behind the plan-generating routine are met.

Original: Experiments show that the goal is reached the vast majority of the time for both the Lunar Lander and the Mountain Car environment (Figure 8), but more detailed examination could be useful.
Revised: Experiments show that the goal is reached the vast majority of the time for both the Lunar Lander and the Mountain Car environment (Figure 8), but more detailed examination could be useful.

Original: Additionally, we could also verify that the goal-picking policy isn’t sensitive to small changes to the state it receives as input.
Revised: Additionally, we could also verify that the goal-picking policy isn’t sensitive to small changes to the state it receives as input.

Original: Training hierarchical agents in the setting where no goal is provided and the agent attempts to maximize the collected reward is a challenging task.
Revised: Training hierarchical agents in the setting where no goal is provided and the agent attempts to maximize the collected reward is a challenging task.

Original: Developing better algorithms to train hierarchical agents would make it possible to tackle more difﬁcult environments and improve the efﬁciency and speed of the training process.
Revised: Developing better algorithms to train hierarchical agents would make it possible to tackle more difﬁcult environments and improve the efﬁciency and speed of the training process.

Original: The experiments done in this paper did not deal with very high-dimensional states such as images.
Revised: The experiments done in this paper did not deal with very high-dimensional states such as images.

Original: Predicting good states is likely very challenging in such environments and determining whether a goal has been reached or not becomes a more difﬁcult question, so new techniques need to be developed for those scenarios, such asfor example re-deﬁning goals so that only the relevant part of the state is included in the goal (and background visuals are ignored, for example).
Revised: Predicting good states is likely very challenging in such environments and determining whether a goal has been reached or not becomes a more difﬁcult question, so new techniques need to be developed for those scenarios, such as re-deﬁning goals so that only the relevant part of the state is included in the goal (and background visuals are ignored, for example).

Original: While the goals give insight into the agent’s behavior, it would be interesting to understand better how these goals are picked.

Original: To do so, we could use already existing explainability techniques, such as distillation or saliency maps.

Original: Our explainability method is orthogonal to other explainability techniques, making it possible to combine it with other techniques, which is a strength of our approach.

Original: E H EURISTIC TO DETECT THE END OF A PLAN
Revised: E H EURISTIC TO DETECT THE END OF A PLAN

Original: One heuristic to determine how many goal-states should be queried when creating the plan is to detect when the goal-states stabilize.
Revised: One heuristic to determine how many goal-states should be queried when creating the plan is to detect when the goal-states stabilize.

Original: For example, when the lunar lander has reached the ground, the next goal should be to stay in the same state.
Revised: For example, when the lunar lander has reached the ground, the next goal should be to stay in the same state.

Original: Similarly, if the mountain car reaches the top of the mountain, the next goal should be to stay at the top of the mountain.
Revised: Similarly, if the mountain car reaches the top of the mountain, the next goal should be to stay at the top of the mountain.

Original: Therefore, if we detect that the state-goal s (cid:48) produced by the goal-picking policy is very similar to the current state s , we may stop querying for new goals.
Revised: Therefore, if we detect that the state-goal s (cid:48) produced by the goal-picking policy is very similar to the current state s , we may stop querying for new goals.