Original: S OFT A CTOR -C RITIC WITH I NHIBITORY N ETWORKS FOR F ASTER R ETRAINING
Revised: S OFT A CTOR -C RITIC WITH I NHIBITORY N ETWORKS FOR F ASTER R ETRAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reusing previously trained models is critical in deep reinforcement learning to speed up training of new agents.
Revised: Reusing previously trained models is critical in deep reinforcement learning to speed up training of new agents.

Original: However, it is unclear how to acquire new skills when objectives and constraints are in conﬂict with previously learned skills.
Revised: However, it is unclear how to acquire new skills when objectives and constraints are in conﬂict with previously learned skills.

Original: Moreover, when retraining, there is an intrinsic conﬂict between exploiting what has already been learned and exploring new skills.
Revised: Moreover, when retraining, there is an intrinsic conﬂict between exploiting what has already been learned and exploring new skills.

Original: In soft actor-critic (SAC) methods, a temperature parameter can be dynamically adjusted to weight the action entropy and balance the explore × exploit trade-off.
Revised: In soft actor-critic (SAC) methods, a temperature parameter can be dynamically adjusted to weight the action entropy and balance the explore × exploit trade-off.

Original: However, controlling a single coefﬁcient can be challenging within the context of retraining, even more so when goals are contradictory.
Revised: However, controlling a single coefﬁcient can be challenging within the context of retraining, even more so when goals are contradictory.

Original: In this work, inspired by neuroscience research, we propose a novel approach using inhibitory networks to allow separate and adaptive state value evaluations, as well as distinct automatic entropy tuning.
Revised: In this work, inspired by neuroscience research, we propose a novel approach using inhibitory networks to allow separate and adaptive state value evaluations, as well as distinct automatic entropy tuning.

Original: Ultimately, our approach allows for controlling inhibition to handle conﬂict between exploiting less risky, acquired behaviors and exploring novel ones to overcome more challenging tasks.
Revised: Ultimately, our approach allows for controlling inhibition to handle conﬂict between exploiting less risky, acquired behaviors and exploring novel ones to overcome more challenging tasks.

Original: We validate our method through experiments in OpenAI Gym environments.
Revised: We validate our method through experiments in OpenAI Gym environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In reinforcement learning (RL), for all but the simplest tasks, agent’s behavior must be optimized with respect to multiple goals and constraints (Sutton & Barto, 2018).
Revised: In reinforcement learning (RL), for all but the simplest tasks, agents behavior must be optimized with respect to multiple goals and constraints (Sutton & Barto, 2018).

Original: It is common  for new objectives and constraints to be added after already training an agent on existing ones.
Revised: It is common in practice for new objectives and constraints to arise after already having trained an agent on existing ones.

Original: Thus, retraining these models is required to account for  additional constraints.
Revised: In order for the behavior of the agent to account for these additional constraints, retraining is required.

Original: This paper addresses the problem of efﬁciently retraining a previously trained agent to acquire a new skill within a similar environment. Within the context of deep RL,the retrainingof anexisting agentwith respect to additional objectives and constraints givesrise to problems in balancing exploitation of previously learned policy/value networks withthe learningof new policy/value networks.
Revised: Within the context of deep RL, retraining an agent to new constraints gives problems in balancing exploitation of previously learned skills with learning new skills. In this paper, we present a method for efﬁciently retraining an agent to acquire new skills in a similar environment.

Original: Many of the existing approaches to solving this type of problem involve the use of hierarchical
Revised: Existing approaches to solving the problem of learning new skills include the use of hierarchical

Original: RL structures (Dayan & Hinton, 1993; Barto & Mahadevan, 2003), such as the options framework (Sutton et al., 1999; Comanici & Precup, 2010), universal value functions (Schaul et al., 2015), option-critic (Bacon et al., 2017), FeUdal networks (Vezhnevets et al., 2017), and data-efﬁcient hierarchical RL (Nachum et al., 2018).
Revised: RL structures (Dayan & Hinton, 1993; Barto & Mahadevan, 2003), such as the options framework (Sutton et al., 1999; Comanici & Precup, 2010), universal value functions (Schaul et al., 2015), option-critic (Bacon et al., 2017), FeUdal networks (Vezhnevets et al., 2017), and data-efﬁcient hierarchical RL (Nachum et al., 2018).

Original: Other methods use multiple policy and value functions each of which are optimized for simple objectives that can then be composed to achieve complex objectives (Van Seijen et al., 2017; Sahni et al., 2017; Haarnoja et al., 2017; Hansen et al., 2020;
Revised: Other methods use multiple policies and value functions each of which are optimized for simple objectives that can then be composed to achieve complex objectives (Van Seijen et al., 2017; Sahni et al., 2017; Haarnoja et al., 2017; Hansen et al., 2020;

Original: Barreto et al., 2020).
Revised: Barreto et al., 2020).

Original: In this paper we propose to address the problem through the use of multiple value functions to provide a complex evaluative input to a single policy network.
Revised: In this paper we propose to address the problem through the use of multiple value functions to provide a complex evaluative input to a single policy network.

Original: By applying different value functions in a state dependent fashion, the reward provided to the policy network during training can remain the same as in prior training when appropriate, and can switch to a different reward when the situation indicates new constraints or goals.
Revised: By applying different value functions in a state dependent fashion, the reward provided to the policy network during training can remain the same as in prior training when appropriate, and can switch to a different reward when the situation indicates new constraints or goals.

Original: The mechanism we propose to govern the output of the composite value function is based upon neuroscientiﬁc research on inhibitory control (Diamond, 2013).
Revised: The mechanism we propose to govern the output of the composite value function is based upon neuroscientiﬁc research on inhibitory control (Diamond, 2013).

Original: The brain uses inhibition to interrupt ongoing goal-directed action when unexpected events or conﬂicts arise.
Revised: The brain uses inhibition to interrupt ongoing goal-directed action when unexpected events or conﬂicts arise.

Original: The horse-race model of
Revised: The horse-race model of

Original: Logan et al.
Revised: Logan et al.

Original: (2015), which describes the behavioral response to a goal and its inhibition in terms of dual processes, is supported by many behavioral and neurobiological studies (Verbruggen &
Revised: (2015), which describes the behavioral response to a goal and its inhibition in terms of dual processes, is supported by many behavioral and neurobiological studies (Verbruggen &

Original: Logan, 2009; Shenoy et al., 2011; Ide et al., 2013; Schall et al., 2017).
Revised: Logan, 2009; Shenoy et al., 2011; Ide et al., 2013; Schall et al., 2017).

Original: We implement this inhibitory concept in soft actor-critic (SAC) algorithm (Haarnoja et al., 2018a) by using an additional value network (inhibitory), as opposed to retraining the previous learned value network (ongoing), to learn the new skill evaluation.
Revised: We implement this inhibitory concept in soft actor-critic (SAC) algorithm (Haarnoja et al., 2018a) by using an additional value network (inhibitory), as opposed to retraining the previously learned value network (ongoing), to learn the new skill evaluation.

Original: Additionally, we propose the use of an inhibitory policy network to control inhibition.
Revised: Additionally, we propose the use of an inhibitory policy network to control inhibition.

Original: We call this SAC with inhibitory networks approach SAC-I. SAC-I is distinct from previous value composition work (Haarnoja et al., 2017; Van Niekerk et al., 2019), since it proposes a speciﬁc mechanism to train and compose value networks and generate a single policy network focused on fast and improved retraining.
Revised: We call this SAC with inhibitory networks approach SAC-I. SAC-I is distinct from previous value composition work (Haarnoja et al., 2017; Van Niekerk et al., 2019), since it proposes a speciﬁc mechanism to train and compose value networks and generate a single policy network focused on fast and improved retraining.

Original: We hypothesize that retraining of RL agents can be accelerated by creating independent and mutually inhibitory evaluative processes that will change the reward function used during learning in a state dependent manner.
Revised: We hypothesize that retraining of RL agents can be accelerated by creating independent and mutually inhibitory evaluative processes that will change the reward function used during learning in a state dependent manner.

Original: SAC provides two important features for the SAC-I: a replay buffer that can be partitioned into episodic memories related to each evaluative-learning process (Botvinick et al., 2019) and an automated entropy estimation (Haarnoja et al., 2018b), which allows computing two separate temperature parameters and exploring actions differently.
Revised: SAC provides two important features for the SAC-I: a replay buffer that can be partitioned into episodic memories related to each evaluative-learning process (Botvinick et al., 2019) and an automated entropy estimation (Haarnoja et al., 2018b), which allows computing two separate temperature parameters and exploring actions differently.

Original: There are two main contributions in this work.
Revised: There are two main contributions in this work.

Original: First , we develop the SAC-I architecture for accelerated retraining, that encompasses the use of inhibitory networks for the control of multiple evaluative networks.
Revised: First , we develop the SAC-I architecture for accelerated retraining, that encompasses the use of inhibitory networks for the control of multiple evaluative networks.

Original: This approach modiﬁes SAC methods by separating the learning process, which includes training multiple value functions, storing episodic replay buffers, estimating distinct temperature parameters, and learning an inhibition policy when necessary (Section 3).
Revised: This approach modiﬁes SAC methods by separating the learning process, which includes training multiple value functions, storing episodic replay buffers, estimating distinct temperature parameters, and learning an inhibition policy when necessary (Section 3).

Original: Second , we provide a detailed validation showing the different components of SAC-I and its improvements over SAC in two modiﬁed environments from OpenAI Gym (Section 4).
Revised: Second , we provide a detailed validation showing the different components of SAC-I and its improvements over SAC in two modiﬁed environments from OpenAI Gym (Section 4).

Original: The
Revised: The

Original: LunarLanderContinuous-v2 with a bomb appearing randomly resembles the classic stop-signal paradigm (Logan et al., 2015; Verbruggen & Logan, 2009) in inhibitory control studies.
Revised: LunarLanderContinuous-v2 with a bomb appearing randomly resembles the classic stop-signal paradigm (Logan et al., 2015; Verbruggen & Logan, 2009) in inhibitory control studies.

Original: A mixed version of the BipedalWalkerHardcore-v3, highlights the out-performance of SAC-I over SAC, which is not able to successfully solve the task.
Revised: A mixed version of the BipedalWalkerHardcore-v3, highlights the out-performance of SAC-I over SAC, which is not able to successfully solve the task.

Original:  B ACKGROUND AND R ELATED W ORK 2. M AXIMUM E NTROPY R EINFORCEMENT L EARNING
Revised:  B ACKGROUND AND R ELATED W ORK 2. M AXIMUM E NTROPY R EINFORCEMENT L EARNING

Original: In RL, knowing the best way to explore while exploiting is non-trivial, environment-dependent, and still an active area of research (Hong et al., 2018).
Revised: In RL, knowing the best way to explore while exploiting is non-trivial, environment-dependent, and still an active area of research (Hong et al., 2018).

Original: Maximum entropy RL theory provides a principled way to address this particular challenge, and has been a key element in many of the recent
Revised: Maximum entropy RL theory provides a principled way to address this particular challenge, and has been a key element in many of the recent

Original: RL advancements, providing improved exploration and faster learning (Thomas, 2014; Schulman et al., 2017; Haarnoja et al., 2017; 2018a;b; Ziebart, 2010).
Revised: RL advancements, providing improved exploration and faster learning (Thomas, 2014; Schulman et al., 2017a; Haarnoja et al., 2017; 2018a;b; Ziebart, 2010).

Original: Given a Markov decision process (MDP) with a set of states S , a set of actions A , a transition function T and a reward function R , forming a tuple < S, A, T, R > (Puterman, 1994), a stochastic policy π : S
Revised: Given a Markov decision process (MDP) with a set of states S , a set of actions A , a transition function T and a reward function R , forming a tuple < S, A, T, R > (Puterman, 1994), a stochastic policy π : S

Original: → A is a mapping from states to probabilities of selecting each possible action, where π ( a | s ) represents the probability of choosing action a given state s .
Revised: → A is a mapping from states to probabilities of selecting each possible action, where π ( a | s ) represents the probability of choosing action a given state s .

Original: In maximum entropy RL, as in an MDP, the goal is to ﬁnd the optimal policy π ∗ that provides the highest expected sum of rewards, while additionally maximizing the entropy of each visited state, leading to the expression (Ziebart, 2010):
Revised: In maximum entropy RL, as in an MDP, the goal is to ﬁnd the optimal policy π ∗ that provides the highest expected sum of rewards, while additionally maximizing the entropy of each visited state, leading to the expression (Ziebart, 2010):

Original: [Equation]
Revised: [Equation]

Original: where α is the temperature parameter that controls the stochasticity of the optimal policy,  and H ( π ) represents the entropy of the policy, Ea ∼ π [ − log ( π ( a | s ))] .
Revised: where α is the temperature parameter that controls the stochasticity of the optimal policy, ρ π is the state-action marginal of the trajectory distribution induced by the policy, r t is a shorthand for the environmental reward r ( s t , a t ) at time t , and H ( π ) represents the entropy of the policy, E [ − log ( π ( a | s ))] .

Original: This approach allows a state-wise balance between exploitation and exploration.
Revised: This approach allows a state-wise balance between exploitation and exploration.

Original: For states with high reward, a low entropy policy is permitted while, for states with low reward, high entropy policies are preferred, leading to greater exploration.
Revised: For states with high reward, a low entropy policy is permitted while, for states with low reward, high entropy policies are preferred, leading to greater exploration.

Original: The discount factor γ is omitted in the equation for simplicity since it leads to a more complex expression for the maximum entropy case (Thomas, 2014).
Revised: The discount factor γ is omitted in the equation for simplicity since it leads to a more complex expression for the maximum entropy case (Thomas, 2014).

Original: But it is required for the convergence of inﬁnite-horizon problems, and it is included in our ﬁnal algorithm.
Revised: But it is required for the convergence of inﬁnite-horizon problems, and it is included in our ﬁnal algorithm.

Original: 2. S OFT A CTOR-C RITIC Soft
Revised: 2. S OFT A CTOR

Revised: -C RITIC

Original:  actor-critic (SAC) (Haarnoja et al., 2018a) is one of the most successful maximum entropy RL methods and has become a common baseline algorithm in most of the RL libraries, outperforming state-of-the-art methods Haarnoja et al.
Revised: Soft actor-critic (SAC) (Haarnoja et al., 2018a) is one of the most successful maximum entropy RL methods and has become a common baseline algorithm in most of the RL libraries, outperforming state-of-the-art methods Haarnoja et al.

Original: (2018a;b).
Revised: (2018a;b).

Original: Like the deep deterministic policy gradient (DDPG) approach (Lillicrap et al., 2016), SAC is a model-free and off-policy method, using a replay buffer, where the policy and value functions are approximated using neural networks.
Revised: Like the deep deterministic policy gradient (DDPG) approach (Lillicrap et al., 2016), SAC is a model-free and off-policy method, using a replay buffer, where the policy and value functions are approximated using neural networks.

Original: In addition, it incorporates a policy entropy term into the objective function facilitating exploration, similar to soft Q-learning (Haarnoja et al., 2017).
Revised: In addition, it incorporates a policy entropy term into the objective function facilitating exploration, similar to soft Q-learning (Haarnoja et al., 2017).

Original: Similar to trust region policy optimization (TRPO) (Schulman et al., 2015) and proximal policy optimization (PPO), SAC uses a stochastic policy and is known to be more stable than DDPG.
Revised: Similar to trust region policy optimization (TRPO) (Schulman et al., 2015) and proximal policy optimization (PPO) Schulman et al. (2017b), SAC uses a stochastic policy and is known to be more stable than DDPG.

Original: In short, SAC combines the best of DDPG (sample efﬁciency) and TRPO/PPO (stability through stochastic policies).
Revised: In short, SAC combines the best of DDPG (sample efﬁciency) and TRPO/PPO (stability through stochastic policies).

Original: As expressed in Equation 1, the SAC policy/actor is trained with the objective of maximizing the expected cumulative reward and the action entropy at a particular state.
Revised: As expressed in Equation 1, the SAC policy/actor is trained with the objective of maximizing the expected cumulative reward and the action entropy at a particular state.

Original: The critic is the soft Q-function and, following the Bellman equation, is expressed by: Q ( s t , a t ) =
Revised: The critic is the soft Q-function and, following the Bellman equation, is expressed by: Q ( s t , a t ) =

Original: r t + γ Eρ π ( s )
Revised: r t + γ E s t +1 ∼ p

Original: [ V ( s t +1 )] , where, ρ π ( s ) represents the state marginal induced by the policy, π ( a | s ) , and the soft value function is parameterized by the Q-function: V ( s t +1 )=
Revised: [ V ( s t +1 )] , where p represents the state transition probability, and the soft value function is parameterized by the Qfunction: V ( s t +1 )

Original:  E [Equation] ∼ 
Revised: = E a t ∼ π

Original: [Equation] [ Q ( s t +1 , a t +1 )
Revised: [ Q ( s t +1 , a t +1 )

Original: − α log π ( a t +1 | s t +1 )] .
Revised: − α log π ( a t +1 | s t +1 )] .

Original: The soft Q-function is trained to minimize the following objective function given by the mean squared error between predicted and observed state-action values:
Revised: The soft Q-function is trained to minimize the following objective function given by the mean squared error between predicted and observed state-action values:

Original: [Equation]
Revised: [Equation]

Original: where D denotes the replay buffer, and ¯ V is the target value function (Mnih et al., 2015).
Revised: where D denotes the replay buffer, and ¯ V is the target value function (Mnih et al., 2015).

Original: Finally, the policy is updated to minimize the KL-divergence between the policy and the exponentiated stateaction value function (this guarantees convergence), and can be expressed by:
Revised: Finally, the policy is updated to minimize the KL-divergence between the policy and the exponentiated stateaction value function (Haarnoja et al., 2018b), and can be expressed by:

Original: [Equation]
Revised: [Equation]

Original: 2. I NHIBITORY 
Revised: 2. I NHIBITORY C ONTROL

Original: [Figure] Inhibitory control, also known as response inhibition, is a critical component of the executive functions and refers to the ability to modify ongoing actions in response to unexpected and dynamically changing task demands (Aron, 2007; Diamond, 2013).
Revised: Inhibitory control, also known as response inhibition, is a critical component of the executive functions and refers to the ability to modify ongoing actions in response to unexpected and dynamically changing task demands (Aron, 2007; Diamond, 2013).

Original: In Shenoy et al.
Revised: In Shenoy et al.

Original: (2011), inhibitory control is formalized as a rational decision-making problem, and a computational model using Bayesian inference and stochastic control tools is proposed and validated by behavioral data from humans and animals.
Revised: (2011), inhibitory control is formalized as a rational decision-making problem, and a computational model using Bayesian inference and stochastic control tools is proposed and validated by behavioral data from humans and animals.

Original: Using a widely adopted paradigm known as the stop-signal task (Logan et al., 2015), authors show that the optimal policy, whether to go or stop, systematically depends on accumulating sensory evidence, which supports the hypothesis that the brain is implementing statistically optimal decision-making (Shenoy et al., 2011).
Revised: Using a widely adopted paradigm known as the stop-signal task (Logan et al., 2015), authors show that the optimal policy, whether to go or stop, systematically depends on accumulating sensory evidence, which supports the hypothesis that the brain is implementing statistically optimal decision-making (Shenoy et al., 2011).

Original: Figure 1 depicts the two processes involved in the stop-signal task.
Revised: Figure 1 depicts the two processes involved in the stop-signal task.

Original: The Go process starts with a go signal followed by a stop signal which triggers the stop process.
Revised: The Go process starts with a go signal followed by a stop signal which triggers the stop process.

Original: The stop process will dominate when its activation is larger than the Go process activation.
Revised: The stop process will dominate when its activation is larger than the Go process activation.

Original: The key assumption is that both processes are stochastically independent, as supported by behavioral studies (Verbruggen & Logan, 2009).
Revised: The key assumption is that both processes are stochastically independent, as supported by behavioral studies (Verbruggen & Logan, 2009).

Original: Stop-signal reaction time (SSRT) is deﬁned as the time necessary to respond to the stop stimulus.
Revised: Stop-signal reaction time (SSRT) is deﬁned as the time necessary to respond to the stop stimulus.

Original: In further work using functional MRI, the anterior cingulate cortex, a region in the brain implicated in a variety of cognitive control functions, is shown to activate proportionally to a Bayesian prediction error between predicted and observed events resembling the temporal-difference methods in reinforcement learning (Ide et al., 2013).
Revised: In further work using functional MRI, the anterior cingulate cortex, a region in the brain implicated in a variety of cognitive control functions, is shown to activate proportionally to a Bayesian prediction error between predicted and observed events resembling the temporal-difference methods in reinforcement learning (Ide et al., 2013).

Original: These previous computational models indicate that when dealing with unexpected events such as the stop signal the brain implements a dual-process model driven by prediction error and responds to the conﬂict in an optimal way.
Revised: These previous computational models indicate that when dealing with unexpected events such as the stop signal the brain implements a dual-process model driven by prediction error and responds to the conﬂict in an optimal way.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: Value function composition .
Revised: Value function composition .

Original: In value function composition (Haarnoja et al., 2017; Van Niekerk et al., 2019), the goal is to model a new task by composing value functions previously trained on sub-tasks.
Revised: In value function composition (Haarnoja et al., 2017; Van Niekerk et al., 2019), the goal is to model a new task by composing value functions previously trained on sub-tasks.

Original: In Todorov (2009), they show that for linearly solvable MDPs (Todorov, 2007), pretrained value function estimators can be optimally composed and solved.
Revised: In Todorov (2009), they show that for linearly solvable MDPs (Todorov, 2007), pretrained value function estimators can be optimally composed and solved.

Original: The composition is a union of tasks, an “OR” composition, and is deﬁned by taking the softmax over the component reward signals.
Revised: The composition is a union of tasks, an “OR” composition, and is deﬁned by taking the softmax over the component reward signals.

Original: Van Niekerk et al.
Revised: Van Niekerk et al.

Original: (2019) extends this result to the standard and entropy regularized RL settings.
Revised: (2019) extends this result to the standard and entropy regularized RL settings.

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2017) deﬁnes a composition rule that approximately solves the intersection of tasks in the entropy regularized setting, an “AND” composition, with the composed signal as an average over the constituent rewards.
Revised: (2017) deﬁnes a composition rule that approximately solves the intersection of tasks in the entropy regularized setting, an “AND” composition, with the composed signal as an average over the constituent rewards.

Original: Despite the similarities, our approach is fundamentally different since it composes a previously learned value function with one which is newly trained, and moreover does not involve combining their value estimates.
Revised: Despite the similarities, our approach is fundamentally different since it composes a previously learned value function with one which is newly trained, and moreover does not involve combining their value estimates.

Original: Hybrid reward architecture .
Revised: Hybrid reward architecture .

Original: In Hybrid Reward Architecture (HRA) (Van Seijen et al., 2017), the goal is to learn a complex task by decomposing its reward, and training separate value functions for each component.
Revised: In Hybrid Reward Architecture (HRA) (Van Seijen et al., 2017), the goal is to learn a complex task by decomposing its reward, and training separate value functions for each component.

Original: The total reward is replaced by an equivalent representation as the sum, an “AND” composition, of decomposed constituent rewards.
Revised: The total reward is replaced by an equivalent representation as the sum, an “AND” composition, of decomposed constituent rewards.

Original: They show that HRA learns more efﬁciently than the deep Q-network algorithm (DQN) (Mnih et al., 2015) when both algorithms have otherwise identical network architectures.
Revised: They show that HRA learns more efﬁciently than the deep Q-network algorithm (DQN) (Mnih et al., 2015) when both algorithms have otherwise identical network architectures.

Original: SAC-I is similar to HRA in the training of multiple Q networks using different rewards, however essentially different since it does not aim to combine the reward values.
Revised: SAC-I is similar to HRA in the training of multiple Q networks using different rewards, however essentially different since it does not aim to combine the reward values.

Original: In our approach , Q networks are trained independently and used to provide specialized action values depending on the state, which is deﬁned by the inhibition rule.
Revised: In our approach, Q networks are trained independently and used to provide specialized action values depending on the state, which is deﬁned by the inhibition rule.

Original: Transfer learning in RL .
Revised: Transfer learning in RL .

Original: Broadly speaking, transfer learning in RL consists of transfering the knowledge gained in one task to improve the learning performance in a related, but different, task (Taylor & Stone, 2009; Lazaric, 2012).
Revised: Broadly speaking, transfer learning in RL consists of transferring the knowledge gained in one task to improve the learning performance in a related, but different task (Taylor & Stone, 2009; Lazaric, 2012).

Original: This knowledge can be some type of learned representation (Rusu et al., 2016b), reward shaping (Brys et al., 2015), demonstration (Schaal, 1996), model dynamics (Ammar et al., 2012), or policy (Rusu et al., 2016a).
Revised: This knowledge can be some type of learned representation (Rusu et al., 2016b), reward shaping (Brys et al., 2015), demonstration (Schaal, 1996), model dynamics (Ammar et al., 2012), or policy (Rusu et al., 2016a).

Original: Our work is within the large ﬁeld of transfer learning, however we are primarily focused on transferring learned value and policy functions among identical aspects of a task, while learning new skills (value functions) and retraining the previous learned policy within the similar environment.
Revised: Our work is within the large ﬁeld of transfer learning, however we are primarily focused on transferring learned value and policy functions among identical aspects of a task, while learning new skills (value functions) and retraining the previously learned policy within the similar environment.

Original:  M ETHODS 3. 
Revised:  M ETHODS 3. SAC-I: SAC WITH I NHIBITORY N ETWORKS

Original: [Figure]

Original: Inhibitory control is traditionally deﬁned as the ability to stop ongoing or planned cognitive or motor processes, overriding impulsive or habitual responses (Aron, 2007).
Revised: Inhibitory control is traditionally deﬁned as the ability to stop ongoing or planned cognitive or motor processes, overriding impulsive or habitual responses (Aron, 2007).

Original: One possible
Revised: One possible

Original: RL implementation is at the executive level by switching between two policies using a hierarchical architecture.
Revised: RL implementation is at the executive level by switching between two policies using a hierarchical architecture.

Original: However, this approach requires having to train multiple policies, and a more complex hierarchical model.
Revised: However, this approach requires having to train multiple policies, and a more complex hierarchical model.

Original: We propose a computationally less expensive approach and do the switching at the Q network level, since it will ultimately drive the change of the habitual response or action.
Revised: We propose a computationally less expensive approach and do the switching at the Q network level, since it will ultimately drive the change of the habitual response or action.

Original: We implement inhibitory control in this broader sense at a higher cognitive level, where the action execution (policy) is updated by the evaluative processes that precede execution.
Revised: We implement inhibitory control in this broader sense at a higher cognitive level, where the action execution (policy) is updated by the evaluative processes that precede execution.

Original: We apply inhibitory control to the SAC algorithm by having multiple, competing value functions (critics) that take turns depending on the current task demand.
Revised: We apply inhibitory control to the SAC algorithm by having multiple, competing value functions (critics) that take turns depending on the current task demand.

Original: Rather than having a Q network that learns the Go task and later the Stop task, we keep the previous Q network that knows to evaluate the Go task and train a new Q-I network that learns the Stop task.
Revised: Rather than having a Q network that learns the Go task and later the Stop task, we keep the previous Q network that knows to evaluate the Go task and train a new Q-I network that learns the Stop task.

Original: Thus, both Q networks are trained independently.
Revised: Thus, both Q networks are trained independently.

Original: Here, the term “inhibitory” refers to a new evaluative process replacing an ongoing one.
Revised: Here, the term “inhibitory” refers to a new evaluative process replacing an ongoing one.

Original: In Figure 2, we depict the SAC policy update with inhibitory networks.
Revised: In Figure 2, we depict the SAC policy update with inhibitory networks.

Original: Go networks estimate ongoing evaluation (which led to pre-trained skills), while the Stop networks estimate the value of the unexpected event, leading the policy to learn new skills.
Revised: Go networks estimate ongoing evaluation (which led to pre-trained skills), while the Stop networks estimate the value of the new event, leading the policy to learn new skills.

Original: Additionally, we introduce the use of an inhibitory policy network to learn the inhibition policy responsible for deciding how and when to use the outcomes from the multiple Q networks.
Revised: Additionally, we introduce the use of an inhibitory policy network to learn the inhibition policy responsible for deciding how and when to use the outcomes from the multiple Q networks.

Original: Alternatively, to avoid having to train an additional policy, a parametric state-based inhibition rule can be implemented using knowledge about the environment or updated constraints.
Revised: Alternatively, to avoid having to train an additional policy, a parametric state-based inhibition rule can be implemented using knowledge about the environment or updated constraints.

Original: We use Q networks to estimate all the value functions, following the implementation in Haarnoja et al.
Revised: We use Q networks to estimate all the value functions, following the implementation in Haarnoja et al.

Original: (2018b), and the clipped double-Q learning trick, introduced in twin delayed DDPG (TD3) (Fujimoto et al., 2018) to avoid their over estimation.
Revised: (2018b), and the clipped double-Q learning trick, introduced in twin delayed DDPG (TD3) (Fujimoto et al., 2018) to avoid their overestimation.

Original: 3. E PISODIC M EMORY THROUGH S ELECTIVE R EPLAY B UFFER
Revised: 3. E PISODIC M EMORY THROUGH S ELECTIVE R EPLAY B UFFER

Original: In neuroscience research, episodic memory refers to the brain’s ability to recollect past experiences, and is an important component of learning (Tulving, 2002).
Revised: In neuroscience research, episodic memory refers to the brain’s ability to recollect past experiences, and is an important component of learning (Tulving, 2002).

Original: In recent years, it has been applied to DRL as a non-parametric framework to retrieve past successful experiences to improve sample efﬁciency (Blundell et al., 2016; Lin et al., 2018; Botvinick et al., 2019) or to avoid catastrophic forgetting (Isele & Cosgun, 2018).
Revised: In recent years, it has been applied to DRL as a non-parametric framework to retrieve past successful experiences to improve sample efﬁciency (Blundell et al., 2016; Lin et al., 2018; Botvinick et al., 2019) or to avoid catastrophic forgetting (Isele & Cosgun, 2018).

Original: In this work, we implement episodic memory in its simplest form; a partition of the state space according to an inhibition rule or policy that yields a partition of the replay buffer.
Revised: In this work, we implement episodic memory in its simplest form; a partition of the state space according to an inhibition rule or policy that yields a partition of the replay buffer.

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: { S R , S I } be a partition of the state space.
Revised: { S R , S I } be a partition of the state space.

Original: The corresponding replay buffer partition is D =
Revised: The corresponding replay buffer partition is D =

Original: {D R , D I } where set membership of a tuple ( s k , a, r k , s (cid:48) ) is parameterized by s k for k ∈ { R, I } where s R ∈ S R
Revised: {D R , D I } where set membership of a tuple ( s k , a, r k , s (cid:48) ) is parameterized by s k for k ∈ { R, I } where s R ∈ S R

Original: and s I ∈ S I .
Revised: and s I ∈ S I .

Revised: For ease of notation, s (cid:48)

Revised: =

Revised: s t +1 , and the next state can belong to any of the partitions, s (cid:48) ∈ S .

Original: Note that inhibitory states have an associated inhibitory reward r I and regular states have the associated reward r R .
Revised: Note that inhibitory states have an associated inhibitory reward r I and regular states have the associated reward r R .

Original: These corresponding rewards are stored in the replay buffers but the replay buffers are not parameterized by them.
Revised: These corresponding rewards are stored in the replay buffers but the replay buffers are not parameterized by them.

Original: The loss function of each Q network is computed using memories from the corresponding replay buffer as expressed by:
Revised: The loss function of each Q network is computed using memories from the corresponding replay buffer as expressed by:

Original: [Equation]
Revised: [Equation]

Original: where, for ease of notation, s (cid:48) =
Revised: for k ∈ { R [Equation]

Original: s t
Revised: I

Original: +1 .
Revised: [Equation] [Equation]

Original: Notice that if we use a single replay buffer that contains both the regular and the inhibitory rewards ( s, a, r R , r I , s (cid:48) ) , and sample a tuple containing the regular ( s, a, r R , 0 , s (cid:48) ) or the inhibitory ( s, a, 0 , r I , s (cid:48) ) rewards depending on the updated Q network, we would have sparser rewards particularly for the Q-I network.
Revised: Notice that if we use a single replay buffer that contains both the regular and the inhibitory rewards ( s, a, r R , r I , s (cid:48) ) , and sample a tuple containing the regular ( s, a, r R , 0 , s (cid:48) ) or the inhibitory ( s, a, 0 , r I , s (cid:48) ) rewards depending on the updated Q network, we would have sparser rewards particularly for the Q-I network.

Original: Therefore, having separate memories promotes faster learning (as shown in Figure 4) and, importantly, also allows the estimation of separate entropy parameters.
Revised: Therefore, having separate memories promotes faster learning (as shown in Figure 4) and, importantly, also allows the estimation of separate entropy parameters.

Original: 3. A UTOMATED D UAL E NTROPY E STIMATION
Revised: 3. A UTOMATED D UAL E NTROPY E STIMATION

Original: The SAC algorithm is sensitive to the α temperature changes depending on the environment, reward scale and training stage, as shown in the initial paper (Haarnoja et al., 2018a).
Revised: The SAC algorithm is sensitive to the α temperature changes depending on the environment, reward scale and training stage, as shown in the initial paper (Haarnoja et al., 2018a).

Original: To address this issue, the same authors propose to automatically adjust the temperature parameter by formulating the problem with a dual objective; maximize entropy while satisfying a minimum entropy constraint (Haarnoja et al., 2018b).
Revised: To address this issue, the same authors propose to automatically adjust the temperature parameter by formulating the problem with a dual objective; maximize entropy while satisfying a minimum entropy constraint (Haarnoja et al., 2018b).

Original: The goal is still maximizing the cumulative expected reward (Equation 1), but the average entropy of the policy is now constrained by a minimum value.
Revised: The goal is still maximizing the cumulative expected reward (Equation 1), but the average entropy of the policy is now constrained by a minimum value.

Original: The full derivation of the dual optimization problem is given in Haarnoja et al.
Revised: The full derivation of the dual optimization problem is given in Haarnoja et al.

Original: (2018b).
Revised: (2018b).

Original: In practice, the optimization is performed recursively as follows: at time t , given the current estimate α t , the optimal policy π ∗ t is estimated as described in Equation 1.
Revised: In practice, the optimization is performed recursively as follows: at time t , given the current estimate α t , the optimal policy π ∗ t is estimated as described in Equation 1.

Original: Subsequently, given π t , the α t is approximated using a neural network.
Revised: Subsequently, given π t , the α t is approximated using a neural network.

Original: While in the standard SAC, there is a single α parameter, in SAC-I, we propose estimating two separate α temperature parameters, α R and α I , to allow distinct action entropy for the previously learned and the new skills, respectively.
Revised: While in the standard SAC, there is a single α parameter, in SAC-I, we propose estimating two separate α temperature parameters, α R and α I , to allow distinct action entropy for the previously learned and the new skills, respectively.

Original: This is implemented by training two separate α networks, with regular states s R ∈ S R distinguished from inhibitory states s I ∈ S I , with losses given by:
Revised: This is implemented by training two separate α networks, with regular states s R ∈ S R distinguished from inhibitory states s I ∈ S I , with losses given by:

Original: [Equation]
Revised: [Equation]

Original: where H 0 is the minimum expected entropy.
Revised: where H 0 is the minimum expected entropy.

Original: The policy loss function is composed of two terms:
Revised: The policy loss function is composed of two terms:

Original: [Equation]
Revised: [Equation]

Original: 3. I NHIBITORY P OLICY N ETWORK
Revised: 3. I NHIBITORY P OLICY N ETWORK

Original: In our proposed SAC-I algorithm (Figure 2), for the cases in which a state-dependent inhibition rule is not deﬁned, we propose training an inhibitory policy network π I .
Revised: In our proposed SAC-I algorithm (Figure 2), for the cases in which a state-dependent inhibition rule is not deﬁned, we propose training an inhibitory policy network π I .

Original: This network can be trained as an automated hard switch or a soft modulator between the regular and inhibitory Q networks.
Revised: This network can be trained as an automated hard switch or a soft modulator between the regular and inhibitory Q networks.

Original: The inhibitory policy network is a stand alone agent with its own Q networks (not shown in Figure 2), however it shares the same replay buffers.
Revised: The inhibitory policy network is a stand-alone agent with its own Q networks (not shown in Figure 2), however it shares the same replay buffers.

Original: The loss functions are the standard ones as deﬁned in Equations 2 and 3.
Revised: The loss functions are the standard ones as deﬁned in Equations 2 and 3.

Original: Ultimately, the goal of the inhibitory policy network is to maximize the environment’s reward by learning to choose between the Go and Stop networks and/or by modulating the Stop network.
Revised: Ultimately, the goal of the inhibitory policy network is to maximize the environment’s reward by learning to choose between the Go and Stop networks and/or by modulating the Stop network.

Original: We show the implementation of both cases next in Section 4.
Revised: We show the implementation of both cases next in Section 4.

Original: SAC-I is summarized in Algorithm 1.
Revised: SAC-I is summarized in Algorithm 1.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: In order to show two different use-cases of SAC-I algorithm, as well as to evaluate it as a way to speed up training during transfer learning, we use continuous tasks from the Box2D simulator,
Revised: In order to show two different use-cases of SAC-I algorithm, as well as to evaluate it as a way to speed up training during transfer learning, we use continuous tasks from the Box2D simulator,

Original: OpenAI Gym (Brockman et al., 2016), LunarLanderContinuous-v2 and BipedalWalkerHardcore-v3.
Revised: OpenAI Gym (Brockman et al., 2016), LunarLanderContinuous-v2 and BipedalWalkerHardcore-v3.

Original: Importantly, we include custom modiﬁcations to them to emulate the scenarios in which retraining 
Revised: Importantly, we include custom modiﬁcations to them to emulate the scenarios in which retraining is necessary.

Revised: By adding a random bomb in the LunarLander task, we show the advantages of using

Original: Initialize Q R 1 , Q R 2 , Q I 1 , Q I 2 , policy π , policy π I , α R and α
Revised: Initialize Q R 1 , Q R 2 , Q I 1 , Q I 2 , policy π , policy π I , α R and α

Original: I networks parameters;
Revised: I networks parameters;

Original: Initialize the target ¯ Q R 1 , ¯ Q R 2 , ¯ Q I 1 and ¯ Q I 2 networks weights;
Revised: Initialize the target ¯ Q R 1 , ¯ Q R 2 , ¯ Q I 1 and ¯ Q I 2 networks weights;

Original: Initialize the replay buffers D R and D I ; for each episode do for each environment step do
Revised: Initialize the replay buffers D R and D I ; for each episode do for each environment step do

Original: Given s t , sample a t from π ( s t ) and ( s t +1 , r t ) from the environment;
Revised: Given s t , sample a t from π ( s t ) and ( s t +1 , r t ) from the environment;

Original: Use an inhibition rule or inhibitory policy π I to classify s t ; if s t ∈ S R , push ( s t , a t , r R t , s t +1 ) to D R ; else if s t ∈ S
Revised: Use an inhibition rule or inhibitory policy π I to classify s t ; if s t ∈ S R , push ( s t , a t , r R t , s t +1 ) to D R ; else if s t ∈ S

Original: I push ( s t , a t , r I t , s t +1 ) to D I ; endfor each gradient step do
Revised: I push ( s t , a t , r I t , s t +1 ) to D I ; endfor each gradient step do

Original: Sample a batch of memories from from {D R , D I } ; for k ∈ { R, I } do
Revised: Sample a batch of memories from from {D R , D I } ; for k ∈ { R, I } do

Original: Update Q k 1 and Q k 2 (Equation 4), α k (Equation 5), and ¯ Q k 1 and ¯ Q k 2 (soft-update); end
Revised: Update Q k 1 and Q k 2 (Equation 4), α k (Equation 5), and ¯ Q k 1 and ¯ Q k 2 (soft-update); end

Original: Update the policy network π (Equation 6);
Revised: Update the policy network π (Equation 6);

Original: If an inhibitory policy network is used, update π I and the associated Q networks; end endis necessary.
Revised: If an inhibitory policy network is used, update π I and the associated Q networks; end end

Original: By adding a random bomb in the LunarLander task, we show the advantages of using

Original: SAC-I, compared to standard SAC, when retraining with conﬂicting goals, similar to what happens in a stop-signal task (Logan et al., 2015), i.e. stopping an ongoing action (to land) whenever an unexpected event occurs (to avoid bomb).
Revised: SAC-I, compared to standard SAC, when retraining with conﬂicting goals, similar to what happens in a stop-signal task (Logan et al., 2015), i.e. stopping an ongoing action (to land) whenever an unexpected event occurs (to avoid bomb).

Original: In the experiments with BipedalWalkerHardcore-v3, we train agents in a simpler version of the task (BipedalWalker-v3), and retrain them in the more complex task.
Revised: In the experiments with BipedalWalkerHardcore-v3, we train agents in a simpler version of the task (BipedalWalker-v3), and retrain them in the more complex task.

Original: We show how SAC-I can help transfer learning and adjust inhibitory control.
Revised: We show how SAC-I can help transfer learning and adjust inhibitory control.

Original: In all the experiments provided in this section, we used ﬁve random seeds to account for the variability during training and compare agents trained across a ﬁxed number of steps.
Revised: In all the experiments provided in this section, we used ﬁve random seeds to account for the variability during training and compare agents trained across a ﬁxed number of steps.

Original: All the hyperparameters are available in Table 1 (Appendix).
Revised: All the hyperparameters are available in Table 1 (Appendix).

Original: 4. L UNAR L ANDER C ONTINUOUS WITH B OMB
Revised: 4. L UNAR L ANDER C ONTINUOUS WITH B OMB

Original: Environment .
Revised: Environment .

Original: The original version of LunarLanderContinuous-v2 is modiﬁed in order to include a bomb that appears randomly within a region above the landing pad (Figure 3).
Revised: The original version of LunarLanderContinuous-v2 is modiﬁed in order to include a bomb that appears randomly within a region above the landing pad (Figure 3).

Original: Like the original version, it includes the environment reward (moving from the top of the frame to the landing pad: 100-140 points, each leg contact: +10, crashing: -100, successful landing: +100, ﬁring engine: -0.
Revised: Like the original version, it includes the environment reward (moving from the top of the frame to the landing pad: 100-140 points, each leg contact: +10, crashing: -100, successful landing: +100, ﬁring engine: -0.

Original: per frame)
Revised: per frame)

Original: , but additionally it includes a penalty for hitting the bomb (-150) and a time penalty (-0.1 per frame) to motivate landing as quickly as possible (like in SST).
Revised: , but additionally it includes a penalty for hitting the bomb (-150) and a time penalty (-0.1 per frame) to motivate landing as quickly as possible (like in SST).

Original: Importantly, the bomb coordinates are included in the observation state only after the bomb appears so the agent does not know about its existence beforehand.
Revised: Importantly, the bomb coordinates are included in the observation state only after the bomb appears so the agent does not know about its existence beforehand.

Original: Further details can be found in Appendix.
Revised: Further details can be found in Appendix.

Original: Experimental design .
Revised: Experimental design .

Original: Initially, a standard SAC agent is trained in the  environment
Revised: Initially, a standard SAC agent is trained in the original environment

Original: LunarLanderContinuous-v2.
Revised: LunarLanderContinuous-v2.

Original: This agent, which we will call “baseline agent”, takes about 250K steps to reach an average cumulative reward of 200 and is able to successfully land in most of the episodes.
Revised: This agent, which we will call “baseline agent”, takes about 250K steps to reach an average cumulative reward of 200 and is able to successfully land in most of the episodes.

Original: All the network weights are transferred to the retrained agent.
Revised: All the network weights are transferred to the retrained agent.

Original: The Q-I network is trained from scratch.
Revised: The Q-I network is trained from scratch.

Original: In this experiment, the inhibition rule/policy works as a switch between the regular and inhibitory Q networks.
Revised: In this experiment, the inhibition rule/policy works as a switch between the regular and inhibitory Q networks.

Original: We show both cases, user-deﬁned inhibition rule and learned inhibitory policy network.
Revised: We show both cases, user-deﬁned inhibition rule and learned inhibitory policy network.

Original: Experiments are performed in a single machine with Intel Core i7-9850H
Revised: Experiments are performed in a single machine with Intel Core i7-9850H

Original: CPU@2.60Hz x 12, Quadro RTX3000, RAM 16GB.
Revised: CPU@2.60Hz x 12, Quadro RTX3000, RAM 16GB.

Original: The average rewards reported in the ﬁgures represent the average of the episode reward, including the bomb penalty, over the last 100 episodes.
Revised: The average rewards reported in the ﬁgures represent the average of the episode reward, including the bomb penalty, over the last 100 episodes.

Original: Agents are trained for a ﬁxed number of 2K episodes, approximately 500K steps.
Revised: Agents are trained for a ﬁxed number of 2K episodes, approximately 500K steps.

Original: Advantage of retraining .
Revised: Advantage of retraining .

Original: In this experiment, we show how the performance of a standard SAC agent is boosted by retraining it using weights from the baseline agent, which already knows how to safely land.
Revised: In this experiment, we show how the performance of a standard SAC agent is boosted by retraining it using weights from the baseline agent, which already knows how to safely land.

Original: In order to learn bomb avoidance skills, we shaped the original reward with an additional penalty given by the expression r bomb.proxy = − 1 e 4 × ( d b − 0 . 3) 4 , where d b is deﬁned as the agent’s distance to the center of the bomb ( x b , y b ) .
Revised: In order to learn bomb avoidance skills, we shaped the original reward with an additional penalty given by the expression r bomb.proxy = − 1 e 4 × ( d b − 0 . 3) 4 , where d b is deﬁned as the agent’s distance to the center of the bomb ( x b , y b ) .

Original: The idea is to have a ﬁeld-type avoidance mechanism and give a penalty proportional to bomb proximity.
Revised: The idea is to have a ﬁeld-type avoidance mechanism and give a penalty proportional to bomb proximity.

Original: Average reward results are depicted in Figure 4, orange and blue lines.
Revised: Average reward results are depicted in Figure 4, orange and blue lines.

Original: For retraining, all the network weights are loaded from the baseline agent and updated in the new task with bomb.
Revised: For retraining, all the network weights are loaded from the baseline agent and updated in the new task with bomb.

Original: The retrained agent (blue line) clearly learns to complete the task (land and avoid the bomb) faster than the agent trained from scratch (orange line).
Revised: The retrained agent (blue line) clearly learns to complete the task (land and avoid the bomb) faster than the agent trained from scratch (orange line).

Original: In about 300K steps, it starts to avoid the bomb and at 500K steps is mostly able to complete the task successfully.
Revised: In about 300K steps, it starts to avoid the bomb and at 500K steps is mostly able to complete the task successfully.

Original: The agent from scratch takes > 1 .
Revised: The agent from scratch takes > 1 .

Original: 25 M steps to learn the task.
Revised: 25 M steps to learn the task.

Original: Effect of using episodic memory and dual alpha in SAC-I .
Revised: Effect of using episodic memory and dual alpha in SAC-I .

Original: We parse out the distinct contributions of different components of SAC-I by training agents with and without episodic memory and the dual α temperature parameters.
Revised: We parse out the distinct contributions of different components of SAC-I by training agents with and without episodic memory and the dual α temperature parameters.

Original: In these versions, we use a user deﬁned inhibition rule given by y > y b and d b < 0 .
Revised: In these versions, we use a user deﬁned inhibition rule given by y > y b and d b < 0 .

Original: If these conditions are met, the inhibitory Q-I network is used with the same shaping used for the SAC agent, r I = r bomb.proxy .
Revised: If these conditions are met, the inhibitory Q-I network is used with the same shaping used for the SAC agent, r I = r bomb.proxy .

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 4.

Original: The SACI vanilla agent (green line) is trained without episodic memory and with a single α parameter.
Revised: The SACI vanilla agent (green line) is trained without episodic memory and with a single α parameter.

Original: The SAC-I with episodic memory (purple line) is trained with separate experience replay buffer for the inhibitory states, but uses only a single entropy parameter α .
Revised: The SAC-I with episodic memory (purple line) is trained with separate experience replay buffer for the inhibitory states, but uses only a single entropy parameter α .

Original: The SAC-I agent with episodic memory and dual α (red line) has faster learning, reaching an average reward of 200 after 300K steps, and highlighting the importance of these components 1 .
Revised: The SAC-I agent with episodic memory and dual α (red line) has faster learning, reaching an average reward of 200 after 300K steps, and highlighting the importance of these components 1 .

Original: SAC-I performance for different conﬂict levels .
Revised: SAC-I performance for different conﬂict levels .

Original: The brain’s inhibitory control is known to be modulated by levels of conﬂict between ongoing processes (Braver et al., 2001).
Revised: The brain’s inhibitory control is known to be modulated by levels of conﬂict between ongoing processes (Braver et al., 2001).

Original: For instance in the SST, changing the frequency of stop signals, which is associated with expectation, alters the response time to go signals.
Revised: For instance in the SST, changing the frequency of stop signals, which is associated with expectation, alters the response time to go signals.

Original: In a similar way, we investigate whether and how the frequency of stop trials (episodes with bomb) impacts the agent’s learning, other than the overall performance which is expected to decrease for increased occurrence of bombs.
Revised: In a similar way, we investigate whether and how the frequency of stop trials (episodes with bomb) impacts the agent’s learning, other than the overall performance which is expected to decrease for increased occurrence of bombs.

Original: We hypothesize that the bomb frequency will impact SAC-I less than SAC because it trains separate Q networks for different skills involved in the task (i.e. landing and bomb avoidance).
Revised: We hypothesize that the bomb frequency will impact SAC-I less than SAC because it trains separate Q networks for different skills involved in the task (i.e. landing and bomb avoidance).

Original: Both SAC and SAC-I agents are trained with varying bomb frequencies, and results are shown in Figure 5.
Revised: Both SAC and SAC-I agents are trained with varying bomb frequencies, and results are shown in Figure 5.

Original: As expected, the overall performance decreases for higher frequency.
Revised: As expected, the overall performance decreases for higher frequency.

Original: SAC-I agent reaches an average reward of 200 in about 200K, 300K and 600K for frequencies 25%, 50% and 75%, respectively.
Revised: SAC-I agent reaches an average reward of 200 in about 200K, 300K and 600K for frequencies 25%, 50% and 75%, respectively.

Original: For the SAC agent, it is clear that bomb frequency not only affects performance but also the training progression.
Revised: For the SAC agent, it is clear that bomb frequency not only affects performance but also the training progression.

Original: For example, a signiﬁcant drop in performance is observed around step 50K for Bomb 75% as well as for 50%, although less.
Revised: For example, a signiﬁcant drop in performance is observed around step 50K for Bomb 75% as well as for 50%, although less.

Original: This likely happens because learning to avoid the bomb interferes with its initial ability to land.
Revised: This likely happens because learning to avoid the bomb interferes with its initial ability to land.

Original: Interestingly, although both agent’s performances converge asymptotically, the difference is larger for Bomb 50% case, in which the uncertainty is the highest.
Revised: Interestingly, although both agent’s performances converge asymptotically, the difference is larger for Bomb 50% case, in which the uncertainty is the highest.

Original: This is likely because SAC and SAC-I agents learn different strategies to avoid the bomb.
Revised: This is likely because SAC and SAC-I agents learn different strategies to avoid the bomb.

Original: While the SAC agent adopts an optimal global strategy accordingly to different bomb frequencies (i.e, if bomb is frequent, slow down when task starts), the SAC-I agent keeps the same landing strategy independent of bomb frequency and learns to avoid the bomb when it appears.
Revised: While the SAC agent adopts an optimal global strategy accordingly to different bomb frequencies (i.e, if bomb is frequent, slow down when task starts), the SAC-I agent keeps the same landing strategy independent of bomb frequency and learns to avoid the bomb when it appears.

Original: Training without reward shaping .
Revised: Training without reward shaping .

Original: Further, we examined the training of SAC and SAC-I agents without using any reward shaping for the bomb avoidance.
Revised: Further, we examined the training of SAC and SAC-I agents without using any reward shaping for the bomb avoidance.

Revised: Both agents use the same reward r = r 0 + r bomb , where r 0 is the original reward and r bomb is a sparse penalty for hitting the bomb.

Original: Results 2 are shown in Figure 6.
Revised: Results 2 are shown in Figure 6.

Original: Surprisingly, the SAC* agent without shaping (yellow line) performed better than the one with, meaning that shaping is negatively impacting its training.
Revised: Surprisingly, the SAC* agent without shaping (yellow line) performed better than the one with, meaning that shaping is negatively impacting its training.

Original: In contrast, the SAC-I* agent without shaping (gray line) successfully learns the task with similar speed to the one with shaping (red line), likely because it keeps separate critic networks to evaluate landing and bomb avoidance.
Revised: In contrast, the SAC-I* agent without shaping (gray line) successfully learns the task with similar speed to the one with shaping (red line), likely because it keeps separate critic networks to evaluate landing and bomb avoidance.

Original: In further experiments, we observed that different reward shaping can boost SAC-I’s performance even more, but not SAC (seetraining results with conservative shaping in  Figure 10, Appendix).
Revised: In further experiments, we observed that different reward shaping can boost SAC-I’s performance even more, but not SAC (see results with conservative shaping in Appendix, Figure 10).

Original: Learning when to inhibit .
Revised: Learning when to inhibit .

Original: In this experiment, we show an agent trained with an inhibitory policy network SAC-I** and compare it to an agent trained with an inhibition rule SAC-I*.
Revised: In this experiment, we show an agent trained with an inhibitory policy network SAC-I** and compare it to an agent trained with an inhibition rule SAC-I*.

Original: The SAC-I* agent (gray line) uses a simple inhibition rule (“whenever bomb appears”) to switch between the regular and inhibitory Q networks (Figure 2).
Revised: The SAC-I* agent (gray line) uses a simple inhibition rule (“whenever bomb appears”) to switch between the regular and inhibitory Q networks (Figure 2).

Original: While, the SAC-I** agent (brown line) learns an inhibition policy, i.e. the best timing to inhibit.
Revised: While, the SAC-I** agent (brown line) learns an inhibition policy, i.e. the best timing to inhibit.

Original: This SAC-I** agent (brown line) performs as good as the SAC-I* and outperforms the SAC agents (Figure 6).
Revised: This SAC-I** agent (brown line) performs as good as the SAC-I* and outperforms the SAC agents (Figure 6).

Original: 4. B IPEDAL W ALKER H ARDCORE - V
Revised: 4. B IPEDAL W ALKER H ARDCORE - V

Original: In this evaluation, we show a different use of inhibitory networks, to control the magnitude of inhibition.
Revised: In this evaluation, we show a different use of inhibitory networks, to control the magnitude of inhibition.

Original: To create the retraining scenario, ﬁrst, we train a standard SAC agent in an easier environment BipedalWalker-v3 (Figure 7a), in which the goal is to walk through a plain terrain, and use its weights to retrain SAC and SAC-I agents in BipedalWalkerHardcore-v3 (Figure 7b-d).
Revised: To create the retraining scenario, ﬁrst, we train a standard SAC agent in an easier environment BipedalWalker-v3 (Figure 7a), in which the goal is to walk through a plain terrain, and use its weights to retrain SAC and SAC-I agents in BipedalWalkerHardcore-v3 (Figure 7b-d).

Original: In the
Revised: In the

Original: BipedalWalker task, agents naturally get a negative “inhibitory” reward whenever they are stuck.
Revised: BipedalWalker task, agents naturally get a negative “inhibitory” reward whenever they are stuck.

Original: We use SAC-I to learn as well as to weight that negative reward.
Revised: We use SAC-I to learn as well as to weight that negative reward.

Original: The BipedalWalkerHardcore-v3 is a challenging task, known to be unsolvable for many of the simpler non-recurrent DRL architectures or model-free RL methods (Wei & Ying, 2021).
Revised: The BipedalWalkerHardcore-v3 is a challenging task, known to be unsolvable for many of the simpler non-recurrent DRL architectures or model-free RL methods (Wei & Ying, 2021).

Original: We solve the task using a standard two-layer dense-network architecture, and adopt two strategies: removing the fall-penalty and creating a cumulative version of the task reward.
Revised: We solve the task using a standard two-layer dense-network architecture, and adopt two strategies: removing the fall-penalty and creating a cumulative version of the task reward.

Original: Otherwise, the task is unsolvable with SAC algorithm (see Figure 11 in Appendix).
Revised: Otherwise, the task is unsolvable with SAC algorithm (see Figure 11 in Appendix).

Original: Learning how to inhibit .
Revised: Learning how to inhibit .

Original: In Figure 8, we show the training performance of SAC and two versions of SAC-I agents learning BipedalWalkerHardcore-v3.
Revised: In Figure 8, we show the training performance of SAC and two versions of SAC-I agents learning BipedalWalkerHardcore-v3.

Original: All agents are retrained from baseline.
Revised: All agents are retrained from baseline.

Original: The SAC agent reward consists of the raw environment reward r 0 , but with a cumulative reward and without the fall penalty 3 , expressed by r = r 0 + r stuck − r fall . 
Revised: The SAC agent reward consists of the raw environment reward r 0 , but with a cumulative reward and without the fall penalty 3 , expressed by r = r 0 + r stuck − r fall . Figure color coding.

Original: Mimicking life, harsh fall penalty seems  block learning.
Revised: Mimicking life, harsh fall penalty seems to block learning.

Original: See Figure 11 in Appendix.
Revised: See Figure 11 in Appendix.

Original: The cumulative reward is deﬁned by r stuck = (cid:80) 5 i =

Revised: The cumulative reward is deﬁned by r stuck = (cid:80) 5 i =

Original: r t − i , accounted if it is negative, i.e. the agent is at a “stuck” state.
Revised: r t − i , accounted if it is negative, i.e. the agent is at a “stuck” state.

Original: The cumulative reward provides a less noisy feedback.
Revised: The cumulative reward provides a less noisy feedback.

Original: The primary goal of SAC-I (standard) is to separately train a Q-I critic network specialized on modeling the new reward structure, i.e. r I =
Revised: The primary goal of SAC-I (standard) is to separately train a Q-I critic network specialized on modeling the new reward structure, i.e. r I =

Original: r 

Revised: r

Original: + r stuck − r fall .
Revised: + r stuck − r fall .

Original: The SAC-I* agent (adaptive) learns an inhibitory policy network to estimate the weight w so that r I ∗ =
Revised: The SAC-I* agent (adaptive) learns an inhibitory policy network to estimate the weight w so that r I ∗ =

Original: r 0 + w × r stuck − r fall .
Revised: r 0 + w × r stuck − r fall .

Original: This SAC-I* agent with adaptive inhibition (green line) outperforms the other agents, and the difference is highlighted in the mixed version of the task (see further Figure 9).
Revised: This SAC-I* agent with adaptive inhibition (green line) outperforms the other agents, and the difference is highlighted in the mixed version of the task (see further Figure 9).

Original: Mixed version of BipedalWalkerHardcore-v3 .
Revised: Mixed version of BipedalWalkerHardcore-v3 .

Original: To replicate the scenario in which there are go and stop episodes, we retrain the baseline agent on a modiﬁed task mixing both the BipedalWalkerv3 (Go trial) and BipedalWalkerHardcore-v3 (Stop trial).
Revised: To replicate the scenario in which there are go and stop episodes, we retrain the baseline agent on a modiﬁed task mixing both the BipedalWalkerv3 (Go trial) and BipedalWalkerHardcore-v3 (Stop trial).

Original: For each episode, we randomly choose in which environment the agent should perform.
Revised: For each episode, we randomly choose in which environment the agent should perform.

Original: Interestingly, this makes the learning even more challenging, because the obstacles are sparser in time and the agent has to explore and exploit while learning the easy and hard versions of the task simultaneously.
Revised: Interestingly, this makes the learning even more challenging, because the obstacles are sparser in time and the agent has to explore and exploit while learning the easy and hard versions of the task simultaneously.

Original: In Figure 9, we show that using inhibitory networks is critical to successfully learn the mixed task.
Revised: In Figure 9, we show that using inhibitory networks is critical to successfully learn the mixed task.

Original: Both agents are retrained using the same reward structure as presented in the previous section.
Revised: Both agents are retrained using the same reward structure as presented in the previous section.

Original: Although the task gets easier as the percent of go trials is increased, from 10% to 30%, we observe that it is more difﬁcult to learn the mixed version of the task since there are less stop trials to learn from (for instance, compare the two plots in the middle column).
Revised: Although the task gets easier as the percent of go trials is increased, from 10% to 30%, we observe that it is more difﬁcult to learn the mixed version of the task since there are less stop trials to learn from (for instance, compare the two plots in the middle column).

Original: Also, we observe some training instability for 70% stop trial, likely because the interference between learning the Stop and Go trials (3rd column).
Revised: Also, we observe some training instability for 70% stop trial, likely because the interference between learning the Stop and Go trials (3rd column).

Original: Interestingly, we observe that the SAC-I* agent with adaptive inhibition has more stability across the stop training sessions.
Revised: Interestingly, we observe that the SAC-I* agent with adaptive inhibition has more stability across the stop training sessions.

Original: For Stop 90%, the averages of the reward standard deviation are 47.5 and 27.5 for SAC-I and SAC-I* agents, respectively.
Revised: For Stop 90%, the averages of the reward standard deviation are 47.5 and 27.5 for SAC-I and SAC-I* agents, respectively.

Original: For Stop 70%, the averages are 56.3 and 31.7, respectively.
Revised: For Stop 70%, the averages are 56.3 and 31.7, respectively.

Original: Finally, we observe that, unlike the SAC agent, only the SAC-I agents are able to successfully learn the stop trials (2nd column).
Revised: Finally, we observe that, unlike the SAC agent, only the SAC-I agents are able to successfully learn the stop trials (2nd column).

Original: Note, they are trained using the same reward structure (previous section).
Revised: Note, they are trained using the same reward structure (previous section).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we draw a parallel between neuroscience research in inhibitory control and reinforcement learning of competing value functions.
Revised: In this work, we draw a parallel between neuroscience research in inhibitory control and reinforcement learning of competing value functions.

Original: We propose an SAC algorithm using inhibitory networks with  particular focus on retraining the agent to acquire a new skill, while still exploiting previously learned abilities.
Revised: We propose an SAC algorithm using inhibitory networks with a particular focus on retraining the agent to acquire a new skill, while still exploiting previously learned abilities.

Original: Through our experiments, we show that SAC-I agents are able to maintain higher rewards from the beginning of retraining since they keep a previous ability compared to retrained SAC agents (Figure 6).
Revised: Through our experiments, we show that SAC-I agents are able to maintain higher rewards from the beginning of retraining since they keep a previous ability compared to retrained SAC agents (Figure 6).

Original: With SAC-I, we advance the use of SAC methods by introducing the use of multiple value networks with respective episodic replay buffers, as well as distinct entropy parameter estimation for skills at different training stages.
Revised: With SAC-I, we advance the use of SAC methods by introducing the use of multiple value networks with respective episodic replay buffers, as well as distinct entropy parameter estimation for skills at different training stages.

Original: In Figure 9, we show SAC-I is able to successfully learn the mixed version of the BipedalWalkerHardcore-v3, otherwise unsolvable with the standard SAC.
Revised: In Figure 9, we show SAC-I is able to successfully learn the mixed version of the BipedalWalkerHardcore-v3, otherwise unsolvable with the standard SAC.

Original: We believe that the experiments and results presented in this paper offer a proof of concept of the advantages using SAC methods with inhibitory networks (SAC-I) for faster retraining.
Revised: We believe that the experiments and results presented in this paper offer a proof of concept of the advantages using SAC methods with inhibitory networks (SAC-I) for faster retraining.

Original: A A PPENDIX B ROADER I MPACTS
Revised: A A PPENDIX B ROADER I MPACTS

Original: Efﬁciency in retraining is useful in situations where training is computationally expensive or can cause damage and wear to instruments like robotic arms.
Revised: Efﬁciency in retraining is useful in situations where training is computationally expensive or can cause damage and wear to instruments like robotic arms.

Original: Our work proposes a method of efﬁcient retraining of agents by reusing trained policies and value function estimators.
Revised: Our work proposes a method of efﬁcient retraining of agents by reusing trained policies and value function estimators.

Original: Ultimately, we contribute to the body of work that prioritizes creative reuse of models.
Revised: Ultimately, we contribute to the body of work that prioritizes creative reuse of models.

Original: [Table]
Revised: [Table]

Original: S UPPLEMENTARY E XPERIMENTS AND R ESULTS
Revised: S UPPLEMENTARY E XPERIMENTS AND R ESULTS

Original: LunarLanderContinuous-v2 with bomb .
Revised: LunarLanderContinuous-v2 with bomb .

Original: The observation space consists of the original 8 states (related to position, angle, velocities and leg contact) plus 4 values indicating the upper left and bottom right coordinates of the bomb zone (a box with a width of ≈ 10% of the screen size).
Revised: The observation space consists of the original 8 states (related to position, angle, velocities and leg contact) plus 4 values indicating the upper left and bottom right coordinates of the bomb zone (a box with a width of ≈ 10% of the screen size).

Original: The agent assumes horizontal coordinates x ∈ [ − 1 , 1] and vertical coordinates y ∈ [0 , 1 . 4] , where y = 0 corresponds to the ground.
Revised: The agent assumes horizontal coordinates x ∈ [ − 1 , 1] and vertical coordinates y ∈ [0 , 1 . 4] , where y = 0 corresponds to the ground.

Original: The bomb is randomly placed with center coordinates x c ∈ [ − 0 . 2 , 0 . 2] and y c ∈ [0 . 1 , 0 . 5] , and appears with a default frequency of 50% (stop trials), which creates an environment with the highest uncertainty.
Revised: The bomb is randomly placed with center coordinates x c ∈ [ − 0 . 2 , 0 . 2] and y c ∈ [0 . 1 , 0 . 5] , and appears with a default frequency of 50% (stop trials), which creates an environment with the highest uncertainty.

Original: In the stop trials, the bomb appears randomly after the agent achieves an altitude between the range [0 . 9 , 1 . 1] .
Revised: In the stop trials, the bomb appears randomly after the agent achieves an altitude between the range [0 . 9 , 1 . 1] .

Original: Importantly, the bomb coordinates are included in the observation state only after the bomb appears so that the agent does not know about its existence beforehand.
Revised: Importantly, the bomb coordinates are included in the observation state only after the bomb appears so that the agent does not know about its existence beforehand.

Original: If the bomb is absent, four dummy negative values are provided in the state.
Revised: If the bomb is absent, four dummy negative values are provided in the state.

Original: The episode ends if the agent hits the bomb (i.e. agent inside the bomb zone) or the ground, successfully lands or reaches the maximum length of 1000 steps.
Revised: The episode ends if the agent hits the bomb (i.e. agent inside the bomb zone) or the ground, successfully lands or reaches the maximum length of 1000 steps.

Original: Conservative reward shaping .
Revised: Conservative reward shaping .

Original: In LunarLanderContinuousv2 with bomb, it is expected that how the inhibitory reward r
Revised: In LunarLanderContinuousv2 with bomb, it is expected that how the inhibitory reward r

Original: I is shaped, as described in the main text, inﬂuences the agents’ training.
Revised: I is shaped, as described in the main text, inﬂuences the agents’ training.

Original: Indeed, we tested a few shaping variants and observed differences.
Revised: Indeed, we tested a few shaping variants and observed differences.

Original: However, results comparing SACI vs. SAC were qualitatively similar to the ones presented so far.
Revised: However, results comparing SACI vs. SAC were qualitatively similar to the ones presented so far.

Original: One potential issue with the current reward shaping to avoid the bomb is that it does not consider any of the aspects involved in landing.
Revised: One potential issue with the current reward shaping to avoid the bomb is that it does not consider any of the aspects involved in landing.

Original: Therefore, we present another variant of the r I which includes some of the aspects of the raw reward such as rewarding getting close to the landing pad, slowing down, and penalizing “aggressive” maneuvers (large velocities and vertical angles).
Revised: Therefore, we present another variant of the r I which includes some of the aspects of the raw reward such as rewarding getting close to the landing pad, slowing down, and penalizing “aggressive” maneuvers (large velocities and vertical angles).

Original: The more conservative reward shaping is given by the sum of the following components
Revised: The more conservative reward shaping is given by the sum of the following components

Original: r I =
Revised: r I =

Original: r x + r y + r angle + r vel , where r x = − 1 / (6 × d x + 0 . 1) + 0 .
Revised: r x + r y + r angle + r vel , where r x = − 1 / (6 × d x + 0 . 1) + 0 .

Original: 77 , r y = − 3 ×
Revised: 77 , r y = − 3 ×

Original: ( d y − 0 . 05) 2 , r angle = − angle 2 , r vel = − 2 × ( v 2 x + v 2 y + v 2 angle ) , and ( d x , d y ) are the horizontal and vertical distances to the bomb, respectively.
Revised: ( d y − 0 . 05) 2 , r angle = − angle 2 , r vel = − 2 × ( v 2 x + v 2 y + v 2 angle ) , and ( d x , d y ) are the horizontal and vertical distances to the bomb, respectively.

Original: In short, the inhibitory reward has maximum value 0 and components with quadratic forms.
Revised: In short, the inhibitory reward has maximum value 0 and components with quadratic forms.

Original: Coefﬁcients are adjusted to have r I < 0 and a balance between the different rewards.
Revised: Coefﬁcients are adjusted to have r I < 0 and a balance between the different rewards.

Original: If the inhibition rule given by conditions d x < 0 .
Revised: If the inhibition rule given by conditions d x < 0 .

Original: 2 and y > y b is met, the Q-I network is used along with the inhibitory reward r I .
Revised: 2 and y > y b is met, the Q-I network is used along with the inhibitory reward r I .

Original: In this version, the bomb avoidance is driven by the horizontal distance d x , independent of d y , making the agent more conservative.
Revised: In this version, the bomb avoidance is driven by the horizontal distance d x , independent of d y , making the agent more conservative.

Original: Results are shown in Figure 10.
Revised: Results are shown in Figure 10.

Original: Unlike SAC, the SAC-I is beneﬁted by the shaping and it offers a mechanism to include additional shaping without interfering previously learned Q networks.
Revised: Unlike SAC, the SAC-I is beneﬁted by the shaping and it offers a mechanism to include additional shaping without interfering previously learned Q networks.

Original: BipedalWalker environments .
Revised: BipedalWalker environments .

Original: In both BipedalWalker-v3 and BipedalWalkerHardcore-v3 tasks, the primary goal is to reach the end of the track using the least amount of energy as possible.
Revised: In both BipedalWalker-v3 and BipedalWalkerHardcore-v3 tasks, the primary goal is to reach the end of the track using the least amount of energy as possible.

Original: The agent gets a dense reward for moving forward (total 300+ points up to the end) and -100 for falling.
Revised: The agent gets a dense reward for moving forward (total 300+ points up to the end) and -100 for falling.

Original: There is a small cost for applying motor torque and moving the joints, so agents with optimal movements score more.
Revised: There is a small cost for applying motor torque and moving the joints, so agents with optimal movements score more.

Original: The observation state has a total of 24 variables split between 14 related to the agent’s body (including the angles and velocities of hull, hip joints, knee joints, vertical and horizontal speeds) and 10 LIDAR sensor readings.
Revised: The observation state has a total of 24 variables split between 14 related to the agent’s body (including the angles and velocities of hull, hip joints, knee joints, vertical and horizontal speeds) and 10 LIDAR sensor readings.

Original: Everything the agent knows about the environment is through the sensors, so it does not know about obstacles or holes until they are scanned by the sensor, neither about its position in the environment since no coordinates are given.
Revised: Everything the agent knows about the environment is through the sensors, so it does not know about obstacles or holes until they are scanned by the sensor, neither about its position in the environment since no coordinates are given.

Original: Note that sensor readings are relative to the agent’s position and angle.
Revised: Note that sensor readings are relative to the agent’s position and angle.

Original: These characteristics make the hardcore version signiﬁcantly more challenging than the plain terrain version.
Revised: These characteristics make the hardcore version signiﬁcantly more challenging than the plain terrain version.

Original: The agent has 2 legs, each leg has 2 joints, and therefore the action space has size 4 with continuous values within the range of [ − 1 , 1] .
Revised: The agent has 2 legs, each leg has 2 joints, and therefore the action space has size 4 with continuous values within the range of [ − 1 , 1] .

Original: Each episode ends whenever the agent reaches the end of the track, falls or reaches the maximum of 2000 steps.
Revised: Each episode ends whenever the agent reaches the end of the track, falls or reaches the maximum of 2000 steps.

Original: Some informal discussion regarding the difﬁculty in solving the task can be found in https://ai.stackexchange.com/questions/13848/has-anyone-beenable-to-solve-openais-hardcore-bipedal-walker-with-their-implem .
Revised: Some informal discussion regarding the difﬁculty in solving the task can be found in https://ai.stackexchange.com/questions/13848/has-anyone-beenable-to-solve-openais-hardcore-bipedal-walker-with-their-implem .

Original: BipedalWalker experimental design .
Revised: BipedalWalker experimental design .

Original: The standard SAC agent is trained in the BipedalWalker-v to be used as a baseline agent in further training in the hardcore version.
Revised: The standard SAC agent is trained in the BipedalWalker-v to be used as a baseline agent in further training in the hardcore version.

Original: Agents are trained with three different seeds for 3K episodes, and the one with the best average reward is selected as the baseline.
Revised: Agents are trained with three different seeds for 3K episodes, and the one with the best average reward is selected as the baseline.

Original: This agent reaches an average over the last 100 episodes above 300+ in about 0.5M steps similar to the results reported in Wei & Ying (2021).
Revised: This agent reaches an average over the last 100 episodes above 300+ in about 0.5M steps similar to the results reported in Wei & Ying (2021).

Original: All the network weights are transferred to the retrained agent.
Revised: All the network weights are transferred to the retrained agent.

Original: The Q-I network is also retrained from previous weights.
Revised: The Q-I network is also retrained from previous weights.

Original: For the BipedalWalkerHardcore-v3, we train agents with ﬁve seeds for 5K episodes, and compute the average reward across runs.
Revised: For the BipedalWalkerHardcore-v3, we train agents with ﬁve seeds for 5K episodes, and compute the average reward across runs.

Original: Effect of removing the fall penalty .
Revised: Effect of removing the fall penalty .

Original: In neuroscience, punishments (negative rewards) are known to produce aversive conditioning or prediction of aversive events that leads to defensive or aggressive Pavlonian actions (Seymour et al., 2007), what in turn can cause fear of failure and procrastination in learning (Martin, 2012).
Revised: In neuroscience, punishments (negative rewards) are known to produce aversive conditioning or prediction of aversive events that leads to defensive or aggressive Pavlonian actions (Seymour et al., 2007), what in turn can cause fear of failure and procrastination in learning (Martin, 2012).

Original: Negative rewards are often successfully used in RL.
Revised: Negative rewards are often successfully used in RL.

Original: However, there are studies showing that intermixing positive and sparse negative rewards might have adverse effects in learning, as reported in a recent work using DDPG in robotic environment Vargas et al.
Revised: However, there are studies showing that intermixing positive and sparse negative rewards might have adverse effects in learning, as reported in a recent work using DDPG in robotic environment Vargas et al.

Original: Moreover, optimally shaping reward is still an open area of research (Hu et al., 2020), and it is likely to depend on the particular DRL method being used.
Revised: Moreover, optimally shaping reward is still an open area of research (Hu et al., 2020), and it is likely to depend on the particular DRL method being used.

Original: Interestingly, in our experiments in BipedalWalkerHardcorev3, we observe that the SAC agents are greatly beneﬁted from removing the fall penalty from the raw environment reward.
Revised: Interestingly, in our experiments in BipedalWalkerHardcorev3, we observe that the SAC agents are greatly beneﬁted from removing the fall penalty from the raw environment reward.

Original: We observe that, with the fall penalty, often the agent learns to simply stop in front of the block by fear of falling (which is likely a local minimum), never exploring enough to overcome the large obstacles.
Revised: We observe that, with the fall penalty, often the agent learns to simply stop in front of the block by fear of falling (which is likely a local minimum), never exploring enough to overcome the large obstacles.

Original: In Figure 11, we illustrate the effects of removing the fall penalty and including the stuck penalty.
Revised: In Figure 11, we illustrate the effects of removing the fall penalty and including the stuck penalty.

Original: Removing the fall penalty (-100) included in original reward/penalty provided by the environment allowed the agent to explore riskier actions to overcome obstacles.
Revised: Removing the fall penalty (-100) included in original reward/penalty provided by the environment allowed the agent to explore riskier actions to overcome obstacles.

Original: The
Revised: The

Original: SAC agent without stuck penalty is also unable to learn the task successfully (purple line).
Revised: SAC agent without stuck penalty is also unable to learn the task successfully (purple line).

Original: It should be noted that retraining from a baseline agent is also critical to speed-up training and make possible solving the task.
Revised: It should be noted that retraining from a baseline agent is also critical to speed-up training and make possible solving the task.

Original: Average reward represents the average over the last 100 episodes of the raw environmental reward.
Revised: Average reward represents the average over the last 100 episodes of the raw environmental reward.

Original: Training from scratch in the mixed version of the BipedalWalkerHardcore-v3 .
Revised: Training from scratch in the mixed version of the BipedalWalkerHardcore-v3 .

Original: In Figure 12, we show the average reward during training for the SAC and SAC-I agents, in the environment with 90% stop trials.
Revised: In Figure 12, we show the average reward during training for the SAC and SAC-I agents, in the environment with 90% stop trials.

Original: Training the SAC-I agent from scratch is slower than retraining from the baseline agent (Figure 10, Stop trial 90%).
Revised: Training the SAC-I agent from scratch is slower than retraining from the baseline agent (Figure 10, Stop trial 90%).

Original: However, the SAC-I agent is still able to solve the hardcore task (Stop trials).
Revised: However, the SAC-I agent is still able to solve the hardcore task (Stop trials).

Original: Meanwhile, the SAC agent is only able to solve the simple task (Go trials).
Revised: Meanwhile, the SAC agent is only able to solve the simple task (Go trials).

Original: The advantage of retraining is evident during the initial training steps.
Revised: The advantage of retraining is evident during the initial training steps.

Original: The retrained SAC-I agent completes the simple task from the beginning of the training (compare with Figure 10, Stop trial 90%).
Revised: The retrained SAC-I agent completes the simple task from the beginning of the training (compare with Figure 10, Stop trial 90%).

Original: The SAC-I agent trained from scratch learns to complete the Go trials only after 1M steps.
Revised: The SAC-I agent trained from scratch learns to complete the Go trials only after 1M steps.