Original:  Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping
Revised: Exploit Reward Shifting in Value-Based Deep-RL: Optimistic Curiosity-Based Exploration and Conservative Exploitation via Linear Reward Shaping

Original: Abstract
Revised: Abstract

Original: In this work, we study the simple yet universally applicable case of reward shaping in value-based Deep Reinforcement Learning (DRL).
Revised: In this work, we study the simple yet universally applicable case of reward shaping in value-based Deep Reinforcement Learning (DRL).

Original: We show that reward shifting in the form of a linear transformation is equivalent to changing the initialization of the Q -function in function approximation.
Revised: We show that reward shifting in the form of a linear transformation is equivalent to changing the initialization of the Q -function in neural approximation.

Original: Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.

Original: Accordingly, conservative exploitation improves offline RL value estimation, and optimistic value estimation improves exploration for online RL.
Revised: Accordingly, conservative exploitation improves offline RL value estimation, and optimistic value estimation improves exploration for online RL.

Original: We validate our insight on a range of RL tasks and show its improvement over baselines:
Revised: We validate our insight on a range of RL tasks and show its improvement over baselines:

Original: (1) In offline RL, the conservative exploitation leads to improved performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to tackle the exploration-exploitation dilemma for better sample efficiency; (3) In discrete control tasks, a negative reward shifting yields an improvement over the curiosity-based exploration method. 
Revised: (1) In offline RL, the conservative exploitation leads to improved performance based on offthe-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to tackle the exploration-exploitation dilemma for better sample efficiency; (3) In discrete control tasks, a negative reward shifting yields an improvement over the curiosity-based exploration method. https://sites.google.com/view/rewardshaping

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Revised: , reward shifting can be seamlessly applied to high-dim tasks including continuous control.

Original:  shaping  in reinforcement learning and has a long-standing history [1, 2], specifying a certain reward to incentivize the learning agent requires domain knowledge and a thorough understanding of the task [3–6].
Revised: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2], specifying a certain reward to incentivize the learning agent requires domain knowledge and a thorough understanding of the task [3–6].

Original: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].
Revised: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].

Original: Although Ng et al.
Revised: Although Ng et al.

Original: [9] theoretically points out that optimal policy remains unchanged under certain form of reward transformation, and in the later work of Wiewiora et al.
Revised: [9] theoretically points out that optimal policy remains unchanged under certain form of reward transformation, and in the later work of Wiewiora et al.

Original: [10] a framework is proposed to guide policies with prior knowledge under tabular setting, how reward shifting accommodates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.
Revised: [10] a framework is proposed to guide policies with prior knowledge under tabular setting, how reward shifting accommodates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.

Original: In this work, we study a special linear transformation, which is the simplest form of reward shaping, in value-based DRL [11–14].
Revised: In this work, we study a special linear transformation, which is the simplest form of reward shaping, in value-based DRL [11–14].

Original: We start with understanding how such a specific kind of reward shaping works in value-based DRL function approximations and show that reward shifting is equivalent to different Q -value initialization, extending previous discovery of [10] to the function approximation setting.
Revised: We start with understanding how such a specific kind of reward shaping works in value-based DRL function approximations and show that reward shifting is equivalent to different Q -value initialization, extending previous discovery of [10] to the function approximation setting.

Original: Figure 1 showcases reward shifting benefits exploration in a maze game and outperforms count-based and ϵ -greedy exploration and learns near optimal value function.
Revised: Figure 1 showcases reward shifting benefits exploration in a maze game and outperforms count-based and ϵ -greedy exploration and learns near optimal value function.

Original: Based on such an equivalence, we introduce the key insight of this work:
Revised: Based on such an equivalence, we introduce the key insight of this work:

Original: A positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to  curiosity-driven exploration.
Revised: A positive reward shifting leads to conservative exploitation, and a negative reward shifting leads to (a new type 2 of) curiosity-driven exploration.

Original: We demonstrate the application of such an insight in three Deep-RL scenarios: (S1) for offline RL , we show that conservative exploitation induced by reward shifting improves learning performance of off-the-shelf algorithms; (S2) for online RL setting , we show multiple value functions with different reward shifting constants can be used to trade-off between exploration and exploitation, thus improve learning efficiency; (S3) for curiosity-driven exploration, we introduce a simple yet crucial adaptation on a prevailing curiosity-based exploration algorithm, the Random Network
Revised: We demonstrate the application of such an insight in three Deep-RL scenarios: (S1) for offline RL , we show that conservative exploitation induced by reward shifting improves learning performance of off-the-shelf algorithms; (S2) for online RL setting , we show multiple value functions with different reward shifting constants can be used to trade-off between exploration and exploitation, thus improve learning efficiency; (S3) for curiosity-driven exploration, we introduce a simple yet crucial adaptation on a prevailing curiosity-based exploration algorithm, the Random Network

Original: Distillation [15], making it compatible with value-based DRL algorithms.
Revised: Distillation [15], making it compatible with value-based DRL algorithms.

Original: Experiments on a diverse set of tasks including continuous and discrete action space control show our method brings substantial improvement over baselines.
Revised: Experiments on a diverse set of tasks including continuous and discrete action space control show our method brings substantial improvement over baselines.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: Analytically, we introduce the key insight that reward shifting is equivalent to diversified Q -value network initialization in value-based DRL, which can be applied to curiosity-driven exploration and conservative exploitation; 2.
Revised: Analytically, we introduce the key insight that reward shifting is equivalent to diversified Q -value network initialization in value-based DRL, which can be applied to curiosity-driven exploration and conservative exploitation; 2.

Original: Practically, we instantiate the key insight to three different scenarios, namely the offline conservative exploitation, sample-efficient continuous control, and curiosity-driven exploration, to contrast the generality of reward shifting; 3.
Revised: Practically, we instantiate the key insight to three different scenarios, namely the offline conservative exploitation, sample-efficient continuous control, and curiosity-driven exploration, to contrast the generality of reward shifting; 3.

Original: Empirically, we demonstrate the effectiveness of the proposed method integrated with multiple off-the-shelf baselines on both continuous and discrete control tasks.
Revised: Empirically, we demonstrate the effectiveness of the proposed method integrated with multiple off-the-shelf baselines on both continuous and discrete control tasks.

Original:  Preliminaries and Related Work 2.1 Online RL
Revised:  Preliminaries and Related Work 2.1 Online RL

Original: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ρ 0 , γ, T } , where S ⊂ R d denotes the d -dim state space, A is the action space (note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ), we consider a deterministic transition dynamics T :
Revised: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ρ 0 , γ, T } , where S ⊂ R d denotes the d -dim state space, A is the action space (note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ), we consider a deterministic transition dynamics T :

Original: S×A (cid:55)→ S and deterministic reward function R : S×A (cid:55)→ R .
Revised: S×A (cid:55)→ S and deterministic reward function R : S×A (cid:55)→ R .

Original: ρ 0 = p ( s 0 )
Revised: ρ 0 = p ( s 0 )

Original: ∈ ∆ ( S ) denotes the initial state distribution.
Revised: ∈ ∆ ( S ) denotes the initial state distribution.

Original: γ is the discount factor and T is the episodic decision horizon.
Revised: γ is the discount factor and T is the episodic decision horizon.

Original: Online RL considers the problem of learning a policy π ∈ Π : S (cid:55)→ ∆ ( A ) (or π ∈ Π : S (cid:55)→
Revised: Online RL considers the problem of learning a policy π ∈ Π : S (cid:55)→ ∆ ( A ) (or π ∈ Π : S (cid:55)→

Original: A with a deterministic policy class), such that the expected cumulative reward E a t ∼ π,s t +1 ∼T ,s 0 ∼ ρ 0 (cid:80)
Revised: A with a deterministic policy class), such that the expected cumulative reward E a t ∼ π,s t +1 ∼T ,s 0 ∼ ρ 0 (cid:80)

Original: Tt =

Revised: Tt =

Original: γ t r t ( s t , a t ) in the Markovian decision process is maximized.
Revised: γ t r t ( s t , a t ) in the Markovian decision process is maximized.

Original: In online RL setting, an agent learns through trials and errors [11], through either on-policy [16–18] or off-policy paradigm [12–14, 19–21].
Revised: In online RL setting, an agent learns through trials and errors [11], through either on-policy [16–18] or off-policy paradigm [12–14, 19–21].

Original: In this work, we focus on the off-policy value-based methods which are in general more sample efficient.
Revised: In this work, we focus on the off-policy value-based methods which are in general more sample efficient.

Original: Specifically, our discussions assume the policy learning is based on a learned Q -value function that approximates the cumulative reward an agent can gain in the following part of an episode.
Revised: Specifically, our discussions assume the policy learning is based on a learned Q -value function that approximates the cumulative reward an agent can gain in the following part of an episode.

Original: Formally, Q ( s t , a t ) =
Revised: Formally, Q ( s t , a t ) =

Original: E π, T (cid:80) Tτ =
Revised: E π, T (cid:80) Tτ =

Original: t γ t r ( s τ , a τ ) , and can be estimated by propagating the Bellman operator B Q ( s, a ) =
Revised: t γ t r ( s τ , a τ ) , and can be estimated by propagating the Bellman operator B Q ( s, a ) =

Original: r ( s, a ) + γ E Q ( s ′ , a ′ ) .
Revised: r ( s, a ) + γ E Q ( s ′ , a ′ ) .

Original: For value-based methods, the (soft-)optimal policy is then produced by
Revised: For value-based methods, the (soft-)optimal policy is then produced by

Original: [Equation]
Revised: [Equation]

Original: ) where Q ∗ is the optimal Q -value function.
Revised: (1) where Q ∗ is the optimal Q -value function.

Original: We can also set the temperature parameter close toto have the deterministic policy class.
Revised: We can also set the temperature parameter close toto have the deterministic policy class.

Original: Simplifying the notion we have π ∗ ( s )
Revised: Simplifying the notion we have π ∗ ( s )

Original: = arg max a Q ∗ ( s, a ) .
Revised: = arg max a Q ∗ [Equation] s, a ) .

Original: Algorithms like DPG [22] can be used to address the intractable analytical argmax issue that arises in continuous action space.
Revised: Algorithms like DPG [22] can be used to address the intractable analytical argmax issue that arises in continuous action space.

Original: We choose to develop our work on top of prevailing baseline algorithms of
Revised: We choose to develop our work on top of prevailing baseline algorithms of

Original:  DQN [13], BCQ [23], CQL [24] and TD3 [14] as a minimal example to isolate the source of gains.
Revised: [Table] DQN [13], BCQ [23], CQL [24] and TD3 [14] as a minimal example to isolate the source of gains.

Original: It should be straightforward to extend the method on top of other learning algorithms.
Revised: It should be straightforward to extend the method on top of other learning algorithms.

Original: [Table]

Original: 2.2 Exploration and the Curiosity-Driven Methods
Revised: 2.2 Exploration and the Curiosity-Driven Methods

Original: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that the agent must simultaneously learn to exploit its accumulated knowledge on the task and explore new states and actions.
Revised: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that the agent must simultaneously learn to exploit its accumulated knowledge on the task and explore new states and actions.

Original: Previous works address the exploration problem from various perspectives: for discrete action space tasks, count-based methods like [28–30] are proposed to motivate the revisiting of under-explored state-action pairs.
Revised: Previous works address the exploration problem from various perspectives: for discrete action space tasks, count-based methods like [28–30] are proposed to motivate the revisiting of under-explored state-action pairs.

Original: Specifically, Choshen et al.
Revised: Specifically, Choshen et al.

Original: [31] extended the idea into general settings by constructing an additional MDP for Exploration-value estimation, as a generalized counter for count-based exploration.
Revised: [31] extended the idea into general settings by constructing an additional MDP for Exploration-value estimation, as a generalized counter for count-based exploration.

Original: To boost exploration in continuous state tasks, curiosity-driven methods are investigated by Houthooft et al.
Revised: To boost exploration in continuous state tasks, curiosity-driven methods are investigated by Houthooft et al.

Original: [32], Pathak et al.
Revised: [32], Pathak et al.

Original: [33], Burda et al.
Revised: [33], Burda et al.

Original: [34, 15], where variety of intrinsic rewards are designed as supplementary to the primal task reward for better exploration.
Revised: [34, 15], where variety of intrinsic rewards are designed as supplementary to the primal task reward for better exploration.

Original: Selfimitate approaches like Oh et al.
Revised: Selfimitate approaches like Oh et al.

Original: [35], Ecoffet et al.
Revised: [35], Ecoffet et al.

Original: [36], Sun et al.
Revised: [36], Sun et al.

Original: [37] repeat success trajectories
Revised: [37] repeat success trajectories

Original: but require extra assumptions on the environment.
Revised: but require extra assumptions on the environment.

Original: DORA [31] constructed an additional MDP to estimate the Exploration-value as a generalised counter for count-based exploration, yet those count-based methods are orthogonal to reward shifting: in intrinsic reward methods, an agent must first experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hard with an arg-max style policy.
Revised: DORA [31] constructed an additional MDP to estimate the Exploration-value as a generalised counter for count-based exploration, yet those count-based methods are orthogonal to reward shifting: in intrinsic reward methods, an agent must first experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hard with an arg-max style policy.

Original: On the other hand, with optimistic initialization, the rarely-visited ( s, a ) pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairs have updated their values with a negatively shifted reward.
Revised: On the other hand, with optimistic initialization, the rarely-visited ( s, a ) pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairs have updated their values with a negatively shifted reward.

Original: From such a perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods to encourage the first visitation of new states.
Revised: From such a perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods to encourage the first visitation of new states.

Original: The works of DIAYN and DADS [38, 39] show that various skills can be developed even without the primal extrinsic reward.
Revised: The works of DIAYN and DADS [38, 39] show that various skills can be developed even without the primal extrinsic reward.

Original: For continuous control tasks, OAC [27] improves the SAC [20] with informative action space noise based on the optimism in face of uncertainty (OFU)
Revised: For continuous control tasks, OAC [27] improves the SAC [20] with informative action space noise based on the optimism in face of uncertainty (OFU)

Original: GAC [44] addresses the exploration issue with a richer functional class for the policy.
Revised: GAC [44] addresses the exploration issue with a richer functional class for the policy.

Original: In the work of Rashid et al.
Revised: In the work of Rashid et al.

Original: [26], the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on specific settings of tabular and discrete control exploration.
Revised: [26], the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on specific settings of tabular and discrete control exploration.

Original: In the work of Osband et al.
Revised: In the work of Osband et al.

Original: [45, 25], ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet those methods are only applicable to discrete control tasks.
Revised: [45, 25], ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet those methods are only applicable to discrete control tasks.

Original: Noted that although the reward shifting can be regarded as a special case of these random priors, it can be distinguished by not changing the optimal
Revised: Noted that although the reward shifting can be regarded as a special case of these random priors, it can be distinguished by not changing the optimal

Original: Q -value, and its flexibility to be plugged into both continuous and discrete control algorithms.
Revised: Q -value, and its flexibility to be plugged into both continuous and discrete control algorithms.

Original: Random Network Distillation (RND) [15] defines the intrinsic reward as the output difference between a fixed neural network ϕ 1 and a trainable network ϕ 2 given state-actions as the inputs.
Revised: Random Network Distillation (RND) [15] defines the intrinsic reward as the output difference between a fixed neural network ϕ 1 and a trainable network ϕ 2 given state-actions as the inputs.

Original: e.g.,
Revised: e.g.,

Original: [Equation]
Revised: [Equation]

Original: where both networks are activated by a sigmoid function.
Revised: where both networks are activated by a sigmoid function.

Original: After optimizing the learnable ϕ 2 to approximate ϕ 1 with seen ( s, a ) pairs 2 , the value of r int ( s, a ) will decay to 0 for such state-action pairs that are frequently visited but remain high for thoase are seldom visited.
Revised: After optimizing the learnable ϕ 2 to approximate ϕ 1 with seen ( s, a ) pairs 3 , the value of r int ( s, a ) will decay to 0 for such state-action pairs that are frequently visited but remain high for thoase are seldom visited.

Original: In this work, we show that exploratory behavior can be achieved simply by shifting the reward function with a constant.
Revised: In this work, we show that exploratory behavior can be achieved simply by shifting the reward function with a constant.

Original: Thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by high function approximation error in the under-explored state-action pairs.
Revised: Thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by high function approximation error in the under-explored state-action pairs.

Original: We demonstrate such insight by showing that RND in its original design is not suitable for value-based methods in developing exploratory behaviors, but integrating
Revised: We demonstrate such insight by showing that RND in its original design is not suitable for value-based methods in developing exploratory behaviors, but integrating

Original: RND with a shifted reward function remarkably improves the learning performance.
Revised: RND with a shifted reward function remarkably improves the learning performance.

Original: 2.3 Offline RL
Revised: 2.3 Offline RL

Original: The offline RL, also known as batch-RL, focuses on the problems where interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset.
Revised: The offline RL, also known as batch-RL, focuses on the problems where interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset.

Original: In those tasks, a fixed buffer B = { s i , a i , r i , s ′ i } i =[ N ] , collected from some unknown behavior policy π β , is provided.
Revised: In those tasks, a fixed buffer B = { s i , a i , r i , s ′ i } i =[ N ] , collected from some unknown behavior policy π β , is provided.

Original: In general, such a dataset can either be generated by rolling out an expert that generates high-quality solutions to the task [23, 46, 47] or a non-expert that executes sub-optimal behaviors [47–51] or be a mixture of both [24].
Revised: In general, such a dataset can either be generated by rolling out an expert that generates high-quality solutions to the task [23, 46, 47] or a non-expert that executes sub-optimal behaviors [47–51] or be a mixture of both [24].

Original: As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation error [23] induced by distributional mismatch [49].
Revised: As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation error [23] induced by distributional mismatch [49].

Original: To address such an issue, a series of algorithms optimize the policy learning under the constraint of distributional similarity [48, 49, 52, 53].
Revised: To address such an issue, a series of algorithms optimize the policy learning under the constraint of distributional similarity [48, 49, 52, 53].

Original: Bharadhwaj et al.
Revised: Bharadhwaj et al.

Original: [24] propose CQL to solve the offline RL tasks with a conservative value estimation.
Revised: [24] propose CQL to solve the offline RL tasks with a conservative value estimation.

Original: Specifically, CQL learns the Q -value estimation by jointly maximizing the Q -values of actions sampled from the behavior offline dataset and minimizing the Q -values of actions sampled with pre-defined prior distributions (e.g., uniform distribution over the action space).
Revised: Specifically, CQL learns the Q -value estimation by jointly maximizing the Q -values of actions sampled from the behavior offline dataset and minimizing the Q -values of actions sampled with pre-defined prior distributions (e.g., uniform distribution over the action space).

Original: As we will show in this work, an alternative approach to have a lower bound for the optimal Q -value function is to use an appropriately shifted reward function.
Revised: As we will show in this work, an alternative approach to have a lower bound for the optimal Q -value function is to use an appropriately shifted reward function.

Original: This idea leads to the direct application of our proposed framework in the offline setting.
Revised: This idea leads to the direct application of our proposed framework in the offline setting.

Original: In general, reward shift can be plugged in many distribution-matching offline-RL algorithms [23, 48, 49, 52] to further improve the performance with conservative Q -value estimation.
Revised: In general, reward shift can be plugged in many distribution-matching offline-RL algorithms [23, 48, 49, 52] to further improve the performance with conservative Q -value estimation.

Original: Table 1 contextualizes reward shifting with respect to related works we discussed above.
Revised: Table 1 contextualizes reward shifting with respect to related works we discussed above.

Original:  A Motivating Example
Revised:  A Motivating Example

Original: We start with two intuitive remarks and a “counter-intuitive” motivating example in this section before introducing our method.
Revised: We start with two intuitive remarks and a “counter-intuitive” motivating example in this section before introducing our method.

Original: Remark 1.
Revised: Remark 1.

Original: Given an MDP M = {S , A , T , R , ρ 0 , γ } , where |A| < ∞ , scaling the reward function with linear transformation, i.e., R k,b
Revised: Given an MDP M = {S , A , T , R , ρ 0 , γ } , where |A| < ∞ , scaling the reward function with linear transformation, i.e., R k,b

Original: = k ·R + b , ∀ k > 0 , b ∈ R , such that r ′ t =
Revised: = k ·R + b , ∀ k > 0 , b ∈ R , such that r ′ t =

Original: kr t + b ∈ R k,b , does not change the optimal policy induced by the corresponding value function Q ∗ k,b ( s, a ) :
Revised: kr t + b ∈ R k,b , does not change the optimal policy induced by the corresponding value function Q ∗ k,b [Equation]

Original: = (cid:80) t γ t r ′ t :
Revised: t [Equation]

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: When |A| = ∞ , scaling the reward function with linear transformation does not change the optimal policy induced by deterministic policy gradient [22], given proper learning rate η ′ =η/k : ∇ θ J ( µ θ ) = E s t [ ∇ [Equation]
Revised: When |A| = ∞ , scaling the reward function with linear transformation does not change the optimal policy induced by deterministic policy gradient [22], given proper learning rate η ′ =

Original: Q

Original: [Equation] ( s t , a t ) | a t = µ [Equation] ∇
Revised: η/k : ∇

Original: [Equation]
Revised: [Equation]

Original: s t

Original: [Equation]

Original: [Equation] [Equation] [Equation] Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimal policy induced by the optimal Q -value function calculated by the shifted reward.
Revised: Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimal policy induced by the optimal Q -value function calculated by the shifted reward.

Original: However, Figurepresents “counter-intuitive” results in a demonstrative exploration task of Grid World.
Revised: However, Figurepresents “counter-intuitive” results in a demonstrative exploration task of Grid World.

Original: In this task, an agent located at the upper left corner of a map needs to explore without reward and reach the goal point located at the lower right corner.
Revised: In this task, an agent located at the upper left corner of a map needs to explore without reward and reach the goal point located at the lower right corner.

Original: A non-trivial reward of +1 will be assigned only when the goal is reached.
Revised: A non-trivial reward of +1 will be assigned only when the goal is reached.

Original: We report learning curves with regard to the episodic success rate in reaching the goal point and the learned Q -values.
Revised: We report learning curves with regard to the episodic success rate in reaching the goal point and the learned Q -values.

Original: In this toy example, we find a negative reward shifting remarkably boosts the learning efficiency of Q -learning, and surpasses the conventional count-based method for exploration.
Revised: In this toy example, we find a negative reward shifting remarkably boosts the learning efficiency of Q -learning, and surpasses the conventional count-based method for exploration.

Original: Moreover, Remark 1 is empirically verified with such a toy example: reward shifting does not change the optimal Q -value as well as its induced policy, but on the contrary accelerates the discovery of (near-)optimal policy.
Revised: Moreover, Remark 1 is empirically verified with such a toy example: reward shifting does not change the optimal Q -value as well as its induced policy, but it does change the learning dynamics and accelerates the discovery of (near-)optimal policy.

Original: In the following of this work, we study the effect of varying constant bias b and fix the scaling factor k = 1 to avoid trivial discussions on the choices of learning rate — there should be no surprise thatchoosing an appropriate learning rate is empirically important.
Revised: In the following of this work, we study the effect of varying constant bias b and fix the scaling factor k = 1 to avoid trivial discussions on the choices of learning rate — there should be no surprise that

Original: We focus on revealing the importance of selecting the universal bias term b in the reward function through the lens of initialization priors in function approximation and show such a bias is generally helpful for both online and offline

Original: 𝑸 ∗ 𝑸 𝓐 𝑏 ! 1 − 𝛾 𝑸 " !
Revised: 𝑸 ∗ 𝑸 𝓐 𝑏 ! 1 − 𝛾 𝑸 " !

Original: ∗ =
Revised: ∗ =

Original: 𝑸 ∗ + " !
Revised: 𝑸 ∗ + " !

Original: $%& "𝑸 !
Revised: $%& "𝑸 !

Original: ≈

Revised: ≈

Original: 𝐚 𝓐 𝑸 ’𝑸 ’ − 𝑸 ∗ < ’𝑸 ’%$ −𝑸 ∗ 𝑭: ’𝑸 ’ → 𝑸 ∗ | s,a,r,s ( )
Revised: 𝐚 𝓐 𝑸 ’𝑸 ’ − 𝑸 ∗ < ’𝑸 ’%$ −𝑸 ∗ 𝑭: ’𝑸 ’ → 𝑸 ∗ | s,a,r,s ( )

Original: 𝐛 𝑸 𝓐 $𝑸 " ! ,$ −𝑸 " !
Revised: 𝐛 𝑸 𝓐 $𝑸 " ! ,$ −𝑸 " !

Original: ∗ < $𝑸 " ! ,$%& −𝑸 " !
Revised: ∗ < $𝑸 " ! ,$%& −𝑸 " !

Original: ∗ 𝑭: ’𝑸 " ! ,’ → 𝑸 " !
Revised: ∗ 𝑭: ’𝑸 " ! ,’ → 𝑸 " !

Original: ∗ | s,a,r,s ( ) 𝐜 − 𝑏 !
Revised: ∗ | s,a,r,s ( ) 𝐜 − 𝑏 !

Original: 1 − 𝛾 𝑸 𝓐’𝑸
Revised: 1 − 𝛾 𝑸 𝓐’𝑸

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  settings.
Revised: choosing an appropriate learning rate is empirically important. We focus on revealing the importance of selecting the universal bias term b in the reward function through the lens of initialization priors in function approximation and show such a bias is generally helpful for both online and offline settings.

Original: In the online settings, it improves learning efficiency, and in the offline settings, it promotes conservative exploitation.
Revised: In the online settings, it improves learning efficiency, and in the offline settings, it promotes conservative exploitation.

Original:  Shifted Priors for Q -value Estimation 4.1 Reward Shift Equals to Different Initialization
Revised:  Shifted Priors for Q -value Estimation 4.1 Reward Shift Equals to Different Initialization

Original: We start by formally introducing notions and the key idea of this work: reward shifting equals different initialization.
Revised: We start by formally introducing notions and the key idea of this work: reward shifting equals different initialization.

Original: We use Figure 2 to illustrate how reward shifting affects function approximation, hence changing the learning dynamics for value-based algorithms.
Revised: We use Figure 2 to illustrate how reward shifting affects function approximation, hence changing the learning dynamics for value-based algorithms.

Original: The original optimal Q -value function is denoted as Q ∗ , and plotted in the figures as Black curves.
Revised: The original optimal Q -value function is denoted as Q ∗ , and plotted in the figures as Black curves.

Original: We then denote the shifted optimal
Revised: We then denote the shifted optimal

Original: Q -value function as Q ∗ b + , which is the Q -value function with the shifted reward r ′ =
Revised: Q -value function as Q ∗ b + , which is the Q -value function with the shifted reward r ′ =

Original: r + b + .
Revised: r + b + .

Original: We use Red curves in figures to denote those shifted Q -value functions.
Revised: We use Red curves in figures to denote those shifted Q -value functions.

Original: In this section we provide analysis with a positive bias b + > 0 to illustrate how positively shifted value function affects function approximation, and coordinates the normally applied near-zero initialized function approximators (e.g., neural networks.
Revised: In this section we provide analysis with a positive bias b + > 0 to illustrate how positively shifted value function affects function approximation, and coordinates the normally applied near-zero initialized function approximators (e.g., neural networks.

Original: Orange curves in the figure) to inspire conservative behaviors.
Revised: Orange curves in the figure) to inspire conservative behaviors.

Original: The discussion of negative biases is elaborated in the next section.
Revised: The discussion of negative biases is elaborated in the next section.

Original: To summarize, shifting the reward function with a positive constant is equivalent to initializing the value network with a smaller value — as the Q -value of unseen state-action pairs during training are much lower than their shifted optimal values, those actions will not be selected in argmax-style policy updates — leading to conservative learning behaviors that benefit policy learning in offline settings
Revised: To summarize, shifting the reward function with a positive constant is equivalent to initializing the value network with a smaller value — as the Q -value of unseen state-action pairs during training are much lower than their shifted optimal values, those actions will not be selected in argmax-style policy updates — leading to conservative learning behaviors that benefit policy learning in offline settings.

Original: . 4.2 (S1) Offline RL: Conservative Exploitation
Revised: 4.2 (S1) Offline RL: Conservative Exploitation

Original: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emerge with a positively shifted reward.
Revised: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emerge with a positively shifted reward.

Original: And such a value estimation empirically lower-bounds the optimal
Revised: And such a value estimation empirically lower-bounds the optimal

Original: Q -value function.
Revised: Q -value function.

Original: As shown in Figure 2(a), a positive constant b + added to the reward function will lead to a universal positively shifted optimal Q -value function, and the gap between the primal
Revised: As shown in Figure 2(a), a positive constant b + added to the reward function will lead to a universal positively shifted optimal Q -value function, and the gap between the primal

Original: Q -value function and the new one is b + 1 − γ .
Revised: Q -value function and the new one is b + 1 − γ .

Original: Optimizing the Q -value approximator with logged data will minimize the difference between the predicted value and the optimal value with observed data.
Revised: Optimizing the Q -value approximator with logged data will minimize the difference between the predicted value and the optimal value with observed data.

Original: Forthe unobserved data, the pessimistically initialized Q -value approximator guarantees the prediction is lower than the positively shifted optimal ones, thus conservative exploitation can be achieved with argmax-style value propagation and action execution on such a value function.
Revised: For

Original: − 𝑏 + 1−𝛾 𝑸
Revised: − 𝑏 + 1−𝛾 𝑸

Original: 𝓐
Revised: 𝓐

Original: ෩𝑸
Revised: ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐
Revised: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐

Original: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =
Revised: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =

Original: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 

Revised: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸

Original: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸
Revised: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸

Original: UB,𝑡
Revised: UB,𝑡

Original: =
Revised: =

Original: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾 ෩𝑸
Revised: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾 ෩𝑸

Original: UB,𝑡 ≥𝑸 ∗ ෩𝑸
Revised: UB,𝑡 ≥𝑸 ∗ ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: ≤𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏
Revised: ≤𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏

Original: + 1−𝛾 
Revised: + 1−𝛾 the unobserved data, the pessimistically initialized Q -value approximator guarantees the prediction is lower than the positively shifted optimal ones, thus conservative exploitation can be achieved with argmax-style value propagation and action execution on such a value function.

Original: 4.3 (Curiosity-Driven) Optimistic Exploration
Revised: 4.3 (Curiosity-Driven) Optimistic Exploration

Original: On the other hand, if we shift the reward function to the negative side, it is equivalent to an optimistic initialization.
Revised: On the other hand, if we shift the reward function to the negative side, it is equivalent to an optimistic initialization.

Original: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.
Revised: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.

Original: With sufficiently small b − (so that | b − | larger than the maximal value of any ( s, a ) -pair), such an upper bound of Q ∗ can be used to conduct curiosity-driven exploration.
Revised: With sufficiently small b − (so that | b − | larger than the maximal value of any ( s, a ) -pair), such an upper bound of Q ∗ can be used to conduct curiosity-driven exploration.

Original: Intuitively, initializing a value network that always predicts a value larger than the true optimal value will lead to curiositydriven exploration behavior, as any visited state will be assigned a relatively smaller value and the argmax-style policy will tend to choose under-explored actions.
Revised: Intuitively, initializing a value network that always predicts a value larger than the true optimal value will lead to curiositydriven exploration behavior, as any visited state will be assigned a relatively smaller value and the argmax-style policy will tend to choose under-explored actions.

Original: Based on the analysis above that (1) a positive constant shift added to the reward function can be used for conservative policy updates and (2) a negative constant shift added to the reward function can be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., the optimistic estimation with b − , and the lower bound, i.e., the conservative estimation with b + of the optimal value function.
Revised: Based on the analysis above that (1) a positive constant shift added to the reward function can be used for conservative policy updates and (2) a negative constant shift added to the reward function can be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., the optimistic estimation with b − , and the lower bound, i.e., the conservative estimation with b + of the optimal value function.

Original: Formally, we use
Revised: Formally, we use

Original: [Equation]
Revised: [Equation]

Original:  to denote the empirical lower bound of value estimation (cf.
Revised: 5) to denote the empirical lower bound of value estimation (cf.

Original: Figure 2(d), Orange line), and
Revised: Figure 2(d), Orange line), and

Original: [Equation]
Revised: [Equation]

Original:  to denote the empirical upper bound of value estimation (cf.
Revised: 6) to denote the empirical upper bound of value estimation (cf.

Original: Figure 3(b), Orange line).
Revised: Figure 3(b), Orange line).

Original: In both notions, t denotes the optimization step.
Revised: In both notions, t denotes the optimization step.

Original: When t = 0 , with a near-zero initialization ˜ Q 0 ≈ 0 , ˜ Q LB , 0 ( s, a ) = − b + 1 − γ is able to lower bound the unknown optimal Q -value given sufficiently large b + .
Revised: When t = 0 , with a near-zero initialization ˜ Q 0 ≈ 0 , ˜ Q LB , 0 ( s, a ) = − b + 1 − γ is able to lower bound the unknown optimal Q -value given sufficiently large b + .

Original: (cf.
Revised: (cf.

Original: Figure 2(d), Green line).
Revised: Figure 2(d), Green line).

Original: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufficiently small b − .
Revised: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufficiently small b − .

Original: Following those notions, we introduce our sample-efficient algorithms for both continuous control and discrete action space respectively.
Revised: Following those notions, we introduce our sample-efficient algorithms for both continuous control and discrete action space respectively.

Original: We propose a practical algorithm for sample-efficient continuous control in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the
Revised: We propose a practical algorithm for sample-efficient continuous control in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the

Original: RND [15], in Section 4.3.2.
Revised: RND [15], in Section 4.3.2.

Original: 4.3.1 (S2) Online RL: Trading-off Exploration and Exploitation with Reward Shift
Revised: 4.3.1 (S2) Online RL: Trading-off Exploration and Exploitation with Reward Shift

Original: According to the principle of optimism in the face of uncertainty (OFU), an exploration bonus that manifests the uncertainty of the Q -value function can be introduced into the value estimation: integrating optimistic exploration with conservative exploitation,
Revised: According to the principle of optimism in the face of uncertainty (OFU), an exploration bonus that manifests the uncertainty of the Q -value function can be introduced into the value estimation: integrating optimistic exploration with conservative exploitation

Original: [Equation]
Revised: [Equation]

Original: where the second term with coefficient β denotes exploration bonus that is composed of uncertainty.
Revised: where the second term with coefficient β denotes exploration bonus that is composed of uncertainty.

Original: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b − ,t ( s, a ) will give near-zero predictions as a consequence of initialization (detailed implications are provided in Appendix C).
Revised: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b − ,t ( s, a ) will give near-zero predictions as a consequence of initialization (detailed implications are provided in Appendix C).

Original: Hence, the explorative bonus becomes − (1 − β ) b + + βb − 1 − γ , which is equivalent to applying another constant reward shift with value of c r
Revised: Hence, the explorative bonus becomes − (1 − β ) b + + βb − 1 − γ , which is equivalent to applying another constant reward shift with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − , formally, we have
Revised: + + βb − , formally, we have

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Assuming we have access to an unbiased estimator for the optimal value function
Revised: Assuming we have access to an unbiased estimator for the optimal value function

Original: Q ∗ , e.g., with Monte-Carlo estimation ˆ Q ∗ =
Revised: Q ∗ , e.g., with Monte-Carlo estimation ˆ Q ∗ =

Original: E (cid:80) t γ t r , and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator, i.e., ϵ 2 t
Revised: E (cid:80) t γ t r , and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator, i.e., ϵ 2 t

Original: = ( ˜ Q t − ˆ Q ∗ ) 2 , ˜ Q t =
Revised: = ( ˜ Q t − ˆ Q ∗ ) 2 , ˜ Q t =

Original: ˜ Q t − 1 − 2 η ( ˜ Q t − 1 − ˆ Q ∗ ) , then the linear combination in Equation (7) is equivalent to a linear combination of the constants with value of c r
Revised: ˜ Q t − 1 − 2 η ( ˜ Q t − 1 − ˆ Q ∗ ) , then the linear combination in Equation (7) is equivalent to a linear combination of the constants with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: The proof can be found in the Appendix B.
Revised: The proof can be found in the Appendix B.

Original: According to Proposition 4.1, a grid search for trading-off between the three hyper-parameters, i.e., the exploration bias b − , the exploitation bias b + and the coefficient β in Equation (7) is trivial as they only lead to a linear combination as c r
Revised: According to Proposition 4.1, a grid search for trading-off between the three hyper-parameters, i.e., the exploration bias b − , the exploitation bias b + and the coefficient β in Equation (7) is trivial as they only lead to a linear combination as c r

Original: = (1 − β ) b
Revised: = (1 − β ) b

Original: + + βb − , indicating that
Revised: + + βb − , indicating that

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Changing the reward shifting constant b is sufficient to trade off between exploration and exploitation.
Revised: Changing the reward shifting constant b is sufficient to trade off between exploration and exploitation.

Original: The corollary says, in principle, a meta-learner can be trained to monitor the learning process and select a proper reward shifting constant automatically [54, 55] to balance exploration and exploitation.
Revised: The corollary says, in principle, a meta-learner can be trained to monitor the learning process and select a proper reward shifting constant automatically [54, 55] to balance exploration and exploitation.

Original: For the ease of exposition, in this work we choose to focus on the simplest yet effective uniform sampling strategy from multiple shift constants, which has been shown as a strong baseline of those meta-learner approaches [56, 57], and leave more complicated meta-learner-based shifting constant adjustment for future investigation.
Revised: For the ease of exposition, in this work we choose to focus on the simplest yet effective uniform sampling strategy from multiple shift constants, which has been shown as a strong baseline of those meta-learner approaches [56, 57], and leave more complicated meta-learner-based shifting constant adjustment for future investigation.

Original: Specifically, we use multiple Q-networks to learn with transition tuples ( s, a, r, s ′ ) sampled from the identical buffer that collects the policy’s historcal interactions with the environment.
Revised: Specifically, we use multiple Q-networks to learn with transition tuples ( s, a, r, s ′ ) sampled from the identical buffer that collects the policy’s historcal interactions with the environment.

Original: In propagating
Revised: In propagating

Original: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with shifted rewards with different constant biases to update their individual Q-networks.
Revised: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with shifted rewards with different constant biases to update their individual Q-networks.

Original: We then uniformly sample one of those learned Q-networks for the optimization of policy networks (e.g., with
Revised: We then uniformly sample one of those learned Q-networks for the optimization of policy networks (e.g., with

Original: DPG [22]).
Revised: DPG [22]).

Original: It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times to collect samples for each value network.
Revised: It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times to collect samples for each value network.

Original: We dub the proposed method Random Reward Shift (RRS), and provide the pseudo-code in
Revised: We dub the proposed method Random Reward Shift (RRS), and provide the pseudo-code in

Original: Algorithm 1 of Appendix D.2.
Revised: Algorithm 1 of Appendix D.2.

Original: 4.3.2 (S3) Compatible Curiosity: Tailored Curiosity-Driven Exploration for Value-Based RL
Revised: 4.3.2 (S3) Compatible Curiosity: Tailored Curiosity-Driven Exploration for Value-Based RL

Original: In previous works, the curiosity-driven exploration methods are always work with policy-based methods.
Revised: In previous works, the curiosity-driven exploration methods are always work with policy-based methods.

Original: In this section, we cast the key insight introduced above to RND [15] — one of the leading algorithms for exploration — to answer why is its vanilla design not suitable for value-based algorithms like Q-learning, and propose a variant of RND that is tailored for DQN.
Revised: In this section, we cast the key insight introduced above to RND [15] — one of the leading algorithms for exploration — to answer why is its vanilla design not suitable for value-based algorithms like Q-learning, and propose a variant of RND that is tailored for DQN.

Original: The vanilla RND use two randomly initialized networks ϕ 1 and ϕ 2 to generate the intrinsic reward r int = | ϕ 1 ( s, a ) − ϕ 2 ( s, a ) | ≥ 0 for exploration.
Revised: The vanilla RND use two randomly initialized networks ϕ 1 and ϕ 2 to generate the intrinsic reward r int = | ϕ 1 ( s, a ) − ϕ 2 ( s, a ) | ≥ 0 for exploration.

Original: During learning, ϕ 2 is a fixed network and the parameters of ϕ 1 is optimized to approximate the output of ϕ 2 for frequently-visited states.
Revised: During learning, ϕ 2 is a fixed network and the parameters of ϕ 1 is optimized to approximate the output of ϕ 2 for frequently-visited states.

Original: We follow
Revised: We follow

Original: Burda et al.
Revised: Burda et al.

Original: [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic reward after t step of optimization.
Revised: [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic reward after t step of optimization.

Original: Specifically, r int , 0 ( s, a ) = 1 , ∀ ( s, a ) — an universally positive bonus is added to the primal reward function at beginning.
Revised: Specifically, r int , 0 ( s, a ) = 1 , ∀ ( s, a ) — an universally positive bonus is added to the primal reward function at beginning.

Original: According to our analysis in previous sections, such a positive reward shift is equivalent to pessimistic initialization and will lead to conservative behaviors in Q -value estimation.
Revised: According to our analysis in previous sections, such a positive reward shift is equivalent to pessimistic initialization and will lead to conservative behaviors in Q -value estimation.

Original: Therefore, the exploration  behaviors at the beginning of learning will be hindered, rather than boosted.
Revised: Therefore, the exploration [Figure] behaviors at the beginning of learning will be hindered, rather than boosted.

Original: To overcome such a conservative behavior induced by the pessimistic initialization, we proposed to use r − int ( s, a ) =
Revised: To overcome such a conservative behavior induced by the pessimistic initialization, we proposed to use r − int ( s, a ) =

Original: | ϕ 1 ( s, a ) − ϕ 2 ( s, a ) | 2 − I, ∀ s, a , where I is a positive constant that assures r − int ≤ 0 is negatively initialized for optimistic exploratory behaviors.
Revised: | ϕ 1 ( s, a ) − ϕ 2 ( s, a ) | 2 − I, ∀ s, a , where I is a positive constant that assures r − int ≤ 0 is negatively initialized for optimistic exploratory behaviors

Original: The Chicken-and-Egg Problem To further understand the difference between the curiosity-driven method in ways of intrinsic reward and our reward shifting-based optimistic initialization, let us consider when do those curiosities work in each case: for intrinsic reward methods, an agent must first experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hard

Original: [Figure]
Revised: [Figure]

Original: [Figure] with an arg-max style policy.
Revised: A New Type of Curiosity To further understand the difference between the curiosity-driven method in ways of intrinsic reward and our reward shifting-based optimistic initialization, let us consider when do those curiosities work in each case: for intrinsic reward methods, an agent must first experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hard with an arg-max style policy.

Original: On the other hand, with optimistic initialization, the rarely-visited ( s, a ) pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairs have updated their values with a negatively shifted reward.
Revised: On the other hand, with optimistic initialization, the rarely-visited ( s, a ) pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairs have updated their values with a negatively shifted reward.

Original: From such a perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods to encourage the first visitation of new states.
Revised: From such a perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods to encourage the first visitation of new states.

Original:  Experiments 5.1 (S1): Offline RL with Conservative Q -value Estimation
Revised:  Experiments 5.1 (S1): Offline RL with Conservative Q -value Estimation

Original: Experiment Setup We start our experiments by demonstrating the effectiveness of reward shifting in offline RL benchmarks.
Revised: Experiment Setup We start our experiments by demonstrating the effectiveness of reward shifting in offline RL benchmarks.

Original: As discussed in Section 4.2, a positive reward shift is equivalent to pessimistic initialization that benefits conservative exploitation.
Revised: As discussed in Section 4.2, a positive reward shift is equivalent to pessimistic initialization that benefits conservative exploitation.

Original: In general, our proposed method can be plugged in to any off-the-shelf offline RL algorithm, we choose to verify the effectiveness and generality of such a conservative Q -value estimation based on BCQ [23] and CQL [24], i.e., both distribution-matching approach and conservative value estimation approaches in offline RL.
Revised: In general, our proposed method can be plugged in to any off-the-shelf offline RL algorithm, we choose to verify the effectiveness and generality of such a conservative Q -value estimation based on BCQ [23] and CQL [24], i.e., both distribution-matching approach and conservative value estimation approaches in offline RL.

Original: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift ( Neg. ), added on either BCQ or CQL.
Revised: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift ( Neg. ), added on either BCQ or CQL.

Original: Results Figure 4 shows our experiment results.
Revised: Results Figure 4 shows our experiment results.

Original: We experiment with both the dataset generated in [23] (Hopper Imitate) and the dataset used in [47] (others), and find in our experiments that learning with the CQL dataset is much more stable.
Revised: We experiment with both the dataset generated in [23] (Hopper Imitate) and the dataset used in [47] (others), and find in our experiments that learning with the CQL dataset is much more stable.

Original: The first two panels show results with BCQ as the backbone algorithm.
Revised: The first two panels show results with BCQ as the backbone algorithm.

Original: We observe a clear performance drop during the training of BCQ as the policy overfits the batched dataset.
Revised: We observe a clear performance drop during the training of BCQ as the policy overfits the batched dataset.

Original: Differently, positive reward shift can alleviate such a problem and outperform BCQ in terms of both best-achieved performance and performance after convergence.
Revised: Differently, positive reward shift can alleviate such a problem and outperform BCQ in terms of both best-achieved performance and performance after convergence.

Original: The following three panels in Figure 4 use CQL as the backbone.
Revised: The following three panels in Figure 4 use CQL as the backbone.

Original: Implementation details and more results can be found in Appendix D.1.
Revised: Implementation details and more results can be found in Appendix D.1.

Original: Take-Away Message In all experiments, shifting the reward with a positive constant improves learning performance, while a negative reward shift impedes to efficient learning — as expected.
Revised: Take-Away Message In all experiments, shifting the reward with a positive constant improves learning performance, while a negative reward shift impedes to efficient learning — as expected.

Original: S2) Online RL with Randomized Priors
Revised: 5.2 (S2): Online RL with Randomized Priors

Original: Experiment Setup We then conduct experiments in the MuJoCo locomotion benchmarks to demonstrate reward shifting improves learning efficiency in the online RL settings.
Revised: Experiment Setup We then conduct experiments in the MuJoCo locomotion benchmarks to demonstrate reward shifting improves learning efficiency in the online RL settings.

Original: As our implementation is based on TD3, we use TD3-based variants as our baselines:
Revised: As our implementation is based on TD3, we use TD3-based variants as our baselines:

Original: The TD3 is trained with default settings according to Fujimoto et al.
Revised: The TD3 is trained with default settings according to Fujimoto et al.

Original: We also include Ensemble TD3 and Bootstrapped TD as baselines due to they are similar to our work in using multiple Q -networks in value estimation. 
Revised: We also include Ensemble TD3 and Bootstrapped TD as baselines due to they are similar to our work in using multiple Q -networks in value estimation. [Figure] p i [Figure]

Revised: d i

Revised: [Figure]

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: [45] but extend it to the continuous control settings.
Revised: [45] but extend it to the continuous control settings.

Original: In continuous control settings, the argmax operator is approximated by the policy network, and multiple policy networks are needed to cooperate with the multiple bootstrapped Q -value networks.
Revised: In continuous control settings, the argmax operator is approximated by the policy network, and multiple policy networks are needed to cooperate with the multiple bootstrapped Q -value networks.

Original: Otherwise, multiple
Revised: Otherwise, multiple

Original: Q -value networks are not independent of each other thus breaking the condition of bootstrapped value estimation.
Revised: Q -value networks are not independent of each other thus breaking the condition of bootstrapped value estimation.

Original: The Ensemble TD3 presents the baseline performance when multiple Q -networks are used for value estimation in TD3, and works as an ablation of our method where all shift priors are set to be 0 .
Revised: The Ensemble TD3 presents the baseline performance when multiple Q -networks are used for value estimation in TD3, and works as an ablation of our method where all shift priors are set to be 0 .

Original: As has been illustrated in Sec.
Revised: As has been illustrated in Sec.

Original: 4.3.1, learning with different reward shifting values is equivalent to learning with optimistic or conservative initialization.
Revised: 4.3.1, learning with different reward shifting values is equivalent to learning with optimistic or conservative initialization.

Original: In our instantiating of RRS, we use 3 Q -networks with different priors.
Revised: In our instantiating of RRS, we use 3 Q -networks with different priors.

Original: We empirically find ± 0 .
Revised: We empirically find ± 0 .

Original: 5 , 0 work universally good for all environments.
Revised: 5 , 0 work universally good for all environments.

Original: Though, future investigation on hyper-parameter may help to further improve the performance.
Revised: Though, future investigation on hyper-parameter may help to further improve the performance.

Original: Results Results are shown in Figure 5.
Revised: Results Results are shown in Figure 5.

Original: RRS outperforms the vanilla TD3 in all five environments and outperforms all baseline methods in most tasks.
Revised: RRS outperforms the vanilla TD3 in all five environments and outperforms all baseline methods in most tasks.

Original: In the experiment of Bootstrapped TD3 and
Revised: In the experiment of Bootstrapped TD3 and

Original: Ensemble TD3, we also use 3 Q -networks for a fair comparison.
Revised: Ensemble TD3, we also use 3 Q -networks for a fair comparison.

Original: Note that there is a trade-off between computational complexity and sample efficiency, i.e., using more Q -networks may further improve the performance at the cost of more computational expenses, as reported in [45].
Revised: Note that there is a trade-off between computational complexity and sample efficiency, i.e., using more Q -networks may further improve the performance at the cost of more computational expenses, as reported in [45].

Original: More implementation details, pseudo-code of RRS, and ablation studies can be found in Appendix D.2.
Revised: More implementation details, pseudo-code of RRS, and ablation studies can be found in Appendix D.2.

Original: Take-Away Message Shifting the reward function can trade-off between exploration and exploitation.
Revised: Take-Away Message Shifting the reward function can trade-off between exploration and exploitation.

Original: The ensemble performance of multiple value networks with random reward shifting drastically improve learning efficiency in continuous control.
Revised: The ensemble performance of multiple value networks with random reward shifting drastically improve learning efficiency in continuous control.

Original: 5.3 (S3): Optimistic Random Network Distillation
Revised: 5.3 (S3): Optimistic Random Network Distillation

Original: [Figure]
Revised: [Figure]

Original: i [Figure] Experiment Setting To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, we benchmark with five discrete exploration tasks, including the classic MountainCar control and four MiniGrid navigation tasks [58], environment details can be found at Appendix D.3.
Revised: Experiment Setting To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, we benchmark with five discrete exploration tasks, including the classic MountainCar control and four MiniGrid navigation tasks [58], environment details can be found at Appendix D.3.

Original: Results Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1 ⃝ DQN and 2 ⃝ RND baselines, we use 3 ⃝ DQN -0.5 to indicate DQN with a − 0 .
Revised: Results Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1 ⃝ DQN and 2 ⃝ RND baselines, we use 3 ⃝ DQN -0.5 to indicate DQN with a − 0 .

Original: 5 reward shift, and 4 ⃝ RND -1.0 5 ⃝ RND -1.5 to indicate RND with − 1 . 0 , − 1 .
Revised: 5 reward shift, and 4 ⃝ RND -1.0 5 ⃝ RND -1.5 to indicate RND with − 1 . 0 , − 1 .

Original: 5 reward shift, respectively.
Revised: 5 reward shift, respectively.

Original: In the following, we use ≻ between methods to indicate the former outperforms the latter.
Revised: In the following, we use ≻ between methods to indicate the former outperforms the latter.

Original: Comparing the results:
Revised: Comparing the results:

Original: reward shifting is equivalent to optimistic initialization, thus boosting exploration 3 ⃝ ≻ 1 ⃝ : a negative reward shift is equivalent to optimistic initialization and helps exploration.
Revised: reward shifting is equivalent to optimistic initialization, thus boosting exploration 3 ⃝ ≻ 1 ⃝ : a negative reward shift is equivalent to optimistic initialization and helps exploration.

Original: RND is effective for value-based exploration — as long as a negative intrinsic reward is used 1 ⃝ ≻ 2 ⃝ : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.
Revised: RND is effective for value-based exploration — as long as a negative intrinsic reward is used 1 ⃝ ≻ 2 ⃝ : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.

Original: 4 ⃝ ≻ 1 ⃝ ; 5 ⃝ ≻ 3 ⃝ : RND is effective for exploration (i.e., improve over DQN) as long as the intrinsic reward bonus is always negative.
Revised: 4 ⃝ ≻ 1 ⃝ ; 5 ⃝ ≻ 3 ⃝ : RND is effective for exploration (i e., improve over DQN) as long as the intrinsic reward bonus is always negative.

Original: reward shifting for optimistic initialization is orthogonal to other exploration methods 5 ⃝ ≻ 4 ⃝ : increasing the magnitude of the negatively shifted reward can further improve the exploration performance — reward shifting can either work in isolation or get combined with other exploration algorithms as they are working 
Revised: reward shifting for optimistic initialization is orthogonal to other exploration methods 5 ⃝ ≻ 4 ⃝ : increasing the magnitude of the negatively shifted reward can further improve the exploration performance — reward shifting can either work in isolation or get combined with other exploration algorithms as they are working orthogonally.

Original: orthogonally.

Original: Take-Away Message Reward shifting is equivalent to optimistic initialization, it can help with exploration in value-based methods.
Revised: Take-Away Message Reward shifting is equivalent to optimistic initialization, it can help with exploration in value-based methods.

Original: Importantly, such an intrinsic motivation is orthogonal to previous count-based methods, such that it can either work in isolation or be combined with conventional curiosity-driven exploration methods.
Revised: Importantly, such an intrinsic motivation is orthogonal to previous count-based methods, such that it can either work in isolation or be combined with conventional curiosity-driven exploration methods.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcement learning algorithms.
Revised: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcement learning algorithms.

Original: Although constant reward shift should not change the optimal policy induced by the optimal value function, in practice such a constant shift does affect the function approximation , and leads to different learning behaviors.
Revised: Although constant reward shift should not change the optimal policy induced by the optimal value function, in practice such a constant shift does affect the function approximation , and leads to different learning behaviors.

Original: Our detailed analysis manifests the fact that a constant reward shift is equivalent to using different initialization in the value function approximation.
Revised: Our detailed analysis manifests the fact that a constant reward shift is equivalent to using different initialization in the value function approximation.

Original: The proposed idea is then verified through a variety of application settings.
Revised: The proposed idea is then verified through a variety of application settings.

Original: Specifically, we show that a negative reward shift leads to curiosity-driven exploration, while a positive reward shift helps conservative exploitation.
Revised: Specifically, we show that a negative reward shift leads to curiosity-driven exploration, while a positive reward shift helps conservative exploitation.

Original: Importantly, our analysis reveals that changing reward shifting constant itself is sufficient in trading-off between exploration and exploitation.
Revised: Importantly, our analysis reveals that changing reward shifting constant itself is sufficient in trading-off between exploration and exploitation.

Original: We empirically verify the effectiveness of the proposed method in a variety of experiments, including better exploitation in offline RL, sample-efficient learning in continuous control benchmarks, and enhanced curiosity-driven exploration in value-based discrete control.
Revised: We empirically verify the effectiveness of the proposed method in a variety of experiments, including better exploitation in offline RL, sample-efficient learning in continuous control benchmarks, and enhanced curiosity-driven exploration in value-based discrete control.

Original: While our experiments demonstrate the performance gain is quite robust to the shifting constant, we would like to point out that the theoretical guidance for such a shifting constant is missing.
Revised: While our experiments demonstrate the performance gain is quite robust to the shifting constant, we would like to point out that the theoretical guidance for such a shifting constant is missing.

Original: Potential solutions may lie in analysis from the perspective of optimization for black-box models, yet it is out of the scope of the current empirical study and left for future research.
Revised: Potential solutions may lie in analysis from the perspective of optimization for black-box models, yet it is out of the scope of the current empirical study and left for future research.

Original:  Acknowledgement
Revised:  Acknowledgement

Original: We thank all anonymous reviewers, ACs, PCs for their efforts and time in the reviewing process and in improving our paper.
Revised: We thank all anonymous reviewers, ACs, PCs for their efforts and time in the reviewing process and in improving our paper.

Original: This work is done with the warm supports from the MMLab members.
Revised: This work is done with the warm supports from the MMLab members.

Original: We acknowledge the insightful discussions with Ziping Xu, the Hot Spring Harbor group and the van der
Revised: We acknowledge the insightful discussions with Ziping Xu, the Hot Spring Harbor group and the van der

Original: Schaar Lab members in improving the presentations of this paper.
Revised: Schaar Lab members in improving the presentations of this paper.

Original: We thank Takuya Kanazawa in pointing out the concurrent work of Dubey et al.
Revised: We thank Takuya Kanazawa in pointing out the concurrent work of Dubey et al.

Original: [59] that also discusses the effects of shifting terms in reward function.
Revised: [59] that also discusses the effects of shifting terms in reward function.

Revised: We thank all insightful comments and discussions during the conference poster sessions that helps us in improving the presentation of the paper.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We discussed in the Conclusion that this work is mainly limited in empirical studies.
Revised: [Yes] We discussed in the Conclusion that this work is mainly limited in empirical studies.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our research studies the general problem of reward shifting in reinforcement learning, aiming at improve learning efficiency.
Revised: Our research studies the general problem of reward shifting in reinforcement learning, aiming at improve learning efficiency.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Please refer to Appendix B 3.
Revised: [Yes] Please refer to Appendix B 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Implementation details can be found at Appendix D. Code is open-sourced at https://github.com/holarissun/RewardShifting (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Implementation details can be found at Appendix D. Code is open-sourced at https://github.com/holarissun/RewardShifting (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Implementation details can be found at Appendix D.1, D.2, and D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] Implementation details can be found at Appendix D.1, D.2, and D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] In our experiments, the results are averaged over 10 runs, and shaded areas indicate the 25% - 75% quantile values.
Revised: [Yes] In our experiments, the results are averaged over 10 runs, and shaded areas indicate the 25% - 75% quantile values.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please refer to Appendix D. 4.
Revised: [Yes] Please refer to Appendix D. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] [23, 24] (b) Did you mention the license of the assets?
Revised: [Yes] [23, 24] (b) Did you mention the license of the assets?

Original: [Yes] MIT license.
Revised: [Yes] MIT license.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] We follow BCQ and CQL to use the dataset they used in the papers.
Revised: [Yes] We follow BCQ and CQL to use the dataset they used in the papers.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: A Extended Related Work
Revised: A Extended Related Work

Original: We extend our related work section on the following related topics as suggested by reviewers:
Revised: We extend our related work section on the following related topics as suggested by reviewers:

Original: A.1 Discussion on Ensembles and Distributional RL
Revised: A.1 Discussion on Ensembles and Distributional RL

Original: In Distributional RL literature [60–64], the distribution of the Q -value, rather than the mean scaler, is estimated.
Revised: In Distributional RL literature [60–64], the distribution of the Q -value, rather than the mean scaler, is estimated.

Original: Distributional-RL focuses on stochastic reward mechanisms and smooth the temporal difference learning in a distributional level, and can be applied to risk-sensitive scenarios [65] where the worst-case performance can be controlled [66].
Revised: Distributional-RL focuses on stochastic reward mechanisms and smooth the temporal difference learning in a distributional level, and can be applied to risk-sensitive scenarios [65] where the worst-case performance can be controlled [66].

Original: In those scenarios, the systematic uncertainty is the crucial issue to address, whereas in our work, we focus on deterministic transition dynamics and use OFU to tackle the epistemic uncertainty in the section of RRS.
Revised: In those scenarios, the systematic uncertainty is the crucial issue to address, whereas in our work, we focus on deterministic transition dynamics and use OFU to tackle the epistemic uncertainty in the section of RRS.

Original: Several previous works discussed ensemble methods for exploration, both for discrete control [67, 68] and for continuous control [69, 70].
Revised: Several previous works discussed ensemble methods for exploration, both for discrete control [67, 68] and for continuous control [69, 70].

Original: However, in our work we show that more exploratory behavior can emerge with the help of reward shifting under only single Q -value network — as a proof of concept that negative reward shift is equivalent to optimistic initialization.
Revised: However, in our work we show that more exploratory behavior can emerge with the help of reward shifting under only single Q -value network — as a proof of concept that negative reward shift is equivalent to optimistic initialization.

Original: A.2 Model-Based Exploration and Uncertainty Estimation
Revised: A.2 Model-Based Exploration and Uncertainty Estimation

Original: Besides the model-free value-based RL methods we focused in our paper, there exist literature like
Revised: Besides the model-free value-based RL methods we focused in our paper, there exist literature like

Original: R-Max [40] that work with model-based methods for better exploration.
Revised: R-Max [40] that work with model-based methods for better exploration.

Original: Moreover, combining multiple neural networks for uncertainty estimation is well-established in supervised learning [71], and similari idea has been explored in the context of RL for discrete control [45, 72].
Revised: Moreover, combining multiple neural networks for uncertainty estimation is well-established in supervised learning [71], and similari idea has been explored in the context of RL for discrete control [45, 72].

Original: In this work, we showcase that a diversified set of reward shifting constants can work as priors for such an ensemble.
Revised: In this work, we showcase that a diversified set of reward shifting constants can work as priors for such an ensemble.

Original: Given the ensemble networks without ground-truth, it is in general hard to disentangle the aleatoric uncertainty from epistemic uncertainty.
Revised: Given the ensemble networks without ground-truth, it is in general hard to disentangle the aleatoric uncertainty from epistemic uncertainty.

Original: Our method works on deterministic reward and transition dynamics to circumvent the discussion of uncertainty stratification.
Revised: Our method works on deterministic reward and transition dynamics to circumvent the discussion of uncertainty stratification.

Original: In the deterministic settings, the source of uncertainty can be solely attributed to the epistemic uncertainty and hence help informative exploration.
Revised: In the deterministic settings, the source of uncertainty can be solely attributed to the epistemic uncertainty and hence help informative exploration.

Original: When it comes to the stochastic environments, the entanglement of aleatoric uncertainty and epistemic uncertainty will make the problem much more difficult [73] as intrinsic motivation methods may get trapped by pursuing the actions that result in high aleatoric uncertainty in certain circumstances (e.g., the Noisy-TV)
Revised: When it comes to the stochastic environments, the entanglement of aleatoric uncertainty and epistemic uncertainty will make the problem much more difficult [73] as intrinsic motivation methods may get trapped by pursuing the actions that result in high aleatoric uncertainty in certain circumstances (e.g., the Noisy-TV)

Original: B Proof of Proposition 4. 
Revised: B Proof of Proposition 4.

Original: Proof.
Revised: Proof.

Original: the estimated Q -value ˆ Q ( s, a ) is composed by the two estimators with function approximation error, defined as ϵ b + ( s, a ) =
Revised: the estimated Q -value ˆ Q ( s, a ) is composed by the two estimators with function approximation error, defined as ϵ b + ( s, a ) =

Original: ˜ Q b + ,t ( s, a ) − b + 1 − γ − Q ∗ ( s, a ) , and ϵ b − ( s, a ) =
Revised: ˜ Q b + ,t ( s, a ) − b + 1 − γ − Q ∗ ( s, a ) , and ϵ b − ( s, a ) =

Original: ˜ Q b − ,t ( s, a ) − b − 1 − γ −
Revised: ˜ Q b − ,t ( s, a ) − b − 1 − γ −

Original: Q ∗ ( s, a ) .
Revised: Q ∗ ( s, a ) .

Original: [Equation]
Revised: [Equation]

Original: where C = (1 − β ) A + βB and the last line requires ˜ Q A, 0 = ˜ Q B, 0 = ˜ Q C, 0 are identical initialization.
Revised: where C = (1 − β ) A + βB and the last line requires ˜ Q A, 0 = ˜ Q B, 0 = ˜ Q C, 0 are identical initialization.

Original: With this notion, Equation (7) can be re-written as
Revised: With this notion, Equation (7) can be re-written as

Original: [Equation]
Revised: [Equation]

Original: 9) where the second line relies on the linear assumption of the approximation error (1 − β ) ϵ b
Revised: 9) where the second line relies on the linear assumption of the approximation error (1 − β ) ϵ b

Original: + ( s, a ) + βϵ b − ( s, a ) .
Revised: + ( s, a ) + βϵ b − ( s, a ) .

Original: We further have (1 − β ) ˜ Q b + + β ˜ Q b − =
Revised: We further have (1 − β ) ˜ Q b + + β ˜ Q b − =

Original: ˜ Q (1 − β ) b + + βb − and ˆ Q ( s, a ) =
Revised: ˜ Q (1 − β ) b + + βb − and ˆ Q ( s, a ) =

Original: ˜ Q c r ( s, a ) , telling us that trading-off between the constant b − used for exploration and the constant b + used for exploitation with the coefficient β is equivalent to use another constant with value of c r
Revised: ˜ Q c r ( s, a ) , telling us that trading-off between the constant b − used for exploration and the constant b + used for exploitation with the coefficient β is equivalent to use another constant with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: C Implications of Assumption in Section 4.3. 
Revised: C Implications of Assumption in Section 4.3.

Original: In our main text, the estimated values for extremely o.o.d.
Revised: In our main text, the estimated values for extremely o.o.d.

Original: samples are assumed to be near zeros.
Revised: samples are assumed to be near zeros.

Original: We provide detailed implications and explanations in this section.
Revised: We provide detailed implications and explanations in this section.

Original: On the one hand, it’s clear that such an assumption holds for the tabular settings, that un-visited state-action pairs have the value in tabular initialization.
Revised: On the one hand, it’s clear that such an assumption holds for the tabular settings, that un-visited state-action pairs have the value in tabular initialization.

Original: On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d.
Revised: On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d.

Original: samples that have the Q -values near zero for function approximation settings.
Revised: samples that have the Q -values near zero for function approximation settings.

Original: Interpolation between those o.o.d.
Revised: Interpolation between those o.o.d.

Original: samples and other state-action pairs will clearly lead to an “in-between” value estimation, which in practice can be achieved with properly regularized neural networks.
Revised: samples and other state-action pairs will clearly lead to an “in-between” value estimation, which in practice can be achieved with properly regularized neural networks.

Original: The key insight we want to emphasize in Section 4.3.1 is that for frequently visited state-action pairs, the value discrepancy with different initialization are small, while for seldomly-visited state-action pairs, the discrepancy are relatively large, enabling the usage of such discrepancy as exploration bonus.
Revised: The key insight we want to emphasize in Section 4.3.1 is that for frequently visited state-action pairs, the value discrepancy with different initialization are small, while for seldomly-visited state-action pairs, the discrepancy are relatively large, enabling the usage of such discrepancy as exploration bonus.

Original: D Implementation Details and Ablation Studies
Revised: D Implementation Details and Ablation Studies

Original: Hardware and Training Time We experiment on a server with 8 TITAN X GPUs and 32 Intel(R)
Revised: Hardware and Training Time We experiment on a server with 8 TITAN X GPUs and 32 Intel(R)

Original: E5-2640 CPUs.
Revised: E5-2640 CPUs.

Original: In general, shifting the reward does not introduce further computation burden except in the continuous control tasks, our method of Random Reward Shift (RRS) requires two additional Q -value networks.
Revised: In general, shifting the reward does not introduce further computation burden except in the continuous control tasks, our method of Random Reward Shift (RRS) requires two additional Q -value networks.

Original: In our PyTorch-based implementation, those additional networks can be easily implemented and optimized in a parallel manner, and the extra computational burden is equivalent to using a √ 3 times wider neural network during optimization.
Revised: In our PyTorch-based implementation, those additional networks can be easily implemented and optimized in a parallel manner, and the extra computational burden is equivalent to using a √ 3 times wider neural network during optimization.

Original: It is worth noting that RRS is computationally much cheaper than the Bootstrapped TD3, where additional policy networks are also needed.
Revised: It is worth noting that RRS is computationally much cheaper than the Bootstrapped TD3, where additional policy networks are also needed.

Original: Network Structure
Revised: Network Structure

Original: Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters.
Revised: Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters.

Original: We implement DQN based on a 3-layer fully connected neural network with 64 hidden units for the Q -value function, using ReLU and linear activation respectively.
Revised: We implement DQN based on a 3-layer fully connected neural network with 64 hidden units for the Q -value function, using ReLU and linear activation respectively.

Original: We use the Adam optimizer with learning rate of 0 . 001 , and use an epsilon-greedy approach as naive exploration strategy.
Revised: We use the Adam optimizer with learning rate of 0 . 001 , and use an epsilon-greedy approach as naive exploration strategy.

Original: In our RND, we use two 4-layer fully connected neural networks with 512 units and ReLU activation in each hidden layer, and a softmax activation for the output layer.
Revised: In our RND, we use two 4-layer fully connected neural networks with 512 units and ReLU activation in each hidden layer, and a softmax activation for the output layer.

Original: Adam optimizer is used for the optimization of the RND networks with learning rate 0 . 0001 .
Revised: Adam optimizer is used for the optimization of the RND networks with learning rate 0 . 0001 .

Original: Our code is provided in the supplementary materials, and will be made public available.
Revised: Our code is provided in the supplementary materials, and will be made public available.

Original: D.1 Offline RL
Revised: D.1 Offline RL

Original: In our experiments, we use a fixed dataset with 10 k offline trainsition tuples for offline RL learning.
Revised: In our experiments, we use a fixed dataset with 10 k offline trainsition tuples for offline RL learning.

Original: Our implementation of BCQ and CQL are both based on the code provided by the authors.
Revised: Our implementation of BCQ and CQL are both based on the code provided by the authors.

Original: The only change we made to verify our insight is to shift the reward by a constant.
Revised: The only change we made to verify our insight is to shift the reward by a constant.

Original: In most environments, we
Revised: In most environments, we

Original: Require Size of mini-batch N , smoothing factor τ > 0 , K reward shift values r ′ k
Revised: Require Size of mini-batch N , smoothing factor τ > 0 , K reward shift values r ′ k

Original: = r + b k ,k = 1 ,...,K .
Revised: = r + b k ,k = 1 ,...,K .

Original: Random initialized policy network π θ , target policy network π θ ′ , θ ′ ← θ .
Revised: Random initialized policy network π θ , target policy network π θ ′ , θ ′ ← θ .

Original: K random initialized Q networks, and corresponding target networks, parameterized by w k ,w ′ k , w ′ k
Revised: K random initialized Q networks, and corresponding target networks, parameterized by w k ,w ′ k , w ′ k

Original: ← w k for k = 1 ,...,K .
Revised: ← w k for k = 1 ,...,K .

Original: (e.g., a ModuleList in PyTorch).
Revised: (e.g., a ModuleList in PyTorch).

Original: for iteration = 1 , 2 ,... do
Revised: for iteration = 1 , 2 ,... do

Original: Uniformly sample one of the K Q -functions, Q w k , for policy update for t = 1 , 2 ,... do # Interaction Run policy π θ , and collect transition tuples ( s t ,a t ,s ′ t ,r t ) .
Revised: Uniformly sample one of the K Q -functions, Q w k , for policy update for t = 1 , 2 ,... do # Interaction Run policy π θ , and collect transition tuples ( s t ,a t ,s ′ t ,r t ) .

Original: Sample a mini-batch of transition tuples { ( s,a,s ′ ,r ) i } Ni =1 .
Revised: Sample a mini-batch of transition tuples { ( s,a,s ′ ,r ) i } Ni =1 .

Original: # Update Q w (in parallel)
Revised: # Update Q w (in parallel)

Original: Calculate the k -
Revised: Calculate the k -

Original: [Table]
Revised: [Table]

Original: target Q value y
Revised: target Q value y

Original: [Table]
Revised: [Table]

Original: ,i =
Revised: ,i =

Original: r i
Revised: r i

Original: [Table]
Revised: [Table]

Original: ′ k
Revised: ′ k

Original: [Table]
Revised: [Table]

Original: i ,
Revised: i ,

Original: [Table]
Revised: [Table]

Original: θ ′ ( s ′ i ))
Revised: θ ′ ( s ′ i ))

Original: Update
Revised: Update

Original: [Table]
Revised: [Table]

Original: with
Revised: with

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: i )
Revised: i )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: end for
Revised: end for

Original: [Table] [Table]
Revised: [Table] [Table]

Original: end for find r ′ =
Revised: end for find r ′ =

Original: r + 8 provides good enough performance.
Revised: r + 8 provides good enough performance.

Original: While in Hopper Medium CQL we find using a smaller positive reward shift r ′ =
Revised: While in Hopper Medium CQL we find using a smaller positive reward shift r ′ =

Original: r + 1 works better than r ′ =
Revised: r + 1 works better than r ′ =

Original: r + 8 , and for Walker Medium CQL, using a larger reward shift of r ′ =
Revised: r + 8 , and for Walker Medium CQL, using a larger reward shift of r ′ =

Original: r + 50 further improves the result with r ′ =
Revised: r + 50 further improves the result with r ′ =

Original: r + 8 .
Revised: r + 8 .

Original: Figure 7 shows different performance under different choices of the reward shift constant.
Revised: Figure 7 shows different performance under different choices of the reward shift constant.

Original: We denote a positive reward shift r ′ =
Revised: We denote a positive reward shift r ′ =

Original: r + 8 as Pos.1 , denote r ′ =
Revised: r + 8 as Pos.1 , denote r ′ =

Original: r + 20 as Pos.2 and denote r ′ =
Revised: r + 20 as Pos.2 and denote r ′ =

Original: r + 50 as
Revised: r + 50 as

Original: Pos.3 for all experiments excetp in the Hopper Medium CQL we use Pos.1 to denote r ′ =
Revised: Pos.3 for all experiments excetp in the Hopper Medium CQL we use Pos.1 to denote r ′ =

Original: r + 1 .
Revised: r + 1 .

Original: In the experiments based on BCQ (first two figures).
Revised: In the experiments based on BCQ (first two figures).

Original: We can observe a uniformly performance improvement with all choices of reward shift constants.
Revised: We can observe a uniformly performance improvement with all choices of reward shift constants.

Original: As the algorithm of CQL has already taken the conservative value estimation into consideration, in the experiments based on CQL, the performance is more closely related to the constant we use.
Revised: As the algorithm of CQL has already taken the conservative value estimation into consideration, in the experiments based on CQL, the performance is more closely related to the constant we use.

Original: Specifically, in Hopper Expert, while using any of the positive reward shift constants improve the learning stability, r ′ =
Revised: Specifically, in Hopper Expert, while using any of the positive reward shift constants improve the learning stability, r ′ =

Original: r + 8 performs better on preserving the learning efficiency during early learning stage.
Revised: r + 8 performs better on preserving the learning efficiency during early learning stage.

Original: For Hopper Medium, we find using larger positive constants hinder the performance.
Revised: For Hopper Medium, we find using larger positive constants hinder the performance.

Original: For Walker Medium, using a larger constant in reward shift performs much better than using a smaller one.
Revised: For Walker Medium, using a larger constant in reward shift performs much better than using a smaller one.

Original: [Figure]
Revised: [Figure]

Original: D.2 Continuous Control
Revised: D.2 Continuous Control

Original: Details of RRS Although we find in the motivating example that a − 5 reward shift is able to remarkably improve the asymptotic performance of TD3, in this work we aim at proposing an uniformly suitable method based on the insight behind the motivating example.
Revised: Details of RRS Although we find in the motivating example that a − 5 reward shift is able to remarkably improve the asymptotic performance of TD3, in this work we aim at proposing an uniformly suitable method based on the insight behind the motivating example.

Original: Therefore we propose to use ± 0 .
Revised: Therefore we propose to use ± 0 .

Original: 5 , 0 as the reward shifting constants.
Revised: 5 , 0 as the reward shifting constants.

Original: We find in experiment that the sampling frequency does not affect the performance.
Revised: We find in experiment that the sampling frequency does not affect the performance.

Original: And in the experiments we follow BDQN [45] to use a fixed value network throughout a whole trajectory.
Revised: And in the experiments we follow BDQN [45] to use a fixed value network throughout a whole trajectory.

Original: i.e., one of the K Q -networks is sampled uniformly after each episode with length of 1000 timesteps.
Revised: i.e., one of the K Q -networks is sampled uniformly after each episode with length of 1000 timesteps.

Original: Intuitively, searching for more suitable reward randomization designs may further improve the performance, yet that is beyond the coverage of this work.
Revised: Intuitively, searching for more suitable reward randomization designs may further improve the performance, yet that is beyond the coverage of this work.

Original: Ablation Studies We experiment with different number of Q -value networks as well as different choices of the random reward shifting ranges.
Revised: Ablation Studies We experiment with different number of Q -value networks as well as different choices of the random reward shifting ranges.

Original: Results are presented in Figure 8.
Revised: Results are presented in Figure 8.

Original: We denote RRS with 7 reward shifting constants ( and therefore also 7 Q -networks) as RRS-7 , and denote RRS with 3 reward shifting constants ( and therefore also 3 Q -networks) as RRS-3 .
Revised: We denote RRS with 7 reward shifting constants ( and therefore also 7 Q -networks) as RRS-7 , and denote RRS with 3 reward shifting constants ( and therefore also 3 Q -networks) as RRS-3 .

Original: The constants following
Revised: The constants following

Original: RRS-3/RRS-7 are the ranges of those random constants.
Revised: RRS-3/RRS-7 are the ranges of those random constants.

Original: Specifically, we use [ − 0 .
Revised: Specifically, we use [ − 0 .

Original: 5] for the
Revised: 5] for the

Original: RRS-3 0.5 settings, [ − 1 . 0 , 0 , 1 . 0] for the RRS-3 1.0 settings, [ − 0 .
Revised: RRS-3 0.5 settings, [ − 1 . 0 , 0 , 1 . 0] for the RRS-3 1.0 settings, [ − 0 .

Original: 5 , − 0 .
Revised: 5 , − 0 .

Original: 33 , − 0 .
Revised: 33 , − 0 .

Original: 33 , 0 . 5] for the RRS-7 0.5 settings and [ − 1 . 0 , − 0 .
Revised: 33 , 0 . 5] for the RRS-7 0.5 settings and [ − 1 . 0 , − 0 .

Original: 67 , − 0 .
Revised: 67 , − 0 .

Original: 67 , 1 . 0] for the RRS-7 1.0 settings.
Revised: 67 , 1 . 0] for the RRS-7 1.0 settings.

Original: According to the experimental results, RRS is not sensitive to hyper-parameters, showing the robustness of the proposed method.
Revised: According to the experimental results, RRS is not sensitive to hyper-parameters, showing the robustness of the proposed method.

Original: We believe further search for those hyper-parameters can further improve the learning efficiency, yet this is off the main scope of this work and therefore left for the future research.
Revised: We believe further search for those hyper-parameters can further improve the learning efficiency, yet this is off the main scope of this work and therefore left for the future research.

Original: [Figure]
Revised: [Figure]

Original: E p
Revised: E p

Original: i s o
Revised: i s o

Original: d i c R
Revised: d i c R

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: D.3 Random Network Distillation
Revised: D.3 Random Network Distillation

Original: Environments In this work, we experiment with five discrete (sparse reward) exploration tasks , namely the MountainCar-v
Revised: Environments In this work, we experiment with five discrete (sparse reward) exploration tasks , namely the MountainCar-v

Original: [Figure]
Revised: [Figure]

Original: and four navigation tasks of MiniGrid suite [58], namely the task of Empty-Random, MultiRoom
Revised: and four navigation tasks of MiniGrid suite [58], namely the task of Empty-Random, MultiRoom

Original: [Figure]
Revised: [Figure]

Original: and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.
Revised: and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.

Original: Figure 9 shows example of different tasks
Revised: Figure 9 shows example of different tasks

Original: [Figure]
Revised: [Figure]

Original: Hyper-Parameter Settings
Revised: Hyper-Parameter Settings

Original: We use a 2 -layer NN with 64 hidden units for Q -networks in DQN and set RND networks to be 3 -layer NN with 512 hidden units.
Revised: We use a 2 -layer NN with 64 hidden units for Q -networks in DQN and set RND networks to be 3 -layer NN with 512 hidden units.

Original: ϵ -greedy exploration is applied to DQN with ϵ decays from 0 .
Revised: ϵ -greedy exploration is applied to DQN with ϵ decays from 0 .

Original: 9 to 0 .
Revised: 9 to 0 .

Original: in the first 1/5 episodes.
Revised: in the first 1/5 episodes.

Original: Size of replaybuffer is set to be 100000 .
Revised: Size of replaybuffer is set to be 100000 .

Original: Ablation Studies We experiment with different reward shifting constants in the discrete control settings.
Revised: Ablation Studies We experiment with different reward shifting constants in the discrete control settings.

Original: We use a relatively large range in choosing constants, i.e., {− 0 .
Revised: We use a relatively large range in choosing constants, i.e., {− 0 .

Original: 05 , − 0 .
Revised: 05 , − 0 .

Original: 15 , − 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 .
Revised: 15 , − 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 .

Original: 5 , − 5 . 0 , − 10 . 0 } .
Revised: 5 , − 5 . 0 , − 10 . 0 } .

Original: Results are presented in Figure 10.
Revised: Results are presented in Figure 10.

Original: In all experiments, using a moderate reward shifting constant like {− 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 . 5 } remarkably improves the learning efficiency.
Revised: In all experiments, using a moderate reward shifting constant like {− 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 . 5 } remarkably improves the learning efficiency.

Original: On the other hand, a too aggressive reward shifting will lead to too much curiosity exploration and hinder the learning efficiency in the limited number of interactions.
Revised: On the other hand, a too aggressive reward shifting will lead to too much curiosity exploration and hinder the learning efficiency in the limited number of interactions.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E Additional Experiments E.1 Necessity of Explorative Behaviors in Maze Tasks
Revised: E Additional Experiments E.1 Necessity of Explorative Behaviors in Maze Tasks

Original: We demonstrate the benefits of reward shifting for deep-exploration tasks and the on-par performance of reward shifting for easy tasks that does not require deep-exploration.
Revised: We demonstrate the benefits of reward shifting for deep-exploration tasks and the on-par performance of reward shifting for easy tasks that does not require deep-exploration.

Original: We experiment on the maze environment and change the size of maze to vary from 2 to 20, denoting as S2, S5, S10, S15, S20, separately.
Revised: We experiment on the maze environment and change the size of maze to vary from 2 to 20, denoting as S2, S5, S10, S15, S20, separately.

Original: Results averaged over 8 runs are reported in Figure 11.
Revised: Results averaged over 8 runs are reported in Figure 11.

Original: We find in easy tasks (S2, S5,
Revised: We find in easy tasks (S2, S5,

Original: S10), both count-based exploration and reward shifting perform similarly to the ϵ -greedy exploration, while on challenging tasks (S15, S20), more explorative behavior encouraged by reward shifting and count-based exploration are important for efficient learning
Revised: S10), both count-based exploration and reward shifting perform similarly to the ϵ -greedy exploration, while on challenging tasks (S15, S20), more explorative behavior encouraged by reward shifting and count-based exploration are important for efficient learning

Original: [Figure]
Revised: [Figure]

Original: E.2 Performance in Challenging Continuous Control Exploration Tasks
Revised: E.2 Performance in Challenging Continuous Control Exploration Tasks

Original: Figure 12 show experiments on (1) the HalfCheetah-SparseReward, where a reward of +1 is provided only when the forward movement of the halfcheetah is larger than 5 unit with regard to the timeframe;
Revised: Figure 12 show experiments on (1) the HalfCheetah-SparseReward, where a reward of +1 is provided only when the forward movement of the halfcheetah is larger than 5 unit with regard to the timeframe;

Original: We note that this environment is different from the SparseHalfCheetah environment that first introduced in VIME [74], as we use different time frames.
Revised: We note that this environment is different from the SparseHalfCheetah environment that first introduced in VIME [74], as we use different time frames.

Original: (We acknowledge and thank the anonymous reviewer FqMf for pointing out this difference.) and (2) the Humanoid, which is a high dimensional continuous control task in the MuJoCo locomotion suite; to verify the effectiveness of reward shifting in exploration.
Revised: (We acknowledge and thank the anonymous reviewer FqMf for pointing out this difference.) and (2) the Humanoid, which is a high dimensional continuous control task in the MuJoCo locomotion suite; to verify the effectiveness of reward shifting in exploration.

Original: In the HalfCheetah-SparseReward environment, the maximum score is 1000 for each
Revised: In the HalfCheetah-SparseReward environment, the maximum score is 1000 for each

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: episode.
Revised: episode.

Original: We use − 0 .
Revised: We use − 0 .

Original: 5 as the negative reward shift for exploration and use 0 .
Revised: 5 as the negative reward shift for exploration and use 0 .

Original: 5 for comparison.
Revised: 5 for comparison.

Original: In
Revised: In

Original: Humanoid, the per-step reward is approximately 5 in previous well-performing agents [14, 20]
Revised: Humanoid, the per-step reward is approximately 5 in previous well-performing agents [14, 20]

Original: [Figure]
Revised: [Figure]

Original: and we hereby use negative shift − 5 and use positive shift 5 , for comparison.
Revised: and we hereby use negative shift − 5 and use positive shift 5 , for comparison.

Original: In HalfCheetah-SparseReward
Revised: In HalfCheetah-SparseReward

Original: [Figure]
Revised: [Figure]

Original: we find a negative reward shift lead to more explorative behavior and improves the learning efficiency while a positive reward shift hinders the learning efficiency.
Revised: we find a negative reward shift lead to more explorative behavior and improves the learning efficiency while a positive reward shift hinders the learning efficiency.

Original: In Humanoid
Revised: In Humanoid

Original: [Figure]
Revised: [Figure]

Original: we find using a reward shift can drastically improve the asymptotic performance by +60%
Revised: we find using a reward shift can drastically improve the asymptotic performance by +60%

Original: [Figure]
Revised: [Figure]

Original: while learning with a positive reward shift retard the learning and converge to a lower performance.
Revised: while learning with a positive reward shift retard the learning and converge to a lower performance.

Original: E.3 Performance in Goal-Conditioned Continuous Control (Robotics) Suite
Revised: E.3 Performance in Goal-Conditioned Continuous Control (Robotics) Suite

Original: To further address the reviewer’s concern on the applicability of reward shifting on challenging continuous control exploration tasks, we benchmark reward shifting on the FetchRobotics suite [8] that is usurally considered to be challenging exploration task in GCRL literature [75, 76].
Revised: To further address the reviewer’s concern on the applicability of reward shifting on challenging continuous control exploration tasks, we benchmark reward shifting on the FetchRobotics suite [8] that is usurally considered to be challenging exploration task in GCRL literature [75, 76].

Original: Figureshows the results we get on FetchReach-v1 and FetchPush-v1 environment.
Revised: Figureshows the results we get on FetchReach-v1 and FetchPush-v1 environment.

Original: We use HER [76] as the backbone algorithms and vary the reward for failure and success in achieving the goals.
Revised: We use HER [76] as the backbone algorithms and vary the reward for failure and success in achieving the goals.

Original: In their default setting, reaching the goal will receive a reward of 0 , otherwise, the agent will receive − 1 as punishment.
Revised: In their default setting, reaching the goal will receive a reward of 0 , otherwise, the agent will receive − 1 as punishment.

Original: In experiments, we find using a positive reward +1 in reaching the goals while using a trivial 0 reward otherwise will drastically hinder the learning efficiency of HER.
Revised: In experiments, we find using a positive reward +1 in reaching the goals while using a trivial 0 reward otherwise will drastically hinder the learning efficiency of HER.

Original: Similar empirical discovery has been reported in Sun et al.
Revised: Similar empirical discovery has been reported in Sun et al.

Original: [37] in the PPO-based learners.
Revised: [37] in the PPO-based learners.

Original: This set of experiments verifies our key insight one more time that explorative behaviors emerge with a negatively shifted reward function, and a positive reward shift leads to conservative behavior.
Revised: This set of experiments verifies our key insight one more time that explorative behaviors emerge with a negatively shifted reward function, and a positive reward shift leads to conservative behavior.

Original: To sum up, our key insight reveals the mechanism of how such an empirically verified heuristic design in GCRL works: a negative reward − 1 (also interpreted as cost) works in the same way as reward shifting to improve exploration.
Revised: To sum up, our key insight reveals the mechanism of how such an empirically verified heuristic design in GCRL works: a negative reward − 1 (also interpreted as cost) works in the same way as reward shifting to improve exploration.