Original: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)
Revised: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).
Revised: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).

Original: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.
Revised: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.

Original: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.

Original: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation beneﬁts the exploration of online RL.
Revised: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation beneﬁts the exploration of online RL.

Original: We validate our insight ona range of RL tasks and show its improvement over baselines:
Revised: We validate our insight ona range of RL tasks and show its improvement over baselines:

Original: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete exploration tasks, a negative reward shiftingbrings an improvement over the previous curiosity-based exploration method.
Revised: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete exploration tasks, a negative reward shiftingbrings an improvement over the previous curiosity-based exploration method.

Original:  Introduction
Revised:  Introduction

Original: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2]
Revised: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2]

Original: specifying a certain reward to incentivize the learning agentrequires domain knowledge and thorough understanding ofthe task [3–6].
Revised: specifying a certain reward to incentivize the learning agentrequires domain knowledge and thorough understanding ofthe task [3–6].

Original: Even with careful design and tuning, learningwith a shaped reward that intends to accelerate learning mayon the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].
Revised: Even with careful design and tuning, learningwith a shaped reward that intends to accelerate learning mayon the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].

Original: Although
Revised: Although

Original: Ng et al.
Revised: Ng et al.

Original: [9] theoretically points the optimal policy remainsunchanged under a special form of reward transformation,and in the later work of Wiewiora et al.
Revised: [9] theoretically points the optimal policy remainsunchanged under a special form of reward transformation,and in the later work of Wiewiora et al.

Original: [10] a framework isproposed to guide policies with prior knowledge under tabular setting, the investigation of how it accommodates recent
Revised: [10] a framework isproposed to guide policies with prior knowledge under tabular setting, the investigation of how it accommodates recent

Original: Deep Reinforcement Learning (DRL) algorithms remainsmuch less explored.
Revised: Deep Reinforcement Learning (DRL) algorithms remainsmuch less explored.

Original: In this work, we focus on the simplest form of rewardshaping, the linear transformation, in value-based DRL [11–14].
Revised: In this work, we study a special linear transformation, whichis the simplest form of reward shaping, in value-based DRL [11–14].

Original: We start with understanding how such a speciﬁc kindof reward shaping works in value-based DRL algorithms. We show that reward shifting, as the simplest reward transformation, is equivalent to manually settingthe initial values of the Q -function estimation, extending previous discovery of [10] to the functionapproximation setting.
Revised: We start with understanding how such aspeciﬁc kind of reward shaping works in value-based DRLfunction approximations and show that reward shifting is equivalent to different Q -value initialization,extending previous discovery of [10] to the function approximation setting.

Original: Figure 1 showcases reward shifting beneﬁts exploration in a maze game andoutperforms count-based and ✏ -greedy exploration.
Revised: Figure 1 showcasesreward shifting beneﬁts exploration in a maze game and outperforms count-based and ✏ -greedyexploration and learns near optimal value function.

Original: Based on such an equivalence, we introduce the key insight of this work:
Revised: Based on such an equivalence, we introduce the key insight of this work:

Original: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.
Revised: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.

Original: We demonstrate the application of such an insight on three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation can lead to improved learning performance based on off-the-shelfalgorithms; (2) for online RL setting, we show multiple value functions with different reward shiftingconstants can be used as a trade-off between exploration and exploitation, thus improving learningefﬁciency; (3) we introduce a simple yet crucial improvement over a prevailing curiosity-basedexploration method, the Random Network Distillation [15], making it compatible with value-based
Revised: We demonstrate the application of such an insight on three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation can lead to improved learning performance based on off-the-shelfalgorithms; (2) for online RL setting, we show multiple value functions with different reward shiftingconstants can be used as a trade-off between exploration and exploitation, thus improving learningefﬁciency; (3) we introduce a simple yet crucial improvement over a prevailing curiosity-basedexploration method, the Random Network Distillation [15], making it compatible with value-based

Original: DRL algorithms.
Revised: DRL algorithms.

Original: The experiments on both continuous and discrete action space control tasks showsour method brings the substantial improvement over previous baselines.
Revised: The experiments on both continuous and discrete action space control tasks showsour method brings the substantial improvement over previous baselines.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network initialization in  deep-RL, which can be used toimprove both curiosity-driven exploration and conservative exploitation;
Revised: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value networkinitialization in value-based deep-RL, which can be applied to curiosity-driven explorationand conservative exploitation;2.

Original: Practically, we present three scenarios motivated by our key insight to show the potentialbeneﬁt of reward shifting, namely the ofﬂine conservative exploitation, the online sample- efﬁcient RL, andthe curiosity-driven exploration;
Revised: Practically, we instantiate the key insight to three different scenarios, namely the ofﬂine conservative exploitation, sample-efﬁcient continuous control, and curiosity-driven exploration,to contrast the generality of reward shifting;

Original: Empirically, we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks.
Revised: Empirically, we demonstrate the effectiveness of the proposed method integrated withoff-the-shelf baselines on both continuous and discrete control tasks.

Original:  Preliminaries and Related Work
Revised:  Preliminaries and Related Work2.1 Online RL

Original: 2.1 Online RL

Original: We follow a standard MDP formulation in the online RL setting, i.e. , M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ), T : S ⇥ A 7! S is the transition dynamics,
Revised: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ), we consider a deterministic transition dynamics

Original: R : S ⇥
Revised: T : S ⇥

Revised: A 7! S and deterministic reward function R : S ⇥

Original: A 7!
Revised: A 7!

Original: R is the reward function.
Revised: R .

Original: ⇢ 0 denotes the initial state  distribution, i.e. , ⇢ 0 = p ( s 0 ) .
Revised: ⇢ 0 denotes the initial state 64 distribution, i.e., ⇢ 0 = p ( s 0 ) .

Original: 64  is the discount factor and T is the episodic decision horizon.
Revised:  is the discount factor and T is the episodic decision horizon.

Original: Online RL considers the problemof learning a policy ⇡ 2 ⇧ : S 7!
Revised: Online RL considers the problem of learning a policy ⇡ 2 ⇧ : S 7!

Original:  ( A ) (or ⇡ 2 ⇧ : S 7!
Revised:  ( A ) (or ⇡ 2 ⇧ : S 7!

Original: A with a  deterministic policy66 class), such that the expected cumulative reward in the Markov decision process is maximized, i.e. ,⇡ [Equation]
Revised: A with a 66 deterministic policy class), such that the expected cumulative reward in the Markov decision processis maximized, i.e., ⇡ = arg max ⇡ E a t ⇠ ⇡ ,s t +1 ⇠ T ,s 0 ⇠ ⇢

Original: P
Revised: P

Original: Tt =

Revised: Tt =

Original:  t r t ( s t , a t ) ,
Revised:  t r t ( s t , a t ) ,

Original: In the online RL setting, an agent normallylearns through trials and errors [11], either with an on-policy paradigm [16–18] or an off-policymanner [13, 12, 19, 20, 14].
Revised: In the online RL setting,an agent normally learns through trials and errors [11], either with an on-policy paradigm [16–18] oran off-policy manner [12–14, 19, 20].

Original: In this work, we focus on the off-policy value-based methods which arein general more sample efﬁcient.
Revised: In this work, we focus on the off-policy value-based methodswhich are in general more sample efﬁcient.

Original: Speciﬁcally, our discussions assume the policy learning is basedon a learned Q -value function, that approximates the cumulative reward an agent can gain in thefollowing part of an episode.
Revised: Speciﬁcally, our discussions assume the policy learning isbased on a learned Q -value function, that approximates the cumulative reward an agent can gain in thefollowing part of an episode.

Original: The Q -value function is deﬁned as Q ( s t , a t ) =
Revised: The Q -value function is deﬁned as Q ( s t , a t ) =

Original: E ⇡ , T
Revised: E ⇡ , T

Original: P
Revised: P

Original: T ⌧ = t  t r ( s ⌧ , a ⌧ ) ,and can be estimated by propagating the Bellman operator B Q ( s, a ) =
Revised: T ⌧ = t  t r ( s ⌧ , a ⌧ ) ,and can be estimated by propagating the Bellman operator B Q ( s, a ) =

Original: r ( s, a ) +  E Q ( s 0 , a 0 ) .
Revised: r ( s, a ) +  E Q ( s 0 , a 0 ) .

Original: Forvalue-based methods, the (soft-)optimal policy is then produced by
Revised: Forvalue-based methods, the (soft-)optimal policy is then produced by

Original: [Equation]
Revised: [Equation]

Original: where Q ⇤ is optimal Q -value function.
Revised: where Q ⇤ is optimal Q -value function.

Original: We can also set the temperature parameter close as 0 tohave the deterministic policy class.
Revised: We can also set the temperature parameter close as 0 tohave the deterministic policy class.

Original: Simplifying the notion we have ⇡ ⇤ ( s ) =
Revised: Simplifying the notion we have ⇡ ⇤ ( s ) =

Original: arg max a Q ⇤ ( s, a ) .
Revised: arg max a Q ⇤ ( s, a ) .

Original: Algorithms like DPG [21] can be used to address the intractable analytical argmax issue arises incontinuous action space.
Revised: Algorithms like DPG [21] can be used to address the intractable analytical argmax issue arises incontinuous action space.

Original: We choose to develop our work on top of prevailing baseline algorithms of
Revised: We choose to develop our work on top of prevailing baseline algorithms of

Original: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.
Revised: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.

Original: Itshould be straightforward to extend the method on top of other learning algorithms.
Revised: Itshould be straightforward to extend the method on top of other learning algorithms.

Original: 2.2 Exploration and the Curiosity-Driven Methods
Revised: 2.2 Exploration and the Curiosity-Driven Methods

Original: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent must learn to exploit its accumulated knowledge on the task while exploring new states andactions.
Revised: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent must learn to exploit its accumulated knowledge on the task while exploring new states andactions.

Original: Previous works address the exploration problem from various perspectives: In the tasks with
Revised: Previous works address the exploration problem from various perspectives: for discrete

Original: [Table]
Revised: [Table]

Original: discrete action space, count-based methods like Bellemare et al [Table] [Table] [Table]
Revised: action space tasks, count-based methods like [27–29] are proposed to motivate the revisiting ofunder-explored state-action pairs.

Original: Ostrovski et [Table] [28], Tang et al
Revised: Speciﬁcally, Choshen et al.

Original: [Table] [29], Choshen et al. [30] are proposed to motivate the policy to explore more on under-explored states. Curiosity-driven methods are investigated by Houthooft et al.
Revised: [30] extended the idea into generalsettings by constructing an additional MDP for Exploration-value estimation, as a generalized counterfor count-based exploration.

Original: [31], Pathak et al. [32], Burda et al.[33, 15], where the intrinsic reward is designed asa supplementary to the primal task reward for betterexploration.
Revised: To boost exploration in continuous state tasks, curiosity-driven methodsare investigated by [31–33, 15], where variety of intrinsic rewards are designed as supplementary tothe primal task reward for better exploration.

Original: Self-imitate approaches like Oh et al.
Revised: Self-imitate approaches like Oh et al.

Original: [34], Ecoffet et al. [35], Sun et al.
Revised: [34], Ecoffet et al.[35], Sun et al.

Original: [36] repeatsuccess trajectories but require extra assumptions on the environment.
Revised: [36] repeat success trajectories but require extra assumptions on the environment.

Original: The work of DIAYN and DADS [37, 38] show that various skills can be developed even without the primal extrinsic reward.
Revised: Thework of DIAYN and DADS [37, 38] show that various skills can be developed even without the primalextrinsic reward.

Original: For continuous control tasks, OAC [26] improves the SAC [20] with informative action space noisebased on the optimism in face of uncertainty (OFU) [39–42].
Revised: For continuous control tasks, OAC [26] improves the SAC [20] with informativeaction space noise based on the optimism in face of uncertainty (OFU) [39–42].

Original: GAC [43] addresses  the explorationissue with a richer functional class for the policy.
Revised: GAC [43] addresses [Table] the exploration issue with a richer functional class for the policy.

Original: In the work of Rashid et al.
Revised: In the work of Rashid et al.

Original: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.
Revised: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.

Original: Inthe work of Osband et al.
Revised: Inthe work of Osband et al.

Original: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.
Revised: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.

Original: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal
Revised: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal

Original: Q -value, and its ﬂexibility to be plugged in to both continuous and discrete control algorithms.
Revised: Q -value, and its ﬂexibility to be plugged in to both continuous and discrete control algorithms.

Original: The Random Network Distillation (RND) [15] proposes to use the  difference between a ﬁxed neuralnetwork  1 and a trainable network  2 to represents the intrinsic reward, e.g. ,
Revised: Random Network Distillation (RND) [15] deﬁnes the intrinsic reward as the output difference betweena ﬁxed neural network  1 and a trainable network  2 given state-actions as the inputs e.g.,

Original: [Equation]
Revised: [Equation]

Original: when outputs of both networks are activated by a sigmoid function, and  2 is optimized to approximate 1 for the visited ( s, a ) pairs 1 .
Revised: when outputs of both networks are activated by a sigmoid function, and  2 is optimized to approximate 1 for the visited ( s, a ) pairs 1 .

Original: Hence, the value of r int ( s, a ) will decay to 0 when such state-actionpairs are visited frequently but remain high for seldom visited pairs.
Revised: Hence, the value of r int ( s, a ) will decay to 0 when such state-actionpairs are visited frequently but remain high for seldom visited pairs.

Original: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.
Revised: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.

Original: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated by  function approximation error.
Revised: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated by high function approximation error in theunder-explored state-action pairs.

Original: We demonstratesuch an insight by showing that RND in its original design is not suitable for value-based methods indeveloping exploratory behaviors, but integrating RND with a shifted reward function remarkablyimproves the learning performance.
Revised: We demonstrate such an insight by showing that RND in itsoriginal design is not suitable for value-based methods in developing exploratory behaviors, butintegrating RND with a shifted reward function remarkably improves the learning performance.

Original: 2.3 Ofﬂine RL
Revised: 2.3 Ofﬂine RL

Original: The ofﬂine RL, also known as batch-RL, focuses on the problems where the interaction with theenvironment is impossible, and the policy can only be optimized based on logged dataset.
Revised: The ofﬂine RL, also known as batch-RL, focuses on the problems where the interaction with theenvironment is impossible, and the policy can only be optimized based on logged dataset.

Original: In thosetasks, a ﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ]  is  provided.
Revised: In thosetasks, a ﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ] , collected from some unknown behavior policy ⇡  , is 118 provided.

Revised: In general, such a dataset can either be generated by rolling out an expert that generateshigh-quality solutions to the task [22, 45, 46] or a non-expert that executes sub-optimal behaviors [47–49, 46, 50] or be a mixture of both [23].

Original: As the agent in the ofﬂine RL setting117 can not correct its potentially biased knowledge through interactions, the most important issue isto address the extrapolation error [22] induced by distributional mismatch [45].
Revised: As the agent in the ofﬂine RL setting can not correctits potentially biased knowledge through interactions, the most important issue is to address theextrapolation error [22] induced by distributional mismatch [48].

Original: To address suchan issue, a series of algorithms optimize the policy learning under the constraint of distributionalsimilarity [45–47].
Revised: To address such an issue, a series ofalgorithms optimize the policy learning under the constraint of distributional similarity [48, 47, 51].

Original: Bharadhwaj et al.
Revised: Bharadhwaj et al.

Original: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.
Revised: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.

Original: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions ( e.g. , uniform distribution over the action space).As we will show in1 Or computed with only states, i.e., r int ( s ) = |  2 ( s )   1 ( s ) | . ! ∗ ! " # ! 1 − & ! " !
Revised: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions (e.g., uniform distribution over the action space).

Original: ∗ = !

Original: ∗ + " !

Original: $%& "# ! ≈0 "

Original: " ! '! ' − !

Original: ∗ < '! '

Original: %$ −!

Original: ∗ ): '! '

Original: → !

Original: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !

Original: ∗ < $! " ! ,$%& −! " !

Original: ∗ ): '! " ! ,' → ! " !

Original: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!

Original: +,,' = '! " ! ,' − # ! 1 − & '!

Original: +,,- = '!

Original: - − # ! 1 − & ( 1   , rather than ˜ Q 0 ⇡ 0 . i.e. , by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value. this work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.
Revised: As we will show inthis work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.

Original: This idea leads to the direct application of our proposedframework in the ofﬂine setting.
Revised: This idea leads to the direct application of our proposedframework in the ofﬂine setting.

Original: In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22, 45–47] to further improve the performance with conservative Q -valueestimation.
Revised: In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22, 48, 47, 51] to further improve the performance with conservative Q -valueestimation.

Original: Table 1 contextualizes reward shifting with respect to related works in both exploration and exploitation motivated RL algorithms.
Revised: Table 1 contextualizes reward shifting with respect to related works we discussed above, morediscussions on related works can be found at Appendix E.

Original:  A Motivating Example
Revised:  A Motivating Example

Original: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing ourproposed method.
Revised: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing our method.

Original: Remark 1.
Revised: Remark 1.

Original: Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 136 with linear transformation, i.e. ,
Revised: Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 139 with linear transformation, i.e.,

Original: R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , does 137 not change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=
Revised: R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , doesnot change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=

Original: P t  t r 0 t :
Revised: [Equation] 0 t :

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: When |A| = 1 , scaling the reward function with linear transformation does not change 139 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k : 
Revised: When |A| = 1 , scaling the reward function with linear transformation does not change 142 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function.
Revised: Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function.

Original: However, Figure 1 presents “counter-intuitive”results that in a demonstrative exploration task of Grid World.
Revised: However, Figure 1 presents “counter-intuitive”results that in a demonstrative exploration task of Grid World.

Original: In this task, an agent is initializedat the upper left corner of the map need to explore the map without reward unless it reaches thegoal point located at the lower right corner, where a positive reward of +1 will be assigned.
Revised: In this task, an agent is initializedat the upper left corner of the map need to explore the map without reward unless it reaches thegoal point located at the lower right corner, where a positive reward of +1 will be assigned.

Original: Wereport learning curves with regard to episodic success rate in reaching the goal point and the learned
Revised: Wereport learning curves with regard to episodic success rate in reaching the goal point and the learned

Original: Q -values.
Revised: Q -values.

Original: In this toy example, we ﬁnd a negative reward shifting remarkably boosts the learningefﬁciency of Q -learning, and surpasses the count-based method that is classic for exploration.
Revised: In this toy example, we ﬁnd a negative reward shifting remarkably boosts the learningefﬁciency of Q -learning, and surpasses the count-based method that is classic for exploration.

Revised: Moreover, Remark 1 is empirically veriﬁed with such a toy example, i.e., reward shifting does notchange the optimal Q -value as well as its induced policy, but on the contrary, accelerating the learningof (near)-optimal policy.

Revised: In the following of this work, we study the effect of varying constant bias b and ﬁx the scaling factork = 1 to avoid trivial discussions on the choices of learning rate — there is no surprise that changingan appropriate learning rate can be empirically important.

Revised: We focus on revealing the importanceof selecting the universal bias term b in the reward function through the lens of initialization priorsin function approximation, and show such a bias is generally helpful for both online and ofﬂinesettings.

Revised: In the online settings it improves learning efﬁciency, and in the ofﬂine settings it promotesconservative exploitation.

Revised:  Shifted Priors for Q -value Estimation

Original: − 𝑏 + 1−𝛾 𝑸
Revised: We start by formally introducing notions and the key idea of this work: reward shifting equals differentinitialization. We use Figure 2 to illustrate how reward shifting affects function approximation, hencechanging the learning dynamics for value-based algorithms. The original optimal Q -value functionis denoted as Q ⇤ , and plotted in the ﬁgures as Black curves.

Revised: We then denote the shifted optimal

Revised: Q -value function as Q ⇤ b + , which is the Q -value function with the shifted reward

Revised: r 0 = r

Revised: + b + .

Revised: Weuse Red curves in ﬁgures to denote those shifted Q -value functions.

Revised: In this section we provide

Revised: ! ∗ ! "

Revised: # ! 1 − & ! " !

Revised: ∗ = !

Revised: ∗ + " !

Revised: $%& "# ! ≈0 "

Revised: " ! '! ' − !

Revised: ∗ < '! '

Revised: %$ −!

Revised: ∗ ): '! '

Revised: → !

Revised: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !

Revised: ∗ < $! " ! ,$%& −! " !

Revised: ∗ ): '! " ! ,' → ! " !

Revised: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!

Revised: +,,' = '! " ! ,' − # ! 1 − & '!

Revised: +,,- = '!

Revised: - − # ! 1 − & ( r 0 = r + b + .

Revised: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized as ˜ Q LB , 0 ⇡ ˜ Q 0  b + 1   , rather than ˜ Q 0 ⇡ 0 .

Revised: i.e., by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value.

Revised: − 𝑏 + 1−𝛾 𝑸

Original: 𝓐
Revised: 𝓐

Original: ෩𝑸
Revised: ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐
Revised: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐

Original: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =
Revised: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =

Original: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 

Revised: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸

Original: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸
Revised: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸

Original: UB,𝑡
Revised: UB,𝑡

Original: =
Revised: =

Original: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾෩𝑸
Revised: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾

Original: UB,𝑡 ≥𝑸 ∗ ෩𝑸 LB,𝑡 ≤𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏 + 1−𝛾
Revised: [Figure]

Original: Moreover, Remark 1 is empirically veriﬁed withsuch a toy example, i.e. , reward shifting does notchange the optimal Q -value as well as its induced policy, but on the contrary, accelerating the learningof (near)-optimal policy. In the following section, we will focus on the scenarios when k = 1 to avoid the trivial (thoughthis can be empirically important) discussions on different learning rates. We reveal the importanceof selecting the universal biasterm, i.e. , b in the reward function, through the lens of initializationpriors in function approximation. Furthermore, we show such a bias term can be utilized not only inthe online RL settings to improve learning efﬁciency but also in the ofﬂine RL settings to conductconservative exploitation with batched data.
Revised: analysis with a positive bias b + > 0 to illustrate how positively shifted value function affects functionapproximation, and coordinates the normally applied near-zero initialized function approximators(e.g., neural networks.

Original: 4.1 Reward Shift Equals to Different Initialization We start by introducing the central idea of this work: reward shift equals different initialization. Speciﬁcally, Figure 2 illustrates the effect of the reward shifting under different settings.
Revised: Orange curves in the ﬁgure) to inspire conservative behaviors.

Original: The blackcurves denote the primal optimal Q -value functions ( e.g. , Q ⇤ ), and the red curves denote the optimal Q -value functions with shifted reward r 0 = r + b + , where b + > 0 is a positive constant bias, andwe will show a different choice of such a bias leads to a different motivation inQ -value estimation. The yellow lines denote Q -value estimators, e.g. , neural network predictions of the corresponding
Revised: The discussionof negative biases is elaborated in the next section.

Original: Q -values. To summarize, shifting the reward function with a positive constant is equivalent to initializing thevalue function network with a smaller value. As the Q -value of unseen state-action pairs duringtraining are much lower than the optimal value, those actions will not be selected in  policy updates,leading to conservative learning behaviors that beneﬁt policy learning in ofﬂine settings [23].
Revised: To summarize, shifting the reward function with a positive constant is equivalent to initializing thevalue function network with a smaller value — as the Q -value of unseen state-action pairs duringtraining are much lower than their shifted optimal value, those actions will not be selected in argmaxstyle policy updates, leading to conservative learning behaviors that beneﬁt policy learning in ofﬂinesettings [23].

Original: 4.2 Conservative Exploitation
Revised: 4.2 Conservative Exploitation

Original: We begin with a natural application in the ofﬂine setting, where a policy doesn’t interact with theenvironment and only learns through a logged data set collected from an unknown behavior policy⇡  , which can either be an expert that generates high-quality solutions to the task [22, 48, 49] or anon-expert that provides actions that are sub-optimal [46, 45, 50, 49, 51] or a mixture of both [23].

Original: Based on the idea we presented in Section 4.1, we introduce a simple yet effective approach forconservative Q -value estimation that learns to heuristically put a  lower bound on the optimal Q -valuefunction in batch settings.
Revised: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emergewith a positively shifted reward. As such a value estimation empirically lower bounds the optimal Q -value function.

Original: As shown in Figure 2(a), a positive constant b + added to the rewardfunction will lead to a uniformly (positively) shifted optimal Q -value function, and the gap betweenthe primal Q -value function and the new one is b + 1   .
Revised: As shown in Figure 2(a), a positive constant b + added to the reward function willlead to a uniformly (positively) shifted optimal Q -value function, and the gap between the primal Q -value function and the new one is b + 1   .

Original: Optimizing the Q -value function with loggeddata ( e.g. , a ﬁxed replay buffer) will minimize the difference between the predicted value and theoptimal value with observed data.
Revised: Optimizing the Q -value function with logged data (e.g.,a ﬁxed replay buffer) will minimize the difference between the predicted value and the optimalvalue with observed data.

Original: For the unobserved data point (in the state-action space S ⇥ A ), thenear-zero initialization guarantees the prediction is lower than the optimal Q -value, thus conservativeexploitation can be conducted with such a value function.
Revised: For the unobserved data point (in the state-action space S ⇥ A ), the near-zero initialization guarantees the prediction is lower than the optimal Q -value, thus conservativeexploitation can be conducted with such a value function.

Original: 4.3 (Curiosity-Driven) Optimistic Exploration
Revised: 4.3 (Curiosity-Driven) Optimistic Exploration

Original: On the other hand, if we shift the reward function to the negative side, it is equivalent to optimisticinitialization.
Revised: On the other hand, if we shift the reward function to the negative side, it is equivalent to optimisticinitialization.

Original: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.
Revised: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.

Original: With sufﬁciently small b  (so that | b  | larger than the maximal value of any  s, a  -pair), such an upperbound of Q ⇤ can be used to conduct curiosity-driven exploration.
Revised: With sufﬁciently small b  (so that | b  | larger than the maximal value of any ( s, a ) -pair), such anupper bound of Q ⇤ can be used to conduct curiosity-driven exploration.

Original: Intuitively, initializing a valuenetwork that predicts value larger or equal than the true value will lead to curiosity-driven exploration,as any visited state will be assigned a low value and the policy that learns to perform the action witha higher value will tend to choose novel actions.
Revised: Intuitively, initializing avalue network that predicts value larger or equal than the true value will lead to curiosity-drivenexploration, as any visited state will be assigned a relatively smaller value and the argmax-style policywill tend to choose under-explored actions.

Original: Based on the discoveries above that (1) a positive constant shift added to the reward function can beused for conservative policy update and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, as shown in Figure 3(b). We are ready to access boththe upper bound, i.e. , the optimistic estimation with b  , and the lower bound, i.e. , the conservativeestimation with b + of the optimal value function.
Revised: Based on the analysis above that (1) a positive constant shift added to the reward function can beused for conservative policy update and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., theoptimistic estimation with b  , and the lower bound, i.e., the conservative estimation with b + of theoptimal value function.

Original: Formally, we use
Revised: Formally, we use

Original: [Equation]
Revised: [Equation]

Original: (5) to denote the empirical lower bound of value estimation (c.f. Figure 2(d), orange line), and 
Revised: 5) to denote the empirical lower bound of value estimation (cf. Figure 2(d), orange line), and [Equation]

Original: [Equation]
Revised: [Equation]

Original: (6) to denote the empirical upper bound of value estimation (c.f. Figure 3(b)).
Revised: to denote the empirical upper bound of value estimation (cf. Figure 3(b)).

Original: In both notions, t denotesthe optimization step.
Revised: In both notions, t denotesthe optimization step.

Original: e.g., when t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 , ˜ Q LB , 0 ( s, a ) =
Revised: e.g., when t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 , ˜ Q LB , 0 ( s, a ) =

Original:  b + 1  is able to lower bound the unknown optimal Q -value given sufﬁciently large b + .
Revised:  b + 1  is able to lower bound the unknown optimal Q -value given sufﬁciently large b + .

Original: (c.f.
Revised: (cf.

Original: Figure 2(d),green line).
Revised: Figure 2(d),green line).

Original: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufﬁciently small b  .
Revised: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufﬁciently small b  .

Original: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous controland discrete action space respectively.
Revised: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous control anddiscrete action space respectively.

Original: We propose a practical algorithm for general continuouscontrol in Sec. 4.3.1, while focusing on a special class of curiosity-driven exploration method, the
Revised: We propose a practical algorithm for sample-efﬁcient continuouscontrol in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the

Original: RND [15], in Sec.
Revised: RND [15], in Section 4.3.2.

Original: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift
Revised: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift

Original: According to the principle of optimism on the face of uncertainty (OFU), an exploration bonus thatmanifests the uncertainty of the Q -value function can be introduced into the value estimation: thebasic idea starts from integrating optimistic exploration with conservative exploitation, i.e. ,
Revised: According to the principle of optimism on the face of uncertainty (OFU), an exploration bonus thatmanifests the uncertainty of the Q -value function can be introduced into the value estimation: thebasic idea starts from integrating optimistic exploration with conservative exploitation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.
Revised: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.

Original: For those under-explored state-action pairs, i.e. , extremely out-of-distribution samples for ourneural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequenceof initialization (detailed implications are provided in Appendix B).
Revised: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neuralnetwork, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequence ofinitialization (detailed implications are provided in Appendix B).

Original: Hence, the explorative bonusbecomes  (1   ) b
Revised: Hence, the explorative bonusbecomes  (1   ) b

Original: + +  b   .
Revised: + +  b   .

Original: Note this is equivalent to applying another constant reward shift with valueof c r = (1   ) b + +  b  .
Revised: Note this is equivalent to applying another constant reward shift with valueof c r = (1   ) b + +  b  .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Assuming we have access to an unbiased estimator for the optimal value function
Revised: Assuming we have access to an unbiased estimator for the optimal value function

Original: Q ⇤ , e.g. , with Monte-Carlo estimation ˆ Q ⇤ =
Revised: Q ⇤ , e.g., with Monte-Carlo estimation ˆ Q ⇤ =

Original: E
Revised: E

Original: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e. , ✏ 2 t
Revised: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e., ✏ 2 t

Original: =
Revised: =

Original: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =
Revised: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =

Original: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r
Revised: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r

Original: =
Revised: =

Original: (1   ) b
Revised: (1   ) b

Original: + +  b  .
Revised: + +  b  .

Original: The proof can be found in the Appendix A.
Revised: The proof can be found in the Appendix A.

Original: According to Proposition 4.1, a grid search for tradingoff between the three hyper-parameters: the exploration bias b  , the exploitation bias b + and the coefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r
Revised: According to Proposition 4.1, a grid search for tradingoff between the three hyper-parameters: the exploration bias b  , the exploitation bias b + and thecoefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r

Original: = (1   ) b + + b  , indicating that
Revised: = (1   ) b + + b  , indicating that

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.
Revised: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.

Original: The corollary indicates that, in principle, a meta-learner can be trained to monitor the learning processand select a proper reward shifting constant automatically [52, 53].
Revised: The corollary indicates that, in principle, a meta-learner can be trained to monitor the learning processand select a proper reward shifting constant automatically [52, 53].

Original: For the ease of exposition, in thiswork we choose to focus on the simplest yet effective uniform sampling strategy from multiple shiftconstants, which has been shown as a strong baseline of those meta-learner approaches [54, 55], andleave more complicated meta-learner-based approaches for future investigation.
Revised: For the ease of exposition, in thiswork we choose to focus on the simplest yet effective uniform sampling strategy from multiple shiftconstants, which has been shown as a strong baseline of those meta-learner approaches [54, 55], andleave more complicated meta-learner-based approaches for future investigation.

Original: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.
Revised: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.

Original: In propagating
Revised: In propagating

Original: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with anew reward with different shifted constant bias in those Q-networks.
Revised: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with anew reward with different shifted constant bias in those Q-networks.

Original: We then uniformly sample oneof those learned Q-networks for the optimization of policy network (with e.g., DPG [21]).
Revised: We then uniformly sample oneof those learned Q-networks for the optimization of policy network (with e.g., DPG [21]).

Original: It is worthnoting that our approach only requires post-hoc revision of the primal reward function, rather thaninteracting with the environment multiple times to collect samples for each value network.
Revised: It is worthnoting that our approach only requires post-hoc revision of the primal reward function, rather thaninteracting with the environment multiple times to collect samples for each value network.

Original: We dubthe proposed method Random Reward Shift (RRS), and provide the pseudo-code in Algorithm 1 of
Revised: We dubthe proposed method Random Reward Shift (RRS), and provide the pseudo-code in Algorithm 1 of

Original: Appendix C.2.
Revised: Appendix C.2.

Original: 4.3.2 Improving Value-Based Curiosity-Driven Exploration 
Revised: 4.3.2 Tailored Curiosity-Driven Exploration for Value-Based Method

Original: The discussion above casts the curiosity-driven exploration method as a special case in the rewardshift equals toinitialization perspective: exploration with RND [15] is equivalent to selecting b + = 0 ,i.e. , using the primal task reward for exploitation, and using b  as a ﬁxed random function over S ⇥ A , i.e. , b  ( s, a ) =
Revised: In previous works, the curiosity-driven exploration methods are always work with policy-basedmethods. In this section, we cast the key insight introduced above to RND [15] — one of theleading algorithms for exploration — to answer why is its vanilla design not suitable for value-basedalgorithms like Q-learning, and propose a variant of RND that is tailored for DQN.

Original:  |  1, 0 ( s, a )   2 ( s, a ) |2 , 8 s, a , where 1 , 0 and  2 are two neural networks 247 with different random initializations.
Revised: The vanilla RND use two randomly initialized networks  1 and  2 to generate the intrinsic rewardr int = |  1 ( s, a )   2 ( s, a ) |  0 for exploration.

Original: While  1 , 0 is learnable,  2 isset to be ﬁxed during learningas a random curiosity prior [24]. Without loss ofgenerality, we can assume 2 = 1 as a constantinitialization, and  1 , 0 = 0 .
Revised: During learning,  2 is a ﬁxed network and theparameters of  1 is optimized to approximate the output of  2 for frequently-visited states. We follow

Original: Then the exploration behavior is fully controlled by the scale of externalreward, which is set to be 1.0 with external reward clipped to [  1 , 1] in [15].
Revised: Burda et al. [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic rewardafter t step of optimization.

Original: The ﬁnal equivalentobjective in this case is to use c r = |  1 ,t ( s, a ) 1 | 2 , t =0 , 1 , 2 , ... . Specially,  1 , 0 = 0 and c r = 1at beginning.
Revised: Speciﬁcally, r int , 0 ( s, a ) = 1 , 8 ( s, a ) — an universally positive bonus isadded to the primal reward function at beginning.

Original: According to our analysis inthe previous section, such a positive  shift directly leads to conservativebehaviors in Q -value estimation, therefore may hinder the exploration behaviors at the beginning,i.e. , when |  1 ,t  1 |  0 .
Revised: According to our analysis in previous sections, such a positive reward shift is equivalent to pessimisticinitialization and will lead to conservative behaviors in Q -value estimation.

Original: The exploration bonus in RND only becomes effective when visiting 256 seldom visited states after the predictions of  1 ,t of frequently-visited states are close to 1 , andhence encouraging stepping into those novel states and discover new knowledge.
Revised: Therefore, the explorationbehaviors at the beginning of learning will be hindered, rather than boosted.

Original: To overcomethe conservative tendency induced by Q -value estimation in RND, we propose to use b   ( s, a ) =| 1 , 0 ( s, a )   2 ( s, a )
Revised: To overcome sucha conservative behavior induced by the pessimistic initialization, we proposed to use r  int ( s, a ) =|

Original:  | 2  I, 8 s, a , where I is a positive constant( e.g. , I = max s,a |  1 , 0 ( s, a )  2 ( s, a ) | 2 ) that assures b  ( s, a ) is negative-initialized for optimistic exploratory behaviors.
Revised:  1 ( s, a )   2 ( s, a ) | 2  I, 8 s, a , where I is a positive constant that assures r  int  0 is negatively 262 initialized for optimistic exploratory behaviors.

Original: It is worth noting that the curiosity introduced by exploratory bonuses like RND focuses on increasingthe temporary Q -value of rarely visited states, which still relies on the visitation behavior itself inrandom exploration. i.e. , without stepping into a non-frequent visited state, the agent will neverreceive the novelty bonus based on the state.
Revised: The Chicken-and-Egg Problem To further understand the difference between the curiosity-drivenmethod in ways of intrinsic reward and our reward shifting-based optimistic initialization, let usconsider when do those curiosities work in each case: for intrinsic reward methods, an agent mustﬁrst experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hardwith an arg-max style policy. On the other hand, with optimistic initialization, the rarely-visited ( s, a )pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairshave updated their values with a negatively shifted reward.

Original: On the other hand, the mechanism of shifting thereward by a constant is an universal optimistic exploration bonus as the curiosity is fused into theinitialization across the entire state-action support.
Revised: From such a perspective, reward shiftingnot only works by itself motivates exploratory behaviors, but can also be seamlessly plugged intointrinsic reward methods to encourage the ﬁrst visitation of new states.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Ofﬂine RL with Conservative Q -value Estimation
Revised: 5.1 Ofﬂine RL with Conservative Q -value Estimation

Original: We ﬁrst demonstrate the proposed method inthe ofﬂine RL setting.
Revised: We start our experiments by demonstrating the effectiveness of reward shifting in ofﬂine RL benchmarks.

Original: Ashas been discussed in Section 4.2,shifting the primal reward function with a positive constant provides a natural way ofconservative exploitation.
Revised: As discussed in Section 4.2, a positive reward shift is equivalent to pessimistic initializationthat beneﬁts conservative exploitation.

Original: By adding a positive reward shift, any visited ( s, a ) saved in the givenreplay buffer B (logged dataset) will be assigned a large reward, while under-explored ( s, a ) pairsremain the low values due to initialization, as illustrated in Figure 2 (d).

Original: Although in general ourproposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm,in this work we demonstrate the effectiveness  of such a conservative Q -value estimation based on BCQ [22] and CQL [23], i.e. , both distribution-matching approach and conservative value estimation approaches inofﬂine RL.
Revised: In general, our proposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm, we choose to verify the effectiveness and generality of such aconservative Q -value estimation based on BCQ [22] and CQL [23], i.e., both distribution-matchingapproach and conservative value estimation approaches in ofﬂine RL.

Original: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.
Revised: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.

Original: Figure 4 shows our experiment results.
Revised: Figure 4 shows our experiment results.

Original: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [49] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.
Revised: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [46] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.

Original: The ﬁrst two panels showresults with BCQ as the backbone algorithm.
Revised: The ﬁrst two panels showresults with BCQ as the backbone algorithm.

Original: We observe clear performance drop when training with
Revised: We observe clear performance drop when training with

Original: BCQ as the policy overﬁts the given dataset in these settings, on the other hand, our proposed rewardshift can alleviate such a problem and outperform BCQ w.r.t.
Revised: BCQ as the policy overﬁts the given dataset in these settings, on the other hand, our proposed rewardshift can alleviate such a problem and outperform BCQ w.r.t.

Original: both best achieved performance andconvergence performance.
Revised: both best achieved performance andconvergence performance.

Original: The following three panels in Figure 4 use CQL as the backbone.
Revised: The following three panels in Figure 4 use CQL as the backbone.

Original: In allexperiments, using a positive reward shift leads to improved learning performance, while a negativereward shift leads to performance decay, as expected.
Revised: In allexperiments, using a positive reward shift leads to improved learning performance, while a negativereward shift leads to performance decay, as expected.

Original: Implementation details and more results can befound in Appendix C.1.
Revised: Implementation details and more results can befound in Appendix C.1.

Original: 5.2 Online RL with Randomized Priors
Revised: 5.2 Online RL with Randomized Priors

Original: We then conduct experiments on the online RL settings.
Revised: We then conduct experiments on the online RL settings.

Original: We demonstrate our proposed method inthe MuJoCo locomotion benchmarks.
Revised: We demonstrate our proposed method inthe MuJoCo locomotion benchmarks.

Original: As our implementation is based on TD3, we use TD3-basedvariants as our baselines:
Revised: As our implementation is based on TD3, we use TD3-basedvariants as our baselines:

Original: The TD3 is trained with default settings according to Fujimoto et al.
Revised: The TD3 is trained with default settings according to Fujimoto et al.[14].

Original: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similar to ourwork in using multiple Q -networks in value estimation.
Revised: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similarto our work in using multiple Q -networks in value estimation.

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: [44] but extend it tothe continuous control settings.
Revised: [44] butextend it to the continuous control settings.

Original: Noting that in the continuous control setting, the argmax operatoris approximated by the policy network, multiple policy networks are needed to cooperate with themultiple bootstrapped Q -value networks.
Revised: In continuous control settings, the argmax operator isapproximated by the policy network, multiple policy networks are needed to cooperate with themultiple bootstrapped Q -value networks.

Original: Otherwise, multiple Q -value networks are not independentof each other thus breaking the condition of bootstrapped value estimation.
Revised: Otherwise, multiple Q -value networks are not independentof each other thus breaking the condition of bootstrapped value estimation.

Original: The Ensemble TD3presents the baseline performance when multiple Q -networks are used for value estimation in TD3,which also works as an ablation of our method when all reward shift priors are set to be 0 .
Revised: The Ensemble TD3presents the baseline performance when multiple Q -networks are used for value estimation in TD3,which also works as an ablation of our method when all reward shift priors are set to be 0 .

Original: As has been illustrated in Sec.
Revised: As has been illustrated in Sec.

Original: 4.3.1, learning with different reward shifting values is equivalent tolearning with optimistic or conservative initialization.
Revised: 4.3.1, learning with different reward shifting values is equivalent tolearning with optimistic or conservative initialization.

Original: In our method of Random Reward Shift (RRS),we use 3 Q -networks with different priors.
Revised: In our method of RRS we use 3 Q -networkswith different priors.

Original: In our experiments, we found ± 0 .
Revised: In our experiments, we found ± 0 .

Original: 5 , 0 work universally asa default setting.
Revised: 5 , 0 work universally as a default setting.

Original: Though, further investigation on hyper-parameter may help to further improve theperformance.
Revised: Though, future investigation on hyper-parameter may help to further improve the performance.

Original: Results are shown in Figure 5.
Revised: Results are shown in Figure 5.

Original: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.
Revised: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.

Revised: In the experiment of Bootstrapped TD3 and Ensemble

Original: In all experiments, we  use 3 Q -networks for a faircomparison.
Revised: TD3, we also use 3 Q -networks for a fair comparison.

Original: Note that there is a trade-off between computational complexity and sample efﬁciency, i.e. , using more Q -networks may further improve the performance at the cost of more computationalexpensive, as reported in [44].
Revised: Note that there is a trade-off betweencomputational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expensive, as reported in [44].

Original: More implementation details, pseudo code of RRS, and ablationstudies can be bound in Appendix C.2.
Revised: More implementationdetails, pseudo code of RRS, and ablation studies can be found in Appendix C.2.

Original: 5.3 Optimistic Random Network Distillation
Revised: 5.3 Optimistic Random Network Distillation

Original: [Figure]
Revised: [Figure]

Original: Finally, we experiment with ﬁve discrete exploration tasks, namely the MountainCar-v0, and fournavigation tasks of MiniGrid suite [56], namely the task of Empty-Random, MultiRoom, and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.
Revised: To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, webenchmark with ﬁve discrete exploration tasks, including the classic MountainCar control and four

Original: More environment details are provided in the Appendix C.3.
Revised: MiniGrid navigation tasks [56], environment details can be found at Appendix C.3.

Original: We compare the vanilla DQN , vanilla RND , as well as improved DQN and RND according to ourproposed insight. DQN-0.5 indicates the results with a  reward shifting of   0  . 
Revised: Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1  DQN and2  RND baselines, we use 3  DQN -0.5 to indicate DQN with a  0 . 5 reward shift, and 4  RND -1.05  RND -1.5 to indicate RND with  1 . 0 ,  1 . 5 reward shift, respectively.

Revised: In the following, we use between methods to indicate the former outperforms the latter.

Original: ComparingDQN-0.5 with the vanilla DQN, our insight of negative reward shift leads to curiosity-driven exploration isagain veriﬁed. RND -1.0 indicates the results when a reward shifting of  1 [-. 
-]
Revised: Comparing the results: reward shifting is equivalent to optimistic initialization, thus boosting exploration 3   1 [-. 
-] : a negative reward shift is equivalent to optimistic initialization and helps exploration.

Original: is added to RND basedon DQN. RND -1.0 improves the performance of RND and DQN in most environments, showing theeffectiveness of the RND-based exploration bonus. Note that for  a fair comparison, RND-1.0 shouldbe compared with the vanilla DQN as the  1 . 0 reward shift just cancels the positive explorationbonus introduced by RND.
Revised: RND is effective for value-based exploration — as long as a negative intrinsic reward is used1   2  : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive 325 intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.4   1  ; 5   3  : RND is effective for exploration (i e., improve over DQN) as long as the intrinsic 327 reward bonus is always negative.

Original: Moreover, we can further improve RND by equipping it with reward-shifting-based curiosity-drivenexploration.{+reward shifting for optimistic initialization is orthogonal to other exploration methods
+}
Revised: {+reward shifting for optimistic initialization is orthogonal to other exploration methods
+}

Original: RND -1.5 ( i.e. , RND with a  1 . 5 reward shift) can be compared with the DQN -0.5,as both receive an -0.5 exploration bonus for unseen states.
Revised:   4  : increasing the magnitude of the negatively shifted reward

Original: We ﬁnd in all experiments that ournegative reward shift can remarkably improve exploration, not only working with RND to improveits performance but also working effectively in isolation.
Revised: can further improve the exploration 330 performance — reward shifting can either work in isolation or get combined with other explorationalgorithms as they are working orthogonally.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.
Revised: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.

Original: Although constant reward shifting should not change the optimal policy inducedby the optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.
Revised: Although constant reward shifting should not change the optimal policy inducedby the optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.

Original: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.
Revised: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.

Original: Theproposed idea is then veriﬁed through a variety of application settings.
Revised: Theproposed idea is then veriﬁed through a variety of application settings.

Original: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.
Revised: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.

Original: Importantly, our analysis manifests that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.
Revised: Importantly, our analysis manifests that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.

Original: We empirically verify theeffectiveness of the proposed method on a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.
Revised: We empirically verify theeffectiveness of the proposed method on a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.

Original: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point out the theoretical guidance for such a shifting constant is missing.
Revised: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point out the theoretical guidance for such a shifting constant is missing.

Original: Analysis fromthe perspective of optimization is an important topic, yetis left for future research as it is out of thescope of the current empirical study.
Revised: Analysis fromthe perspective of optimization is an important topic, yet it is out of the scope of the current empiricalstudy and left for future research.