Original: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)
Revised: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).
Revised: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).

Original: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.
Revised: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.

Original: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.

Original: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation beneﬁts the exploration of online RL.
Revised: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation improves exploration for online RL.

Original: We validate our insight ona range of RL tasks and show its improvement over baselines:
Revised: We validate our insight ona range of RL tasks and show its improvement over baselines:

Original: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete exploration tasks, a negative reward shiftingbrings an improvement over theprevious curiosity-based exploration method.
Revised: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete control tasks, a negative reward shiftingyields an improvement over the curiosity-based exploration method.

Original:  Introduction
Revised:  Introduction

Original: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2] specifying a certain reward to incentivize the learning agentrequires domain knowledge and  thorough understanding ofthe task [3–6].
Revised: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2],specifying a certain reward to incentivize the learning agentrequires domain knowledge and a thorough understandingof the task [3–6].

Original: Even with careful design and tuning, learningwith a shaped reward that intends to accelerate learning mayon the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].
Revised: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learningmay on the contrary hinder the learning performance byincurring the sub-optimal behaviors of the agent [7, 8].

Original: Although Ng et al.
Revised: Although Ng et al.

Original: [9] theoretically points the optimal policy remainsunchanged under a special form of reward transformation,and in the later work of Wiewiora et al. [10] a framework isproposed to guide policies with prior knowledge under tabular setting,the investigation of how it accommodates recent Deep Reinforcement Learning (DRL) algorithms remainsmuch less explored.
Revised: [9] theoretically points out that optimalpolicy remains unchanged under certain form of rewardtransformation, and in the later work of Wiewiora et al.[10] a framework is proposed to guide policies with priorknowledge under tabular setting, how reward shifting accommodates recent Deep Reinforcement Learning (DRL)algorithms remains much less explored.

Original: In this work, we study a special linear transformation, whichis the simplest form of reward shaping, in value-based
Revised: In this work, we study a special linear transformation, whichis the simplest form of reward shaping, in value-based

Original: DRL [11–14].
Revised: DRL [11–14].

Original: We start with understanding how such aspeciﬁc kind of reward shaping works in value-based DRLfunction approximations and show that reward shifting is equivalent to different Q -value initialization,extending previous discovery of [10] to the function approximation setting.
Revised: We start with understanding how such aspeciﬁc kind of reward shaping works in value-based DRLfunction approximations and show that reward shifting is equivalent to different Q -value initialization,extending previous discovery of [10] to the function approximation setting.

Original: Figure 1 showcasesreward shifting beneﬁts exploration in a maze game and outperforms count-based and ✏ -greedyexploration and learns near optimal value function.
Revised: Figure 1 showcasesreward shifting beneﬁts exploration in a maze game and outperforms count-based and ✏ -greedyexploration and learns near optimal value function.

Original: Based on such an equivalence, we introduce the key insight of this work:
Revised: Based on such an equivalence, we introduce the key insight of this work:

Original: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.
Revised: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.

Original: We demonstrate the application of such an insight on three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation can lead to improved learning performance based on off-the-shelfalgorithms; (2) for online RL setting, we show multiple value functions with different reward shiftingconstants can be used as a trade-off between exploration and exploitation, thus improving learningefﬁciency; (3) we introduce a simple yet crucial improvement over a prevailing curiosity-basedexploration method, the Random Network Distillation [15], making it compatible with value-based
Revised: We demonstrate the application of such an insight in three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation induced by reward shifting improves learning performance ofoff-the-shelf algorithms; (2) for online RL setting, we show multiple value functions with differentreward shifting constants can be used to trade-off between exploration and exploitation, thus improvelearning efﬁciency; (3) we introduce a simple yet crucial adaptation on a prevailing curiosity-basedexploration algorithm, the Random Network Distillation [15], making it compatible with value-based

Original: DRL algorithms.
Revised: DRL algorithms.

Original: The experiments on both continuous and discrete action space control tasks showsour method bringsthe substantial improvement overprevious baselines.
Revised: Experiments on a diverse set of tasks including continuous and discrete action spacecontrol show our method brings substantial improvement over baselines.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value networkinitialization in value-based deep-RL, which can be applied to curiosity-driven explorationand conservative exploitation;2.
Revised: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network initialization in value-based DRL, which can be applied to curiosity-driven exploration andconservative exploitation;

Original: Practically, we instantiate the key insight to three different scenarios, namely the ofﬂine conservative exploitation, sample-efﬁcient continuous control, and curiosity-driven exploration,to contrast the generality of reward shifting;
Revised: Practically, we instantiate the key insight to three different scenarios, namely the ofﬂine conservative exploitation, sample-efﬁcient continuous control, and curiosity-driven exploration, to contrast the generality of reward shifting;

Original: Empirically, we demonstrate the effectiveness of the proposed method integrated withoff-the-shelf baselines on both continuous and discrete control tasks.
Revised: Empirically, we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks.

Original:  Preliminaries and Related Work2.1 Online RL
Revised:  Preliminaries and Related Work

Revised: 2.1 Online RL

Original: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ), we consider a deterministic transition dynamics 
Revised: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ), we consider a deterministic transition dynamics T :

Original: T : S ⇥
Revised: S ⇥

Original: A 7! S and deterministic reward function R : S ⇥A 7!
Revised: A 7! S and deterministic reward function R : S ⇥

Original:  R . ⇢ 0denotes the initial state 64 distribution, i.e., ⇢ 0 = p ( s 0{+
+} ) .
Revised: A 7! R . ⇢ 0 = p ( s {+
+} ) 2  ( S ) denotes the initial 64 state distribution.

Original:  is the discount factor and T is the episodic decision horizon.
Revised:  is the discount factor and T is the episodic decision horizon.

Original: Online RL considers the problem of learning a policy ⇡ 2 ⇧ : S 7!
Revised: Online RL considersthe problem of learning a policy ⇡ 2 ⇧ : S 7!

Original:  ( A ) (or ⇡ 2 ⇧ : S 7!
Revised:  ( A ) (or ⇡ 2 ⇧ : S 7!

Original: A with a 66 deterministic policy class), such that the expected cumulative rewardin the Markov decision processis maximized, i.e., ⇡ = arg max ⇡ E a t ⇠ ⇡ ,s t +1 ⇠ T ,s 0 ⇠ ⇢
Revised: A with a deterministicpolicy class), such that the expected cumulative reward E a t ⇠ ⇡ ,s t +1 ⇠ T ,s 0 ⇠ ⇢

Original: P
Revised: P

Original: Tt =

Revised: Tt =

Original:  t r t ( s t , a t ) ,
Revised:  t r t ( s t , a t ) inthe Markovian decision process is maximized.

Original: Inthe online RL setting,an agentnormally learns through trials and errors [11],  eitherwith an on-policyparadigm [16–18] oran off-policy manner [12–14, 19, 20].
Revised: In online RL setting, an agent learns through trialsand errors [11], through either on-policy [16–18] or off-policy paradigm [12–14, 19, 20].

Original: In this work, we focus on the off-policy value-based methodswhich are in general more sample efﬁcient.
Revised: In thiswork, we focus on the off-policy value-based methods which are in general more sample efﬁcient.

Original: Speciﬁcally, our discussions assume the policy learning isbased on a learned Q -value function, that approximates the cumulative reward an agent can gain in thefollowing part of an episode.
Revised: Speciﬁcally, our discussions assume the policy learning is based on a learned Q -value functionthat approximates the cumulative reward an agent can gain in the following part of an episode.

Original: The Q -value function is deﬁned as Q ( s t , a t ) =
Revised: Formally, Q ( s t , a t ) =

Original: E ⇡ , T
Revised: E ⇡ , T

Original: P
Revised: P

Original: T ⌧ = t  t r ( s ⌧ , a ⌧ ) ,and can be estimated by propagating the Bellman operator B Q ( s, a ) =
Revised: T ⌧ = t  t r ( s ⌧ , a ⌧ ) , and can be estimated by propagating the Bellmanoperator B Q ( s, a ) =

Original: r ( s, a ) +  E Q ( s 0 , a 0 ) .
Revised: r ( s, a ) +  E Q ( s 0 , a 0 ) .

Original: Forvalue-based methods, the (soft-)optimal policy is then produced by
Revised: For value-based methods, the (soft-)optimal policy isthen produced by

Original: [Equation]
Revised: [Equation]

Original: where Q ⇤ is  optimal Q -value function.
Revised: where Q ⇤ is the optimal Q -value function.

Original: We can also set the temperature parameter close as 0 tohave the deterministic policy class.
Revised: We can also set the temperature parameter close to 0to have the deterministic policy class.

Original: Simplifying the notion we have ⇡ ⇤ ( s ) =
Revised: Simplifying the notion we have ⇡ ⇤ ( s ) =

Original: arg max a Q ⇤ ( s, a ) .
Revised: arg max a Q ⇤ ( s, a ) .

Original: Algorithms like DPG [21] can be used to address the intractable analytical argmax issue  arises incontinuous action space.
Revised: Algorithms like DPG [21] can be used to address the intractable analytical argmax issue that arises incontinuous action space.

Original: We choose to develop our work on top of prevailing baseline algorithms of
Revised: We choose to develop our work on top of prevailing baseline algorithms of

Original: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.
Revised: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.

Original: Itshould be straightforward to extend the method on top of other learning algorithms.
Revised: Itshould be straightforward to extend the method on top of other learning algorithms.

Original: 2.2 Exploration and the Curiosity-Driven Methods
Revised: 2.2 Exploration and the Curiosity-Driven Methods

Original: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent must  learn to exploit its accumulated knowledge on the task while exploring new states andactions.
Revised: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent must simultaneously learn to exploit its accumulated knowledge on the task and explore newstates and actions.

Original: Previous works address the exploration problem from various perspectives: for discrete
Revised: Previous works address the exploration problem from various perspectives: fordiscrete action space tasks, count-based methods like [27–29] are proposed to motivate the revisiting

Original: [Table]
Revised: [Table]

Original: action space tasks, count-based methods like [27–29] are proposed to motivate the revisiting ofunder-explored state-action pairs.
Revised: of under-explored state-action pairs.

Original: Speciﬁcally, Choshen et al.
Revised: Speciﬁcally, Choshen et al.

Original: [30] extended the idea into generalsettings by constructing an additional MDP for Exploration-value estimation, as a generalized counterfor count-based exploration.
Revised: [30] extended the idea into generalsettings by constructing an additional MDP for Exploration-value estimation, as a generalized counterfor count-based exploration.

Original: To boost exploration in continuous state tasks, curiosity-driven methodsare investigated by [31–33, 15], where variety of intrinsic rewards are designed as supplementary tothe primal task reward for better exploration.
Revised: To boost exploration in continuous state tasks, curiosity-driven methodsare investigated by Houthooft et al. [31], Pathak et al. [32], Burda et al. [33, 15], where variety ofintrinsic rewards are designed as supplementary to the primal task reward for better exploration.

Original: Self-imitate approaches like Oh et al.
Revised: Selfimitate approaches like Oh et al.

Original: [34], Ecoffet et al.[35], Sun et al.
Revised: [34], Ecoffet et al. [35], Sun et al.

Original: [36] repeat success trajectories but require extra assumptions on the environment.
Revised: [36] repeat success trajectoriesbut require extra assumptions on the environment.

Original: Thework of DIAYN and DADS [37, 38] show that various skills can be developed even without the primalextrinsic reward.
Revised: The works of DIAYN and DADS [37, 38] showthat various skills can be developed even without the primal extrinsic reward.

Original: For continuous control tasks, OAC [26] improves the SAC [20] with informativeaction space noise based on the optimism in face of uncertainty (OFU) [39–42].
Revised: For continuous controltasks, OAC [26] improves the SAC [20] with informative action space noise based on the optimism inface of uncertainty (OFU) [39–42].

Original: GAC [43] addresses[Table] the exploration issue with a richer functional class for the policy.
Revised: GAC [43] addresses the exploration issue with a richer functionalclass for the policy.

Original: In the work of Rashid et al.
Revised: In the work of Rashid et al.

Original: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.
Revised: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.

Original: Inthe work of Osband et al.
Revised: Inthe work of Osband et al.

Original: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.
Revised: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.

Original: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal
Revised: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal

Original: Q -value, and its ﬂexibility to be plugged in to both continuous and discrete control algorithms.
Revised: Q -value, and its ﬂexibility to be plugged into both continuous and discrete control algorithms.

Original: Random Network Distillation (RND) [15] deﬁnes the intrinsic reward as the output difference betweena ﬁxed neural network  1 and a trainable network  2 given state-actions as the inputs e.g.,
Revised: Random Network Distillation (RND) [15] deﬁnes the intrinsic reward as the output difference betweena ﬁxed neural network  1 and a trainable network  2 given state-actions as the inputs e.g.,

Original: [Equation]
Revised: [Equation]

Original: when outputs of both networks are activated by a sigmoid function, and  2 is optimized to approximate 1 for the visited ( s, a ) pairs 1 . Hence, the value of r int ( s, a ) will decay to 0 when such state-actionpairs  arevisited frequently  but remain high for  seldom visited pairs.
Revised: where both networks are activated by a sigmoid function. After optimizing the learnable  2 toapproximate  1 with seen ( s, a ) pairs 1 , the value of r int ( s, a ) will decay to 0 for such state-actionpairs that are frequently visited but remain high for thoase are seldom visited.

Original: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.
Revised: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.

Original: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated by high function approximation error in theunder-explored state-action pairs.
Revised: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated by high function approximation error in theunder-explored state-action pairs.

Original: We demonstrate suchan insight by showing that RND in itsoriginal design is not suitable for value-based methods in developing exploratory behaviors, butintegrating RND with a shifted reward function remarkably improves the learning performance.
Revised: We demonstrate such insight by showing that RND in its originaldesign is not suitable for value-based methods in developing exploratory behaviors, but integrating RND with a shifted reward function remarkably improves the learning performance.

Original: 2.3 Ofﬂine RL
Revised: 2.3 Ofﬂine RL

Original: The ofﬂine RL, also known as batch-RL, focuses on the problems wherethe interaction with theenvironment is impossible, and the policy can only be optimized based on  logged dataset.
Revised: The ofﬂine RL, also known as batch-RL, focuses on the problems where interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset.

Original: In thosetasks, a ﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ] , collected from some unknown behavior policy ⇡  , is118 provided.
Revised: In those tasks, aﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ] , collected from some unknown behavior policy ⇡  , is provided.

Original:  In general, such a dataset can either be generated by rolling out an expert that generateshigh-quality solutions to the task [22, 45, 46] or a non-expert that executes sub-optimal behaviors [47–49, 46, 50] or be a mixture of both [23].
Revised: 119 In general, such a dataset can either be generated by rolling out an expert that generates high-qualitysolutions to the task [22, 45, 46] or a non-expert that executes sub-optimal behaviors [46–50] or bea mixture of both [23].

Original: As the agent in the ofﬂine RL setting can not correctits potentially biased knowledge through interactions, the most important issue is to address theextrapolation error [22] induced by distributional mismatch [48].
Revised: As the agent in the ofﬂine RL setting can not correct its potentially biasedknowledge through interactions, the most important issue is to address the extrapolation error [22]induced by distributional mismatch [48].

Original: To address such an issue, a series ofalgorithms optimize the policy learning under the constraint of distributional similarity [48, 47, 51].
Revised: To address such an issue, a series of algorithms optimizethe policy learning under the constraint of distributional similarity [47, 48, 51].

Original: Bharadhwaj et al.
Revised: Bharadhwaj et al.

Original: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.
Revised: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.

Original: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions (e.g., uniform distribution over the action space).
Revised: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions (e.g., uniform distribution over the action space).

Original: As we will show inthis work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.
Revised: As we will show inthis work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.

Original: This idea leads to the direct application of our proposedframework in the ofﬂine setting.
Revised: This idea leads to the direct application of our proposedframework in the ofﬂine setting.

Original: In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22,48, 47,  51] to further improve the performance with conservative Q -valueestimation.
Revised: In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22, 47, 48, 51] to further improve the performance with conservative Q -valueestimation.

Original: Table 1 contextualizes reward shifting with respect to related works we discussed above, morediscussions on related works can be found at Appendix E.
Revised: Table 1 contextualizes reward shifting with respect to related works we discussed above, morediscussions on related works can be found at Appendix E.

Original:  A Motivating Example
Revised:  A Motivating Example

Original: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing our method.
Revised: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing our method.

Original: Remark 1.
Revised: Remark 1.

Original: Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 139 with linear transformation, i.e.,
Revised: Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 140 with linear transformation, i.e.,

Original: R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , doesnot change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=
Revised: R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , doesnot change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=

Original: [Equation]
Revised: [Equation]

Original:  0 t :
Revised:  t r 0 t :

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: When |A| = 1 , scaling the reward function with linear transformation does not change 142 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k : 
Revised: When |A| = 1 , scaling the reward function with linear transformation does not change 143 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k : r ✓ J ( µ ✓ ) = E s t [ r a [Equation]

Revised: ⇤

Original: [Equation]
Revised: [Equation]

Revised: s t

Revised: [Equation]

Revised: a t

Revised: [Equation]

Revised: | a t = µ ✓

Revised: [Equation]

Revised: s t )

Revised: r ✓ µ ✓

Revised: [Equation]

Revised: ] =

Revised: E s t [ r a

Revised: [Equation]

Revised: ⇤ k

Original: [Equation]
Revised: [Equation]

Revised: b

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: a t

Revised: [Equation]

Revised: | a t = µ ✓

Revised: [Equation]

Revised: s t

Revised: [Equation]

Original:  Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function.
Revised: ✓ µ ✓ [Equation] ] /k, (4) Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function calculated by the shifted reward.

Original: However, Figure 1 presents “counter-intuitive”results that in a demonstrative exploration task of Grid World.
Revised: However, Figure 1presents “counter-intuitive” results in a demonstrative exploration task of Grid World.

Original: In this task, an agent is initializedat the upper left corner of the map need to explorethe map without reward unless it reaches thegoal point located at the lower right corner, where a positive reward of +1 will be assigned.
Revised: In this task, anagent located at the upper left corner of a map needs to explore without reward and reach the goalpoint located at the lower right corner. A non-trivial reward of +1 will be assigned only when the goalis reached.

Original: Wereport learning curves with regard to  episodic success rate in reaching the goal point and the learned Q -values.
Revised: We report learning curves with regard to the episodic success rate in reaching the goalpoint and the learned Q -values.

Original: In this toy example, we ﬁnd a negative reward shifting remarkably boosts the learningefﬁciency of Q -learning, and surpasses the  count-based method that is classic for exploration.
Revised: In this toy example, we ﬁnd a negative reward shifting remarkablyboosts the learning efﬁciency of Q -learning, and surpasses the conventional count-based method forexploration.

Original: Moreover, Remark 1 is empirically veriﬁed with such a toy example, i.e., reward shifting does notchange the optimal Q -value as well as its induced policy, but on the contrary, accelerating the learningof (near)-optimal policy.
Revised: Moreover, Remark 1 is empirically veriﬁed with such a toy example: reward shiftingdoes not change the optimal Q -value as well as its induced policy, but on the contrary accelerates thediscovery of (near-)optimal policy.

Original: In the following of this work, we study the effect of varying constant bias b and ﬁx the scaling factork = 1 to avoid trivial discussions on the choices of learning rate — there is no surprise that changingan appropriate learning rate can be empirically important.
Revised: In the following of this work, we study the effect of varying constant bias b and ﬁx the scaling factork = 1 to avoid trivial discussions on the choices of learning rate — there should be no surprise thatchoosing an appropriate learning rate is empirically important.

Original: We focus on revealing the importanceof selecting the universal bias term b in the reward function through the lens of initialization priorsin function approximation, and show such a bias is generally helpful for both online and ofﬂinesettings.
Revised: We focus on revealing the importanceof selecting the universal bias term b in the reward function through the lens of initialization priorsin function approximation and show such a bias is generally helpful for both online and ofﬂinesettings.

Original: In the online settings it improves learning efﬁciency, and in the ofﬂine settings it promotesconservative exploitation.
Revised: In the online settings, it improves learning efﬁciency, and in the ofﬂine settings, it promotesconservative exploitation.

Original:  Shifted Priors for Q -value Estimation
Revised:  Shifted Priors for Q -value Estimation

Original: 4.1 Reward Shift Equals to Different Initialization
Revised: 4.1 Reward Shift Equals to Different Initialization

Original: We start by formally introducing notions and the key idea of this work: reward shifting equals differentinitialization.
Revised: We start by formally introducing notions and the key idea of this work: reward shifting equals differentinitialization.

Original: We use Figure 2 to illustrate how reward shifting affects function approximation, hencechanging the learning dynamics for value-based algorithms.
Revised: We use Figure 2 to illustrate how reward shifting affects function approximation, hencechanging the learning dynamics for value-based algorithms.

Original: The original optimal Q -value functionis denoted as Q ⇤ , and plotted in the ﬁgures as Black curves.
Revised: The original optimal Q -value functionis denoted as Q ⇤ , and plotted in the ﬁgures as Black curves.

Original: We then denote the shifted optimal
Revised: We then denote the shifted optimal

Original: Q -value function as Q ⇤ b + , which is the Q -value function with the shifted reward
Revised: Q -value function as Q ⇤ b + , which is the Q -value function with the shifted reward

Original: r 0 = r
Revised: r 0 = r

Original: + b + .
Revised: + b + .

Original: Weuse Red curves in ﬁgures to denote those shifted Q -value functions.
Revised: Weuse Red curves in ﬁgures to denote those shifted Q -value functions.

Original: In this section we provide
Revised: In this section we provide

Original: ! ∗ ! "
Revised: ! ∗ ! "

Original: # ! 1 − & ! " !
Revised: # ! 1 − & ! " !

Original: ∗ = !
Revised: ∗ = !

Original: ∗ + " !
Revised: ∗ + " !

Original: $%& "# ! ≈0 "
Revised: $%& "# ! ≈0 "

Original: " ! '! ' − !
Revised: " ! '! ' − !

Original: ∗ < '! '
Revised: ∗ < '! '

Original: %$ −!
Revised: %$ −!

Original: ∗ ): '! '
Revised: ∗ ): '! '

Original: → !
Revised: → !

Original: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !
Revised: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !

Original: ∗ < $! " ! ,$%& −! " !
Revised: ∗ < $! " ! ,$%& −! " !

Original: ∗ ): '! " ! ,' → ! " !
Revised: ∗ ): '! " ! ,' → ! " !

Original: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!
Revised: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!

Original: +,,' = '! " ! ,' − # ! 1 − & '!
Revised: +,,' = '! " ! ,' − # ! 1 − & '!

Original: +,,- = '!
Revised: +,,- = '!

Original: - − # ! 1 − & ( r 0 = r + b + .
Revised: - − # ! 1 − & ( r 0 = r + b + .

Original: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized as ˜ Q LB , 0 ⇡ ˜ Q 0  b + 1   , rather than ˜ Q 0 ⇡ 0 .
Revised: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized as ˜ Q LB , 0 ⇡ ˜ Q 0  b + 1   , rather than ˜ Q 0 ⇡ 0 .

Original: i.e., by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value.
Revised: i.e., by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value.

Original: − 𝑏 + 1−𝛾 𝑸
Revised: − 𝑏 + 1−𝛾 𝑸

Original: 𝓐
Revised: 𝓐

Original: ෩𝑸
Revised: ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐
Revised: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐

Original: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =
Revised: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =

Original: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 

Revised: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸

Original: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾෩𝑸
Revised: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾

Revised: [Figure]

Original: UB,𝑡
Revised: UB,𝑡

Original: =
Revised: [Figure]

Original: ෩𝑸 𝑏 − ,𝑡 −𝑏 − 1−𝛾
Revised: − , [Figure] −

Original: [Figure]
Revised: [Figure]

Original: analysis with a positive bias b + > 0 to illustrate how positively shifted value function affects functionapproximation, and coordinates the normally applied near-zero initialized function approximators(e.g., neural networks.
Revised: analysis with a positive bias b + > 0 to illustrate how positively shifted value function affects functionapproximation, and coordinates the normally applied near-zero initialized function approximators(e.g., neural networks.

Original: Orange curves in the ﬁgure) to inspire conservative behaviors.
Revised: Orange curves in the ﬁgure) to inspire conservative behaviors.

Original: The discussionof negative biases is elaborated in the next section.
Revised: The discussionof negative biases is elaborated in the next section.

Original: To summarize, shifting the reward function with a positive constant is equivalent to initializing thevaluefunction network with a smaller value — as the Q -value of unseen state-action pairs duringtraining are much lower than their shifted optimal value, those actions will not be selected in argmaxstyle policy updates, leading to conservative learning behaviors that beneﬁt policy learning in ofﬂinesettings [23].
Revised: To summarize, shifting the reward function with a positive constant is equivalent to initializing thevalue network with a smaller value — as the Q -value of unseen state-action pairs during training aremuch lower than their shifted optimal value, those actions will not be selected in argmax-style policyupdates — leading to conservative learning behaviors that beneﬁt policy learning in ofﬂine settings.

Original: 4.2 Conservative Exploitation
Revised: 4.2 Conservative Exploitation

Original: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emergewith a positively shifted reward.
Revised: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emergewith a positively shifted reward.

Original: As such a value estimation empirically lower bounds the optimal
Revised: And such a value estimation empirically lower-bounds the optimal

Original: Q -value function.
Revised: Q -value function.

Original: As shown in Figure 2(a), a positive constant b + added to the reward function willlead to a uniformly (positively) shifted optimal Q -value function, and the gap between the primal
Revised: As shown in Figure 2(a), a positive constant b + added to the reward functionwill lead to a universal positively shifted optimal Q -value function, and the gap between the primal

Original: Q -value function and the new one is b + 1   .
Revised: Q -value function and the new one is b + 1   .

Original: Optimizing the Q -value function with logged data (e.g.,a ﬁxed replay buffer) will minimize the difference between the predicted value and the optimalvalue with observed data.
Revised: Optimizing the Q -value approximator with logged data willminimize the difference between the predicted value and the optimal value with observed data.

Original: For the unobserved data point (in the state-action space S ⇥ A ), the near-zero initialization guarantees the prediction is lower than the  optimal Q -value, thus conservativeexploitation can be conducted with such a value function.
Revised: Forthe unobserved data, the pessimistically initialized Q -value approximator guarantees the prediction islower than the positively shifted optimal ones, thus conservative exploitation can be achieved withargmax-style value propagation and action execution on such a value function.

Original: 4.3 (Curiosity-Driven) Optimistic Exploration
Revised: 4.3 (Curiosity-Driven) Optimistic Exploration

Original: On the other hand, if we shift the reward function to the negative side, it is equivalent to  optimisticinitialization.
Revised: On the other hand, if we shift the reward function to the negative side, it is equivalent to an optimisticinitialization.

Original: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.
Revised: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.

Original: With sufﬁciently small b  (so that | b  | larger than the maximal value of any ( s, a ) -pair), such anupper bound of Q ⇤ can be used to conduct curiosity-driven exploration.
Revised: With sufﬁciently small b  (so that | b  | larger than the maximal value of any ( s, a ) -pair), such anupper bound of Q ⇤ can be used to conduct curiosity-driven exploration.

Original: Intuitively, initializing avalue network that  predicts  value largeror equal than the true  value will lead to curiosity-drivenexploration, as any visited state will be assigned a relatively smaller value and the argmax-style policywill tend to choose under-explored actions.
Revised: Intuitively, initializing avalue network that always predicts a value larger than the true optimal value will lead to curiositydriven exploration behavior, as any visited state will be assigned a relatively smaller value and theargmax-style policy will tend to choose under-explored actions.

Original: Based on the analysis above that (1) a positive constant shift added to the reward function can beused for conservative policy update and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., theoptimistic estimation with b  , and the lower bound, i.e., the conservative estimation with b + of theoptimal value function.
Revised: Based on the analysis above that (1) a positive constant shift added to the reward function can be 198 used for conservative policy updates and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., theoptimistic estimation with b  , and the lower bound, i.e., the conservative estimation with b + of theoptimal value function.

Original: Formally, we use
Revised: Formally, we use

Original: [Equation]
Revised: [Equation]

Original: 5) to denote the empirical lower bound of value estimation (cf.
Revised: 5) to denote the empirical lower bound of value estimation (cf.

Original: Figure 2(d), orange line), and
Revised: Figure 2(d), Orange line), and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  to denote the empirical upper bound of value estimation (cf.
Revised: 6) to denote the empirical upper bound of value estimation (cf.

Original: Figure 3(b)).
Revised: Figure 3(b), Orange line).

Original: In both notions, t denotesthe optimization step.
Revised: In bothnotions, t denotes the optimization step.

Original: e.g., when t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 , ˜ Q LB , 0 ( s, a ) =
Revised: When t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 ,˜ Q LB , 0 ( s, a ) =

Original:  b + 1  is able to lower bound the unknown optimal Q -value given sufﬁciently large b + .
Revised:  b + 1   is able to lower bound the unknown optimal Q -value given sufﬁciently largeb + .

Original: (cf.
Revised: (cf.

Original: Figure 2(d),green line).
Revised: Figure 2(d), Green line).

Original: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufﬁciently small b  .
Revised: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with asufﬁciently small b  .

Original: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous control anddiscrete action space respectively.
Revised: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous control anddiscrete action space respectively.

Original: We propose a practical algorithm for sample-efﬁcient continuouscontrol in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the
Revised: We propose a practical algorithm for sample-efﬁcient continuouscontrol in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the

Original: RND [15], in Section 4.3.2.
Revised: RND [15], in Section 4.3.2.

Original: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift
Revised: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift

Original: According to the principle of optimism on the face of uncertainty (OFU), an exploration bonus thatmanifests the uncertainty of the Q -value function can be introduced into the value estimation: thebasic idea starts from integrating optimistic exploration with conservative exploitation,i.e.,
Revised: According to the principle of optimism in the face of uncertainty (OFU), an exploration bonus 214 that manifests the uncertainty of the Q -value function can be introduced into the value estimation:integrating optimistic exploration with conservative exploitation,

Original: [Equation]
Revised: [Equation]

Original: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.
Revised: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.

Original: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neuralnetwork, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequence ofinitialization (detailed implications are provided in Appendix B).
Revised: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neuralnetwork, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequence ofinitialization (detailed implications are provided in Appendix B).

Original: Hence, the explorative bonusbecomes  (1   ) b
Revised: Hence, the explorative bonusbecomes  (1   ) b

Original: + +  b   . Note this is equivalent to applying another constant reward shift with valueof c r = (1   ) b + +  b  .
Revised: + +  b   , which is equivalent to applying another constant reward shift with value ofc r = (1   ) b + +  b  , formally, we have

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Assuming we have access to an unbiased estimator for the optimal value function
Revised: Assuming we have access to an unbiased estimator for the optimal value function

Original: Q ⇤ , e.g., with Monte-Carlo estimation ˆ Q ⇤ =
Revised: Q ⇤ , e.g., with Monte-Carlo estimation ˆ Q ⇤ =

Original: E
Revised: E

Original: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e., ✏ 2 t
Revised: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e., ✏ 2 t

Original: =
Revised: =

Original: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =
Revised: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =

Original: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r
Revised: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r

Original: =
Revised: =

Original: (1   ) b
Revised: (1   ) b

Original: + +  b  .
Revised: + +  b  .

Original: The proof can be found in the Appendix A.
Revised: The proof can be found in the Appendix A.

Original: According to Proposition 4.1, a grid search for tradingoff between the three hyper-parameters: the exploration bias b  , the exploitation bias b + and thecoefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r
Revised: According to Proposition 4.1, a grid search for trading-offbetween the three hyper-parameters, i.e., the exploration bias b  , the exploitation bias b + and the coefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r

Original: = (1   ) b + + b  , indicating that
Revised: = (1   ) b + + b  , indicating that

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.
Revised: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.

Original: The corollary indicates that, in principle, a meta-learner can be trained to monitor the learning processand select a proper reward shifting constant automatically [52, 53].
Revised: The corollary says, in principle, a meta-learner can be trained to monitor the learning process andselect a proper reward shifting constant automatically [52, 53] to balance exploration and exploitation.

Original: For the ease of exposition, in thiswork we choose to focus on the simplest yet effective uniform sampling strategy from multiple shiftconstants, which has been shown as a strong baseline of those meta-learner approaches [54, 55], andleave more complicated meta-learner-based approaches for future investigation.
Revised: For the ease of exposition, in this work we choose to focus on the simplest yet effective uniformsampling strategy from multiple shift constants, which has been shown as a strong baseline of thosemeta-learner approaches [54, 55], and leave more complicated meta-learner-based shifting constantadjustment for future investigation.

Original: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.
Revised: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.

Original: In propagating
Revised: In propagating

Original: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with anew reward with differentshifted constant bias in those Q-networks.
Revised: Q-values through the temporal difference loss, the primal recorded reward value r is replacedwith shifted rewards with different constant biases to update their individual Q-networks.

Original: We then uniformly sample oneof those learned Q-networks for the optimization of policy network (with e.g., DPG [21]).
Revised: We thenuniformly sample one of those learned Q-networks for the optimization of policy networks (e.g., with DPG [21]).

Original: It is worthnoting that our approach only requires post-hoc revision of the primal reward function, rather thaninteracting with the environment multiple times to collect samples for each value network.
Revised: It is worth noting that our approach only requires post-hoc revision of the primal rewardfunction, rather than interacting with the environment multiple times to collect samples for each valuenetwork.

Original: We dubthe proposed method Random Reward Shift (RRS), and provide the pseudo-code inAlgorithm 1 of
Revised: We dub the proposed method Random Reward Shift (RRS), and provide the pseudo-code in

Original:  Appendix C.2.
Revised: Algorithm 1 of Appendix C.2.

Original: 4.3.2 Tailored Curiosity-Driven Exploration for Value-Based Method
Revised: 4.3.2 Tailored Curiosity-Driven Exploration for Value-Based Method

Original: In previous works, the curiosity-driven exploration methods are always work with policy-basedmethods.
Revised: In previous works, the curiosity-driven exploration methods are always work with policy-basedmethods.

Original: In this section, we cast the key insight introduced above to RND [15] — one of theleading algorithms for exploration — to answer why is its vanilla design not suitable for value-basedalgorithms like Q-learning, and propose a variant of RND that is tailored for DQN.
Revised: In this section, we cast the key insight introduced above to RND [15] — one of theleading algorithms for exploration — to answer why is its vanilla design not suitable for value-basedalgorithms like Q-learning, and propose a variant of RND that is tailored for DQN.

Original: The vanilla RND use two randomly initialized networks  1 and  2 to generate the intrinsic rewardr int = |  1 ( s, a )   2 ( s, a ) |  0 for exploration.
Revised: The vanilla RND use two randomly initialized networks  1 and  2 to generate the intrinsic rewardr int = |  1 ( s, a )   2 ( s, a ) |  0 for exploration.

Original: During learning,  2 is a ﬁxed network and theparameters of  1 is optimized to approximate the output of  2 for frequently-visited states.
Revised: During learning,  2 is a ﬁxed network and theparameters of  1 is optimized to approximate the output of  2 for frequently-visited states.

Original: We follow
Revised: We follow

Original: Burda et al.
Revised: Burda et al.

Original: [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic rewardafter t step of optimization.
Revised: [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic rewardafter t step of optimization.

Original: Speciﬁcally, r int , 0 ( s, a ) = 1 , 8 ( s, a ) — an universally positive bonus isadded to the primal reward function at beginning.
Revised: Speciﬁcally, r int , 0 ( s, a ) = 1 , 8 ( s, a ) — an universally positive bonus isadded to the primal reward function at beginning.

Original: According to our analysis in previous sections, such a positive reward shift is equivalent to pessimisticinitialization and will lead to conservative behaviors in Q -value estimation.
Revised: According to our analysis in previous sections, such a positive reward shift is equivalent to pessimisticinitialization and will lead to conservative behaviors in Q -value estimation.

Original: Therefore, the explorationbehaviors at the beginning of learning will be hindered, rather than boosted.
Revised: Therefore, the explorationbehaviors at the beginning of learning will be hindered, rather than boosted.

Original: To overcome sucha conservative behavior induced by the pessimistic initialization, we proposed to use r  int ( s, a ) =|
Revised: To overcome sucha conservative behavior induced by the pessimistic initialization, we proposed to use r  int ( s, a ) =|

Original:  1 ( s, a )   2 ( s, a ) | 2  I, 8 s, a , where I is a positive constant that assures r  int  0 is negatively 262 initialized for optimistic exploratory behaviors.
Revised:  1 ( s, a )   2 ( s, a ) | 2  I, 8 s, a , where I is a positive constant that assures r  int  0 is negatively 264 initialized for optimistic exploratory behaviors.

Original: The Chicken-and-Egg Problem To further understand the difference between the curiosity-drivenmethod in ways of intrinsic reward and our reward shifting-based optimistic initialization, let usconsider when do those curiosities work in each case: for intrinsic reward methods, an agent mustﬁrst experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hardwith an arg-max style policy.
Revised: The Chicken-and-Egg Problem To further understand the difference between the curiosity-drivenmethod in ways of intrinsic reward and our reward shifting-based optimistic initialization, let usconsider when do those curiosities work in each case: for intrinsic reward methods, an agent mustﬁrst experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hardwith an arg-max style policy.

Original: On the other hand, with optimistic initialization, the rarely-visited ( s, a )pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairshave updated their values with a negatively shifted reward.
Revised: On the other hand, with optimistic initialization, the rarely-visited ( s, a )pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairshave updated their values with a negatively shifted reward.

Original: From such a perspective, reward shiftingnot only works by itself motivates exploratory behaviors, but can also be seamlessly plugged intointrinsic reward methods to encourage the ﬁrst visitation of new states.
Revised: From such a perspective, reward shiftingnot only works by itself motivates exploratory behaviors, but can also be seamlessly plugged intointrinsic reward methods to encourage the ﬁrst visitation of new states.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Ofﬂine RL with Conservative Q -value Estimation
Revised: 5.1 Ofﬂine RL with Conservative Q -value Estimation

Original: We start our experiments by demonstrating the effectiveness of reward shifting in ofﬂine RL benchmarks.
Revised: We start our experiments by demonstrating the effectiveness of reward shifting in ofﬂine RL benchmarks.

Original: As discussed in Section 4.2, a positive reward shift is equivalent to pessimistic initializationthat beneﬁts conservative exploitation.
Revised: As discussed in Section 4.2, a positive reward shift is equivalent to pessimistic initializationthat beneﬁts conservative exploitation.

Original: In general, our proposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm, we choose to verify the effectiveness and generality of such aconservative Q -value estimation based on BCQ [22] and CQL [23], i.e., both distribution-matchingapproach and conservative value estimation approaches in ofﬂine RL.
Revised: In general, our proposed method can be plugged in to anyoff-the-shelf ofﬂine RL algorithm, we choose to verify the effectiveness and generality of such a conservative Q -value estimation based on BCQ [22] and CQL [23], i.e., both distribution-matchingapproach and conservative value estimation approaches in ofﬂine RL.

Original: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.
Revised: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.

Original: Figure 4 shows our experiment results.
Revised: Figure 4 shows our experiment results.

Original: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [46] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.
Revised: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [46] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.

Original: The ﬁrst two panels showresults with BCQ as the backbone algorithm.
Revised: The ﬁrst two panels showresults with BCQ as the backbone algorithm.

Original: We observe  clear performance drop when training with BCQ as the policy overﬁts the given dataset in these settings, on the other hand, our proposed rewardshift can alleviate such a problem and outperform BCQ w.r.t. both best achieved performance andconvergence performance.
Revised: We observe a clear performance drop during the trainingof BCQ as the policy overﬁts the batched dataset. Differently, positive reward shift can alleviate sucha problem and outperform BCQ in terms of both best-achieved performance and performance afterconvergence.

Original: The following three panels in Figure 4 use CQL as the backbone.
Revised: The following three panels in Figure 4 use CQL as the backbone.

Original: In allexperiments, using a positive reward shift leads to improved learning performance, while a negativereward shift leads to performance decay, as expected.
Revised: In all experiments,shifting the reward with a positive constant improves learning performance, while a negative rewardshift impedes to efﬁcient learning — as expected.

Original: Implementation details and more results can befound in Appendix C.1.
Revised: Implementation details and more results can befound in Appendix C.1.

Original: 5.2 Online RL with Randomized Priors
Revised: 5.2 Online RL with Randomized Priors

Original: We then conduct experiments on the online RL settings.We demonstrate our proposed method inthe MuJoCo locomotion benchmarks.
Revised: We then conduct experiments in the MuJoCo locomotion benchmarks to demonstrate reward shiftingimproves learning efﬁciency in the online RL settings.

Original: As our implementation is based on TD3, we use TD3-basedvariants as our baselines:
Revised: As our implementation is based on TD3,we use TD3-based variants as our baselines:

Original: The TD3 is trained with default settings according to Fujimoto et al.[14].
Revised: The TD3 is trained with default settings accordingto Fujimoto et al.

Original: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similarto our work in using multiple Q -networks in value estimation.
Revised: We also include Ensemble TD3 and Bootstrapped TD3 as baselinesdue to they are similar to our work in using multiple Q -networks in value estimation.

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: [44] butextend it to the continuous control settings.
Revised: [44] but extend it to the continuous control settings.

Original: In continuous control settings, the argmax operator isapproximated by the policy network,  multiple policy networks are needed to cooperate with themultiple bootstrapped Q -value networks.
Revised: In continuous control settings, theargmax operator is approximated by the policy network, and multiple policy networks are needed tocooperate with the multiple bootstrapped Q -value networks.

Original: Otherwise, multiple Q -value networks are not independentof each other thus breaking the condition of bootstrapped value estimation.
Revised: Otherwise, multiple Q -value networksare not independent of each other thus breaking the condition of bootstrapped value estimation.

Original: The Ensemble TD3presents the baseline performance when multiple Q -networks are used for value estimation in TD3,which also works as an ablation of our method when allreward shift priors are set to be 0 .
Revised: The Ensemble TD3 presents the baseline performance when multiple Q -networks are used for valueestimation in TD3, and works as an ablation of our method where all shift priors are set to be 0 .

Original: As has been illustrated in Sec.
Revised: As has been illustrated in Sec.

Original: 4.3.1, learning with different reward shifting values is equivalent tolearning with optimistic or conservative initialization.
Revised: 4.3.1, learning with different reward shifting values is equivalentto learning with optimistic or conservative initialization.

Original: In our method of RRS we use 3 Q -networkswith different priors.
Revised: In our instantiating of RRS, we use3 Q -networks with different priors.

Original: In our experiments, we found ± 0 .
Revised: We empirically ﬁnd ± 0 .

Original: 5 , 0 work universally as a default setting.
Revised: 5 , 0 work universally good for allenvironments.

Original: Though, future investigation on hyper-parameter may help to further improve the performance.
Revised: Though, future investigation on hyper-parameter may help to further improve theperformance.

Original: Results are shown in Figure 5.
Revised: Results are shown in Figure 5.

Original: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.
Revised: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.

Original: In the experiment of Bootstrapped TD3 and Ensemble
Revised: In the experiment of Bootstrapped TD3 and Ensemble

Original: TD3, we also use 3 Q -networks for a fair comparison.
Revised: TD3, we also use 3 Q -networks for a fair comparison.

Original: Note that there is a trade-off betweencomputational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expensive, as reported in [44].
Revised: Note that there is a trade-off betweencomputational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expenses, as reported in [44].

Original: More implementationdetails, pseudo code of RRS, and ablation studies can be found in Appendix C.2.
Revised: More implementationdetails, pseudo-code of RRS, and ablation studies can be found in Appendix C.2.

Original: 5.3 Optimistic Random Network Distillation
Revised: 5.3 Optimistic Random Network Distillation

Original: [Figure]

Original: To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, webenchmark with ﬁve discrete exploration tasks, including the classic MountainCar control and four
Revised: To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, webenchmark with ﬁve discrete exploration tasks, including the classic MountainCar control and four

Original: MiniGrid navigation tasks [56], environment details can be found at Appendix C.3.
Revised: MiniGrid navigation tasks [56], environment details can be found at Appendix C.3.

Original: Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1  DQN and2  RND baselines, we use 3  DQN -0.5 to indicate DQN with a  0 .
Revised: Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1  DQN and2  RND baselines, we use 3  DQN -0.5 to indicate DQN with a  0 .

Original: 5 reward shift, and 4  RND -1.05  RND -1.5 to indicate RND with  1 . 0 ,  1 .
Revised: 5 reward shift, and 4  RND -1.05  RND -1.5 to indicate RND with  1 . 0 ,  1 .

Original: 5 reward shift, respectively.
Revised: 5 reward shift, respectively.

Original: In the following, we use between methods to indicate the former outperforms the latter.
Revised: In the following, we use between methods to indicate the former outperforms the latter.

Original: Comparing the results:
Revised: Comparing the results:

Original: reward shifting is equivalent to optimistic initialization, thus boosting exploration
Revised: reward shifting is equivalent to optimistic initialization, thus boosting exploration

Original: 3   1  : a negative reward shift is equivalent to optimistic initialization and helps exploration.
Revised: 3   1  : a negative reward shift is equivalent to optimistic initialization and helps exploration.

Original: RND is effective for value-based exploration — as long as a negative intrinsic reward is used1   2  : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive 325 intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.4   1  ; 5   3  : RND is effective for exploration (i
Revised: RND is effective for value-based exploration — as long as a negative intrinsic reward is used1   2  : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive 328 intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.4   1  ; 5   3  : RND is effective for exploration (i

Original: e., improve over DQN) as long as the intrinsic 327 reward bonus is always negative.
Revised: e., improve over DQN) as long as the intrinsic 330 reward bonus is always negative.

Original: reward shifting for optimistic initialization is orthogonal to other exploration methods

Revised: reward shifting for optimistic initialization is orthogonal to other exploration methods

Original:   4  : increasing the magnitude of the negativelyshifted reward
Revised:   4  : increasing the magnitude of the negatively

Original:  can further improve the exploration 330 performance — reward shifting can either work in isolation or get combined with other explorationalgorithms as they are working orthogonally.
Revised: shifted reward can further improve the exploration 333 performance — reward shifting can either work in isolation or get combined with other explorationalgorithms as they are working orthogonally.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.
Revised: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.

Original: Although constant reward shifting should not change the optimal policy inducedby the optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.
Revised: Although constant reward shift should not change the optimal policy induced bythe optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.

Original: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.
Revised: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.

Original: Theproposed idea is then veriﬁed through a variety of application settings.
Revised: Theproposed idea is then veriﬁed through a variety of application settings.

Original: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.
Revised: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.

Original: Importantly, our analysis manifests that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.
Revised: Importantly, our analysis reveals that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.

Original: We empirically verify theeffectiveness of the proposed method on a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.
Revised: We empirically verify theeffectiveness of the proposed method in a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.

Original: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point out  the theoretical guidance for such a shifting constant is missing.
Revised: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point out that the theoretical guidance for such a shifting constant is missing.

Original: Analysis fromthe perspective of optimization is an important topic, yet it is out of the scope of the current empiricalstudy and left for future research.
Revised: Potentialsolutions may lie in analysis from the perspective of optimization for black-box models, yet it is outof the scope of the current empirical study and left for future research.