Original: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)
Revised: Exploiting Reward Shifting in Value-Based Deep RL Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).
Revised: In this work, we study the simple yet universally applicable case of reward shapingin value-based Deep Reinforcement Learning (DRL).

Original: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.
Revised: We show that reward shiftingin the form of the linear transformation is equivalent to changing the initialization ofthe Q -function in function approximation.

Original: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bringthe key insight that a positive reward shifting leads to conservative exploitation,while a negative reward shifting leads to curiosity-driven exploration.

Original: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation improves exploration for online RL.
Revised: Accordingly,conservative exploitation improves ofﬂine RL value estimation, and optimisticvalue estimation beneﬁts the exploration of online RL.

Original: We validate our insight ona range of RL tasks and show its improvement over baselines:
Revised: We validate our insight ona range of RL tasks and show its improvement over baselines:

Original: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete control tasks, a negative reward shiftingyields an improvement over the  curiosity-based exploration method.
Revised: (1) In ofﬂine RL,the conservative exploitation leads to improved performance based on off-the-shelfalgorithms; (2) In online continuous control, multiple value functions with differentshifting constants can be used to tackle the exploration-exploitation dilemma forbetter sample efﬁciency; (3) In discrete exploration tasks, a negative reward shiftingbrings an improvement over the previous curiosity-based exploration method.

Original:  Introduction
Revised:  Introduction

Original: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2],specifying a certain reward to incentivize the learning agentrequires domain knowledge anda thorough understandingof the task [3–6].
Revised: While reward shaping is a well-established practice in reinforcement learning and has a long-standing history [1, 2] specifying a certain reward to incentivize the learning agentrequires domain knowledge and thorough understanding ofthe task [3–6].

Original: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learningmay on the contrary hinder the learning performance byincurring the sub-optimal behaviors of the agent [7, 8].
Revised: Even with careful design and tuning, learningwith a shaped reward that intends to accelerate learning mayon the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent [7, 8].

Original: Although Ng et al.
Revised: Although Ng et al.

Original: [9] theoretically points out that optimalpolicy remains unchanged under certain form of rewardtransformation, and in the later work of Wiewiora et al.[10] a framework is proposed to guide policies with priorknowledge under tabular setting,  how reward shifting accommodates recent Deep Reinforcement Learning (DRL)algorithms remains much less explored.
Revised: [9] theoretically points the optimal policy remainsunchanged under a special form of reward transformation,and in the later work of Wiewiora et al. [10] a framework isproposed to guide policies with prior knowledge under tabular setting, the investigation of how it accommodates recent Deep Reinforcement Learning (DRL) algorithms remainsmuch less explored.

Original: In this work, we study a special linear transformation, whichis the simplest form of reward shaping, in value-based DRL [11–14].
Revised: In this work, we focus on the simplest form of rewardshaping, the linear transformation, in value-based DRL [11–14].

Original: We start with understanding how such aspeciﬁc kind of reward shaping works in value-based DRLfunction approximations and show that reward shifting is equivalent to different Q -value initialization,extending previous discovery of [10] to the function approximation setting.
Revised: We start with understanding how such a speciﬁc kindof reward shaping works in value-based DRL algorithms. We show that reward shifting, as the simplest reward transformation, is equivalent to manually settingthe initial values of the Q -function estimation, extending previous discovery of [10] to the functionapproximation setting.

Original: Figure 1 showcasesreward shifting beneﬁts exploration in a maze game and outperforms count-based and ✏ -greedyexploration and learns near optimal value function.
Revised: Figure 1 showcases reward shifting beneﬁts exploration in a maze game andoutperforms count-based and ✏ -greedy exploration.

Original: Based on such an equivalence, we introduce the key insight of this work:
Revised: Based on such an equivalence, we introduce the key insight of this work:

Original: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.
Revised: A positive reward shifting leads to conservative exploitation, while a negative reward shiftingleads to curiosity-driven exploration.

Original: We demonstrate the application of such an insight in three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation induced by reward shifting improves learning performance ofoff-the-shelf algorithms; (2) for online RL setting, we show multiple value functions with differentreward shifting constants can be used to trade-off between exploration and exploitation, thus improvelearning efﬁciency; (3) we introduce a simple yet crucial adaptation on a prevailing curiosity-basedexploration algorithm, the Random Network Distillation [15], making it compatible with value-based
Revised: We demonstrate the application of such an insight on three Deep-RL scenarios: (1) for ofﬂine RL, weshow that conservative exploitation can lead to improved learning performance based on off-the-shelfalgorithms; (2) for online RL setting, we show multiple value functions with different reward shiftingconstants can be used as a trade-off between exploration and exploitation, thus improving learningefﬁciency; (3) we introduce a simple yet crucial improvement over a prevailing curiosity-basedexploration method, the Random Network Distillation [15], making it compatible with value-based

Original: DRL algorithms.
Revised: DRL algorithms.

Original: Experiments on a diverse set of tasks including continuous and discrete action spacecontrol show our method brings  substantial improvement over  baselines.
Revised: The experiments on both continuous and discrete action space control tasks showsour method brings the substantial improvement over previous baselines.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network
Revised: We introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network

Original: initialization in value-based DRL, which can be applied to  curiosity-driven exploration andconservative exploitation;
Revised: initialization in deep-RL, which can be used to improve both curiosity-driven exploration and conservative exploitation;

Original: Practically, we instantiate the key insight to three different scenarios, namely the ofﬂine conservative exploitation, sample-efﬁcient continuous control, andcuriosity-driven exploration, to contrast the generality of reward shifting;
Revised: Practically, we present three scenarios motivated by our key insight to show the potentialbeneﬁt of reward shifting, namely the ofﬂine conservative exploitation, the online sample- efﬁcient RL, and the curiosity-driven exploration;

Original: Empirically, we demonstrate the effectiveness of the proposed method integrated with
Revised: Empirically, we demonstrate the effectiveness of the proposed method integrated with

Original: off-the-shelf baselines on both continuous and discrete control tasks.
Revised: off-the-shelf baselines on both continuous and discrete control tasks.

Original:  Preliminaries and Related Work
Revised:  Preliminaries and Related Work

Original: 2.1 Online RL
Revised: 2.1 Online RL

Original: We follow a standard MDP formulation in the online RL setting, i.e., M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ),we consider a deterministic transition dynamics T : 
Revised: We follow a standard MDP formulation in the online RL setting, i.e. , M = {S , A , T , R , ⇢ 0 ,  , T } ,where S ⇢ R d denotes the d -dim state space, A is the action space (note for discrete action space|A| < 1 and for continuous control |A| = 1 ), T : S ⇥ A 7! S is the transition dynamics,

Original:  S ⇥
Revised: R : S ⇥

Original: A 7!S and deterministic reward function R : S ⇥
Revised: A 7! R is the reward function.

Original: A 7! R . ⇢ 0 = p ( s [-
 ) 2  ( S-] ) denotes the initial 64 state distribution.
Revised: ⇢ 0 denotes the initial state distribution, i.e. , ⇢ 0 = p ( s [-
 ) 2  ( S-]0 ) .

Original:   is the discount factor and T is the episodic decision horizon.
Revised: 64  is the discount factor and T is the episodic decision horizon.

Original: Online RL considersthe problem of learning a policy ⇡ 2 ⇧ : S 7!
Revised: Online RL considers the problemof learning a policy ⇡ 2 ⇧ : S 7!

Original:  ( A ) (or ⇡ 2 ⇧ : S 7!
Revised:  ( A ) (or ⇡ 2 ⇧ : S 7!

Original: A with a deterministicpolicy class), such that the expected cumulative reward E a t ⇠ ⇡ ,s t +1 ⇠ T ,s 0 ⇠ ⇢
Revised: A with a deterministic policy 66 class), such that the expected cumulative reward in the Markov decision process is maximized, i.e. ,⇡ [Equation]

Original: P
Revised: P

Original: Tt =

Revised: Tt =

Original:  t r t ( s t , a t ) inthe Markovian decision process is maximized.
Revised:  t r t ( s t , a t ) ,

Original: In  online RL setting, an agent learns through trialsand errors [11],through either  on-policy  [16–18] or off-policy paradigm [12–14, 19, 20].
Revised: In the online RL setting, an agent normallylearns through trials and errors [11], either with an on-policy paradigm [16–18] or an off-policymanner [13, 12, 19, 20, 14].

Original: In thiswork, we focus on the off-policy value-based methods which are in general more sample efﬁcient.
Revised: In this work, we focus on the off-policy value-based methods which arein general more sample efﬁcient.

Original: Speciﬁcally, our discussions assume the policy learning is based on a learned Q -value functionthat approximates the cumulative reward an agent can gain in the following part of an episode.
Revised: Speciﬁcally, our discussions assume the policy learning is basedon a learned Q -value function, that approximates the cumulative reward an agent can gain in thefollowing part of an episode.

Original: Formally, Q ( s t , a t ) =
Revised: The Q -value function is deﬁned as Q ( s t , a t ) =

Original: E ⇡ , T
Revised: E ⇡ , T

Original: P
Revised: P

Original: T ⌧ = t  t r ( s ⌧ , a ⌧ ) , and can be estimated by propagating the Bellmanoperator B Q ( s, a ) =
Revised: T ⌧ = t  t r ( s ⌧ , a ⌧ ) ,and can be estimated by propagating the Bellman operator B Q ( s, a ) =

Original: r ( s, a ) +  E Q ( s 0 , a 0 ) .
Revised: r ( s, a ) +  E Q ( s 0 , a 0 ) .

Original: For value-based methods, the (soft-)optimal policy isthen produced by
Revised: Forvalue-based methods, the (soft-)optimal policy is then produced by

Original: [Equation]
Revised: [Equation]

Original: where Q ⇤ isthe optimal Q -value function.
Revised: where Q ⇤ is optimal Q -value function.

Original: We can also set the temperature parameter close to 0to have the deterministic policy class.
Revised: We can also set the temperature parameter close as 0 tohave the deterministic policy class.

Original: Simplifying the notion we have ⇡ ⇤ ( s ) =
Revised: Simplifying the notion we have ⇡ ⇤ ( s ) =

Original: arg max a Q ⇤ ( s, a ) .
Revised: arg max a Q ⇤ ( s, a ) .

Original: Algorithms like DPG [21] can be used to address the intractable analytical argmax issuethat arises incontinuous action space.
Revised: Algorithms like DPG [21] can be used to address the intractable analytical argmax issue arises incontinuous action space.

Original: We choose to develop our work on top of prevailing baseline algorithms of
Revised: We choose to develop our work on top of prevailing baseline algorithms of

Original: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.
Revised: DQN [13], BCQ [22], CQL [23] and TD3 [14] as a minimal example to isolate the source of gains.

Original: Itshould be straightforward to extend the method on top of other learning algorithms.
Revised: Itshould be straightforward to extend the method on top of other learning algorithms.

Original: 2.2 Exploration and the Curiosity-Driven Methods
Revised: 2.2 Exploration and the Curiosity-Driven Methods

Original: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent mustsimultaneously learn to exploit its accumulated knowledge on the task and explore newstates and actions.
Revised: One of the most important issues in online RL is the exploration-exploitation dilemma [11] that theagent must learn to exploit its accumulated knowledge on the task while exploring new states andactions.

Original: Previous works address the exploration problem from various perspectives: fordiscrete action space tasks, count-based methods like [27–29] are proposed to motivate the revisiting
Revised: Previous works address the exploration problem from various perspectives: In the tasks with

Original: [Table]
Revised: [Table]

Original: of under-explored state-action pairs.
Revised: discrete action space, count-based methods like Bellemare et al [Table] [Table] [Table]

Revised: Ostrovski et

Revised: [Table]

Revised: [28], Tang et al

Original: Speciﬁcally, Choshen et al.
Revised: [Table] [29], Choshen et al.

Original: [30] extended the idea into generalsettings by constructing an additional MDP for Exploration-value estimation, as a generalized counterfor count-based exploration.
Revised: [30] are proposed to motivate the policy to explore more on under-explored states.

Original: To boost exploration in continuous state tasks, curiosity-driven methodsare investigated by Houthooft et al.
Revised: Curiosity-driven methods are investigated by Houthooft et al.

Original: [31], Pathak et al.
Revised: [31], Pathak et al.

Original: [32], Burda et al. [33, 15], where variety ofintrinsic rewards are designed as  supplementary to the primal task reward for better exploration.
Revised: [32], Burda et al.[33, 15], where the intrinsic reward is designed as a supplementary to the primal task reward for betterexploration.

Original: Selfimitate approaches like Oh et al.
Revised: Self-imitate approaches like Oh et al.

Original: [34], Ecoffet et al.
Revised: [34], Ecoffet et al.

Original: [35], Sun et al.
Revised: [35], Sun et al.

Original: [36] repeat success trajectoriesbut require extra assumptions on the environment.
Revised: [36] repeatsuccess trajectories but require extra assumptions on the environment.

Original: The works of DIAYN and DADS [37, 38] showthat various skills can be developed even without the primal extrinsic reward.
Revised: The work of DIAYN and DADS [37, 38] show that various skills can be developed even without the primal extrinsic reward.

Original: For continuous controltasks, OAC [26] improves the SAC [20] with informative action space noise based on the optimism inface of uncertainty (OFU) [39–42].
Revised: For continuous control tasks, OAC [26] improves the SAC [20] with informative action space noisebased on the optimism in face of uncertainty (OFU) [39–42].

Original: GAC [43] addresses the exploration issue with a richer functionalclass for the policy.
Revised: GAC [43] addresses the explorationissue with a richer functional class for the policy.

Original: In the work of Rashid et al.
Revised: In the work of Rashid et al.

Original: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.
Revised: [25], the problematic pessimistic initialization is addressed for betterexploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration.

Original: Inthe work of Osband et al.
Revised: Inthe work of Osband et al.

Original: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.
Revised: [44, 24], ensemble models with diverse initialization and randomized priorsare used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet thosemethods are only applicable to discrete control tasks.

Original: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal
Revised: Noted that although the reward shifting can beregarded as a special case of these random priors, it can be distinguished by not changing the optimal

Original: Q -value, and its ﬂexibility to be plugged into both continuous and discrete control algorithms.
Revised: Q -value, and its ﬂexibility to be plugged in to both continuous and discrete control algorithms.

Original:  Random Network Distillation (RND) [15] deﬁnes the intrinsic reward as theoutput difference betweena ﬁxed neural network  1 and a trainable network  2 given state-actions as the inputs e.g.,
Revised: The Random Network Distillation (RND) [15] proposes to use the difference between a ﬁxed neuralnetwork  1 and a trainable network  2 to represents the intrinsic reward, e.g. ,

Original: [Equation]
Revised: [Equation]

Original: where both networks are activated by a sigmoid function. After optimizing the learnable  2 toapproximate  1 with seen ( s, a ) pairs 1 , the value of r int ( s, a ) will decay to 0 for such state-actionpairsthat arefrequently visited  but remain high forthoase are seldom visited.
Revised: when outputs of both networks are activated by a sigmoid function, and  2 is optimized to approximate 1 for the visited ( s, a ) pairs 1 . Hence, the value of r int ( s, a ) will decay to 0 when such state-actionpairs are visited frequently but remain high for seldom visited pairs.

Original: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.
Revised: In this work, we show that exploratory behavior can be achieved simply by shifting the rewardfunction with a constant.

Original: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated byhigh function approximation error in theunder-explored state-action pairs.
Revised: Thus our method is orthogonal to those previous approaches in the sensethat our intrinsic exploration behavior is motivated by function approximation error.

Original: We demonstrate such insight by showing that RND in its originaldesign is not suitable for value-based methods in developing exploratory behaviors, but integrating RND with a shifted reward function remarkably improves the learning performance.
Revised: We demonstratesuch an insight by showing that RND in its original design is not suitable for value-based methods indeveloping exploratory behaviors, but integrating RND with a shifted reward function remarkablyimproves the learning performance.

Original: 2.3 Ofﬂine RL
Revised: 2.3 Ofﬂine RL

Original: The ofﬂine RL, also known as batch-RL, focuses on the problems where  interaction with the environment is impossible, and the policy can only be optimized based onthe logged dataset.
Revised: The ofﬂine RL, also known as batch-RL, focuses on the problems where the interaction with theenvironment is impossible, and the policy can only be optimized based on logged dataset.

Original: In those tasks, aﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ], collected from some unknown behavior policy ⇡  , is provided.
Revised: In thosetasks, a ﬁxed buffer B = { s i , a i , r i , s 0 i } i =[ N ] is provided.

Original: 119 In general, such a dataset can either be generated by rolling out an expert that generates high-qualitysolutions to the task [22, 45, 46] or a non-expert that executes sub-optimal behaviors [46–50] or bea mixture of both [23].

Original: As the agent in the ofﬂine RL setting  can not correct its potentially biasedknowledge through interactions, the most important issue is to address the extrapolation error [22]induced by distributional mismatch [48].
Revised: As the agent in the ofﬂine RL setting 117 can not correct its potentially biased knowledge through interactions, the most important issue isto address the extrapolation error [22] induced by distributional mismatch [45].

Original: To address such an issue, a series of algorithms optimizethe policy learning under the constraint of distributional similarity [47, 48, 51].
Revised: To address suchan issue, a series of algorithms optimize the policy learning under the constraint of distributionalsimilarity [45–47].

Original: Bharadhwaj et al.
Revised: Bharadhwaj et al.

Original: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.
Revised: [23] propose CQL to solve the ofﬂine RL tasks with a conservative value estimation.

Original: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions (e.g., uniform distribution over the action space).

Original: As we will show inthis work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.

Original: This idea leads to the direct application of our proposedframework in the ofﬂine setting.

Original: In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22, 47, 48, 51] to further improve the performance with conservative Q -valueestimation.

Original: Table 1 contextualizes reward shifting with respect to related works we discussed above, morediscussions on related works can be found at Appendix E.

Original: A Motivating Example

Original: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing our method.

Original: Remark 1.

Original: Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 140 with linear transformation, i.e.,

Original: R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , doesnot change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=

Original: [Equation]

Original:  t r 0 t :

Original: [Equation]

Original: Remark 2. When |A| = 1 , scaling the reward function with linear transformation does not change 143 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k : r ✓ J ( µ ✓ ) = E s t [ r a [Equation]
Revised: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actionssampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled withpre-deﬁned prior distributions ( e.g. , uniform distribution over the action space).

Original: ⇤

Original: [Equation]

Original: s t

Original: [Equation]

Original: a t

Original: [Equation]

Original: | a t = µ ✓

Original: [Equation]

Original: s t )

Original: r ✓ µ ✓

Original: [Equation]

Original: ] =

Original: E s t [ r a

Original: [Equation]

Original: ⇤ k

Original: [Equation]

Original: b

Original: [Equation]

Original: t

Original: [Equation]

Original: a t

Original: [Equation]

Original: | a t = µ ✓

Original: [Equation]

Original: s t

Original: [Equation]

Original: ✓ µ ✓

Original: [Equation]

Original: ] /k, (4)

Original: Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function calculated by the shifted reward.

Original: However, Figure 1presents “counter-intuitive” results in a demonstrative exploration task of Grid World.

Original: In this task, anagent located at the upper left corner of a map needs to explore without reward and reach the goalpoint located at the lower right corner.

Original: A non-trivial reward of +1 will be assigned only when the goalis reached.

Original: We report learning curves with regard to the episodic success rate in reaching the goalpoint and the learned Q -values.

Original: In this toy example, we ﬁnd a negative reward shifting remarkablyboosts the learning efﬁciency of Q -learning, and surpasses the conventional count-based method forexploration.

Original: Moreover, Remark 1 is empirically veriﬁed with such a toy example: reward shiftingdoes not change the optimal Q -value as well as its induced policy, but on the contrary accelerates thediscovery of (near-)optimal policy.

Original: In the following of this work, we study the effect of varying constant bias b and ﬁx the scaling factork = 1 to avoid trivial discussions on the choices of learning rate — there should be no surprise thatchoosing an appropriate learning rate is empirically important.

Original: We focus on revealing the importanceof selecting the universal bias term b in the reward function through the lens of initialization priorsin function approximation and show such a bias is generally helpful for both online and ofﬂinesettings.

Original: In the online settings, it improves learning efﬁciency, and in the ofﬂine settings, it promotesconservative exploitation.

Original: Shifted Priors for Q -value Estimation

Original: 4.1 Reward Shift Equals to Different Initialization

Original: We start by formally introducing notions and the key idea of this work: reward shifting equals differentinitialization.

Original: We use Figure 2 to illustrate how reward shifting affects function approximation, hencechanging the learning dynamics for value-based algorithms.

Original: The original optimal Q -value functionis denoted as Q ⇤ , and plotted in the ﬁgures as Black curves.

Original: We then denote the shifted optimal

Original: Q -value function as Q ⇤ b + , which is the Q -value function with the shifted reward

Original:  r 0 = r + b + .Weuse Red curves in ﬁgures to denote those shifted Q -value functions. In this section we provide
Revised: As we will show in1 Or computed with only states, i.e., r int ( s ) = |  2 ( s )   1 ( s ) | .

Original: ! ∗ ! "
Revised: ! ∗ ! "

Original: # ! 1 − & ! " !
Revised: # ! 1 − & ! " !

Original: ∗ = !
Revised: ∗ = !

Original: ∗ + " !
Revised: ∗ + " !

Original: $%& "# ! ≈0 "
Revised: $%& "# ! ≈0 "

Original: " ! '! ' − !
Revised: " ! '! ' − !

Original: ∗ < '! '
Revised: ∗ < '! '

Original: %$ −!
Revised: %$ −!

Original: ∗ ): '! '
Revised: ∗ ): '! '

Original: → !
Revised: → !

Original: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !
Revised: ∗ | s,a,r,s ( ) # ! " $! " ! ,$ −! " !

Original: ∗ < $! " ! ,$%& −! " !
Revised: ∗ < $! " ! ,$%& −! " !

Original: ∗ ): '! " ! ,' → ! " !
Revised: ∗ ): '! " ! ,' → ! " !

Original: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!
Revised: ∗ | s,a,r,s ( ) ' − # ! 1 − & ! "'!

Original: +,,' = '! " ! ,' − # ! 1 − & '!
Revised: +,,' = '! " ! ,' − # ! 1 − & '!

Original: +,,- = '!
Revised: +,,- = '!

Original: - − # ! 1 − & (r 0 = r + b + .
Revised: - − # ! 1 − & (

Original: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized as ˜ Q LB , 0 ⇡ ˜ Q 0  b + 1   , rather than ˜ Q 0 ⇡ 0 . 
Revised: 1   , rather than ˜ Q 0 ⇡ 0 . i.e. , by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value. this work, an alternative approach to have a lower bound for the optimal Q -value function is to usean appropriately shifted reward function.

Original: i.e., by shifting the  reward with proper positive value b + , we are able to initialize the Q -value network that lower-bounds the optimal Q -value.
Revised: This idea leads to the direct application of our proposedframework in the ofﬂine setting. In general, reward shift can be plugged in many distribution-matchingofﬂine-RL algorithms [22, 45–47] to further improve the performance with conservative Q -valueestimation. Table 1 contextualizes reward shifting with respect to related works in both exploration and exploitation motivated RL algorithms.  A Motivating Example

Original: − 𝑏 + 1−𝛾 𝑸
Revised: We start with two intuitive remarks and a “counter-intuitive” motivating example in this sectionbefore introducing our proposed method. Remark 1. Given an MDP M = {S , A , T , R , ⇢ 0 ,  } , where |A| < 1 , scaling the reward function 136 with linear transformation, i.e. , R k,b = k ·R + b , 8 k > 0 , b 2 R , such that r 0 t = kr t + b 2 R k,b , does 137 not change the optimal policy induced by the corresponding value function Q ⇤ k,b ( s, a ) :=

Revised: P t  t r 0 t :

Revised: [Equation]

Revised: Remark 2.

Revised: When |A| = 1 , scaling the reward function with linear transformation does not change 139 the optimal policy induced by deterministic policy gradient [21], given proper learning rate ⌘ 0 = ⌘ /k :

Revised: [Equation]

Revised: Remark 1 and Remark 2 declare the fact that constant reward shifting does not affect the optimalpolicy induced by the optimal Q -value function.

Revised: However, Figure 1 presents “counter-intuitive”results that in a demonstrative exploration task of Grid World.

Revised: In this task, an agent is initializedat the upper left corner of the map need to explore the map without reward unless it reaches thegoal point located at the lower right corner, where a positive reward of +1 will be assigned.

Revised: Wereport learning curves with regard to episodic success rate in reaching the goal point and the learned

Revised: Q -values.

Revised: In this toy example, we ﬁnd a negative reward shifting remarkably boosts the learningefﬁciency of Q -learning, and surpasses the count-based method that is classic for exploration.

Revised: − 𝑏 + 1−𝛾 𝑸

Original: 𝓐
Revised: 𝓐

Original: ෩𝑸
Revised: ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐
Revised: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐

Original: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =
Revised: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =

Original: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 

Revised: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸

Original: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 
Revised: ≈ 𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸

Original: [Figure]

Original: UB,𝑡
Revised: UB,𝑡

Original: [Figure]
Revised: = ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾 ෩𝑸 UB,𝑡 ≥𝑸 ∗ ෩𝑸

Original: − ,
Revised: LB,𝑡

Original: [Figure] − 
Revised: ≤𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏 + 1−𝛾

Original: [Figure] analysis with  a positive bias b + > 0 to illustrate how positively shifted value function affects functionapproximation, and coordinates the normally applied near-zero initialized function approximators(e.g., neural networks.
Revised: Moreover, Remark 1 is empirically veriﬁed with such a toy example, i.e. , reward shifting does notchange the optimal Q -value as well as its induced policy, but on the contrary, accelerating the learningof (near)-optimal policy.

Original: Orange curves in the ﬁgure) to inspire conservative behaviors.
Revised: In the following section, we will focus on the scenarios when k = 1 to avoid the trivial (thoughthis can be empirically important) discussions on different learning rates. We reveal the importanceof selecting the universal bias term, i.e. , b in the reward function, through the lens of initializationpriors in function approximation. Furthermore, we show such a bias term can be utilized not only inthe online RL settings to improve learning efﬁciency but also in the ofﬂine RL settings to conductconservative exploitation with batched data.  Shifted Priors for Q -value Estimation

Original: The discussionof negative biases is elaborated in the next section.
Revised: We start by introducing the central idea of this work: reward shift equals different initialization.

Original: To summarize, shifting the reward function with  a positive constant is equivalent to initializing thevalue network with a smaller value — as the Q -value of unseen state-action pairs during training aremuch lower than their shifted optimal value, those actions will not be selected in argmax-style policyupdates — leading to conservative learning behaviors that beneﬁt policy learning in ofﬂine settings.
Revised: Speciﬁcally, Figure 2 illustrates the effect of the reward shifting under different settings. The blackcurves denote the primal optimal Q -value functions ( e.g. , Q ⇤ ), and the red curves denote the optimal Q -value functions with shifted reward r 0 = r + b + , where b + > 0 is a positive constant bias, andwe will show a different choice of such a bias leads to a different motivation in Q -value estimation.

Revised: The yellow lines denote Q -value estimators, e.g. , neural network predictions of the corresponding

Revised: Q -values.

Revised: To summarize, shifting the reward function with a positive constant is equivalent to initializing thevalue function network with a smaller value.

Revised: As the Q -value of unseen state-action pairs duringtraining are much lower than the optimal value, those actions will not be selected in policy updates,leading to conservative learning behaviors that beneﬁt policy learning in ofﬂine settings [23].

Original: 4.2 Conservative Exploitation
Revised: 4.2 Conservative Exploitation

Revised: We begin with a natural application in the ofﬂine setting, where a policy doesn’t interact with theenvironment and only learns through a logged data set collected from an unknown behavior policy⇡  , which can either be an expert that generates high-quality solutions to the task [22, 48, 49] or anon-expert that provides actions that are sub-optimal [46, 45, 50, 49, 51] or a mixture of both [23].

Original: According to the key insight we presented in Section 4.1, conservative Q -value estimation can emergewith a positively shifted reward. And such a value estimation empirically lower-bounds the optimal Q -value function.
Revised: Based on the idea we presented in Section 4.1, we introduce a simple yet effective approach forconservative Q -value estimation that learns to heuristically put a lower bound on the optimal Q -valuefunction in batch settings.

Original: As shown in Figure 2(a), a positive constant b + added to the reward functionwill lead to a universal positively shifted optimal Q -value function, and the gap between the primal Q -value function and the new one is b + 1   .
Revised: As shown in Figure 2(a), a positive constant b + added to the rewardfunction will lead to a uniformly (positively) shifted optimal Q -value function, and the gap betweenthe primal Q -value function and the new one is b + 1   .

Original: Optimizing the Q -value approximator with logged data willminimize the difference between the predicted value and the optimal value with observed data.
Revised: Optimizing the Q -value function with loggeddata ( e.g. , a ﬁxed replay buffer) will minimize the difference between the predicted value and theoptimal value with observed data.

Original: Forthe unobserved data, the pessimistically initialized Q -value approximator guarantees the prediction islower than thepositively shifted optimal ones, thus conservative exploitation can be achieved withargmax-style value propagation and action execution on such a value function.
Revised: For the unobserved data point (in the state-action space S ⇥ A ), thenear-zero initialization guarantees the prediction is lower than the optimal Q -value, thus conservativeexploitation can be conducted with such a value function.

Original: 4.3 (Curiosity-Driven) Optimistic Exploration
Revised: 4.3 (Curiosity-Driven) Optimistic Exploration

Original: On the other hand, if we shift the reward function to the negative side, it is equivalent toan optimisticinitialization.
Revised: On the other hand, if we shift the reward function to the negative side, it is equivalent to optimisticinitialization.

Original: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.
Revised: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration.

Original: With sufﬁciently small b  (so that | b  | larger than the maximal value of any( s, a) -pair), such anupper bound of Q ⇤ can be used to conduct curiosity-driven exploration.
Revised: With sufﬁciently small b  (so that | b  | larger than the maximal value of any s, a -pair), such an upperbound of Q ⇤ can be used to conduct curiosity-driven exploration.

Original: Intuitively, initializing avalue network thatalways predictsa value larger  than the trueoptimal value will lead to curiositydriven exploration behavior, as any visited state will be assigned a relatively smaller value and theargmax-style policy  will tend to choose under-explored actions.
Revised: Intuitively, initializing a valuenetwork that predicts value larger or equal than the true value will lead to curiosity-driven exploration,as any visited state will be assigned a low value and the policy that learns to perform the action witha higher value will tend to choose novel actions.

Original: Based on the analysis above that (1) a positive constant shift added to the reward function can be 198 used for conservative policy updates and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, we are ready to access both the upper bound, i.e., theoptimistic estimation with b  , and the lower bound, i.e., the conservative estimation with b + of theoptimal value function.
Revised: Based on the discoveries above that (1) a positive constant shift added to the reward function can beused for conservative policy update and (2) a negative constant shift added to the reward functioncan be used for curiosity-driven exploration, as shown in Figure 3(b). We are ready to access boththe upper bound, i.e. , the optimistic estimation with b  , and the lower bound, i.e. , the conservativeestimation with b + of the optimal value function.

Original: Formally, we use
Revised: Formally, we use

Original: [Equation]
Revised: [Equation]

Original: 5) to denote the empirical lower bound of value estimation (cf. Figure 2(d), Orange line), and
Revised: (5) to denote the empirical lower bound of value estimation (c.f. Figure 2(d), orange line), and

Original: [Equation]
Revised: [Equation]

Original: 6) to denote the empirical upper bound of value estimation (cf. Figure 3(b), Orange line).
Revised: (6) to denote the empirical upper bound of value estimation (c.f. Figure 3(b)).

Original: In bothnotions, t denotes the optimization step.
Revised: In both notions, t denotesthe optimization step.

Original: When t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 ,˜ Q LB , 0 ( s, a ) =
Revised: e.g., when t = 0 , with a near-zero initialization ˜ Q 0 ⇡ 0 , ˜ Q LB , 0 ( s, a ) =

Original:  b + 1   is able to lower bound the unknown optimal Q -value given sufﬁciently largeb + .
Revised:  b + 1  is able to lower bound the unknown optimal Q -value given sufﬁciently large b + .

Original: (cf.
Revised: (c.f.

Original: Figure 2(d), Green line).
Revised: Figure 2(d),green line).

Original: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with asufﬁciently small b  .
Revised: Similarly, Q -value is upper-bounded by ˜ Q UB , 0 ( s, a ) with a sufﬁciently small b  .

Original: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous control anddiscrete action space respectively.
Revised: Following those notions, we introduce our sample-efﬁcient algorithms for both continuous controland discrete action space respectively.

Original: We propose a practical algorithm for sample-efﬁcient continuouscontrol in Section 4.3.1, and focus on a special class of curiosity-driven exploration method, the
Revised: We propose a practical algorithm for general continuouscontrol in Sec. 4.3.1, while focusing on a special class of curiosity-driven exploration method, the

Original: RND [15], in Section 4.3.2.
Revised: RND [15], in Sec.

Original: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift
Revised: 4.3.1 Trading-off Exploration and Exploitation with Reward Shift

Original: According to the principle of optimism in the face of uncertainty (OFU), an exploration bonus 214 that manifests the uncertainty of the Q -value function can be introduced into the value estimation:integrating optimistic exploration with conservative exploitation, 
Revised: According to the principle of optimism on the face of uncertainty (OFU), an exploration bonus thatmanifests the uncertainty of the Q -value function can be introduced into the value estimation: thebasic idea starts from integrating optimistic exploration with conservative exploitation, i.e. ,

Original: [Equation]
Revised: [Equation]

Original: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.
Revised: where the second term with coefﬁcient  denotes exploration bonus that is composed of uncertainty.

Original: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neuralnetwork, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequence ofinitialization (detailed implications are provided in Appendix B).
Revised: For those under-explored state-action pairs, i.e. , extremely out-of-distribution samples for ourneural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b  ,t ( s, a ) will give near-zero predictions as a consequenceof initialization (detailed implications are provided in Appendix B).

Original: Hence, the explorative bonusbecomes  (1   ) b
Revised: Hence, the explorative bonusbecomes  (1   ) b

Original: + +  b   , which is equivalent to applying another constant reward shift with value ofc r = (1   ) b + +  b  , formally, we have
Revised: + +  b   . Note this is equivalent to applying another constant reward shift with valueof c r = (1   ) b + +  b  .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Assuming we have access to an unbiased estimator for the optimal value function
Revised: Assuming we have access to an unbiased estimator for the optimal value function

Original: Q ⇤ , e.g., with Monte-Carlo estimation ˆ Q ⇤ =
Revised: Q ⇤ , e.g. , with Monte-Carlo estimation ˆ Q ⇤ =

Original: E
Revised: E

Original: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e., ✏ 2 t
Revised: P t  t r , and the optimization is based on minimizingthe MSE between the unbiased estimator and the function approximator, i.e. , ✏ 2 t

Original: =
Revised: =

Original: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =
Revised: ( ˜ Q t  ˆ Q ⇤ ) 2 ,˜ Q t =

Original: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r
Revised: ˜ Q t  1  2 ⌘ ( ˜ Q t  1  ˆ Q ⇤ ) , then the linear combination in Equation (7) is equivalent to a linearcombination of the constants with value of c r

Original: =
Revised: =

Original: (1   ) b
Revised: (1   ) b

Original: + +  b  .
Revised: + +  b  .

Original: The proof can be found in the Appendix A.
Revised: The proof can be found in the Appendix A.

Original: According to Proposition 4.1, a grid search for trading-offbetween the three hyper-parameters, i.e., the exploration bias b  , the exploitation bias b + and the coefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r
Revised: According to Proposition 4.1, a grid search for tradingoff between the three hyper-parameters: the exploration bias b  , the exploitation bias b + and the coefﬁcient  in Equation (7) is trivial as they only lead to a linear combination as c r

Original: = (1   ) b + + b  , indicating that
Revised: = (1   ) b + + b  , indicating that

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.
Revised: Changing the reward shifting constant b is sufﬁcient to trade off between explorationand exploitation.

Original: The corollary says, in principle, a meta-learner can be trained to monitor the learning process andselect a proper reward shifting constant automatically [52, 53] to balance exploration and exploitation.
Revised: The corollary indicates that, in principle, a meta-learner can be trained to monitor the learning processand select a proper reward shifting constant automatically [52, 53].

Original: For the ease of exposition, in this work we choose to focus on the simplest yet effective uniformsampling strategy from multiple shift constants, which has been shown as a strong baseline of thosemeta-learner approaches [54, 55], and leave more complicated meta-learner-based shifting constantadjustment for future investigation.
Revised: For the ease of exposition, in thiswork we choose to focus on the simplest yet effective uniform sampling strategy from multiple shiftconstants, which has been shown as a strong baseline of those meta-learner approaches [54, 55], andleave more complicated meta-learner-based approaches for future investigation.

Original: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.
Revised: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s 0 ) sampled from theidentical buffer that collects the policy’s historcal interactions with the environment.

Original: In propagating
Revised: In propagating

Original: Q-values through the temporal difference loss, the primal recorded reward value r is replacedwith shifted rewards with different  constant biases to update their individual Q-networks.
Revised: Q-values through the temporal difference loss, the primal recorded reward value r is replaced with anew reward with different shifted constant bias in those Q-networks.

Original: We thenuniformly sample one of those learned Q-networks for the optimization of policy networks (e.g., with DPG [21]).
Revised: We then uniformly sample oneof those learned Q-networks for the optimization of policy network (with e.g., DPG [21]).

Original: It is worth noting that our approach only requires post-hoc revision of the primal rewardfunction, rather than interacting with the environment multiple times to collect samples for each valuenetwork.
Revised: It is worthnoting that our approach only requires post-hoc revision of the primal reward function, rather thaninteracting with the environment multiple times to collect samples for each value network.

Original: We dub the proposed method Random Reward Shift (RRS), and provide the pseudo-code in 
Revised: We dubthe proposed method Random Reward Shift (RRS), and provide the pseudo-code in Algorithm 1 of

Original: Algorithm 1 of Appendix C.2.
Revised: Appendix C.2.

Original: 4.3.2 Tailored Curiosity-Driven Explorationfor Value-Based Method
Revised: 4.3.2 Improving Value-Based Curiosity-Driven Exploration

Original: In previous works, the curiosity-driven exploration methods are always work with policy-basedmethods. In this section, we cast the key insight introduced above to  RND [15]— one of theleading algorithms for exploration — to answer why is its vanilla design not suitable for value-basedalgorithms like Q-learning, and propose a variant of RND that is tailored for DQN.
Revised: The discussion above casts the curiosity-driven exploration method as a special case in the rewardshift equals to initialization perspective: exploration with RND [15] is equivalent to selecting b + = 0 ,i.e. , using the primal task reward for exploitation, and using b  as a ﬁxed random function over S ⇥ A , i.e. , b  ( s, a ) =

Original: The vanilla RND use two randomly initialized networks  1 and  2 to generate the intrinsic rewardr int = |  1  ( s, a )   2 ( s, a ) |    0 for exploration.
Revised: |  1 , 0 ( s, a )   2 ( s, a ) | 2 , 8 s, a , where  1 , 0 and  2 are two neural networks 247 with different random initializations.

Original: During learning,  2 is a ﬁxed network and theparameters of  1 is optimized to approximate the output of   2 for frequently-visited states. We follow
Revised: While  1 , 0 is learnable,  2 is set to be ﬁxed during learningas a random curiosity prior [24]. Without loss of generality, we can assume  2 = 1 as a constantinitialization, and  1 , 0 = 0 .

Original: Burda et al. [15] to bound the intrinsic reward below 1 and use r int ,t to denote the intrinsic rewardafter t step of optimization.
Revised: Then the exploration behavior is fully controlled by the scale of externalreward, which is set to be 1.0 with external reward clipped to [  1 , 1] in [15].

Original: Speciﬁcally, r int , 0 ( s, a )  =  1 , 8 ( s, a ) — an universally positive bonus isadded to the primal reward function at beginning.
Revised: The ﬁnal equivalentobjective in this case is to use c r = |  1 ,t ( s, a )  1 | 2 , t = 0 , 1 , 2 , ... . Specially,  1 , 0 = 0 and c r = 1at beginning.

Original: According to our analysis in  previous sections, such a positivereward shift is equivalent to pessimisticinitialization and will lead to conservative behaviors in Q -value estimation.
Revised: According to our analysis in the previous section, such a positive shift directly leads to conservativebehaviors in Q -value estimation, therefore may hinder the exploration behaviors at the beginning,i.e. , when |  1 ,t  1 |  0 .

Original: Therefore, the explorationbehaviors at the beginning of learning will be hindered, rather than boosted.
Revised: The exploration bonus in RND only becomes effective when visiting 256 seldom visited states after the predictions of  1 ,t of frequently-visited states are close to 1 , andhence encouraging stepping into those novel states and discover new knowledge.

Original: To overcome sucha conservative behavior induced by the pessimistic initialization, we proposed to use r int ( s, a ) =| 
Revised: To overcomethe conservative tendency induced by Q -value estimation in RND, we propose to use b  ( s, a ) =|  1 , 0 ( s, a )   2 ( s, a )

Original:  1 ( s, a )   2 ( s, a ) | 2  I, 8 s, a , where I is a positive constant  that assures r  int  0 is negatively 264 initialized for optimistic exploratory behaviors.
Revised: | 2  I, 8 s, a , where I is a positive constant ( e.g. , I = max s,a |  1 , 0 ( s, a )  2 ( s, a ) | 2 ) that assures b  ( s, a ) is negative-initialized for optimistic exploratory behaviors.

Original: The Chicken-and-Egg Problem To further understand the difference between the curiosity-drivenmethod in ways of intrinsic reward and our reward shifting-based optimistic initialization, let usconsider when do those curiosities work in each case: for intrinsic reward methods, an agent mustﬁrst experience a new ( s, a ) pair before receiving a high intrinsic reward — this is extremely hardwith an arg-max style policy. On the other hand, with optimistic initialization, the rarely-visited ( s, a )pairs will naturally have higher Q -values before experiencing it — as the frequently-visited pairshave updated their values with a negatively shifted reward.
Revised: It is worth noting that the curiosity introduced by exploratory bonuses like RND focuses on increasingthe temporary Q -value of rarely visited states, which still relies on the visitation behavior itself inrandom exploration. i.e. , without stepping into a non-frequent visited state, the agent will neverreceive the novelty bonus based on the state.

Original: From such a perspective, reward shiftingnot only works by itself motivates exploratory behaviors, but can also be seamlessly plugged intointrinsic reward methods to encourage the ﬁrst visitation of new states.
Revised: On the other hand, the mechanism of shifting thereward by a constant is an universal optimistic exploration bonus as the curiosity is fused into theinitialization across the entire state-action support.

Original:  Experiments
Revised:  Experiments

Original: 5.1 Ofﬂine RL with Conservative Q -value Estimation
Revised: 5.1 Ofﬂine RL with Conservative Q -value Estimation

Original: We start our experiments by demonstrating the effectiveness of reward shifting in  ofﬂine RL benchmarks.
Revised: We ﬁrst demonstrate the proposed method in the ofﬂine RL setting.

Original: As  discussed in Section 4.2,  a positive reward shift is equivalent to pessimistic initializationthat beneﬁts conservative exploitation.
Revised: As has been discussed in Section 4.2, shifting the primal reward function with a positive constant provides a natural way ofconservative exploitation.

Revised: By adding a positive reward shift, any visited ( s, a ) saved in the givenreplay buffer B (logged dataset) will be assigned a large reward, while under-explored ( s, a ) pairsremain the low values due to initialization, as illustrated in Figure 2 (d).

Original: In general, our proposed method can be plugged in to anyoff-the-shelf ofﬂine RL algorithm,  we choose to verify the effectivenessand generality of such a conservative Q -value estimation based on BCQ [22] and CQL [23], i.e., both distribution-matchingapproach and conservative value estimation approaches in ofﬂine RL.
Revised: Although in general ourproposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm, in this work we demonstrate the effectiveness of such a conservative Q -value estimation based on BCQ [22] and CQL [23], i.e. , both distribution-matching approach and conservative value estimation approaches inofﬂine RL.

Original: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.
Revised: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift( Neg. ), added on either BCQ or CQL.

Original: Figure 4 shows our experiment results.
Revised: Figure 4 shows our experiment results.

Original: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [46] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.
Revised: We experiment withboth the dataset generated in [22] (Hopper Imitate) and the dataset used in [49] (others), and ﬁnd inour experiments that learning with the CQL dataset is much more stable.

Original: The ﬁrst two panels showresults with BCQ as the backbone algorithm.
Revised: The ﬁrst two panels showresults with BCQ as the backbone algorithm.

Original: We observea clear performance drop during the trainingof BCQ as the policy overﬁts the batched dataset. Differently, positive reward shift can alleviate sucha problem and outperform BCQ in terms of both best-achieved performance and performance afterconvergence.
Revised: We observe clear performance drop when training with BCQ as the policy overﬁts the given dataset in these settings, on the other hand, our proposed rewardshift can alleviate such a problem and outperform BCQ w.r.t. both best achieved performance andconvergence performance.

Original: The following three panels in Figure 4 use CQL as the backbone.
Revised: The following three panels in Figure 4 use CQL as the backbone.

Original: In all experiments,shifting the reward with a positive constant improves learning performance, while a negative rewardshift impedes to efﬁcient learning — as expected.
Revised: In allexperiments, using a positive reward shift leads to improved learning performance, while a negativereward shift leads to performance decay, as expected.

Original: Implementation details and more results can befound in Appendix C.1.
Revised: Implementation details and more results can befound in Appendix C.1.

Original: 5.2 Online RL with Randomized Priors
Revised: 5.2 Online RL with Randomized Priors

Original: We then conduct experiments in the MuJoCo locomotion benchmarks to demonstrate reward shiftingimproves learning efﬁciency in the online RL settings. 
Revised: We then conduct experiments on the online RL settings. We demonstrate our proposed method inthe MuJoCo locomotion benchmarks.

Original: As our implementation is based on TD3,we use TD3-based variants as our baselines:
Revised: As our implementation is based on TD3, we use TD3-basedvariants as our baselines:

Original: The TD3 is trained with default settings accordingto Fujimoto et al.
Revised: The TD3 is trained with default settings according to Fujimoto et al.

Original: We also include Ensemble TD3 and Bootstrapped TD3 as baselinesdue to they are similar to our work in using multiple Q -networks in value estimation.
Revised: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similar to ourwork in using multiple Q -networks in value estimation.

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: [44] but extend it to the continuous control settings.
Revised: [44] but extend it tothe continuous control settings.

Original: In continuous control settings, theargmax operator is approximated by the policy network,and multiple policy networks are needed tocooperate with the multiple bootstrapped Q -value networks.
Revised: Noting that in the continuous control setting, the argmax operatoris approximated by the policy network, multiple policy networks are needed to cooperate with themultiple bootstrapped Q -value networks.

Original: Otherwise, multiple Q -value networksare not independent of each other thus breaking the condition of bootstrapped value estimation.
Revised: Otherwise, multiple Q -value networks are not independentof each other thus breaking the condition of bootstrapped value estimation.

Original: The Ensemble TD3 presents the baseline performance when multiple Q -networks are used for valueestimation in TD3, and works as an ablation of our method where all  shift priors are set to be 0 .
Revised: The Ensemble TD3presents the baseline performance when multiple Q -networks are used for value estimation in TD3,which also works as an ablation of our method when all reward shift priors are set to be 0 .

Original: As has been illustrated in Sec.
Revised: As has been illustrated in Sec.

Original: 4.3.1, learning with different reward shifting values is equivalentto learning with optimistic or conservative initialization.
Revised: 4.3.1, learning with different reward shifting values is equivalent tolearning with optimistic or conservative initialization.

Original: In our instantiating of RRS, we use3 Q -networks with different priors.
Revised: In our method of Random Reward Shift (RRS),we use 3 Q -networks with different priors.

Original: We empirically ﬁnd ± 0 .
Revised: In our experiments, we found ± 0 .

Original: 5 , 0 work universally good for allenvironments.
Revised: 5 , 0 work universally asa default setting.

Original: Though, future investigation on hyper-parameter may help to further improve theperformance.
Revised: Though, further investigation on hyper-parameter may help to further improve theperformance.

Original: Results are shown in Figure 5.
Revised: Results are shown in Figure 5.

Original: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.
Revised: RRS outperforms the vanilla TD3 in all ﬁve environments andoutperforms all baseline methods in most tasks.

Original: In the experiment of Bootstrapped TD3 and Ensemble

Original: TD3, wealso use 3 Q -networks for a fair comparison.
Revised: In all experiments, we use 3 Q -networks for a faircomparison.

Original: Note that there is a trade-off betweencomputational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expenses, as reported in [44].
Revised: Note that there is a trade-off between computational complexity and sample efﬁciency, i.e. , using more Q -networks may further improve the performance at the cost of more computationalexpensive, as reported in [44].

Original: More implementationdetails, pseudo-code of RRS, and ablation studies can be found in Appendix C.2.
Revised: More implementation details, pseudo code of RRS, and ablationstudies can be bound in Appendix C.2.

Original: 5.3 Optimistic Random Network Distillation
Revised: 5.3 Optimistic Random Network Distillation

Original: To demonstrate the key insight and effectiveness of reward shifting in optimistic exploration, webenchmark with ﬁve discrete exploration tasks, including the classic MountainCar control and four
Revised: [Figure] Finally, we experiment with ﬁve discrete exploration tasks, namely the MountainCar-v0, and fournavigation tasks of MiniGrid suite [56], namely the task of Empty-Random, MultiRoom, and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.

Original: MiniGrid navigation tasks [56], environment details can be found at Appendix C.3.
Revised: More environment details are provided in the Appendix C.3.

Original: Figure 6 presents the results of 5 different methods for comparison: besides the as-is 1  DQN and2  RND baselines, we use 3  DQN -0.5 to indicate DQN  with a 0 . 5 reward shift, and 4  RND -1.05  RND -1.5 to indicate RND with 1 . 0,  1 .5 reward shift, respectively.
Revised: We compare the vanilla DQN , vanilla RND , as well as improved DQN and RND according to ourproposed insight. DQN -0.5 indicates the results with a reward shifting of  0 .

Original: In the following, we use between methods to indicate the former outperforms the latter.

Original: Comparing  the results: reward shifting is equivalent to optimistic initialization, thus boosting exploration 3   1  : anegative reward shift is equivalent to optimistic initialization and helps exploration.{+shifting of  1 . 
+}
Revised: Comparing DQN-0.5 with the vanilla DQN, our insight of negative reward shift leads to curiosity-driven exploration isagain veriﬁed. RND -1.0 indicates the results when a reward {+shifting of  1 . 
+}

Original: RND is effective for value-based exploration — as long as a negative intrinsic reward is used1   2  : vanilla RND with positive intrinsic reward is always worse than DQN: adding a positive 328 intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.4   1  ; 5   3  : RND is effective for exploration (i e., improve over DQN) as long as the intrinsic 330 reward bonus is always negative.
Revised: is added to RND basedon DQN. RND -1.0 improves the performance of RND and DQN in most environments, showing theeffectiveness of the RND-based exploration bonus. Note that for a fair comparison, RND -1.0 shouldbe compared with the vanilla DQN as the  1 . 0 reward shift just cancels the positive explorationbonus introduced by RND.

Original: [-reward shifting for optimistic initialization is orthogonal to other exploration methods
-]
Revised: [-reward shifting for optimistic initialization is orthogonal to other exploration methods
-]Moreover, we can further improve RND by equipping it with reward-shifting-based curiosity-drivenexploration.

Original:    4  : increasing the magnitude of the negatively
Revised: RND -1.5 ( i.e. , RND with a  1 . 5 reward shift) can be compared with the DQN -0.5,as both receive an -0.5 exploration bonus for unseen states.

Original: shifted reward  can further improve the exploration 333 performance — reward shifting can either work in isolation or get combined with other explorationalgorithms as they are working orthogonally.
Revised: We ﬁnd in all experiments that ournegative reward shift can remarkably improve exploration, not only working with RND to improveits performance but also working effectively in isolation.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.
Revised: In this work, we studied how reward shifting affects policy learning in value-based deep reinforcementlearning algorithms.

Original: Although constant reward shift should not change the optimal policy induced bythe optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.
Revised: Although constant reward shifting should not change the optimal policy inducedby the optimal value function, in practice such a constant shift does affect the function approximation ,and leads to different learning behaviors.

Original: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.
Revised: Our detailed analysis manifests the fact that a constantreward shift is equivalent to using different initialization in the value function approximation.

Original: Theproposed idea is then veriﬁed through a variety of application settings.
Revised: Theproposed idea is then veriﬁed through a variety of application settings.

Original: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.
Revised: Speciﬁcally, we show thata negative reward shift leads to curiosity-driven exploration, while a positive reward shift helpsconservative exploitation.

Original: Importantly, our analysis reveals that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.
Revised: Importantly, our analysis manifests that changing reward shifting constantitself is sufﬁcient in trading-off between exploration and exploitation.

Original: We empirically verify theeffectiveness of the proposed method in a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.
Revised: We empirically verify theeffectiveness of the proposed method on a variety of experiments, including better exploitation inofﬂine RL, sample-efﬁcient learning in continuous control benchmarks, and enhanced curiosity-drivenexploration in value-based discrete control.

Original: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point outthat the theoretical guidance for such a shifting constant is missing.
Revised: While our experiments demonstrate the performance gain is quite robust to the shifting constant, wewould like to point out the theoretical guidance for such a shifting constant is missing.

Original: Potentialsolutions may lie in analysis from the perspective of optimization for black-box models, yet  it is outof the scope of the current empirical study and left for future research.
Revised: Analysis fromthe perspective of optimization is an important topic, yet is left for future research as it is out of thescope of the current empirical study.