Original: R EWARD S HIFTING FOR O PTIMISTIC E XPLORATION AND C ONSERVATIVE E XPLOITATION
Revised: R EWARD S HIFTING FOR O PTIMISTIC E XPLORATION AND C ONSERVATIVE E XPLOITATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work, we study the simple yet universally applicable case of reward shaping, the linear transformation, in value-based Deep Reinforcement Learning.
Revised: In this work, we study the simple yet universally applicable case of reward shaping, the linear transformation, in value-based Deep Reinforcement Learning.

Original: We show that reward shifting, as the simplest linear reward transformation, is equivalent to changing initialization of the Q -function in function approximation.
Revised: We show that reward shifting, as the simplest linear reward transformation, is equivalent to changing initialization of the Q -function in function approximation.

Original: Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.

Original: In this case, a conservative exploitation improves ofﬂine RL value estimation, and the optimistic value estimation beneﬁts the exploration of online RL.
Revised: In this case, a conservative exploitation improves ofﬂine RL value estimation, and the optimistic value estimation beneﬁts the exploration of online RL.

Original: We verify our insight on a range of tasks: (1) In ofﬂine
Revised: We verify our insight on a range of tasks: (1) In ofﬂine

Original: RL, the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efﬁciency; (3) In online RL with discrete action space, a negative reward shifting brings an improvement over the previous curiosity-based exploration method.
Revised: RL, the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efﬁciency; (3) In online RL with discrete action space, a negative reward shifting brings an improvement over the previous curiosity-based exploration method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While reward shaping is a well-established practice in reinforcement learning applications and has a long-standing history (Randløv & Alstrøm, 1998; Laud, 2004), specifying a certain reward to incentivize the learning agent requires domain knowledge and deep understanding of the task (Vinyals et al., 2019;
Revised: While reward shaping is a well-established practice in reinforcement learning applications and has a long-standing history (Randløv & Alstrøm, 1998; Laud, 2004), specifying a certain reward to incentivize the learning agent requires domain knowledge and deep understanding of the task (Vinyals et al., 2019;

Original: Akkaya et al., 2019; Berner et al., 2019; Elbarbari et al., 2021).
Revised: Akkaya et al., 2019; Berner et al., 2019; Elbarbari et al., 2021).

Original: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent (Florensa et al., 2017; Plappert et al., 2018).
Revised: Even with careful design and tuning, learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent (Florensa et al., 2017; Plappert et al., 2018).

Original: Although Ng et al.
Revised: Although Ng et al.

Original: (1999) theoretically points the optimal policy will keep unchanged under a special form of reward transformation, and in the later work of Wiewiora et al.
Revised: (1999) theoretically points the optimal policy will keep unchanged under a special form of reward transformation, and in the later work of Wiewiora et al.

Original: (2003) a framework is proposed to guide policies with prior knowledge under tabular setting, the investigation of how it accommodates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.
Revised: (2003) a framework is proposed to guide policies with prior knowledge under tabular setting, the investigation of how it accommodates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.

Original: In this work, we focus on the simplest case of reward shaping, the linear transformation, in valuebased DRL (Sutton & Barto, 1998; Lillicrap et al., 2015; Mnih et al., 2015; Fujimoto et al., 2018b).
Revised: In this work, we focus on the simplest case of reward shaping, the linear transformation, in valuebased DRL (Sutton & Barto, 1998; Lillicrap et al., 2015; Mnih et al., 2015; Fujimoto et al., 2018b).

Original: We start with understanding how such a speciﬁc kind of reward shaping works in value-based DRL algorithms.
Revised: We start with understanding how such a speciﬁc kind of reward shaping works in value-based DRL algorithms.

Original: We show that reward shifting, as the simplest reward transformation, is equivalent to engineering initialization of the Q -function estimation, extending previous discovery of (Wiewiora et al., 2003) to the function approximation settings.
Revised: We show that reward shifting, as the simplest reward transformation, is equivalent to engineering initialization of the Q -function estimation, extending previous discovery of (Wiewiora et al., 2003) to the function approximation settings.

Original: Based on such an equivalence, we bring the key insight of this work: a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.
Revised: Based on such an equivalence, we bring the key insight of this work: a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration.

Original: We demonstrate the application of such an insight to three downstream tasks: (1) for ofﬂine RL, we show that conservative exploitation can lead to improved learning performance based on off-the-shelf algorithms; (2) for online RL setting, we show multiple value functions with different reward shifting constants can be used as a trade-off between exploration and exploitation, thus improving learning efﬁciency; (3) ﬁnally, we introduce a simple yet crucial improvement over a prevailing curiosity-based exploration method,
Revised: We demonstrate the application of such an insight to three downstream tasks: (1) for ofﬂine RL, we show that conservative exploitation can lead to improved learning performance based on off-the-shelf algorithms; (2) for online RL setting, we show multiple value functions with different reward shifting constants can be used as a trade-off between exploration and exploitation, thus improving learning efﬁciency; (3) ﬁnally, we introduce a simple yet crucial improvement over a prevailing curiosity-based exploration method,

Original: the Random
Revised: the Random

Original: Network Distillation (Burda et al., 2018b), making it compatible with value-based DRL algorithms.
Revised: Network Distillation (Burda et al., 2018b), making it compatible with value-based DRL algorithms.

Original: We evaluate our idea on various tasks, including both continuous and discrete action space control, resulting in substantial improvements over previous baselines.
Revised: We evaluate our idea on various tasks, including both continuous and discrete action space control, resulting in substantial improvements over previous baselines.

Original: Our contributions can be summarized as follows
Revised: Our contributions can be summarized as follows

Original: we introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network initialization, which can be used to boost both curiosity-driven exploration and conservative exploitation; 2. motivated by our key insight, we present three scenarios where the reward shifting can beneﬁt, namely the ofﬂine conservative exploitation, the online sample-efﬁcient RL, and the curiosity-driven exploration; 3.
Revised: we introduce the key insight that reward shifting is equivalent to diversiﬁed Q -value network initialization, which can be used to boost both curiosity-driven exploration and conservative exploitation; 2. motivated by our key insight, we present three scenarios where the reward shifting can beneﬁt, namely the ofﬂine conservative exploitation, the online sample-efﬁcient RL, and the curiosity-driven exploration; 3.

Original: we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks.
Revised: we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks.

Original:  P RELIMINARIES 2. O NLINE RL
Revised:  P RELIMINARIES 2. O NLINE RL

Original: We follow a standard MDP formulation in the online RL settings, i.e., M = {S , A , T , R , ρ 0 , γ, T } , where S ⊂ R d denotes the d -dim state space, A is the action space (note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ), T : S × A (cid:55)→ S is the transition dynamics,
Revised: We follow a standard MDP formulation in the online RL settings, i.e., M = {S , A , T , R , ρ 0 , γ, T } , where S ⊂ R d denotes the d -dim state space, A is the action space (note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ), T : S × A (cid:55)→ S is the transition dynamics,

Original: R : S × A (cid:55)→ R is the reward function.
Revised: R : S × A (cid:55)→ R is the reward function.

Original: ρ 0 denotes the initial state distribution, i.e., ρ 0 = p ( s 0 ) .
Revised: ρ 0 denotes the initial state distribution, i.e., ρ 0 = p ( s 0 ) .

Original: γ is the discount factor and T is the episodic decision.
Revised: γ is the discount factor and T is the episodic decision.

Original: Online RL considers the problem of learning a policy π ∈ Π : S (cid:55)→ ∆ A (or π ∈ Π : S (cid:55)→ A with a deterministic policy class), such that the expected cumulative reward in the Markov decision process is maximized, i.e.,
Revised: Online RL considers the problem of learning a policy π ∈ Π : S (cid:55)→ ∆ A (or π ∈ Π : S (cid:55)→ A with a deterministic policy class), such that the expected cumulative reward in the Markov decision process is maximized, i.e.,

Original: [Equation]
Revised: [Equation]

Original: In the online RL setting, an agent normally learns through trials and errors (Sutton & Barto, 1998), either with an on-policy paradigm (Schulman et al., 2015; 2017; Cobbe et al., 2021) or an off-policy manner (Mnih et al., 2015;
Revised: In the online RL setting, an agent normally learns through trials and errors (Sutton & Barto, 1998), either with an on-policy paradigm (Schulman et al., 2015; 2017; Cobbe et al., 2021) or an off-policy manner (Mnih et al., 2015;

Original: Lillicrap et al., 2015; Wang et al., 2016;
Revised: Lillicrap et al., 2015; Wang et al., 2016;

Original: Haarnoja et al., 2018; Fujimoto et al., 2018b).
Revised: Haarnoja et al., 2018; Fujimoto et al., 2018b).

Original: In this work, we focus on the off-policy value-based methods which are in general more sample efﬁcient.
Revised: In this work, we focus on the off-policy value-based methods which are in general more sample efﬁcient.

Original: Speciﬁcally, our discussions assume the policy learning is based on a learned
Revised: Speciﬁcally, our discussions assume the policy learning is based on a learned

Original: Q -value function
Revised: Q -value function

Original: , that approximates the cumulative reward an agent can gain in the following part of an episode.
Revised: , that approximates the cumulative reward an agent can gain in the following part of an episode.

Original: The Q -value function is deﬁned as Q ( s t , a t ) =
Revised: The Q -value function is deﬁned as Q ( s t , a t ) =

Original: E π, T (cid:80) Tτ =
Revised: E π, T (cid:80) Tτ =

Original: t γ t r ( s τ , a τ ) , and can be approximated through the Bellman Operator B Q ( s, a ) =
Revised: t γ t r ( s τ , a τ ) , and can be approximated through the Bellman Operator B Q ( s, a ) =

Original: r ( s, a ) + γ E Q ( s (cid:48) , a (cid:48) ) .
Revised: r ( s, a ) + γ E Q ( s (cid:48) , a (cid:48) ) .

Original: For value-based methods, the (soft-)optimal policy is then produced by
Revised: For value-based methods, the (soft-)optimal policy is then produced by

Original: [Equation]
Revised: [Equation]

Original: where Q ∗ is optimal Q -value function.
Revised: where Q ∗ is optimal Q -value function.

Original: We can also set the temperature parameter close as 0 to have the deterministic policy class.
Revised: We can also set the temperature parameter close as 0 to have the deterministic policy class.

Original: Simplifying the notion we have π ( s ) =
Revised: Simplifying the notion we have π ( s ) =

Original: arg max a Q ∗ ( s, a ) .
Revised: arg max a Q ∗ ( s, a ) .

Original: Algorithms like DPG (Silver et al., 2014) can be used to address the intractable analytical argmax issue arises in continuous action space.
Revised: Algorithms like DPG (Silver et al., 2014) can be used to address the intractable analytical argmax issue arises in continuous action space.

Original: We develop our work on top of prevailing baseline algorithms of
Revised: We develop our work on top of prevailing baseline algorithms of

Original: DQN (Mnih et al., 2015), BCQ (Fujimoto et al., 2018a), and TD3 (Fujimoto et al., 2018b), and it will be easy to extend to other baseline algorithms.
Revised: DQN (Mnih et al., 2015), BCQ (Fujimoto et al., 2018a), and TD3 (Fujimoto et al., 2018b), and it will be easy to extend to other baseline algorithms.

Original: 2. E XPLORATION AND THE C URIOSITY -D RIVEN M ETHODS
Revised: 2. E XPLORATION AND THE C URIOSITY -D RIVEN M ETHODS

Original: One of the most important issues in online RL is the exploration-exploitation dilemma (Sutton &
Revised: One of the most important issues in online RL is the exploration-exploitation dilemma (Sutton &

Original: Barto, 1998) that the agent must learn to exploit its accumulated knowledge on the task while exploring new states and actions.
Revised: Barto, 1998) that the agent must learn to exploit its accumulated knowledge on the task while exploring new states and actions.

Original: Plenty of previous works address the exploration problem from various perspectives: In the tasks with discrete action space, count-based methods like Bellemare et al.
Revised: Plenty of previous works address the exploration problem from various perspectives: In the tasks with discrete action space, count-based methods like Bellemare et al.

Original: ; Ostrovski et al.
Revised: ; Ostrovski et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: (2017) are proposed to motivate the policy to explore more on under-explored states.
Revised: (2017) are proposed to motivate the policy to explore more on under-explored states.

Original: Curiosity-driven methods are investigated by Houthooft et al.
Revised: Curiosity-driven methods are investigated by Houthooft et al.

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: (2018a;b), where the intrinsic reward is designed as a supplementary to the primal task reward for better exploration.
Revised: (2018a;b), where the intrinsic reward is designed as a supplementary to the primal task reward for better exploration.

Original: Self-imitate approaches like Oh et al.
Revised: Self-imitate approaches like Oh et al.

Original: ; Ecoffet et al.
Revised: ; Ecoffet et al.

Original: ; Sun et al.
Revised: ; Sun et al.

Original: (2019) repeat success trajectories but require extra assumptions on the environment.
Revised: (2019) repeat success trajectories but require extra assumptions on the environment.

Original: The work of DIAYN and DADS (Eysenbach et al., 2018; Sharma et al., 2019) show that various skills can be developed even without the primal extrinsic reward.
Revised: The work of DIAYN and DADS (Eysenbach et al., 2018; Sharma et al., 2019) show that various skills can be developed even without the primal extrinsic reward.

Original: For continuous control tasks, OAC (Ciosek et al., 2019) improves the SAC (Haarnoja et al., 2018) with informative action space noise based on the optimism in face of uncertainty (OFU)
Revised: For continuous control tasks, OAC (Ciosek et al., 2019) improves the SAC (Haarnoja et al., 2018) with informative action space noise based on the optimism in face of uncertainty (OFU)

Original: (Brafman & Tennenholtz, 2002; Jaksch et al., 2010; Azar et al., 2017; Jin et al., 2018).
Revised: (Brafman & Tennenholtz, 2002; Jaksch et al., 2010; Azar et al., 2017; Jin et al., 2018).

Original: GAC (Tessler et al., 2019) addresses the exploration issue with a richer functional class for the policy.
Revised: GAC (Tessler et al., 2019) addresses the exploration issue with a richer functional class for the policy.

Original: In the recent work of Rashid et al.
Revised: In the recent work of Rashid et al.

Original: , the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration. 
Revised: , the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on speciﬁc settings of tabular and discrete control exploration. In the work of

Revised: Osband et al.

Revised: (2016; 2018), ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet those methods are only applicable to discrete control tasks.

Revised: Noted that although the reward shifting can be regarded as a special case of these random priors, it can be distinguished by not changing the optimal Q-value, and ﬂexible to be plugged in to both continuous and discrete control algorithms.

Original: The Random Network Distillation (RND) (Burda et al., 2018b) propose to use the difference between a ﬁxed neural network φ 1 and a trainable network φ 2 to represents the intrinsic reward, e.g.,
Revised: The Random Network Distillation (RND) (Burda et al., 2018b) propose to use the difference between a ﬁxed neural network φ 1 and a trainable network φ 2 to represents the intrinsic reward, e.g.,

Original: [Equation]
Revised: [Equation]

Original: when outputs of both networks are activated by a sigmoid function, and φ 2 is optimized to approximate φ 1 for the visited ( s, a ) pairs.
Revised: when outputs of both networks are activated by a sigmoid function, and φ 2 is optimized to approximate φ 1 for the visited ( s, a ) pairs.

Original: Henceforth, the value of r int ( s, a ) will decay to 0 when such state-action pairs are visited frequently but remain high for seldom visited pairs.
Revised: Henceforth, the value of r int ( s, a ) will decay to 0 when such state-action pairs are visited frequently but remain high for seldom visited pairs.

Original: In this work, we show that exploratory behavior can be achieved simply by shifting the reward function with a constant, thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by function approximation error.
Revised: In this work, we show that exploratory behavior can be achieved simply by shifting the reward function with a constant, thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by function approximation error.

Original: We demonstrate such an insight by showing that RND, with its original design, is not suitable for value-based methods in developing exploratory behaviors, but integrating RND with a shifted reward function can remarkably improve the learning performance.
Revised: We demonstrate such an insight by showing that RND, with its original design, is not suitable for value-based methods in developing exploratory behaviors, but integrating RND with a shifted reward function can remarkably improve the learning performance.

Original: 2. O FFLINE RL
Revised: 2. O FFLINE RL

Original: The ofﬂine RL, also known as batch-RL, focuses on the problems where the interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset.
Revised: The ofﬂine RL, also known as batch-RL, focuses on the problems where the interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset.

Original: In those tasks, a ﬁxed buffer B = { s i , a i , r i , s (cid:48) i } i =[ N ] is provided.
Revised: In those tasks, a ﬁxed buffer B = { s i , a i , r i , s (cid:48) i } i =[ N ] is provided.

Original: As the agent in the ofﬂine RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation error (Fujimoto et al., 2018a) induced by distributional mismatch (Kumar et al., 2019).
Revised: As the agent in the ofﬂine RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation error (Fujimoto et al., 2018a) induced by distributional mismatch (Kumar et al., 2019).

Original: To address such an issue, a series of algorithms optimize the policy learning under the constraint of distributional similarity (Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020).
Revised: To address such an issue, a series of algorithms optimize the policy learning under the constraint of distributional similarity (Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020).

Original: Bharadhwaj et al.
Revised: Bharadhwaj et al.

Original: (2020) proposed CQL to solve the ofﬂine RL tasks with a conservative value estimation.
Revised: (2020) proposed CQL to solve the ofﬂine RL tasks with a conservative value estimation.

Original: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actions sampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled with pre-deﬁned prior distributions (e.g., uniform distribution over the action space).
Revised: Speciﬁcally, CQL learns the Q -value estimation by jointly maximizing the Q -values of actions sampled from the behavior ofﬂine dataset and minimizing the Q -values of actions sampled with pre-deﬁned prior distributions (e.g., uniform distribution over the action space).

Original: As we will show in this work, an alternative approach to have a lower bound for the optimal Q -value function is to use an appropriately shifted reward function.
Revised: As we will show in this work, an alternative approach to have a lower bound for the optimal Q -value function is to use an appropriately shifted reward function.

Original: This idea leads to the direct application of our proposed framework in the ofﬂine setting.
Revised: This idea leads to the direct application of our proposed framework in the ofﬂine setting.

Original: In general, reward shift can be plugged in many distribution-matching ofﬂine-RL algorithms (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020) to further improve the performance with conservative Q -value estimation.
Revised: In general, reward shift can be plugged in many distribution-matching ofﬂine-RL algorithms (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020) to further improve the performance with conservative Q -value estimation.

Original:  A M OTIVATING E XAMPLE
Revised:  A M OTIVATING E XAMPLE

Original: [Figure]
Revised: [Figure]

Original: We start with a motivating example that may look counterintuitive: in the prevailing continuous control environment of Humanoid, a three-dimensional bipedal robot is simulated based on the MuJoCo engine.
Revised: We start with a motivating example that may look counterintuitive: in the prevailing continuous control environment of Humanoid, a three-dimensional bipedal robot is simulated based on the MuJoCo engine.

Original: The learning objective is to control the robot to walk forward as fast as possible, without falling over.
Revised: The learning objective is to control the robot to walk forward as fast as possible, without falling over.

Original: There are two types of positive reward signals: (1) to encourage forward-moving, i.e., the reward is proportional to the horizontal displacement, and (2) to encourage the robot to avoid falling over, i.e., a binary alive bonus whether the mass center of the robot is higher than some certain threshold.{+𝑨 𝑸 − 𝑏 + 1 − 𝛾 𝑸 ∗ 𝑸 𝑸 𝑸 𝑨 𝑨 𝑨 𝑏 + 1 − 𝛾 𝑸 𝑏 + ∗ = 𝑸 ∗ + 𝑏 + 1−𝛾 ෩𝑸 
+}
Revised: There are two types of positive reward signals: (1) to encourage forward-moving, i.e., the reward is proportional to the horizontal displacement, and (2) to encourage the robot to {+𝑨 𝑸 − 𝑏 + 1 − 𝛾 𝑸 ∗ 𝑸 𝑸 𝑸 𝑨 𝑨 𝑨 𝑏 + 1 − 𝛾 𝑸 𝑏 + ∗ = 𝑸 ∗ + 𝑏 + 1−𝛾 ෩𝑸 
+}

Original: Although the design of (2) is intended to provide a curriculum for the agent, we show in Figure 1 that such a curriculum, on the contrary, hinders the learning performance of such a bipedal robot to run fast. [Figure]
Revised: ≈෩𝑸 𝑡 − 𝑸 ∗ < ෩𝑸 𝑡−1 −𝑸 ∗ ෩𝑸 𝑏 + ,𝑡 − 𝑸 𝑏 + ∗ < ෩𝑸 𝑏 + ,𝑡−1 − 𝑸 𝑏 + ∗ 𝑭: ෩𝑸 𝑡 → 𝑸 ∗ | s,a,r,s ′ 𝑁 𝑭: ෩𝑸 𝑏 + ,𝑡 → 𝑸 𝑏 + ∗ | s,a,r,s ′

Original: 𝑸
Revised: 𝑁

Original: [Figure]
Revised: [Figure]

Original:  𝑏 [Figure] 𝑏 
Revised: LB,𝑡 = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1 −

Original: [Figure]
Revised: [Figure]

Original: ෩𝑸

Original: [Figure]
Revised: [Figure]

Original: 𝑸
Revised: 𝑏

Original: [Figure]

Original: −𝑸

Original: [Figure]
Revised: [Figure]

Original: 𝑏
Revised: 𝐛

Original: [Figure]
Revised: [Figure]

Original: 𝑭: ෩𝑸 𝑡 → 𝑸 ∗ | s,a,r,s ′ 𝑁
Revised: avoid falling over, i.e., a binary alive bonus whether the mass center of the robot is higher than some certain threshold.

Original: [Figure]
Revised: Although the design of (2) is intended to provide a curriculum for the agent, we show in Figure 1 that such a curriculum, on the contrary, hinders the learning performance of such a bipedal robot to run fast.

Original: The orange curve, TD3 w/o Alive Bonus, shows the smoothed learning curve of a TD3 agent trained with the alive bonus set to be 0 ; the blue curve, TD3 w/ Alive Bonus, shows the averaged learning curve of a vanilla TD3 agent trained with the alive bonus set to be 5 as default.
Revised: The orange curve, TD3 w/o Alive Bonus, shows the smoothed learning curve of a TD3 agent trained with the alive bonus set to be 0 ; the blue curve, TD3 w/ Alive Bonus, shows the averaged learning curve of a vanilla TD3 agent trained with the alive bonus set to be 5 as default.

Original: Noted that for both approaches, we use identically the same primal task in policy evaluation to make sure the curves are comparable.
Revised: Noted that for both approaches, we use identically the same primal task in policy evaluation to make sure the curves are comparable.

Original: In general curriculum learning improves learning efﬁciency (Graves et al., 2017; Matiisen et al., 2019; Portelas et al., 2020), however, in the environment of Humanoid, such a curriculum design, which aims at helping the agent learning to run after knowing how to avoid falling over, hinders the learning efﬁciency and asymptotic performance.
Revised: In general curriculum learning improves learning efﬁciency (Graves et al., 2017; Matiisen et al., 2019; Portelas et al., 2020), however, in the environment of Humanoid, such a curriculum design, which aims at helping the agent learning to run after knowing how to avoid falling over, hinders the learning efﬁciency and asymptotic performance.

Original: Remark 1.
Revised: Remark 1.

Original: Given an MDP M = {S , A , T , R , ρ 0 , γ, T } , where |A| < ∞ , scaling the reward function with linear transformation, i.e., R k,b
Revised: Given an MDP M = {S , A , T , R , ρ 0 , γ, T } , where |A| < ∞ , scaling the reward function with linear transformation, i.e., R k,b

Original: = k ·R + b , ∀ k > 0 , b ∈ R , do not change the optimal policy induced by
Revised: = k ·R + b , ∀ k > 0 , b ∈ R , do not change the optimal policy induced by

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Given an MDP M = {S , A , T , R , ρ 0 , γ, T } , where |A| = ∞ , scaling the reward function with linear transformation, i.e., R k,b
Revised: Given an MDP M = {S , A , T , R , ρ 0 , γ, T } , where |A| = ∞ , scaling the reward function with linear transformation, i.e., R k,b

Original: = k ·R + b , ∀ k > 0 , b ∈ R , do not change the optimal policy induced by deterministic policy gradient (Silver et al., 2014), given proper learning rate:
Revised: = k ·R + b , ∀ k > 0 , b ∈ R , do not change the optimal policy induced by deterministic policy gradient (Silver et al., 2014), given proper learning rate:

Original: [Equation]
Revised: [Equation]

Original: In the following, we will focus on the scenarios when k = 1 to avoid the trivial (though maybe empirically important) discussions on different learning rates.
Revised: In the following, we will focus on the scenarios when k = 1 to avoid the trivial (though maybe empirically important) discussions on different learning rates.

Original: We reveal the importance of selecting the universal bias term, i.e., b in the reward function, through the lens of initialization priors in function approximation.
Revised: We reveal the importance of selecting the universal bias term, i.e., b in the reward function, through the lens of initialization priors in function approximation.

Original: We further show such a bias term can be utilized not only in the online RL settings to improve learning efﬁciency but also in the ofﬂine RL settings to conduct conservative exploitation with batched data.
Revised: We further show such a bias term can be utilized not only in the online RL settings to improve learning efﬁciency but also in the ofﬂine RL settings to conduct conservative exploitation with batched data.

Original:  S HIFTED P RIORS FOR Q-V ALUE E STIMATION 4. R EWARD S HIFT E QUALS TO D IFFERENT I NITIALIZATION
Revised:  S HIFTED P RIORS FOR Q-V ALUE E STIMATION 4. R EWARD S HIFT E QUALS TO D IFFERENT I NITIALIZATION

Original: We start by introducing the central idea of this work: reward shift equals different initialization.
Revised: We start by introducing the central idea of this work: reward shift equals different initialization.

Original: Speciﬁcally, we illustrate the basic idea with Figure 2.
Revised: Speciﬁcally, we illustrate the basic idea with Figure 2.

Original: The black curves denote the primal optimal
Revised: The black curves denote the primal optimal

Original: Q value functions (e.g and  Q  reward r (cid:48)
Revised: Q -value functions (e.g., Q ∗ ), and the red curves denote the optimal Q -value functions with shifted reward r (cid:48)

Original: =
Revised: =

Original:  + b + , where b + > 0 is a positive constant bias, and we will show a different choice of such a  leads to a different motivation in Q 
Revised: r + b + , where b + > 0 is a positive constant bias, and we will show a different choice of such a bias leads to a different motivation in Q -value estimation.

Revised: The yellow lines denote Q -value estimators, e.g., neural network predictions of the corresponding Q -values.

Original: (a) shift the reward function with a positive bias term b + will lead to an uniformly increased Q -value function, namely Q ∗ b + = Q ∗ + b + 1 − γ , during learning, a neural network estimator ˜ Q initialized with ˜ Q 0 ≈ 0 is optimized to approximate the Q -value functions (e.g., through Temporal Difference or Monte Carlo estimation).
Revised: (a) shift the reward function with a positive bias term b + will lead to an uniformly increased Q -value function, namely Q ∗ b + = Q ∗ + b + 1 − γ , during learning, a neural network estimator ˜ Q initialized with ˜ Q 0 ≈ 0 is optimized to approximate the Q -value functions (e.g., through Temporal Difference or Monte Carlo estimation).

Original: (b) for any value-based RL algorithm, the value optimization step can be regarded as a function F that minimizes the difference between the estimated Q -value function ˜ Q t and the optimal oneQ ∗ , given the interaction experience with the environment (e
Revised: (b) for any value-based RL algorithm, the value optimization step can be regarded as a function F that minimizes the difference between the estimated Q -value function ˜ Q t and the optimal one

Original: g., a replay buffer B for off-policy methods).
Revised: Q ∗ , given the interaction experience with the environment (e.g., a replay buffer B for off-policy methods).

Original: (c) similarly, the optimizer given the same interactive experience (e g., replay buffer B ) will learn to minimize the difference between Q -value function ˜ Q b + ,t and the optimal one Q ∗ b + , after re-labeling the rewards in the buffer by r (cid:48)
Revised: (c) similarly, the optimizer given the same interactive experience (e.g., replay buffer B ) will learn to minimize the difference between Q -value function ˜ Q b + ,t and the optimal one Q ∗ b + , after re-labeling the rewards in the buffer by r (cid:48)

Original: =
Revised: =

Original: r + b + .
Revised: r + b + .

Original: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized at ˜ Q 0 ≈ 0 − b + 1 − γ , rather than ˜ Q 0 ≈ 0 .
Revised: (d) according to Remark 2, the optimization conducted in (c) is equivalent to (b) with the neural network Q -value estimator initialized at ˜ Q 0 ≈ 0 − b + 1 − γ , rather than ˜ Q 0 ≈ 0 .

Original: i.e., by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lowerbounded the optimal Q -value.
Revised: i.e., by shifting the reward with proper positive value b + , we are able to initialize the Q -value network that lowerbounded the optimal Q -value.

Original: To summarize, shifting the reward function with a positive constant is equivalent to initializing the value function network with a smaller value, hence during training, the Q -value of unseen stateaction pair is far lower than the optimal value and hence will not be selected in policy update, leading to conservative learning behavior.
Revised: To summarize, shifting the reward function with a positive constant is equivalent to initializing the value function network with a smaller value, hence during training, the Q -value of unseen stateaction pair is far lower than the optimal value and hence will not be selected in policy update, leading to conservative learning behavior.

Original: 4. C ONSERVATIVE E XPLOITATION
Revised: 4. C ONSERVATIVE E XPLOITATION

Original: We begin with a natural application in the ofﬂine setting, where a policy doesn’t interact with the environment and only learns through a logged data set collected from an unknown behavior policy π β , which can either be an expert that generates high-quality solutions to the task (Fujimoto et al., 2018a;
Revised: We begin with a natural application in the ofﬂine setting, where a policy doesn’t interact with the environment and only learns through a logged data set collected from an unknown behavior policy π β , which can either be an expert that generates high-quality solutions to the task (Fujimoto et al., 2018a;

Original: Zhang et al., 2020; Fu et al., 2020) or a non-expert that provides actions that are sub-optimal (Wu et al., 2019; Kumar et al., 2019; Agarwal et al., 2020; Fu et al., 2020;
Revised: Zhang et al., 2020; Fu et al., 2020) or a non-expert that provides actions that are sub-optimal (Wu et al., 2019; Kumar et al., 2019; Agarwal et al., 2020; Fu et al., 2020;

Original: Jarrett et al., 2021) or a mixture of both (Bharadhwaj et al., 2020).
Revised: Jarrett et al., 2021) or a mixture of both (Bharadhwaj et al., 2020).

Original: Based on the basic idea we presented in Section 4.1, we introduce a simple yet effective approach for conservative Q-value estimation that learns to heuristically put a lower bound on the optimal
Revised: Based on the basic idea we presented in Section 4.1, we introduce a simple yet effective approach for conservative Q-value estimation that learns to heuristically put a lower bound on the optimal

Original: Q-value function in batch settings.
Revised: Q-value function in batch settings.

Original: Hark back to Figure 2(a), a positive constant b + added to the reward function will lead to a uniformly (positively) shifted optimal Q -value function, and the gap between the primal Q -value function and the new one is b + 1 − γ .
Revised: Hark back to Figure 2(a), a positive constant b + added to the reward function will lead to a uniformly (positively) shifted optimal Q -value function, and the gap between the primal Q -value function and the new one is b + 1 − γ .

Original: Optimizing the Q -value function with logged data (e.g., a ﬁxed replay buffer) will minimize the difference between the predicted value and the optimal value with observed data.
Revised: Optimizing the Q -value function with logged data (e.g., a ﬁxed replay buffer) will minimize the difference between the predicted value and the optimal value with observed data.

Original: For the unobserved data point (in the state-action space
Revised: For the unobserved data point (in the state-action space

Original: S ×A ), the near-zero initialization guarantees the prediction is lower than the optimal Q -value, thus conservative exploitation can be conducted with such a value function.
Revised: S ×A ), the near-zero initialization guarantees the prediction is lower than the optimal Q -value, thus conservative exploitation can be conducted with such a value function.

Original: 4. (C URIOSITY -D RIVEN ) O PTIMISTIC E XPLORATION
Revised: 4. (C URIOSITY -D RIVEN ) O PTIMISTIC E XPLORATION

Original: On the other hand, if we shift the reward function to the negative side, it is equivalent to optimistic initialization.
Revised: On the other hand, if we shift the reward function to the negative side, it is equivalent to optimistic initialization.

Original: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration: while adding a negative constant value b − on the reward function lead to negatively shiftedoptimal Q -value function Q ∗ b − (Figure 3 (a)), minimizing the difference between a Q -value approximator and the optimal Q -value will enable calculating an upper-bound estimation for Q ∗ , as shown in Figure 3(b). With sufﬁciently large b − (so that b − larger than the maximal value of any s, a -pair), such an upper bound of Q ∗ can be used to conduct curiosity-driven exploration. Intuitively, initializing a value network that predicts value larger or equal than the true value will lead to curiosity-driven exploration, as any visited state will be assigned a low value and the policy that learns to perform the action with a higher value will tend to choose novel actions.
Revised: Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven exploration: while adding a negative constant value b − on the reward function lead to negatively shifted

Original: − 𝑏 + 1−𝛾 𝑸
Revised: − 𝑏 + 1−𝛾 𝑸

Original: 𝓐
Revised: 𝓐

Original: ෩𝑸
Revised: ෩𝑸

Original: LB,𝑡
Revised: LB,𝑡

Original: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐
Revised: = ෩𝑸 𝑏 + ,𝑡 − 𝑏 + 1−𝛾 𝐜 𝐚 𝐛 𝑸 ∗ 𝑸 𝓐

Original: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =
Revised: 𝑏 − 1−𝛾 𝑸 𝑏 − ∗ =

Original: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 0 ≈𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸
Revised: 𝑸 ∗ + 𝑏 − 1−𝛾 ෪𝑸 0 ≈𝑸 ∗ 𝑸 𝓐 − 𝑏 − 1−𝛾 ෩𝑸

Original: UB,𝑡
Revised: UB,𝑡

Original: =
Revised: =

Original: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾 ෩𝑸
Revised: ෩𝑸 𝑏 − ,𝑡 − 𝑏 − 1−𝛾 ෩𝑸

Original: UB,𝑡 ≥ 𝑸 ∗ ෩𝑸 LB,𝑡
Revised: UB,𝑡 ≥ 𝑸 ∗ ෩𝑸 LB,𝑡

Original: ≤ 𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏
Revised: ≤ 𝑸 ∗ − 𝑏 − 1−𝛾 − 𝑏

Original: + 1−𝛾 
Revised: + 1−𝛾 optimal value function Q ∗ b − (Figure 3 (a)), minimizing the difference between a Q -value approximator and the optimal Q -value will enable calculating an upper-bound estimation for Q ∗ , as shown in Figure 3(b). With sufﬁciently large b − (so that b − larger than the maximal value of any s, a -pair), such an upper bound of Q ∗ can be used to conduct curiosity-driven exploration. Intuitively, initializing a value network that predicts value larger or equal than the true value will lead to curiosity-driven exploration, as any visited state will be assigned a low value and the policy that learns to perform the action with a higher value will tend to choose novel actions.

Original: Based on the discoveries above that (1) a positive constant shift added to the reward function can be used for conservative policy update, as shown in Figure 3(c) and (2) a negative constant shift added to the reward function can be used for curiosity-driven exploration, as shown in Figure 3(b).
Revised: Based on the discoveries above that (1) a positive constant shift added to the reward function can be used for conservative policy update, as shown in Figure 3(c) and (2) a negative constant shift added to the reward function can be used for curiosity-driven exploration, as shown in Figure 3(b).

Original: We are ready to access both the upper bound, i.e., the optimistic estimation with b − , and the lower bound, i.e., the conservative estimation with b + of the optimal value function.
Revised: We are ready to access both the upper bound, i.e., the optimistic estimation with b − , and the lower bound, i.e., the conservative estimation with b + of the optimal value function.

Original: Henceforth, we are ready to introduce our sample-efﬁcient algorithms for both continuous control and discrete action space respectively.
Revised: Henceforth, we are ready to introduce our sample-efﬁcient algorithms for both continuous control and discrete action space respectively.

Original: We propose a practical algorithm for general continuous control in Sec.
Revised: We propose a practical algorithm for general continuous control in Sec.

Original: 4.3.1, while focusing on a special class of curiosity-driven exploration method, the RND, in Sec.
Revised: 4.3.1, while focusing on a special class of curiosity-driven exploration method, the RND, in Sec.

Original: 4.3. S AMPLE -E FFICIENT C ONTINUOUS C ONTROL WITH R EWARD S HIFT
Revised: [Figure] [Figure] FFICIENT C ONTINUOUS C ONTROL WITH R EWARD S HIFT

Original: Based on the principle of optimism on the face of uncertainty (OFU), we introduce a exploration bonus that manifests the uncertainty of the Q -value function.
Revised: Based on the principle of optimism on the face of uncertainty (OFU), we introduce a exploration bonus that manifests the uncertainty of the Q -value function.

Original: Speciﬁcally, the basic idea starts from integrating optimistic exploration with conservative exploitation, i.e.,
Revised: Speciﬁcally, the basic idea starts from integrating optimistic exploration with conservative exploitation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the second term with coefﬁcient β denotes exploration bonus that is composed of uncertainty.
Revised: where the second term with coefﬁcient β denotes exploration bonus that is composed of uncertainty.

Original: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b − ,t ( s, a ) will give near-zero predictions as a consequence of initialization (detailed implications are provided in Appendix B).
Revised: For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for our neural network, both ˜ Q b + ,t ( s, a ) and ˜ Q b − ,t ( s, a ) will give near-zero predictions as a consequence of initialization (detailed implications are provided in Appendix B).

Original: henceforth, the explorative bonus becomes − (1 − β ) b + + βb − 1 − γ .
Revised: henceforth, the explorative bonus becomes − (1 − β ) b + + βb − 1 − γ .

Original: Note this is equivalent to applying another constant reward shift with value of c r
Revised: Note this is equivalent to applying another constant reward shift with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: For the explored state-action pairs close to the samples from the replay buffer, we have
Revised: For the explored state-action pairs close to the samples from the replay buffer, we have

Original: Proposition 1.
Revised: Proposition 1.

Original: Assuming we have access to an unbiased estimator for the optimal value function
Revised: Assuming we have access to an unbiased estimator for the optimal value function

Original: Q ∗ , e.g., with Monte-Carlo estimation ˆ Q ∗ =
Revised: Q ∗ , e.g., with Monte-Carlo estimation ˆ Q ∗ =

Original: E (cid:80) t γr t , and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator, i.e., (cid:15) 2 t =
Revised: E (cid:80) t γr t , and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator, i.e., (cid:15) 2 t =

Original: ( ˜ Q t − ˆ Q ∗ ) 2 , ˜ Q t =
Revised: ( ˜ Q t − ˆ Q ∗ ) 2 , ˜ Q t =

Original: ˜ Q t − 1 − 2 η ( ˜ Q t − 1 − ˆ Q ∗ ) , then combining the linear combination in Equation (6) is equivalent to using a linear combination of the constants with value of c r
Revised: ˜ Q t − 1 − 2 η ( ˜ Q t − 1 − ˆ Q ∗ ) , then combining the linear combination in Equation (6) is equivalent to using a linear combination of the constants with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: According to Proposition 1, a grid search for trading-off between the three hyper-parameters: the exploration bias b − , the exploitation bias b + and the coefﬁcient in Equation (6) is trivial as they only lead to a linear combination as c r
Revised: According to Proposition 1, a grid search for trading-off between the three hyper-parameters: the exploration bias b − , the exploitation bias b + and the coefﬁcient in Equation (6) is trivial as they only lead to a linear combination as c r

Original: = (1 − β ) b
Revised: = (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: In principle, a meta-learner can be trained to monitor the learning process and select a proper constant automatically (Graves et al., 2017;
Revised: In principle, a meta-learner can be trained to monitor the learning process and select a proper constant automatically (Graves et al., 2017;

Original: Matiisen et al., 2019; Portelas et al., 2020).
Revised: Matiisen et al., 2019; Portelas et al., 2020).

Original: In this work we focus on a simple yet effective uniform sampling strategy from multiple shift constants, which is shown as a strong baseline in Graves et al.
Revised: In this work we focus on a simple yet effective uniform sampling strategy from multiple shift constants, which is shown as a strong baseline in Graves et al.

Original: (2017); Matiisen et al.
Revised: (2017); Matiisen et al.

Original: and leave more complicated meta-learner-based approaches in future investigation.
Revised: and leave more complicated meta-learner-based approaches in future investigation.

Original: [Figure] Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s (cid:48) ) sampled from the identical buffer that collects the policy’s interaction history with the environment.
Revised: Speciﬁcally, we use multiple Q-networks to learn with transition tuples ( s, a, r, s (cid:48) ) sampled from the identical buffer that collects the policy’s interaction history with the environment.

Original: For each
Revised: For each

Original: Q-network, the primal r is replaced with a new reward with shifted constant bias for temporal difference updates.
Revised: Q-network, the primal r is replaced with a new reward with shifted constant bias for temporal difference updates.

Original: Those learned Q-networks are sampled uniformly during the training of policy network.
Revised: Those learned Q-networks are sampled uniformly during the training of policy network.

Original: It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times to collect samples for each value network [Figure]
Revised: It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times to collect samples for each value network.

Original: 4.3. I MPROVING V ALUE -B ASED C URIOSITY -D RIVEN E XPLORATION
Revised: 4.3. I MPROVING V ALUE -B ASED C URIOSITY -D RIVEN E XPLORATION

Original: The discussion above casts the curiosity-driven exploration method as a special case in the reward shift equals to initialization perspective: exploration with RND (Burda et al., 2018b) is equivalent to selecting b + = 0 , i.e., using the primal task reward for exploitation, and using b − as a ﬁxed random function over S×A , i.e., b − ( s, a ) =
Revised: The discussion above casts the curiosity-driven exploration method as a special case in the reward shift equals to initialization perspective: exploration with RND (Burda et al., 2018b) is equivalent to selecting b + = 0 , i.e., using the primal task reward for exploitation, and using b − as a ﬁxed random function over S×A , i.e., b − ( s, a ) =

Original: | φ 1 , 0 ( s, a ) − φ 2 ( s, a )
Revised: | φ 1 , 0 ( s, a ) − φ 2 ( s, a )

Original: | 2 , ∀ s, a , where φ 1 , 0 and φ 2 are two neural networks with different random initializations.
Revised: | 2 , ∀ s, a , where φ 1 , 0 and φ 2 are two neural networks with different random initializations.

Original: While φ 1 , 0 is learnable, φ 2 is set to be ﬁxed during learning as a random curiosity prior (Osband et al., 2018).
Revised: While φ 1 , 0 is learnable, φ 2 is set to be ﬁxed during learning as a random curiosity prior (Osband et al., 2018).

Original: Without loss of generality, we can assume φ 2 = 1 as a constant initialization, and φ 1 , 0 = 0 .
Revised: Without loss of generality, we can assume φ 2 = 1 as a constant initialization, and φ 1 , 0 = 0 .

Original: Then the exploration behavior is fully controlled by the scale of external reward, which is set to be 1.0 with external reward clipped to [ − 1 , 1] in (Burda et al., 2018b).
Revised: Then the exploration behavior is fully controlled by the scale of external reward, which is set to be 1.0 with external reward clipped to [ − 1 , 1] in (Burda et al., 2018b).

Original: The ﬁnal equivalent objective in this case is to use c r = | φ 1 ,t
Revised: The ﬁnal equivalent objective in this case is to use c r = | φ 1 ,t

Original: ( s, a ) − 

Revised: ( s, a ) −

Original: | 2 , t = 0 , 1 , 2 , ... .
Revised: | 2 , t = 0 , 1 , 2 , ... .

Original: Specially, φ 1 , 0 = 0 and c r = 1 at beginning.
Revised: Specially, φ 1 , 0 = 0 and c r = 1 at beginning.

Original: According to our analysis in the previous section, such a positive shift directly leads to conservative behaviors in Q -value estimation, therefore may hinder the exploration behaviors at the beginning, i.e., when | φ 1 ,t − 1 | (cid:29) 0 .
Revised: According to our analysis in the previous section, such a positive shift directly leads to conservative behaviors in Q -value estimation, therefore may hinder the exploration behaviors at the beginning, i.e., when | φ 1 ,t − 1 | (cid:29) 0 .

Original: The exploration bonus in RND only becomes effective when visiting seldom visited states after the predictions of φ 1 ,t of frequently-visited states are close to 1 , and hence encouraging stepping into those novel states and discover new knowledge.
Revised: The exploration bonus in RND only becomes effective when visiting seldom visited states after the predictions of φ 1 ,t of frequently-visited states are close to 1 , and hence encouraging stepping into those novel states and discover new knowledge.

Original: To overcome the conservative tendency induced by Q -value estimation in RND, we propose to use b − ( s, a ) = | φ 1 , 0 ( s, a ) − φ 2 ( s, a ) | 2 −
Revised: To overcome the conservative tendency induced by Q -value estimation in RND, we propose to use b − ( s, a ) = | φ 1 , 0 ( s, a ) − φ 2 ( s, a ) | 2 −

Original: I, ∀ s, a , where I is a positive constant (e.g., I = max s,a | φ 1 , 0 ( s, a ) − φ 2 ( s, a ) | ) that assures b − ( s, a ) is negative-initialized for optimistic exploratory behaviors
Revised: I, ∀ s, a , where I is a positive constant (e.g., I = max s,a | φ 1 , 0 ( s, a ) − φ 2 ( s, a ) | ) that assures b − ( s, a ) is negative-initialized for optimistic exploratory behaviors.

Original: It is worth noting that the curiosity introduced by exploratory bonuses like RND focuses on increasing the temporary Q -value of rarely visited states, which still relies on the visitation behavior itself in random exploration.
Revised: It is worth noting that the curiosity introduced by exploratory bonuses like RND focuses on increasing the temporary Q -value of rarely visited states, which still relies on the visitation behavior itself in random exploration.

Original: i.e., without stepping into a non-frequent visited state, the agent will never receive the novelty bonus based on the state.
Revised: i.e., without stepping into a non-frequent visited state, the agent will never receive the novelty bonus based on the state.

Original: On the other hand, the mechanism of shifting the reward by a constant is an universal optimistic exploration bonus as the curiosity is fused into the initialization across the entire state-action support.
Revised: On the other hand, the mechanism of shifting the reward by a constant is an universal optimistic exploration bonus as the curiosity is fused into the initialization across the entire state-action support.

Original:  E XPERIMENTS 5. O FFLINE R EINFORCEMENT L EARNING WITH C ONSERVATIVE Q- VALUE E STIMATION
Revised:  E XPERIMENTS 5. O FFLINE R EINFORCEMENT L EARNING WITH C ONSERVATIVE Q- VALUE E STIMATION

Original: We ﬁrst demonstrate the proposed method in the ofﬂine RL setting.
Revised: We ﬁrst demonstrate the proposed method in the ofﬂine RL setting.

Original: As has been discussed in Section 4.2, shifting the primal reward function with a positive constant provides a natural way of conservative exploitation.
Revised: As has been discussed in Section 4.2, shifting the primal reward function with a positive constant provides a natural way of conservative exploitation.

Original: By adding a positive reward shift, any visited ( s, a ) saved in the given replay buffer B (logged dataset) will be assigned a large reward, while under-explored ( s, a ) pairs remain have the low values due to initialization, as illustrated in Figure 2 (d).
Revised: By adding a positive reward shift, any visited ( s, a ) saved in the given replay buffer B (logged dataset) will be assigned a large reward, while under-explored ( s, a ) pairs remain have the low values due to initialization, as illustrated in Figure 2 (d).

Original: Although in general our proposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm, in this work we demonstrate the effectiveness of such a conservative Q -value estimation based on BCQ (Fujimoto et al., 2018a) and CQL (Bharadhwaj et al., 2020), i.e., both distribution-matching approach and conservative value estimation approaches in ofﬂine RL.
Revised: Although in general our proposed method can be plugged in to any off-the-shelf ofﬂine RL algorithm, in this work we demonstrate the effectiveness of such a conservative Q -value estimation based on BCQ (Fujimoto et al., 2018a) and CQL (Bharadhwaj et al., 2020), i.e., both distribution-matching approach and conservative value estimation approaches in ofﬂine RL.

Original: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift ( Neg. ), added on either BCQ or CQL.
Revised: To verify our insight, we experiment with both positive reward shift ( Pos. ) and negative reward shift ( Neg. ), added on either BCQ or CQL.

Original: Figure 4 shows our experiment results.
Revised: Figure 4 shows our experiment results.

Revised: We experiment with

Original: We experiment with both the dataset generated in BCQ (Hopper Imitate BCQ) and the dataset used in CQL (Fu et al., 2020) (others), and ﬁnd in our experiments that learning with the CQL dataset is much more stable.
Revised: [Figure] both the dataset generated in BCQ (Hopper Imitate BCQ) and the dataset used in CQL (Fu et al., 2020) (others) [Figure] and ﬁnd in our experiments that learning with the CQL dataset is much more stable.

Original: the ﬁrst two ﬁgures show results with BCQ as the backbone algorithm, where the former shows results on the vanilla BCQ dataset, while the latter shows results on the CQL dataset.
Revised: the ﬁrst two ﬁgures show results with BCQ as the backbone algorithm, where the former shows results on the vanilla BCQ dataset, while the latter shows results on the CQL dataset.

Original: The following three experiments in Figure 4 use CQL as the backbone.
Revised: The following three experiments in Figure 4 use CQL as the backbone.

Original: In all experiments, using a positive reward shift leads to improved learning performance, while a negative reward shift leads to performance decay, as expected.
Revised: In all experiments, using a positive reward shift leads to improved learning performance, while a negative reward shift leads to performance decay, as expected.

Original: Implementation details and ablation study can be found in Appendix C.1.
Revised: Implementation details and ablation study can be found in Appendix C.1.

Original: 5. O NLINE R EINFORCEMENT L EARNING WITH R ANDOMIZED P RIORS
Revised: 5. O NLINE R EINFORCEMENT L EARNING WITH R ANDOMIZED P RIORS

Original: We then conduct experiments on the online RL settings.
Revised: We then conduct experiments on the online RL settings.

Original: We demonstrate our proposed method in the MuJoCo locomotion benchmarks.
Revised: We demonstrate our proposed method in the MuJoCo locomotion benchmarks.

Original: As our implementation is based on TD3, we use TD3-based variants as our baselines:
Revised: As our implementation is based on TD3, we use TD3-based variants as our baselines:

Original: The TD3 is trained with default settings according to Fujimoto et al.
Revised: The TD3 is trained with default settings according to Fujimoto et al.

Original: (2018b).
Revised: (2018b).

Original: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similar to our work in using multiple Q -networks in value estimation.
Revised: We also include Ensemble TD3 and Bootstrapped TD3 as baselines due to they are similar to our work in using multiple Q -networks in value estimation.

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: but extend it to the continuous control settings.
Revised: but extend it to the continuous control settings.

Original: Noting that in the continuous control setting, the argmax operator is approximated by the policy network, multiple policy networks are needed to cooperate with the multiple bootstrapped Q -value networks.
Revised: Noting that in the continuous control setting, the argmax operator is approximated by the policy network, multiple policy networks are needed to cooperate with the multiple bootstrapped Q -value networks.

Original: Otherwise, multiple Q -value networks are not independent of each other thus breaking the condition of bootstrapped value estimation.
Revised: Otherwise, multiple Q -value networks are not independent of each other thus breaking the condition of bootstrapped value estimation.

Original: The
Revised: The

Original: Ensemble TD3 presents the baseline performance when multiple Q -networks are used for value estimation in TD3, which also works as an ablation of our method when all reward shift priors are set to be 0 .
Revised: Ensemble TD3 presents the baseline performance when multiple Q -networks are used for value estimation in TD3, which also works as an ablation of our method when all reward shift priors are set to be 0 .

Original: As has been illustrated in Sec.
Revised: As has been illustrated in Sec.

Original: 4.3.1, learning with different reward shifting values is equivalent to learning with optimistic or conservative initialization.
Revised: 4.3.1, learning with different reward shifting values is equivalent to learning with optimistic or conservative initialization.

Original: In our method of Random
Revised: In our method of Random

Original: Reward Shift (RRS), we use 3 Q -networks with different priors.
Revised: Reward Shift (RRS), we use 3 Q -networks with different priors.

Original: In our experiments, we found ± 0 .
Revised: In our experiments, we found ± 0 .

Original: 5 , 0 works universally as a default setting.
Revised: 5 , 0 works universally as a default setting.

Original: Though, further investigation on hyper-parameter may help to further improve the performance.
Revised: Though, further investigation on hyper-parameter may help to further improve the performance.

Original: Results are shown in Figure 5.
Revised: Results are shown in Figure 5.

Original: RRS outperforms the vanilla TD3 in all ﬁve environments and outperforms all baseline methods in most tasks.
Revised: RRS outperforms the vanilla TD3 in all ﬁve environments and outperforms all baseline methods in most tasks.

Original: In all experiments, we use 3 Q -networks for a fair comparison.
Revised: In all experiments, we use 3 Q -networks for a fair comparison.

Original: Note that there is a trade-off between computational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expensive, as reported in (Osband et al., 2016).
Revised: Note that there is a trade-off between computational complexity and sample efﬁciency, i.e., using more Q -networks may further improve the performance at the cost of more computational expensive, as reported in (Osband et al., 2016).

Original: More implementation details and pseudo code of RRS can be found in Appendix C.2. And ablation studies can be bound in Appendix ?? .
Revised: More implementation details, pseudo code of RRS, and ablation studies can be bound in Appendix C.2. [Figure]

Original: 5. O PTIMISTIC R ANDOM N ETWORK D ISTILLATION
Revised: 5. O PTIMISTIC R ANDOM N ETWORK D ISTILLATION

Revised: Finally

Original: Finally, we experiment with ﬁve discrete exploration tasks, namely the MountainCar-v0, and four navigation tasks of MiniGrid suite (Chevalier-Boisvert et al., 2018), namely the task of EmptyRandom, MultiRoom, and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.
Revised: [Figure] we experiment with ﬁve discrete exploration tasks, namely the MountainCar-v0, and four navigation tasks of MiniGrid suite (Chevalier-Boisvert et al., 2018), namely the task of EmptyRandom, MultiRoom [Figure] and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.

Original: More environment details are provided in the Appendix C.3.
Revised: More environment details are provided in the Appendix C. [Figure]

Original: We compare the vanilla DQN , vanilla RND , as well as improved DQN and RND according to our proposed insight.
Revised: We compare the vanilla DQN , vanilla RND , as well as improved DQN and RND according to our proposed insight.

Original: DQN -0.5 indicates the results with a reward shifting of − 0 .
Revised: DQN -0.5 indicates the results with a reward shifting of − 0 .

Original: Comparing DQN0.5 with the vanilla DQN, our insight of negative reward shift leads to curiosity-driven exploration is [Figure]
Revised: Comparing DQN0.5 with the vanilla DQN, our insight of negative reward shift leads to curiosity-driven exploration is again veriﬁed.

Original: [Figure]{+RND -1.0 indicates the results when a reward shifting of − 1 . 
 is added to RND based on DQN. RND -1.0 improves the performance of RND and DQN in most environments, showing the effectiveness of the RND-based exploration bonus. Note that for a fair comparison, RND1.0 should be compared with the vanilla DQN as the − 1 . 0 reward shift just cancels the positive exploration bonus introduced by RND.+}
Revised: {+RND -1.0 indicates the results when a reward shifting of − 1 . 
 is added to RND based on DQN. RND -1.0 improves the performance of RND and DQN in most environments, showing the effectiveness of the RND-based exploration bonus. Note that for a fair comparison, RND1.0 should be compared with the vanilla DQN as the − 1 . 0 reward shift just cancels the positive exploration bonus introduced by RND.+}

Original: Moreover we can further improve RND by equipping it with reward-shifting-based curiosity-driven exploration.
Revised: Moreover, we can further improve RND by equipping it with reward-shifting-based curiosity-driven exploration.

Original: RND -1.5 (i e., RND with a − 1 .
Revised: RND -1.5 (i.e., RND with a − 1 .

Original: 5 reward shift) can be compared with the DQN -0.5, as both receive an -0.5 exploration bonus for unseen states.
Revised: 5 reward shift) can be compared with the DQN -0.5, as both receive an -0.5 exploration bonus for unseen states.

Original: We ﬁnd in all experiments that our negative reward shift can remarkably improve exploration, not only working with RND to improve its performance but also work effectively in isolation.
Revised: We ﬁnd in all experiments that our negative reward shift can remarkably improve exploration, not only working with RND to improve its performance but also work effectively in isolation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we study how reward shifting affects policy learning in value-based deep reinforcement learning algorithms.
Revised: In this work, we study how reward shifting affects policy learning in value-based deep reinforcement learning algorithms.

Original: Although constant reward shifting should not change the optimal policy induced by the optimal value function, in practice such a constant shift does affect the function approximation.
Revised: Although constant reward shifting should not change the optimal policy induced by the optimal value function, in practice such a constant shift does affect the function approximation.

Original: Our detailed analysis manifests the fact that a constant reward shift is equivalent to using different initialization in the value function approximation.
Revised: Our detailed analysis manifests the fact that a constant reward shift is equivalent to using different initialization in the value function approximation.

Original: Speciﬁcally, we show that a negative reward shift leads to curiosity-driven exploration, while a positive reward shift helps conservative exploitation.
Revised: Speciﬁcally, we show that a negative reward shift leads to curiosity-driven exploration, while a positive reward shift helps conservative exploitation.

Original: The proposed idea is then veriﬁed through a variety of application scenarios, including ofﬂine RL, sample-efﬁcient continuous control, and curiosity-driven exploration in value-based methods.
Revised: The proposed idea is then veriﬁed through a variety of application scenarios, including ofﬂine RL, sample-efﬁcient continuous control, and curiosity-driven exploration in value-based methods.

Original: Thus reward shifting is a simple yet effective technique that deserves further investigation. 
Revised: Thus reward shifting is a simple yet effective technique that deserves further investigation. [Figure]

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: In this work  we study how the linear reward shaping in reinforcement learning beneﬁts ofﬂine conservative estimation, online continuous control, and curiosity-driven exploration in discrete control tasks.
Revised: In this work [Figure] we study how the linear reward shaping in reinforcement learning beneﬁts ofﬂine conservative estimation, online continuous control, and curiosity-driven exploration in discrete control tasks.

Original: Although in our work we experimented on a variety of benchmark environments, there are plenty of real-world applications:
Revised: Although in our work we experimented on a variety of benchmark environments, there are plenty of real-world applications:

Original: Improving the learning stability as well as the asymptotic performance of ofﬂine RL empowers the application of RL in scenarios where interaction with the environment is extremely expensive or unethical, e g., healthcare, robotics, ﬁnance, etc.
Revised: Improving the learning stability as well as the asymptotic performance of ofﬂine RL empowers the application of RL in scenarios where interaction with the environment is extremely expensive or unethical, e g., healthcare, robotics, ﬁnance, etc.

Original: Moreover, our discussions on continuous and discrete control tasks open up a promising direction in pursuance of sample-efﬁcient learning without introducing heavy extra computational burdens.
Revised: Moreover, our discussions on continuous and discrete control tasks open up a promising direction in pursuance of sample-efﬁcient learning without introducing heavy extra computational burdens.

Original: Large-scale applications of our method can help improving efﬁciency in exploration, which is always conducted through computing additional curiosity networks explicitly in previous works.
Revised: Large-scale applications of our method can help improving efﬁciency in exploration, which is always conducted through computing additional curiosity networks explicitly in previous works

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We include our code in the supplementary materials.
Revised: We include our code in the supplementary materials.

Original: More details for our experiments on ofﬂine
Revised: More details for our experiments on ofﬂine

Original: RL, continuous control and discrete control can be found in Appendix C.1, Appendix C. and
Revised: RL, continuous control and discrete control can be found in Appendix C.1, Appendix C. and

Original: Appendix C.3, separately.
Revised: Appendix C.3, separately.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: the estimated Q -value ˆ Q ( s, a ) is composed by the two estimators with function approximation error, deﬁned as (cid:15) b
Revised: the estimated Q -value ˆ Q ( s, a ) is composed by the two estimators with function approximation error, deﬁned as (cid:15) b

Original: + ( s, a ) =
Revised: + ( s, a ) =

Original: ˜ Q b + ,t ( s, a ) − b + 1 − γ − Q ∗ ( s, a ) and (cid:15) b − ( s, a ) =
Revised: ˜ Q b + ,t ( s, a ) − b + 1 − γ − Q ∗ ( s, a ) and (cid:15) b − ( s, a ) =

Original: ˜ Q b − ,t ( s, a ) − b − 1 − γ − Q ∗ ( s, a ) .
Revised: ˜ Q b − ,t ( s, a ) − b − 1 − γ − Q ∗ ( s, a ) .

Original: (1 − β ) (cid:15) A,t + β(cid:15) B,t = 2 η (1 − β ) ˆ Q ∗ + (1 − β )(1 − 2 η ) ˜ Q A,t + 2 ηβ ˆ Q ∗ + β (1 − 2 η ) ˜ Q B,t = 2 η ˆ Q ∗ + (1 − 2 η )
Revised: [Equation]

Original: [(1 − β ) ˜ Q A,t + β ˜ Q B,t ] = [-
 η ˆ Q ∗ + (1 − 2 η )[(1 − β )(1 − 2 η ) t ˜ Q A, 0 + 4(1 − β ) η 2 1 − (1 − 2 η ) t ˆ-]
Revised: = [-
 η ˆ Q ∗ + (1 − 2 η )[(1 − β )(1 − 2 η ) t ˜ Q A, 0 + 4(1 − β ) η 2 1 − (1 − 2 η ) t ˆ-][Equation]

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Revised: t =

Original: [Equation]
Revised: [Equation]

Original: t
Revised: η ˆ

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] −
Revised: t ] =

Original: [Equation]
Revised: [Equation]

Original: ] = 2 η ˆQ ∗ + (1 − 2 η )[(1 − 2 η ) t ((1 − β ) ˜ Q A
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ˜ Q B

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: = (cid:15) C,t (7)
Revised: [Equation] [Equation] [Equation] [Equation]

Original: where C = (1 − β ) A + βB and the last line requires ˜ Q A, 0 = ˜ Q B, 0 = ˜ Q C, 0 are identical initialization.
Revised: where C = (1 − β ) A + βB and the last line requires ˜ Q A, 0 = ˜ Q B, 0 = ˜ Q C, 0 are identical initialization.

Original: With this notion, Equation (6) can be re-written as
Revised: With this notion, Equation (6) can be re-written as

Original: [Equation]
Revised: [Equation]

Original: where the second line relies on the linear assumption of the approximation error (1 − β ) (cid:15) b
Revised: where the second line relies on the linear assumption of the approximation error (1 − β ) (cid:15) b

Original: + ( s, a ) + β(cid:15) b − ( s, a ) .
Revised: + ( s, a ) + β(cid:15) b − ( s, a ) .

Original: We further have (1 − β ) ˜ Q b + + β ˜ Q b − =
Revised: We further have (1 − β ) ˜ Q b + + β ˜ Q b − =

Original: ˜ Q (1 − β ) b + + βb − and ˆ Q ( s, a ) =
Revised: ˜ Q (1 − β ) b + + βb − and ˆ Q ( s, a ) =

Original: ˜ Q c r ( s, a ) , telling us that trading-off between the constant b − used for exploration and the constant b + used for exploitation with the coefﬁcient β is equivalent to use another constant with value of c r
Revised: ˜ Q c r ( s, a ) , telling us that trading-off between the constant b − used for exploration and the constant b + used for exploitation with the coefﬁcient β is equivalent to use another constant with value of c r

Original: =
Revised: =

Original: (1 − β ) b
Revised: (1 − β ) b

Original: + + βb − .
Revised: + + βb − .

Original: B I MPLICATIONS OF A SSUMPTION IN S EC . 4.3. 
Revised: B I MPLICATIONS OF A SSUMPTION IN S EC . 4.3.

Original: In our main text, the estimated values for extremely o.o.d.
Revised: In our main text, the estimated values for extremely o.o.d.

Original: samples are assumed to be near zeros.
Revised: samples are assumed to be near zeros.

Original: We provide detailed implications and explanations in this section.
Revised: We provide detailed implications and explanations in this section.

Original: On the one hand, it’s clear that such an assumption holds for the tabular settings, that un-visited state-action pairs have the value in tabular initialization.
Revised: On the one hand, it’s clear that such an assumption holds for the tabular settings, that un-visited state-action pairs have the value in tabular initialization.

Original: On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d.
Revised: On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d.

Original: samples that have the Q-values near zero for function approximation settings.
Revised: samples that have the Q-values near zero for function approximation settings.

Original: Interpolation between those o.o.d.
Revised: Interpolation between those o.o.d.

Original: samples and other state-action pairs will clearly lead to an “in-between” value estimation, which in practice can be achieved with properly regularized neural networks.
Revised: samples and other state-action pairs will clearly lead to an “in-between” value estimation, which in practice can be achieved with properly regularized neural networks.

Original: The key insight we want to emphasize in Sec.
Revised: The key insight we want to emphasize in Sec.

Original: 4.3.1 is that for frequently visited state-action pairs, the value discrepancy with different initialization are small, while for seldomly-visited state-action pairs, the discrepancy are relatively large, enabling the usage of such discrepancy as exploration bonus.
Revised: 4.3.1 is that for frequently visited state-action pairs, the value discrepancy with different initialization are small, while for seldomly-visited state-action pairs, the discrepancy are relatively large, enabling the usage of such discrepancy as exploration bonus.

Original: C I MPLEMENTATION D ETAILS A ND A BLATION S TUDIES
Revised: C I MPLEMENTATION D ETAILS A ND A BLATION S TUDIES

Original: Hardware and Training Time We experiment on a server with 8 TITAN X GPUs and 32 Intel(R)
Revised: Hardware and Training Time We experiment on a server with 8 TITAN X GPUs and 32 Intel(R)

Original: E5-2640 CPUs.
Revised: E5-2640 CPUs.

Original: In general, shifting the reward does not introduce further computation burden except in the continuous control tasks, our method of Random Reward Shift (RRS) requires two additional
Revised: In general, shifting the reward does not introduce further computation burden except in the continuous control tasks, our method of Random Reward Shift (RRS) requires two additional

Original: Q -value networks.
Revised: Q -value networks.

Original: In our PyTorch-based implementation, those additional networks can be easily implemented and optimized in a parallel manner, and the extra computational burden is equivalent to using a √ 3 times wider neural network during optimization.
Revised: In our PyTorch-based implementation, those additional networks can be easily implemented and optimized in a parallel manner, and the extra computational burden is equivalent to using a √ 3 times wider neural network during optimization.

Original: It is worth noting that RRS is computationally much cheaper than the Bootstrapped TD3, where additional policy networks are also needed.
Revised: It is worth noting that RRS is computationally much cheaper than the Bootstrapped TD3, where additional policy networks are also needed.

Original: Network 
Revised: Network Structure

Original: Structure Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters.
Revised: Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters.

Original: We implement DQN based on a 3-layer fully connected neural network with 64 hidden units for the Q -value function, using ReLU and linear activation respectively.
Revised: We implement DQN based on a 3-layer fully connected neural network with 64 hidden units for the Q -value function, using ReLU and linear activation respectively.

Original: We use the Adam optimizer with learning rate of 0 . 001 , and use an epsilongreedy approach as naive exploration strategy.
Revised: We use the Adam optimizer with learning rate of 0 . 001 , and use an epsilongreedy approach as naive exploration strategy.

Original: In our RND, we use two 4-layer fully connected neural networks with 512 units and ReLU activation in each hidden layer, and a softmax activation for the output layer.
Revised: In our RND, we use two 4-layer fully connected neural networks with 512 units and ReLU activation in each hidden layer, and a softmax activation for the output layer.

Original: Adam optimizer is used for the optimization of the RND networks with learning rate 0 . 0001 .
Revised: Adam optimizer is used for the optimization of the RND networks with learning rate 0 . 0001 .

Original: Our code is provided in the supplementary materials, and will be made public available.
Revised: Our code is provided in the supplementary materials, and will be made public available.

Original: C. O FFLINE RL
Revised: C. O FFLINE RL

Original: In our experiments, we use a ﬁxed dataset with 10 k ofﬂine trainsition tuples for ofﬂine RL learning.
Revised: In our experiments, we use a ﬁxed dataset with 10 k ofﬂine trainsition tuples for ofﬂine RL learning.

Original: Our implementation of BCQ and CQL are both based on the code provided by the authors.
Revised: Our implementation of BCQ and CQL are both based on the code provided by the authors.

Original: The only change we made to verify our insight is to shift the reward by a constant.
Revised: The only change we made to verify our insight is to shift the reward by a constant.

Original: In most environments, we ﬁnd r (cid:48)
Revised: In most environments, we ﬁnd r (cid:48)

Original: =
Revised: =

Original: r + 8 provides good enough performance.
Revised: r + 8 provides good enough performance.

Original: While in Hopper Medium CQL we ﬁnd using a smaller positive reward shift r (cid:48)
Revised: While in Hopper Medium CQL we ﬁnd using a smaller positive reward shift r (cid:48)

Original: =
Revised: =

Original: r + 1 works better than r (cid:48)
Revised: r + 1 works better than r (cid:48)

Original: =
Revised: =

Original: r + 8 , and for Walker Medium CQL, using a larger reward shift of r (cid:48)
Revised: r + 8 , and for Walker Medium CQL, using a larger reward shift of r (cid:48)

Original: =
Revised: =

Original: r + 50 further improves the result with r (cid:48)
Revised: r + 50 further improves the result with r (cid:48)

Original: =
Revised: =

Original: r + 8 .
Revised: r + 8 .

Original: Figure 7 shows different performance under different choices of the reward shift constant.
Revised: Figure 7 shows different performance under different choices of the reward shift constant.

Original: We denote a positive reward shift r (cid:48)
Revised: We denote a positive reward shift r (cid:48)

Original: =
Revised: =

Original: r + 8 as Pos.1 , denote r (cid:48)
Revised: r + 8 as Pos.1 , denote r (cid:48)

Original: =
Revised: =

Original: r + 20 as Pos.2 and denote r (cid:48) =
Revised: r + 20 as Pos.2 and denote r (cid:48) =

Original: r + 50 as
Revised: r + 50 as

Original: Pos.3 for all experiments excetp in the Hopper Medium CQL we use Pos.1 to denote r (cid:48)
Revised: Pos.3 for all experiments excetp in the Hopper Medium CQL we use Pos.1 to denote r (cid:48)

Original: =
Revised: =

Original: r + 1 .
Revised: r + 1 .

Original: In the experiments based on BCQ (ﬁrst two ﬁgures).
Revised: In the experiments based on BCQ (ﬁrst two ﬁgures).

Original: We can observe a uniformly performance improvement with all choices of reward shift constants.
Revised: We can observe a uniformly performance improvement with all choices of reward shift constants.

Original: As the algorithm of CQL has already taken the conservative value estimation into consideration, in the experiments based on CQL, the performance is more closely related to the constant we use.
Revised: As the algorithm of CQL has already taken the conservative value estimation into consideration, in the experiments based on CQL, the performance is more closely related to the constant we use.

Original: Speciﬁcally, in Hopper Expert, while using any of the positive reward shift constants improve the learning stability, r (cid:48)
Revised: Speciﬁcally, in Hopper Expert, while using any of the positive reward shift constants improve the learning stability, r (cid:48)

Original: =
Revised: =

Original: r + 8 performs better on preserving the learning efﬁciency during early learning stage.
Revised: r + 8 performs better on preserving the learning efﬁciency during early learning stage.

Original: For Hopper Medium, we ﬁnd using larger positive constants hinder the performance.
Revised: For Hopper Medium, we ﬁnd using larger positive constants hinder the performance.

Original: For Walker Medium, using a larger constant in reward shift performs much better than using a smaller one.
Revised: For Walker Medium, using a larger constant in reward shift performs much better than using a smaller one.

Original:  C. C ONTINUOUS C ONTROLPseudo-Code for
Revised: [Figure] C. C ONTINUOUS C ONTROL

Original:  Random Reward Shift
Revised: Pseudo-Code for Random Reward Shift

Original: The pseudo-code of RRS is provided in Algorithm1.
Revised: The pseudo-code of RRS is provided in Algorithm

Original: [Figure]
Revised: [Figure]

Original: • the size of mini-batch N , smoothing factor τ > 0 , K reward shift values

Original: [Figure]

Original: [Equation]

Original: [Figure] • Random initialized policy network π θ , target policy network π [Figure] Details of RRS Although we ﬁnd in the motivating example that a − 5 reward shift is able to remarkably improve the asymptotic performance of TD3, in this work we aim at proposing an uniformly suitable method based on the insight behind the motivating example.
Revised: Details of RRS Although we ﬁnd in the motivating example that a − 5 reward shift is able to remarkably improve the asymptotic performance of TD3, in this work we aim at proposing an uniformly suitable method based on the insight behind the motivating example.

Original: Therefore we propose to use ± 0 .
Revised: Therefore we propose to use ± 0 .

Original: 5 , 0 as the reward shifting constants.
Revised: 5 , 0 as the reward shifting constants.

Original: We ﬁnd in experiment that the sampling frequency does not affect the performance.
Revised: We ﬁnd in experiment that the sampling frequency does not affect the performance.

Original: And in the experiments we follow BDQN Osband et al.
Revised: And in the experiments we follow BDQN Osband et al.

Original: (2016) to use a ﬁxed value network throughout a whole trajectory.
Revised: (2016) to use a ﬁxed value network throughout a whole trajectory.

Original: i.e., one of the K Q -networks is sampled uniformly after each episode with length of 1000 timesteps.
Revised: i.e., one of the K Q -networks is sampled uniformly after each episode with length of 1000 timesteps.

Original: Intuitively, searching for more suitable reward randomization designs may further improve the performance, yet that is beyond the coverage of this work.
Revised: Intuitively, searching for more suitable reward randomization designs may further improve the performance, yet that is beyond the coverage of this work.

Original: Ablation Studies We experiment with different number of Q -value networks as well as different choices of the random reward shifting ranges.
Revised: Ablation Studies We experiment with different number of Q -value networks as well as different choices of the random reward shifting ranges.

Original: Results are presented in Figure 8.
Revised: Results are presented in Figure 8.

Original: We denote RRS with 7 reward shifting constants ( and therefore also 7 Q -networks) as RRS7 , and denote RRS with 3 reward shifting constants ( and therefore also 3 Q -networks) as RRS-3 .
Revised: We denote RRS with 7 reward shifting constants ( and therefore also 7 Q -networks) as RRS- 7 , and denote RRS with 3 reward shifting constants ( and therefore also 3 Q -networks) as RRS-3 .

Original: The constants following RRS-3/RRS-7 are the ranges of those random con- stants.
Revised: The constants following RRS-3/RRS-7 are the ranges of those random constants.

Original: Speciﬁcally, we use [ − 0 .
Revised: Speciﬁcally, we use [ − 0 .

Original: 5] for the RRS-3 0.5 settings, [ − 1 . 0 , 0 , 1 . 0]
Revised: 5] for the RRS-3 0.5 settings, [ − 1 . 0 , 0 , 1 . 0]

Original: for the
Revised: for the

Original: RRS-3 1.0 settings, [ − 0 .
Revised: RRS-3 1.0 settings, [ − 0 .

Original: 5 , − 0 .
Revised: 5 , − 0 .

Original: 33 , − 0 .
Revised: 33 , − 0 .

Original: 33 , 0 . 5] for the RRS-7 0.5 settings and [ − 1 . 0 , − 0 .
Revised: 33 , 0 . 5] for the RRS-7 0.5 settings and [ − 1 . 0 , − 0 .

Original: 67 , − 0 .
Revised: 67 , − 0 .

Original: 67 , 1 . 0] for the RRS-7 1.0 settings.
Revised: 67 , 1 . 0] for the RRS-7 1.0 settings.

Original: According to the experimental results, RRS is not sensitive to hyper-parameters, showing the robustness of the proposed method.
Revised: According to the experimental results, RRS is not sensitive to hyper-parameters, showing the robustness of the proposed method.

Original: We believe further search for those hyper-parameters can further improve the learning efﬁciency, yet this is off the main scope of this work and therefore left for the future research.
Revised: We believe further search for those hyper-parameters can further improve the learning efﬁciency, yet this is off the main scope of this work and therefore left for the future research.

Original: [Figure]
Revised: [Figure]

Original: C. R ANDOM N ETWORK D ISTILLATION
Revised: C. R ANDOM N ETWORK D ISTILLATION

Original: Environments In this work, we experiment with ﬁve discrete (sparse reward) exploration tasks , namely the MountainCar-v0, and four navigation tasks of MiniGrid suite (Chevalier-Boisvert et al., 2018), namely the task of Empty-Random, MultiRoom, and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.
Revised: Environments In this work, we experiment with ﬁve discrete (sparse reward) exploration tasks , namely the MountainCar-v [Figure] and four navigation tasks of MiniGrid suite (Chevalier-Boisvert et al., 2018), namely the task of Empty-Random, MultiRoom [Figure]

Revised: and FourRooms, to verify our insight on improving RND for value-based curiosity-driven exploration.

Original: Figure 9 shows example of different tasks is
Revised: Figure 9 shows example of different tasks.

Original: Ablation Studies We experiment with different reward shifting constants in the discrete control settings.
Revised: Ablation Studies We experiment with different reward shifting constants in the discrete control settings.

Original: We use a relatively large range in choosing constants, i e., {− 0 .
Revised: We use a relatively large range in choosing constants, i.e., {− 0 .

Original: 05 , − 0 .
Revised: 05 , − 0 .

Original: 15 , − 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 .
Revised: 15 , − 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 .

Original: 5 , − 5 . 0 , − 10 . 0 } .
Revised: 5 , − 5 . 0 , − 10 . 0 } .

Original: Results are presented in Figure 10.
Revised: Results are presented in Figure 10.

Original: In all experiments, using a moderate reward shifting constant like {− 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 . 5 } remarkably[Figure] improves the learning efﬁciency.
Revised: In all experiments, using a moderate reward shifting constant like {− 1 . 0 , − 1 . 5 , − 2 . 0 , − 2 . 5 } remarkably improves the learning efﬁciency.

Original: On the other hand, a too aggressive reward shifting will lead to too much curiosity exploration and hinder the learning efﬁciency in the limited number of interactions
Revised: On the other hand, a too aggressive reward shifting will lead to too much curiosity exploration and hinder the learning efﬁciency in the limited number of interactions.

Original: [Figure]
Revised: [Figure]