Original: O PTIMISTIC E XPLORATION WITH B ACKWARD B OOTSTRAPPED B ONUS FOR D EEP R EINFORCEMENT L EARNING
Revised: O PTIMISTIC E XPLORATION WITH B ACKWARD B OOTSTRAPPED B ONUS FOR D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Optimism in the face of uncertainty is a principled approach for provably efﬁcient exploration for reinforcement learning in tabular and linear settings.
Revised: Optimism in the face of uncertainty is a principled approach for provably efﬁcient exploration for reinforcement learning in tabular and linear settings.

Original: However, such an approach is challenging in developing practical exploration algorithms for Deep Reinforcement Learning (DRL).
Revised: However, such an approach is challenging in developing practical exploration algorithms for Deep Reinforcement Learning (DRL).

Original: To address this problem, we propose an Optimistic Exploration algorithm with Backward Bootstrapped Bonus (OEB3) for DRL by following these two principles.
Revised: To address this problem, we propose an Optimistic Exploration algorithm with Backward Bootstrapped Bonus (OEB3) for DRL by following these two principles.

Original: OEB3 is built on bootstrapped deep
Revised: OEB3 is built on bootstrapped deep

Original: Q -learning, a non-parametric posterior sampling method for temporally-extended exploration.
Revised: Q -learning, a non-parametric posterior sampling method for temporally-extended exploration.

Original: Based on such a temporally-extended exploration, we construct an
Revised: Based on such a temporally-extended exploration, we construct an

Original: UCB-bonus indicating the uncertainty of Q -functions.
Revised: UCB-bonus indicating the uncertainty of Q -functions.

Original: The UCB-bonus is further utilized to estimate an optimistic Q -value, which encourages the agent to explore the scarcely visited states and actions to reduce uncertainty.
Revised: The UCB-bonus is further utilized to estimate an optimistic Q -value, which encourages the agent to explore the scarcely visited states and actions to reduce uncertainty.

Original: In the estimation of
Revised: In the estimation of

Original: Q -function, we adopt an episodic backward update strategy to propagate the future uncertainty to the estimated Q -function consistently.
Revised: Q -function, we adopt an episodic backward update strategy to propagate the future uncertainty to the estimated Q -function consistently.

Original: Extensive evaluations show that OEB3 outperforms several state-of-the-art exploration approaches in Mnist maze and 49 Atari games.
Revised: Extensive evaluations show that OEB3 outperforms several state-of-the-art exploration approaches in MNIST maze and 49 Atari games.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In Reinforcement learning (RL) (Sutton & Barto, 2018) formalized by the Markov decision process (MDP), an agent aims to maximize the long-term reward by interacting with the unknown environment.
Revised: In Reinforcement learning (RL) (Sutton & Barto, 2018) formalized by the Markov decision process (MDP), an agent aims to maximize the long-term reward by interacting with the unknown environment.

Original: The agent takes actions according to the knowledge of experiences, which leads to the fundamental problem of the exploration-exploitation dilemma.
Revised: The agent takes actions according to the knowledge of experiences, which leads to the fundamental problem of the exploration-exploitation dilemma.

Original: An agent may choose the best decision given current information or acquire more information by exploring the poorly understood states and actions.
Revised: An agent may choose the best decision given current information or acquire more information by exploring the poorly understood states and actions.

Original: Exploring the environment may sacriﬁce immediate rewards but potentially improves future performance.
Revised: Exploring the environment may sacriﬁce immediate rewards but potentially improves future performance.

Original: The exploration strategy is crucial for the RL agent to ﬁnd the optimal policy.
Revised: The exploration strategy is crucial for the RL agent to ﬁnd the optimal policy.

Original: The theoretical RL offers various provably efﬁcient exploration methods in tabular and linear MDPs with the basic value iteration algorithm: least-squares value iteration (LSVI).
Revised: The theoretical RL offers various provably efﬁcient exploration methods in tabular and linear MDPs with the basic value iteration algorithm: least-squares value iteration (LSVI).

Original: Optimism in the face of uncertainty (Auer & Ortner, 2007; Jin et al., 2018) is a principled approach.
Revised: Optimism in the face of uncertainty (Auer & Ortner, 2007; Jin et al., 2018) is a principled approach.

Original: In tabular cases, the optimism-based methods incorporate upper conﬁdence bound (UCB) into the value functions as bonuses and attain the optimal worst-case regret (Azar et al., 2017; Jaksch et al., 2010; Dann &
Revised: In tabular cases, the optimism-based methods incorporate upper conﬁdence bound (UCB) into the value functions as bonuses and attain the optimal worst-case regret (Azar et al., 2017; Jaksch et al., 2010; Dann &

Original: Brunskill, 2015).
Revised: Brunskill, 2015).

Original: Randomized value function based on posterior sampling chooses actions according to the randomly sampled statistically plausible value functions and is known to achieve near-optimal worst-case and Bayesian regrets (Osband & Van Roy, 2017; Russo, 2019).
Revised: Randomized value function based on posterior sampling chooses actions according to the randomly sampled statistically plausible value functions and is known to achieve near-optimal worst-case and Bayesian regrets (Osband & Van Roy, 2017; Russo, 2019).

Original: Recently, the theoretical analyses in tabular cases are extended to linear MDP where the transition and the reward function are assumed to be linear.
Revised: Recently, the theoretical analyses in tabular cases are extended to linear MDP where the transition and the reward function are assumed to be linear.

Original: In linear cases, optimistic LSVI (Jin et al., 2020) attains a near-optimal worst-case regret by using a designed bonus, which is provably efﬁcient.
Revised: In linear cases, optimistic LSVI (Jin et al., 2020) attains a near-optimal worst-case regret by using a designed bonus, which is provably efﬁcient.

Original: Randomized LSVI (Zanette et al., 2020) also attains a near-optimal worst-case regret.
Revised: Randomized LSVI (Zanette et al., 2020) also attains a near-optimal worst-case regret.

Original: Although the analyses in tabular and linear cases provide attractive approaches for efﬁcient exploration, these principles are still challenging in developing a practical exploration algorithm for 
Revised: Although the analyses in tabular and linear cases provide attractive approaches for efﬁcient exploration, these principles are still challenging in developing a practical exploration algorithm for Deep

Original: Deep Reinforcement Learning (DRL) (Mnih et al., 2015; Fujimoto et al., 2018), which achieves human-level performance in large-scale tasks such as Atari games(Machado et al., 2018) and robotic tasks (Andrychowicz et al., 2020).
Revised: Reinforcement Learning (DRL) (Mnih et al., 2015), which achieves human-level performance in large-scale tasks such as Atari games and robotic tasks.

Original: For example, in linear cases, the bonus in optimistic LSVI (Jin et al., 2020) and nontrivial noise in randomized LSVI (Zanette et al., 2020) are tailored to linear models (Abbasi-Yadkori et al., 2011), and are incompatible with practically powerful function approximations such as neural networks.
Revised: For example, in linear cases, the bonus in optimistic LSVI (Jin et al., 2020) and nontrivial noise in randomized LSVI (Zanette et al., 2020) are tailored to linear models (Abbasi-Yadkori et al., 2011), and are incompatible with practically powerful function approximations such as neural networks.

Original: To address this problem, we propose the Optimistic Exploration algorithm with Backward Bootstrapped Bonus (OEB3) for DRL.
Revised: To address this problem, we propose the Optimistic Exploration algorithm with Backward Bootstrapped Bonus (OEB3) for DRL.

Original: OEB3 is an instantiation of optimistic LSVI (Jin et al., 2020) in
Revised: OEB3 is an instantiation of optimistic LSVI (Jin et al., 2020) in

Original: DRL by using a general-purpose UCB-bonus to provide an optimistic Q -value and a randomized value function to perform temporally-extended exploration.
Revised: DRL by using a general-purpose UCB-bonus to provide an optimistic Q -value and a randomized value function to perform temporally-extended exploration.

Original: We propose an UCB-bonus that represents the disagreement of bootstrapped Q -functions (Osband et al., 2016) to measure the epistemic uncertainty of the unknown optimal value function.
Revised: We propose an UCB-bonus that represents the disagreement of bootstrapped Q -functions (Osband et al., 2016) to measure the epistemic uncertainty of the unknown optimal value function.

Original: Q -value added by UCB-bonus becomes an optimistic Q + function that is higher than Q for scarcely visited state-action pairs and remains close to Q for frequently visited ones.
Revised: Q -value added by UCB-bonus becomes an optimistic Q + function that is higher than Q for scarcely visited state-action pairs and remains close to Q for frequently visited ones.

Original: The optimistic Q + function encourages the agent to explore the states and actions with high UCB-bonuses, signifying scarcely visited areas or meaningful events in completing a task.
Revised: The optimistic Q + function encourages the agent to explore the states and actions with high UCB-bonuses, signifying scarcely visited areas or meaningful events in completing a task.

Original: We propose an extension of episodic backward update (Lee et al., 2019) to propagate the future uncertainty to the estimated action-value function consistently within an episode.
Revised: We propose an extension of episodic backward update (Lee et al., 2019) to propagate the future uncertainty to the estimated action-value function consistently within an episode.

Original: The backward update also enables OEB3 to perform highly sample-efﬁcient training.
Revised: The backward update also enables OEB3 to perform highly sample-efﬁcient training.

Original: Comparing to existing count-based and curiosity-driven exploration methods (Taiga et al., 2020),
Revised: Comparing to existing count-based and curiosity-driven exploration methods (Taiga et al., 2020),

Original: OEB3 has several beneﬁts.
Revised: OEB3 has several beneﬁts.

Original: (1) We utilize intrinsic rewards to produce optimistic value function and also take advantage of bootstrapped Q -learning to perform temporally-consistent exploration while existing methods do not combine these two principles.
Revised: (1) We utilize intrinsic rewards to produce optimistic value function and also take advantage of bootstrapped Q -learning to perform temporally-consistent exploration while existing methods do not combine these two principles.

Original: The UCB-bonus measures the disagreement of Q -values, which considers the long-term uncertainty in an episode rather than the single-step uncertainty used in most bonus-based methods (Pathak et al., 2019; Burda et al., 2019b).
Revised: The UCB-bonus measures the disagreement of Q -values, which considers the long-term uncertainty in an episode rather than the single-step uncertainty used in most bonus-based methods (Pathak et al., 2019; Burda et al., 2019b).

Original: Meanwhile, the UCB-bonus is computed without introducing additional modules compared to bootstrapped DQN.
Revised: Meanwhile, the UCB-bonus is computed without introducing additional modules compared to bootstrapped DQN.

Original: (3) We provide a theoretical analysis showing that OEB3 has theoretical consistency with optimistic
Revised: (3) We provide a theoretical analysis showing that OEB3 has theoretical consistency with optimistic

Original: LSVI in linear cases.
Revised: LSVI in linear cases.

Original: (4) Extensive evaluations show that OEB3 outperforms several state-
Revised: (4) Extensive evaluations show that OEB3 outperforms several state-of-the-art exploration approaches in MNIST maze and 49 Atari games

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we introduce bootstrapped DQN (Osband et al., 2016), which we utilize in OEB3 for temporarily-extended exploration.
Revised: In this section, we introduce bootstrapped DQN (Osband et al., 2016), which we utilize in OEB3 for temporarily-extended exploration.

Original: We further introduce optimistic LSVI (Jin et al., 2020), which we instantiate via DRL and propose OEB3.
Revised: We further introduce optimistic LSVI (Jin et al., 2020), which we instantiate via DRL and propose OEB3.

Original: 2. B OOTSTRAPPED DQN
Revised: 2. B OOTSTRAPPED DQN

Original: We consider an episodic MDP represented as ( S , A , T, P , r ) ,
Revised: We consider an episodic MDP represented as ( S , A , T, P , r ) ,

Original: where T ∈ Z
Revised: where T ∈ Z

Original: + is the episode length, S is the state space,
Revised: + is the episode length, S is the state space,

Original: A is the action space, r is the reward function, and P is the unknown dynamics.
Revised: A is the action space, r is the reward function, and P is the unknown dynamics.

Original: In each timestep, the agent obtains the current state s t , takes an action a t , interacts with the environment, receives a reward r t , and updates to the next state s t +1 .
Revised: In each timestep, the agent obtains the current state s t , takes an action a t , interacts with the environment, receives a reward r t , and updates to the next state s t +1 .

Original: The action-value function Q π ( s t , a t ) :
Revised: The action-value function Q π ( s t , a t ) :

Original: =
Revised: =

Original: E π (cid:2) (cid:80) T − 1 i =
Revised: E π (cid:2) (cid:80) T − 1 i =

Original: t γ i − t r i ] represents the expected cumulative reward starting from state s t , taking action a t , and thereafter following policy π ( a t
Revised: t γ i − t r i ] represents the expected cumulative reward starting from state s t , taking action a t , and thereafter following policy π ( a t

Original: | s t ) until the end of the episode.
Revised: | s t ) until the end of the episode.

Original: γ ∈ [0 , 1) is the discount factor.
Revised: γ ∈ [0 , 1) is the discount factor.

Original: The optimal value function Q ∗ = max π Q π , and the optimal action a ∗ = arg max a ∈A Q ∗ ( s, a ) .
Revised: The optimal value function Q ∗ = max π Q π , and the optimal action a ∗ = arg max a ∈A Q ∗ ( s, a ) .

Original: Deep Q -Network (DQN) uses a deep neural network with parameters θ to approximate the Q -function.
Revised: Deep Q -Network (DQN) uses a deep neural network with parameters θ to approximate the Q -function.

Original: The loss function takes the form of L ( θ ) =
Revised: The loss function takes the form of L ( θ ) =

Original: E [( y t − Q ( s t , a t ; θ )) 2 (cid:12)(cid:12) ( s t , a t , r t , s t +1 ) ∼ D ] ,
Revised: E [( y t − Q ( s t , a t ; θ )) 2 (cid:12)(cid:12) ( s t , a t , r t , s t +1 ) ∼ D ] ,

Original: where y t =
Revised: where y t =

Original: r t + γ max a (cid:48) Q ( s t +1 , a (cid:48) ; θ − ) is the target value, and θ − is the parameter of the target network.
Revised: r t + γ max a (cid:48) Q ( s t +1 , a (cid:48) ; θ − ) is the target value, and θ − is the parameter of the target network.

Original: The agent accumulates experiences ( s t , a t , r t , s t +1 ) in a replay buffer D and samples mini-batches in training.
Revised: The agent accumulates experiences ( s t , a t , r t , s t +1 ) in a replay buffer D and samples mini-batches in training.

Original: Bootstrapped DQN (Osband et al., 2016; 2018) is a non-parametric posterior sampling method, which maintains K estimations of Q -values to represent the posterior distribution of the randomized value function.
Revised: Bootstrapped DQN (Osband et al., 2016; 2018) is a non-parametric posterior sampling method, which maintains K estimations of Q -values to represent the posterior distribution of the randomized value function.

Original: Bootstrapped DQN uses a multi-head network that contains a shared convolution network and K heads.
Revised: Bootstrapped DQN uses a multi-head network that contains a shared convolution network and K heads.

Original: Each head deﬁnes a Q k -function.
Revised: Each head deﬁnes a Q k -function.

Original: Bootstrapped DQN diversiﬁes different Q k by using different random initialization and individual target networks.
Revised: Bootstrapped DQN diversiﬁes different Q k by using different random initialization and individual target networks.

Original: The loss for training Q k is
Revised: The loss for training Q k is

Original: [Equation]
Revised: [Equation]

Original: The k -th head Q k ( s, a ; θ k ) is trained with its own target network Q k ( s, a ; θ k − ) .
Revised: The k -th head Q k ( s, a ; θ k ) is trained with its own target network Q k ( s, a ; θ k − ) .

Original: If k -th head is sampled at the start of an episode when interacting with the environment, the agent will follow Q k to choose actions in the whole episode, which provides temporally-consistent exploration for DRL.
Revised: If k -th head is sampled at the start of an episode when interacting with the environment, the agent will follow Q k to choose actions in the whole episode, which provides temporally-consistent exploration for DRL.

Original: 1: Initialize: Λ t ← λ · I and w h ←2: for episode m = 0 to M − 1 do 3:
Revised: 1: Initialize: Λ t ← λ · I and w h ←2: for episode m = 0 to M − 1 do 3:

Original: Receive the initial state s4: for step t = 0 to T − 1 do 5:
Revised: Receive the initial state s4: for step t = 0 to T − 1 do 5:

Original: Take action a t = arg max a ∈A Q t ( s t , a ) and observe s t +1 .
Revised: Take action a t = arg max a ∈A Q t ( s t , a ) and observe s t +1 .

Original: 6: end for 7: for step t =
Revised: 6: end for 7: for step t =

Original: T − 1 to 0 do 8: Λ t ← (cid:80)
Revised: T − 1 to 0 do 8: Λ t ← (cid:80)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I 9: w
Revised: I 9: w

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: max a Q t +1 ( x τt +1 , a )] 10:
Revised: max a Q t +1 ( x τt +1 , a )] 10:

Original: Q t ( · , · ) = min { w (cid:62) t φ (
Revised: Q t ( · , · ) = min { w (cid:62) t φ (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: [ φ ( · , · ) (cid:62) Λ − 1 t φ ( · , · )] 1 / 2 , T } 11: end for 12: end for 2.
Revised: [ φ ( · , · ) (cid:62) Λ − 1 t φ ( · , · )] 1 / 2 , T } 11: end for 12: end for 2.

Original: O PTIMISTIC LSVI
Revised: O PTIMISTIC LSVI

Original: Optimistic LSVI (Jin et al., 2020) uses an optimistic Q -value with LSVI in linear MDP.
Revised: Optimistic LSVI (Jin et al., 2020) uses an optimistic Q -value with LSVI in linear MDP.

Original: We denote the feature map of the state-action pair as φ : S ×A → R d .
Revised: We denote the feature map of the state-action pair as φ : S ×A → R d .

Original: The transition kernel and reward function are assumed to be linear in φ .
Revised: The transition kernel and reward function are assumed to be linear in φ .

Original: Optimistic LSVI shown in Algorithm 1 consists of two parts.
Revised: Optimistic LSVI shown in Algorithm 1 consists of two parts.

Original: In the ﬁrst part (line 3-6), the agent executes the policy according to Q t for an episode.
Revised: In the ﬁrst part (line 3-6), the agent executes the policy according to Q t for an episode.

Original: In the second part (line 7-11), the parameter w t of Q -function is updated in closed-form by following the regularized least-squares problem
Revised: In the second part (line 7-11), the parameter w t of Q -function is updated in closed-form by following the regularized least-squares problem

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (cid:3)

Original: [Equation]
Revised: [Equation]

Original: where m is the number of episodes, and τ is the episodic index.
Revised: where m is the number of episodes, and τ is the episodic index.

Original: The least-squares problem has the explicit solution w t = Λ − 1 t (cid:80) mτ =

Revised: The least-squares problem has the explicit solution w t = Λ − 1 t (cid:80) mτ =

Original: φ ( x τt , a τt ) (cid:2) r t ( x τt , a τt ) + max a Q t +1 ( x τt +1 , a ) (cid:3) (line 9), where Λ t is the Gram matrix.
Revised: φ ( x τt , a τt ) (cid:2) r t ( x τt , a τt ) + max a Q t +1 ( x τt +1 , a ) (cid:3) (line 9), where Λ t is the Gram matrix.

Original: Then the value function is estimated by Q t ( s, a ) ≈ w (cid:62) t φ ( s, a ) .
Revised: Then the value function is estimated by Q t ( s, a ) ≈ w (cid:62) t φ ( s, a ) .

Original: Optimistic LSVI uses a bonus α [ φ ( s, a ) (cid:62) Λ − 1 t φ ( s, a )] 1 / 2 (line 10) to measure the uncertainty of state-action pairs.
Revised: Optimistic LSVI uses a bonus α [ φ ( s, a ) (cid:62) Λ − 1 t φ ( s, a )] 1 / 2 (line 10) to measure the uncertainty of state-action pairs.

Original: We can intuitively consider u :=
Revised: We can intuitively consider u :=

Original: ( φ (cid:62) Λ − 1 t φ ) − 1 as a pseudo count of state-action pair by projecting the total features that have observed ( Λ t ) to the direction of the curresponding feature φ .
Revised: ( φ (cid:62) Λ − 1 t φ ) − 1 as a pseudo count of state-action pair by projecting the total features that have observed ( Λ t ) to the direction of the curresponding feature φ .

Original: Thus, the bonus α / √ u represents the uncertainty along the direction of φ .
Revised: Thus, the bonus α / √ u represents the uncertainty along the direction of φ .

Original: By adding the bonus to Q -value, we obtain an optimistic value function Q + , which serves as an upper bound of
Revised: By adding the bonus to Q -value, we obtain an optimistic value function Q + , which serves as an upper bound of

Original: Q to encourage exploration.
Revised: Q to encourage exploration.

Original: The bonus in each step is propagated from the end of the episode by the backward update of Q -value (line 7-11), which follows the principle of dynamic programming.
Revised: The bonus in each step is propagated from the end of the episode by the backward update of Q -value (line 7-11), which follows the principle of dynamic programming.

Original: Theoretical analysis (Jin et al., 2020) shows that optimistic LSVI achieves a near-optimal worst-case regret of ˜ O ( √ d 3 T 3 L 3 ) with proper selections of α and λ , where L is the total number of steps.
Revised: Theoretical analysis (Jin et al., 2020) shows that optimistic LSVI achieves a near-optimal worst-case regret of ˜ O ( √ d 3 T 3 L 3 ) with proper selections of α and λ , where L is the total number of steps.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Optimistic LSVI (Jin et al., 2020) provides an atractive approach for efﬁcient exploration.
Revised: Optimistic LSVI (Jin et al., 2020) provides an atractive approach for efﬁcient exploration.

Original: Nevertheless, developing a practical exploration algorithm for DRL is challenging, since (i) the UCB-bonus utilized by Optimistic LSVI is tailored for linear MDPs, and (ii) optimistic LSVI utilizes backward update of Q -functions (line 7-11 in Alg. 1) to aggregate uncertainty.
Revised: Nevertheless, developing a practical exploration algorithm for DRL is challenging, since (i) the UCB-bonus utilized by Optimistic LSVI is tailored for linear MDPs, and (ii) optimistic LSVI utilizes backward update of Q -functions (line 7-11 in Alg. 1) to aggregate uncertainty.

Original: To this end, we propose the following approaches, which are the building blocks of OEB
Revised: To this end, we propose the following approaches, which are the building blocks of OEB

Original: • We propose a general-purpose UCB-bonus for optimistic exploration.
Revised: • We propose a general-purpose UCB-bonus for optimistic exploration.

Original: More speciﬁcally, we utilize bootstrapped DQN to construct a general-purpose UCB-bonus, which is theoretically consistent with optimistic LSVI for linear MDPs.
Revised: More speciﬁcally, we utilize bootstrapped DQN to construct a general-purpose UCB-bonus, which is theoretically consistent with optimistic LSVI for linear MDPs.

Original: We refer to Section 3.1 for the details.
Revised: We refer to Section 3.1 for the details.

Original: • We propose a sample-efﬁcient learning algorithm to integrate bootstrapped DQN and UCB-bonus into the backward update, which faithfully follows the principle of dynamic programming.
Revised: • We propose a sample-efﬁcient learning algorithm to integrate bootstrapped DQN and UCB-bonus into the backward update, which faithfully follows the principle of dynamic programming.

Original: More speciﬁcally, we extend Episodic Backward Update (EBU)
Revised: More speciﬁcally, we extend Episodic Backward Update (EBU)

Original: (Lee et al., 2019) from ordinary Qlearning to bootstrapped Q -learning, which we abbreviate by BEBU (Bootstrapped EBU).
Revised: (Lee et al., 2019) from ordinary Qlearning to bootstrapped Q -learning, which we abbreviate by BEBU (Bootstrapped EBU).

Original: BEBU allows sample-efﬁcient learning and fast propagation of the future uncertainty to the estimated Qvalue consistently.
Revised: BEBU allows sample-efﬁcient learning and fast propagation of the future uncertainty to the estimated Qvalue consistently.

Original: We further propose OEB3 by combining BEBU and the UCB-bonus obtained via bootstrapped Q -learning.
Revised: We further propose OEB3 by combining BEBU and the UCB-bonus obtained via bootstrapped Q -learning.

Original: We refer to Section 3.2 for the details.
Revised: We refer to Section 3.2 for the details.

Original: (a)   1   4 1.
Revised: (a)   1   4 1.

Original: (b)   1   4 1.
Revised: (b)   1   4 1.

Original: g g + (c)
Revised: g g + (c)

Original: 3. UCB-B ONUS IN OEB 
Revised: 3. UCB-B ONUS IN OEB

Original: Optimistic exploration uses an optimistic action-value function Q + to incentive exploration by adding a bonus term to the ordinary Q -value.
Revised: Optimistic exploration uses an optimistic action-value function Q + to incentive exploration by adding a bonus term to the ordinary Q -value.

Original: Thus Q + serves an upper bound of ordinary Q .
Revised: Thus Q + serves an upper bound of ordinary Q .

Original: The bonus term represents the epistemic uncertainty that results from lacking experiences of the corresponding states and actions.
Revised: The bonus term represents the epistemic uncertainty that results from lacking experiences of the corresponding states and actions.

Original: In this paper, we propose an UCB-bonus B ( s t , a t ) by measuring the disagreement of the bootstrapped Q -values { Q k ( s t , a t ) }
Revised: In this paper, we propose an UCB-bonus B ( s t , a t ) by measuring the disagreement of the bootstrapped Q -values { Q k ( s t , a t ) }

Original: Kk =1 of the state-action pair ( s t , a t ) in bootstrapped DQN, which takes the following form,
Revised: Kk =1 of the state-action pair ( s t , a t ) in bootstrapped DQN, which takes the following form,

Original: [Equation]
Revised: [Equation]

Original: where ¯ Q ( s t , a t ) is the mean of bootstrapped Q -values.
Revised: where ¯ Q ( s t , a t ) is the mean of bootstrapped Q -values.

Original: The following theorem establishes connection between the UCB-bonus deﬁned in Eq.
Revised: The following theorem establishes connection between the UCB-bonus deﬁned in Eq.

Original: (3) and the bonus in optimistic-LSVI.
Revised: (3) and the bonus in optimistic-LSVI.

Original: Theorem 1 (Informal Version of Theorem 2) .
Revised: Theorem 1 (Informal Version of Theorem 2) .

Original: In linear function approximation, the UCB-bonus
Revised: In linear function approximation, the UCB-bonus

Original: B ( s t , a t ) in OEB3 is equivalent to the bonus-term [ φ (cid:62) t Λ − 1 t φ t ] 1 / 2 in optimistic-LSVI, where Λ t ← (cid:80) mτ =

Revised: B ( s t , a t ) in OEB3 is equivalent to the bonus-term [ φ (cid:62) t Λ − 1 t φ t ] 1 / 2 in optimistic-LSVI, where Λ t ← (cid:80) mτ =

Original: φ ( x τt , a τt ) φ ( x τt , a τt ) (cid:62) + λ · I , and m is the current episode.
Revised: φ ( x τt , a τt ) φ ( x τt , a τt ) (cid:62) + λ · I , and m is the current episode.

Original: We refer to Appendix A for the proof and a detailed discussion.
Revised: We refer to Appendix A for the proof and a detailed discussion.

Original: Implementing the UCB-bonus
Revised: Implementing the UCB-bonus

Original: B ( s t , a t ) deﬁned in Eq.
Revised: B ( s t , a t ) deﬁned in Eq.

Original: (3) in DRL is desirable for exploration for the following reasons.
Revised: (3) in DRL is desirable for exploration for the following reasons.

Original: • Bootstrapped DQN is a non-parametric posterior sampling method, which can be implemented via deep neural networks (Osband et al., 2019).
Revised: • Bootstrapped DQN is a non-parametric posterior sampling method, which can be implemented via deep neural networks (Osband et al., 2019).

Original: • The UCB-bonus B ( s t , a t ) deﬁned in Eq.
Revised: • The UCB-bonus B ( s t , a t ) deﬁned in Eq.

Original: (3) quantiﬁes the epistemic uncertainty of a speciﬁc state-action pair adequately.
Revised: (3) quantiﬁes the epistemic uncertainty of a speciﬁc state-action pair adequately.

Original: More speciﬁcally, due to the non-convexity nature of optimizing neural network and independent initialization, if ( s t , a t ) is scarcely visited, the UCB-bonus B ( s t , a t ) obtained via bootstrapped DQN is high.
Revised: More speciﬁcally, due to the non-convexity nature of optimizing neural network and independent initialization, if ( s t , a t ) is scarcely visited, the UCB-bonus B ( s t , a t ) obtained via bootstrapped DQN is high.

Original: Moreover, the UCB-bonus converges to zero asymptotically as the sample size increases to inﬁnity.
Revised: Moreover, the UCB-bonus converges to zero asymptotically as the sample size increases to inﬁnity.

Original: • The UCB-bonus is computed in a batch when performing experience replay, which is more efﬁcient than other optimistic methods that change the action-selection scheme in each timestep (Chen et al., 2017; Nikolov et al., 2019).
Revised: • The UCB-bonus is computed in a batch when performing experience replay, which is more efﬁcient than other optimistic methods that change the action-selection scheme in each timestep (Chen et al., 2017; Nikolov et al., 2019).

Original: The optimistic Q + is obtained by summing up the UCB-bonus and the estimated Q -function, which takes the following form,
Revised: The optimistic Q + is obtained by summing up the UCB-bonus and the estimated Q -function, which takes the following form,

Original: [Equation]
Revised: [Equation]

Original: We use a simple regression task with neural networks to illustrate the proposed UCB-bonus, as shown in Figure 1.
Revised: We use a simple regression task with neural networks to illustrate the proposed UCB-bonus, as shown in Figure 1.

Original: We use 20 neural networks with the same network architecture to solve the same regression problem.
Revised: We use 20 neural networks with the same network architecture to solve the same regression problem.

Original: Each network contains three residual blocks with two fully-connected layers each.
Revised: Each network contains three residual blocks with two fully-connected layers each.

Original: According to (Osband et al., 2016), the difference between the outcome of ﬁtting the neural networks is a result of different initialization.
Revised: According to (Osband et al., 2016), the difference between the outcome of ﬁtting the neural networks is a result of different initialization.

Original: For a single input x , the networks yield different estimations { g i ( x ) } 20 i =1 as shown in Figure 1.
Revised: For a single input x , the networks yield different estimations { g i ( x ) } 20 i =1 as shown in Figure 1.

Original: It follows from Figure 1(a) that the estimations { g i ( x ) } 20 i =1 behave similar in the region with rich observations, resulting in small disagreement of the estimations, but vary in the region with scarce observations, resulting in large disagreement of the estimations.
Revised: It follows from Figure 1(a) that the estimations { g i ( x ) } 20 i =1 behave similar in the region with rich observations, resulting in small disagreement of the estimations, but vary in the region with scarce observations, resulting in large disagreement of the estimations.

Original: In Figure 1(b), we illustrate the conﬁdence bound of the regression results ¯ g ( x ) ± ˜ σ ( g i ( x )) and
Revised: In Figure 1(b), we illustrate the conﬁdence bound of the regression results ¯ g ( x ) ± ˜ σ ( g i ( x )) and

Original: ¯ g ( x ) ± 2˜ σ ( g i ( x )) , where ¯ g ( x ) and ˜ σ ( g i ( x )) are the mean and standard deviation of the estimations.
Revised: ¯ g ( x ) ± 2˜ σ ( g i ( x )) , where ¯ g ( x ) and ˜ σ ( g i ( x )) are the mean and standard deviation of the estimations.

Original: The standard deviation ˜ σ ( g i ( x )) captures the epistemic uncertainty of regression results.
Revised: The standard deviation ˜ σ ( g i ( x )) captures the epistemic uncertainty of regression results.

Original: Figure 1(c) shows the optimistic estimation as g + ( x ) =
Revised: Figure 1(c) shows the optimistic estimation as g + ( x ) =

Original: ¯ g ( x ) + ˜ σ ( g i ( x )) by adding the uncertainty measured by standard deviation.
Revised: ¯ g ( x ) + ˜ σ ( g i ( x )) by adding the uncertainty measured by standard deviation.

Original: It can be seen that the optimistic estimation g + is close to ¯ g in the region with rich observations, and higher than ¯ g in the region with scarce observations.
Revised: It can be seen that the optimistic estimation g + is close to ¯ g in the region with rich observations, and higher than ¯ g in the region with scarce observations.

Original: In DRL, the bootstrapped Q -functions { Q k ( s t , a t ) } Kk =1 obtained by ﬁtting the target Q -function perform similarly as { g i ( x ) } 20 i =1 in the regression task.
Revised: In DRL, the bootstrapped Q -functions { Q k ( s t , a t ) } Kk =1 obtained by ﬁtting the target Q -function perform similarly as { g i ( x ) } 20 i =1 in the regression task.

Original: A higher UCB-bonus B ( s t , a t ) :=
Revised: A higher UCB-bonus B ( s t , a t ) :=

Original: ˜ σ ( Q k ( s t , a t )) indicates a higher epistemic uncertainty of the action-value function with ( s t , a t ) .
Revised: ˜ σ ( Q k ( s t , a t )) indicates a higher epistemic uncertainty of the action-value function with ( s t , a t ) .

Original: The estimated
Revised: The estimated

Original: Q -function augmented with the UCB-bonus yields Q + , which produces optimistic estimation for novel state-action pairs and behaves similar to the target Q -function in areas that are well explored by the agent.
Revised: Q -function augmented with the UCB-bonus yields Q + , which produces optimistic estimation for novel state-action pairs and behaves similar to the target Q -function in areas that are well explored by the agent.

Original: Hence, the optimistic extimation Q + incentives the agent to explore the potentially informative states and actions efﬁciently.
Revised: Hence, the optimistic extimation Q + incentives the agent to explore the potentially informative states and actions efﬁciently.

Original: 3. B OOTSTRAPPED E PISODIC B ACKWARD U PDATE (BEBU)
Revised: 3. B OOTSTRAPPED E PISODIC B ACKWARD U PDATE (BEBU)

Original: There are two major reasons for updating the action-value function through BEBU in OEB3.
Revised: There are two major reasons for updating the action-value function through BEBU in OEB3.

Original: • The backward propagation utilizes a complete trajectory from the replay buffer.
Revised: • The backward propagation utilizes a complete trajectory from the replay buffer.

Original: Such an approach allows OEB3 to infer the long-term effect of decision making from the replay buffer.
Revised: Such an approach allows OEB3 to infer the long-term effect of decision making from the replay buffer.

Original: In contrast, DQN and Bootstrapped DQN utilize the replay buffer by random sampling one-step transitions, which loses the information containing such a long-term effect (Lee et al., 2019).
Revised: In contrast, DQN and Bootstrapped DQN utilize the replay buffer by random sampling one-step transitions, which loses the information containing such a long-term effect (Lee et al., 2019).

Original: • The backward propagation is required to propagate the future uncertainty to the estimated actionvalue function consistently via UCB-bonus within an episode.
Revised: • The backward propagation is required to propagate the future uncertainty to the estimated actionvalue function consistently via UCB-bonus within an episode.

Original: For instance, let t 2 >
Revised: For instance, let t 2 >

Original: t 1 be the indices of two steps in an episode.
Revised: t 1 be the indices of two steps in an episode.

Original: If the update of Q t 2 occurs after the update of Q t 1 within an episode, then it can occur that the uncertainty propagated into Q t 1 is not consistent with the uncertainty that Q t 2 contains.
Revised: If the update of Q t 2 occurs after the update of Q t 1 within an episode, then it can occur that the uncertainty propagated into Q t 1 is not consistent with the uncertainty that Q t 2 contains.

Original: To integrate the UCB-bonus into bootstrapped Q -learning, we propose a novel Q -target in update by adding the bonus term in both the immediate reward and the next- Q value.
Revised: To integrate the UCB-bonus into bootstrapped Q -learning, we propose a novel Q -target in update by adding the bonus term in both the immediate reward and the next- Q value.

Original: The proposed Q -target needs to be suitable for BEBU in training.
Revised: The proposed Q -target needs to be suitable for BEBU in training.

Original: Formally, the Q -target for updating Q k is deﬁned as
Revised: Formally, the Q -target for updating Q k is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where a (cid:48) =
Revised: where a (cid:48) =

Original: arg max a Q k ( s t +1 , a ; θ k − ) .
Revised: arg max a Q k ( s t +1 , a ; θ k − ) .

Original: The choice of a (cid:48) is determined by the target Q -value without considering the bonus.
Revised: The choice of a (cid:48) is determined by the target Q -value without considering the bonus.

Original: The immediate reward is added by B ( s t , a t ; θ ) with a factor α 1 , where the bonus B is computed by bootstrapped Q -network with parameter θ .
Revised: The immediate reward is added by B ( s t , a t ; θ ) with a factor α 1 , where the bonus B is computed by bootstrapped Q -network with parameter θ .

Original: The next- Q value is added by 1 a (cid:48) (cid:54) =
Revised: The next- Q value is added by 1 a (cid:48) (cid:54) =

Original: a t +1 ˜ B k
Revised: a t +1 ˜ B k

Original: ( s t +1 , a (cid:48) ; θ − ) with factor α 2 , where the bonus ˜ B k is computed by the target network with parameter θ − .
Revised: ( s t +1 , a (cid:48) ; θ − ) with factor α 2 , where the bonus ˜ B k is computed by the target network with parameter θ − .

Original: We assign different bonus ˜ B k of next- Q value to different heads, since the choice of action a (cid:48) are different among the heads.
Revised: We assign different bonus ˜ B k of next- Q value to different heads, since the choice of action a (cid:48) are different among the heads.

Original: Meanwhile, we assign the same bonus B of immediate reward for all the heads.
Revised: Meanwhile, we assign the same bonus B of immediate reward for all the heads.

Original: We introduce the indicator function 1 a (cid:48) (cid:54) =
Revised: We introduce the indicator function 1 a (cid:48) (cid:54) =

Original: a t +1 to suit the backward update of
Revised: a t +1 to suit the backward update of

Original: Q -values.
Revised: Q -values.

Original: More speciﬁcally, in the t -th step, the action-value function Q k is updated optimistically at the state-action pair ( s t +1 , a t +1 ) due to the backward update.
Revised: More speciﬁcally, in the t -th step, the action-value function Q k is updated optimistically at the state-action pair ( s t +1 , a t +1 ) due to the backward update.

Original: Thus, we ignore the bonus of next- Q value in the update of Q k when a (cid:48) is equal to a t +1 .
Revised: Thus, we ignore the bonus of next- Q value in the update of Q k when a (cid:48) is equal to a t +1 .

Original: We use an example to explain the process of backward update.
Revised: We use an example to explain the process of backward update.

Original: We store and sample the experiences in episodes, and perform update in episodes rather than uniformly sampled transitions.
Revised: We store and sample the experiences in episodes, and perform update in episodes rather than uniformly sampled transitions.

Original: We consider an episode containing three time steps, ( s 0 , a 0 ) → ( s 1 , a 1 )
Revised: We consider an episode containing three time steps, ( s 0 , a 0 ) → ( s 1 , a 1 )

Original: → ( s 2 , a 2 ) .
Revised: → ( s 2 , a 2 ) .

Original: We thus update the Q -value in the head k in the backward manner, namely Q ( s 2 , a 2 ) → Q ( s 1 , a 1 ) → Q ( s 0 , a 0 ) from the end of the episode.
Revised: We thus update the Q -value in the head k in the backward manner, namely Q ( s 2 , a 2 ) → Q ( s 1 , a 1 ) → Q ( s 0 , a 0 ) from the end of the episode.

Original: We describe the process as follows.
Revised: We describe the process as follows.

Original: First, we update Q ( s 2 , a 2 ) ← r ( s 2 , a 2 ) + α 1 B ( s 2 , a 2 ) .
Revised: First, we update Q ( s 2 , a 2 ) ← r ( s 2 , a 2 ) + α 1 B ( s 2 , a 2 ) .

Original: Note that in the last time step, we do not need to consider the next- Q value.
Revised: Note that in the last time step, we do not need to consider the next- Q value.

Original: Then, Q ( s 1 , a 1 ) ← [ r ( s 1 , a 1 ) + α 1 B ( s 1 , a 1 )]
Revised: Then, Q ( s 1 , a 1 ) ← [ r ( s 1 , a 1 ) + α 1 B ( s 1 , a 1 )]

Original: + [ Q ( s 2 , a (cid:48) )
Revised: + [ Q ( s 2 , a (cid:48) )

Original: + α 2 1 a
Revised: + α 2 1 a

Original: (cid:48)
Revised: (cid:48)

Original: (cid:54) = a 2 ˜ B ( s 2 , a (cid:48) )] by following Eq.
Revised: (cid:54) = a 2 ˜ B ( s 2 , a (cid:48) )] by following Eq.

Original: (5), where a (cid:48) =
Revised: (5), where a (cid:48) =

Original: arg max a Q ( s 2 , a ) .
Revised: arg max a Q ( s 2 , a ) .

Original: Since Q ( s 2 , a 2 ) is updated optimistically in step 1, we ignore the bonus-term ˜ B in next- Q value when a (cid:48)
Revised: Since Q ( s 2 , a 2 ) is updated optimistically in step 1, we ignore the bonus-term ˜ B in next- Q value when a (cid:48)

Original: = a 2 .
Revised: = a 2 .

Original: The UCB-bonus is augmented in update by adding B and ˜ B to the immediate reward and next- Q value, respectively.
Revised: The UCB-bonus is augmented in update by adding B and ˜ B to the immediate reward and next- Q value, respectively.

Original: The update of Q ( s 0 , a 0 ) follows the same principle.
Revised: The update of Q ( s 0 , a 0 ) follows the same principle.

Original: The optimistic Q -value is Q ( s 0 , a 0 ) ←
Revised: The optimistic Q -value is Q ( s 0 , a 0 ) ←

Original: [Equation]
Revised: [Equation]

Original: In practice, the episodic update typically leads to instability in DRL due to the strong correlation in consecutive transitions.
Revised: In practice, the episodic update typically leads to instability in DRL due to the strong correlation in consecutive transitions.

Original: Hence, we propose a diffusion factor β ∈ [0 , 1] in BEBU to prevent instability as in Lee et al.
Revised: Hence, we propose a diffusion factor β ∈ [0 , 1] in BEBU to prevent instability as in Lee et al.

Original: The Q -value is therefore computed as the weighted sum of the current value and the back-propagated estimation scaled with factor β .
Revised: The Q -value is therefore computed as the weighted sum of the current value and the back-propagated estimation scaled with factor β .

Original: We consider an episodic experience that contains
Revised: We consider an episodic experience that contains

Original: T transitions, denoted by E = { S , A , R , S (cid:48) } , where S = { s 0 , . . .
Revised: T transitions, denoted by E = { S , A , R , S (cid:48) } , where S = { s 0 , . . .

Original: , s T − 1 } , A
Revised: , s T − 1 } , A

Original: = { a 0 , . . .
Revised: = { a 0 , . . .

Original: , a T − 1 } ,
Revised: , a T − 1 } ,

Original: R = { r 0 , . . .
Revised: R = { r 0 , . . .

Original: , r T − 1 } and S (cid:48)
Revised: , r T − 1 } and S (cid:48)

Original: = { s 1 , . . .
Revised: = { s 1 , . . .

Original: , s T } .
Revised: , s T } .

Original: We initialize a Q -table ˜Q ∈ R K ×|A|× T by Q ( · ; θ − ) to store the next Q -values of all the next states S (cid:48) and valid actions for K heads.
Revised: We initialize a Q -table ˜Q ∈ R K ×|A|× T by Q ( · ; θ − ) to store the next Q -values of all the next states S (cid:48) and valid actions for K heads.

Original: We initialize y ∈ R K × T to store the Q -target for K heads and T steps.
Revised: We initialize y ∈ R K × T to store the Q -target for K heads and T steps.

Original: We use bootstrapped Q -network with parameters θ to compute the bonus B =
Revised: We use bootstrapped Q -network with parameters θ to compute the bonus B =

Original: [ B ( s 0 , a 0 ) , . . .
Revised: [ B ( s 0 , a 0 ) , . . .

Original: , B ( s T − 1 , a T − 1 )] for immediate reward, and use the target network with parameters θ − to compute bonus ˜B k =
Revised: , B ( s T − 1 , a T − 1 )] for immediate reward, and use the target network with parameters θ − to compute bonus ˜B k =

Original: [ ˜ B k ( s 1 , a (cid:48) 1 ) , . . . ,
Revised: [ ˜ B k ( s 1 , a (cid:48) 1 ) , . . . ,

Original: ˜ B k ( s T , a (cid:48) T )] for next- Q value in each head, where a (cid:48) t = arg max a Q k ( s t , a ; θ k − ) .
Revised: ˜ B k ( s T , a (cid:48) T )] for next- Q value in each head, where a (cid:48) t = arg max a Q k ( s t , a ; θ k − ) .

Original: The bonus vector B ∈ R T is the same for all Q -heads, while ˜B ∈ R K × T contains different values for different heads because the choices of a (cid:48) t are different.
Revised: The bonus vector B ∈ R T is the same for all Q -heads, while ˜B ∈ R K × T contains different values for different heads because the choices of a (cid:48) t are different.

Original: In the training of head k , we initialize the Q -target in the last step by y [ k, T − 1]
Revised: In the training of head k , we initialize the Q -target in the last step by y [ k, T − 1]

Original: = R T − 1 + α 1 B T − 1 .
Revised: = R T − 1 + α 1 B T − 1 .

Original: We then perform a recursive backward update to get all Q -target values.
Revised: We then perform a recursive backward update to get all Q -target values.

Original: The elements of ˜Q [ k, a t +1 , t ] for step t in head k is updated by using its corresponding Q -target y [ k, t + 1] with the diffusion factor as follows,
Revised: The elements of ˜Q [ k, a t +1 , t ] for step t in head k is updated by using its corresponding Q -target y [ k, t + 1] with the diffusion factor as follows,

Original: [Equation]
Revised: [Equation]

Original: Then, we update y [ k, t ] in the previous time step based on the newly updated t -th column of ˜Q [ k ] as follows,
Revised: Then, we update y [ k, t ] in the previous time step based on the newly updated t -th column of ˜Q [ k ] as follows,

Original: [Equation]
Revised: [Equation]

Original: where a (cid:48) =
Revised: where a (cid:48) =

Original: arg max a ˜Q [ k, a, t ] .
Revised: arg max a ˜Q [ k, a, t ] .

Original: In practice, we construct a matrix ˜A =
Revised: In practice, we construct a matrix ˜A =

Original: arg max a ˜Q [ · , a, · ] ∈ R K × T to gather all the actions a (cid:48) that correspond to the next- Q .
Revised: arg max a ˜Q [ · , a, · ] ∈ R K × T to gather all the actions a (cid:48) that correspond to the next- Q .

Original: We then construct a mask matrix M ∈
Revised: We then construct a mask matrix M ∈

Original: R K × T to store the information whether ˜A is identical to the executed action in the corresponding timestep or not.
Revised: R K × T to store the information whether ˜A is identical to the executed action in the corresponding timestep or not.

Original: The bonus of next- Q is the element-wise product of M and ˜B with factor α 2 .
Revised: The bonus of next- Q is the element-wise product of M and ˜B with factor α 2 .

Original: After the backward update, we compute the Q -value of ( S , A ) as Q = Q ( S , A ; θ ) ∈ R K × T .
Revised: After the backward update, we compute the Q -value of ( S , A ) as Q = Q ( S , A ; θ ) ∈ R K × T .

Original: The loss function takes the form of L ( θ )
Revised: The loss function takes the form of L ( θ )

Original: =
Revised: =

Original: E (cid:2) ( y − Q ) 2 | ( s t , a t , r t , s t +1 )
Revised: E (cid:2) ( y − Q ) 2 | ( s t , a t , r t , s t +1 )

Original: ∈ E, E ∼ D (cid:3) , where the episodic experience E is sampled from replay buffer to perform gradient descent.
Revised: ∈ E, E ∼ D (cid:3) , where the episodic experience E is sampled from replay buffer to perform gradient descent.

Original: The gradient of all heads can be computed simultaneously via BEBU.
Revised: The gradient of all heads can be computed simultaneously via BEBU.

Original: We refer the full algorithm of OEB3 to Appendix B.
Revised: We refer the full algorithm of OEB3 to Appendix B.

Revised: The theory in optimistic-LSVI requires a strong linear assumption in the transition dynamics.

Revised: To make it works empirically, we make several adjustments in the implementation details.

Revised: First, in each training step of optimistic-LSVI, all historical samples are utilized to update the weight of

Original: We highlight that UCB-bonuses are propagated from the end of  the episode along with the  update of Q -functions through BEBU, which faithfully follows the backward update of optimistic LSVI in 
Revised: Q -function and calculate the conﬁdence bonus. While in OEB3, we use samples from a batch of episodic trajectories from the replay buffer in each training step. Such a difference in implementation is imposed to achieve computational efﬁciency. Second, in OEB3, the target-network has a relatively low update frequency, whereas, in Optimistic LSVI, the target Q function is updated in each iteration.

Revised: Such implementation techniques are commonly used in most existing (off-policy) DRL algorithms.

Revised: We use BEBU to propagate the future uncertainty in an episode, which is an extension of EBU (Lee et al., 2019).

Original: Algorithm 1 to guarantee the theoretical consistency.
Revised: Compared to EBU, BEBU requires extra tensors to store the UCB-bonus for immediate reward and next- Q value, which are integrated to propagate uncertainties. Meanwhile, integrating uncertainty into BEBU needs special design by using the mask. The previous works do not propagate the future uncertainty and, therefore, does not capture the core beneﬁt of utilizing UCB-bonus for the exploration of MDPs. We highlight that OEB3 propagates future uncertainty in a time-consistent manner based on BEBU, which exploits the theoretical analysis established by Jin et al.

Original: The backward update signiﬁcantly improves the sample-efﬁciency by allowing bonuses and delayed rewards to propagatedirectly through transitions of the sampled episode, which we demonstrate in the sequel.
Revised: The backward update signiﬁcantly improves the sample-efﬁciency by allowing bonuses and delayed rewards to propagate through transitions of the sampled episode, which we demonstrate in the sequel.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: One practical principle for exploration in DRL is maintaining epistemic uncertainty of action-value functions and learning to reduce the uncertainty.
Revised: One practical principle for exploration in DRL is maintaining epistemic uncertainty of action-value functions and learning to reduce the uncertainty.

Original: Epistemic uncertainty appears due to the missing knowledge of the environment, and disappears with the progress of exploration.
Revised: Epistemic uncertainty appears due to the missing knowledge of the environment, and disappears with the progress of exploration.

Original: Bootstrapped
Revised: Bootstrapped

Original: DQN (Osband et al., 2016; 2018) samples Q -values from the randomized value function to encourage exploration through Thompson sampling.
Revised: DQN (Osband et al., 2016; 2018) samples Q -values from the randomized value function to encourage exploration through Thompson sampling.

Original: Chen et al.
Revised: Chen et al.

Revised: (2017) proposes to use the standard-deviation of bootstrapped Q-functions to measure the uncertainty.

Revised: While the uncertainty measurement is similar to that of OEB3, our method is different from Chen et al.

Original: (2017) shows that optimistic action-selection has better performance than randomly sampled Q -heads.
Revised: (2017) in the following aspects. First, our approach propagates the uncertainty through time by the backward update, which allows for deep exploration for the MDPs. Second, Chen et al. (2017) does not use the bonus in the update of Q -functions.

Original: Information-Directed Sampling (IDS) (Nikolov et al., 2019) is based on bootstrapped DQN, which chooses actions by balancing the instantaneous regret and information gain.
Revised: The bonus is computed when taking the actions. Third, we establish a theoretical connection between the UCB-bonus and the bonus in optimistic-LSVI. SUNRISE (Lee et al., 2020) extends Chen et al.

Revised: (2017) to continuous control through conﬁdence reward and weighted Bellman backup.

Revised: Information-Directed Sampling (IDS) (Nikolov et al., 2019) is based on bootstrapped DQN, and chooses actions by balancing the instantaneous regret and information gain.

Original: OAC (Ciosek et al., 2019) uses two Q -networks to get lower and upper bounds of Q -value to perform exploration in continuous control tasks.
Revised: OAC (Ciosek et al., 2019) uses two Q -networks to get lower and upper bounds of Q -value to perform exploration in continuous control tasks.

Original: These methods seek to estimate the epistemic uncertaintywhen performing action selection and choose the optimistic actions.
Revised: These methods seek to estimate the epistemic uncertainty and choose the optimistic actions.

Original: In contrast, we use the uncertainty of value function to construct intrinsic rewards and perform backward update, which propagates future uncertainty to the estimated Q -value.
Revised: In contrast, we use the uncertainty of value function to construct intrinsic rewards and perform backward update, which propagates future uncertainty to the estimated Q -value.

Original: Besides, our method is more time efﬁcient, since we compute the UCB-bonus in  a batch rather than in each time step.
Revised: Uncertainty Bellman Equation (UBE) and Exploration (O’Donoghue et al., 2018) proposes an upper bound on the variance of the posterior of the Q -values, which are further utilized for optimism in exploration. Bayesian-DQN (Azizzadenesheli et al., 2018) modiﬁes the linear regression of the last layer in Q -network and uses Bayesian Linear Regression (BLR) instead, which estimates an posterior of the Q -function. These methods use parametric distributions to describe the posterior while OEB uses a non-parametric method to estimate the conﬁdence bonus. UBE and BLR require inverting a large matrix in training.

Original: Previous methods also utilize the epistemic uncertainty of dynamics through Bayesian posterior (Ratzlaff et al., 2020) and ensembles (Pathak et al., 2019).
Revised: Previous methods also utilize the epistemic uncertainty of dynamics through Bayesian posterior (Ratzlaff et al., 2020) and ensembles (Pathak et al., 2019).

Original: Nevertheless, these works consider single-step uncertainty in transitions, while our method considers the long-term uncertainty in an episode.
Revised: Nevertheless, these works consider single-step uncertainty, while we consider the long-term uncertainty in an episode.

Original: To measure the novelty of states for constructing count-based intrinsic rewards, previous methods use density model (Bellemare et al., 2016;
Revised: To measure the novelty of states for constructing count-based intrinsic rewards, previous methods use density model (Bellemare et al., 2016;

Original: Ostrovski et al., 2017), static hashing (Tang et al., 2017;
Revised: Ostrovski et al., 2017), static hashing (Tang et al., 2017;

Original: Choi et al., 2019; Rashid et al., 2020), episodic curiosity (Savinov et al., 2019; Badia et al., 2020), representation changes (Raileanu & Rockt¨aschel, 2020), curiosity-bottleneck (Kim et al., 2019b), information gain (Houthooft et al., 2016) and random network distillation (RND) (Burda et al., 2019b).
Revised: Choi et al., 2019; Rashid et al., 2020), episodic curiosity (Savinov et al., 2019; Badia et al., 2020), representation changes (Raileanu & Rockt ¨ aschel, 2020), curiosity-bottleneck (Kim et al., 2019b), information gain (Houthooft et al., 2016) and RND (Burda et al., 2019b).

Original: The curiosity-driven exploration based on prediction-error of environment models such as 
Revised: The curiosity-driven exploration based on prediction-error of environment models such as ICM (Pathak et al., 2017;

Original: ICM (Pathak et al., 2017; Burda et al., 2019a), EMI (Kim et al., 2019a), self-aware agent (Haber et al., 2018), and learning progress (Kim et al., 2020) enable the agents to explorethe environment in a self-supervised manner.
Revised: Burda et al., 2019a), EMI (Kim et al., 2019a), variational dynamics (Bai et al., 2020) and learning progress (Kim et al., 2020) enable the agents to explore in a self-supervised manner.

Original: According to a recent empirical research (Taiga et al., 2020), althoughthe existing bonus-based methods show promising results in hard exploration tasks likeMontezuma’s
Revised: According Taiga et al. (2020), although bonus-based methods show promising results in hard exploration tasks like

Original:  Revenge, these methods do not perform well on other Atari games.
Revised: Montezuma’s Revenge, they do not perform well on other Atari games.

Original: Meanwhile, NoisyNet (Fortunato et al., 2018) performs signiﬁcantly better than popular bonus-based methods evaluated by the whole Atari suit.
Revised: Meanwhile, NoisyNet (Fortunato et al., 2018) performs signiﬁcantly better than popular bonus-based methods evaluated by the whole Atari suit.

Original: Taiga et al.
Revised: Taiga et al.

Original: (2020) suggests that the real pace of progress in exploration may have been obfuscated by good results on several hard exploration games.
Revised: (2020) suggests that the real pace of progress in exploration may have been obfuscated by good results on several hard exploration games.

Original: We follow this principle and evaluate OEB3 on the whole Atari suit with 49 games.
Revised: We follow this principle and evaluate OEB3 on the whole Atari suit with 49 games.

Revised: Beyond model-free methods, model-based RL also uses optimism for planning and exploration (Nix & Weigend, 1994).

Original: E XPERIMENTAL R ESULTS 5. E NVIRONMENTAL S ETUP
Revised: Model-assisted RL (Kalweit & Boedecker, 2017) uses ensembles to make use of artiﬁcial data only in cases of high uncertainty. Buckman et al. (2018) uses ensemble dynamics and Q-functions to use model rollouts when they do not cause large errors.

Revised: Planning to explore (Sekar et al., 2020) seeks out future uncertainty by integrating uncertainty to Dreamer (Hafner et al., 2020).

Revised: Ready Policy One (Ball et al., 2020) optimizes policies for both reward and model uncertainty reduction.

Revised: Noise-Augmented RL (Pacchiano et al., 2020) uses statistical bootstrap to generalize the optimistic posterior sampling (Agrawal & Jia, 2017) to DRL.

Revised: Hallucinated UCRL (Curi et al., 2020) reduces optimistic exploration to exploitation by enlarging the control space.

Revised: The model-based RL needs to estimate the posterior of dynamics, while OEB3 relies on the posterior of Q -functions.

Revised:  E XPERIMENTAL R ESULTS 5. E NVIRONMENTAL S ETUP

Original: We evaluate the algorithms in high-dimensional image-based tasks, including Mnist Maze and
Revised: We evaluate the algorithms in high-dimensional image-based tasks, including MNIST Maze and

Original: Atari games.
Revised: Atari games.

Original: We refer Appendix C for the experiments on Mnist Maze, and discuss the experiments on Atari games in the sequel.
Revised: We refer Appendix C for the experiments on MNIST Maze, and discuss the experiments on Atari games in the sequel.

Original: Comparing OEB3 with bootstrapped DQN-based baselines directly incur performance bias, since OEB3 uses backward update for training, which is different from bootstrapped DQN.
Revised: Comparing OEB3 with bootstrapped DQN-based baselines directly incur performance bias, since OEB3 uses backward update for training, which is different from bootstrapped DQN.

Original: To this end, we re-implement all bootstrapped DQN-based baselines with BEBUbased update.
Revised: To this end, we re-implement all bootstrapped DQN-based baselines with BEBUbased update.

Original: In our experiments, we compare the following methods: (1) OEB3 .
Revised: In our experiments, we compare the following methods: (1) OEB3 .

Original: The proposed optimistic exploration method with UCB-bonus and backward update in this work.
Revised: The proposed optimistic exploration method with UCB-bonus and backward update in this work.

Original: (2) BEBU .
Revised: (2) BEBU .

Original: The re-implementation of bootstrapped DQN (Osband et al., 2016) with BEBU-based training.
Revised: The re-implementation of bootstrapped DQN (Osband et al., 2016) with BEBU-based training.

Original: (3) BEBUUCB .
Revised: (3) BEBU-UCB .

Original: BEBU with optimistic action-selection according to the upper bound of Q (Chen et al., 2017).
Revised: BEBU with optimistic action-selection according to the upper bound of Q (Chen et al., 2017).

Original: (4) BEBU-IDS .
Revised: (4) BEBU-IDS .

Original: Integrating homoscedastic IDS (Nikolov et al., 2019) into BEBU.
Revised: Integrating homoscedastic IDS (Nikolov et al., 2019) into BEBU without distributional RL.

Revised: We refer to Appendix B for the algorithmic comparison between all methods.

Original: According to EBU (Lee et al., 2019), the backward update is signiﬁcantly more sample-efﬁcient than ordinary Q -learning by using 20M training frames to achieve the mean human-normalized score of DQN with 200M training frames.
Revised: According to EBU (Lee et al., 2019), the backward update is signiﬁcantly more sample-efﬁcient than ordinary Q -learning by using 20M training frames to achieve the mean human-normalized score of DQN with 200M training frames.

Original: We follow this setting by training all BEBU-based methods with 20M frames.
Revised: We follow this setting by training all BEBU-based methods with 20M frames.

Original: In our experiments, 20M frames in OEB3 is sufﬁcient to produce strong empirical results and achieve competitive results with several baselines in 200M frames.
Revised: In our experiments, 20M frames in OEB3 is sufﬁcient to produce strong empirical results and achieve competitive results with several baselines in 200M frames.

Original: After training, an ensemble policy by a majority vote of Q -heads is used for 30 no-op  evaluation (Osband et al., 2016).
Revised: After training, an ensemble policy by a majority vote of Q -heads is used for 30 no-op evaluation. The majority-vote combines all the heads into a single ensemble policy, which follows the same evaluation method as in Osband et al.

Original: We use the popular human-normalized score Score 
Revised: We use the popular human-normalized score Score [Table] Score

Original: [Figure]
Revised: [Table]

Original:  Score random | to perform the comparison.
Revised: − Score random | to perform the comparison.

Revised: In tackling Atari games, Osband et al.

Revised: (2016) observes that the bootstrapping does not contribute much in performance.

Revised: Empirically, Bootstrapped DQN uses the same samples to train all Q -heads in each training step.

Revised: This empirical simpliﬁcation is also adopted by Chen et al.

Revised: Osband et al.

Original:  [Figure]
Revised: ; Nikolov et al. We use such an simpliﬁcation for OEB3 and all Bootstrapped DQN-based baselines. [Figure]

Original: We set α 1 and α 2 to the same value of 0 .
Revised: We set α 1 and α 2 to the same value of 0 .

Original: 5 × 10 −by searching coarsely.
Revised: 5 × 10 − 4 by searching coarsely.

Original: We use diffusion factor β = 0 .
Revised: We use diffusion factor β = 0 .

Original: 5 for all methods by following Lee et al.
Revised: 5 for all methods by following Lee et al.

Original: OEB3 requires much less training time than BEBU-UCB and BEBU-IDS because the conﬁdence bound used in BEBU-UCB and regret-information ratio used in BEBU-IDS are both computed in each time step when interacting with the environment, while the UCBbonuses in OEB3 are calculated in episodes when performing batch training.
Revised: OEB3 requires much less training time than BEBUUCB and BEBU-IDS because the conﬁdence bound used in BEBU-UCB and regret-information ratio used in BEBU-IDS are both computed in each time step when interacting with the environment, while the UCB-bonuses in OEB3 are calculated in episodes when performing batch training.

Original: The number of environmental interaction steps L 1 are typically set to be much larger than the number of training steps L 2 (e.g., in DQN, L 1 ≈ 4 L 2 ).
Revised: The number of environmental interaction steps L 1 are typically set to be much larger than the number of training steps L 2 (e.g., in DQN, L 1 ≈ 4 L 2 ).

Original: We refer to Appendix D for the detailed speciﬁcations.
Revised: We refer to Appendix D for the detailed speciﬁcations.

Original: The code is available at https://bit.ly/33jv1ab .
Revised: The code is available at https://bit.ly/33jv1ab .

Original: [Table]

Original: 5. R ESULT C OMPARISON
Revised: 5. R ESULT C OMPARISON

Original: In Table 1, we additionally report the performance of DQN (Mnih et al., 2015), NoisyNet (Fortunato et al., 2018), Bootstrapped DQN (Osband et al., 2016), and IDS (Nikolov et al., 2019) in 200M training frames.
Revised: In Table 1, we additionally report the performance of DQN (Mnih et al., 2015), NoisyNet (Fortunato et al., 2018), Bootstrapped DQN (Osband et al., 2016), and IDS (Nikolov et al., 2019) in 200M training frames.

Original: We choose NoisyNet as a baseline since it is shown (Taiga et al., 2020) to perform substantially better than existing bonus-based methods evaluated by the whole Atari suit (instead of several hard exploration games), including CTS-counts (Bellemare et al., 2016), PixelCNN-counts (Ostrovski et al., 2017), RND (Burda et al., 2019b), and ICM (Pathak et al., 2017).
Revised: We choose NoisyNet as a baseline since it is shown (Taiga et al., 2020) to perform substantially better than existing bonus-based methods evaluated by the whole Atari suit (instead of several hard exploration games), including CTS-counts (Bellemare et al., 2016), PixelCNN-counts (Ostrovski et al., 2017), RND (Burda et al., 2019b), and ICM (Pathak et al., 2017).

Original: According to 
Revised: UBE (O’Donoghue et al., 2018) uses a parametric method to describe the posterior of Q -values, which are utilized for optimism in exploration. We also use UBE as a baseline.

Original:  Table 1, BootDQN-IDS performs better than NoisyNet.
Revised: According to Table 1, BootDQN-IDS performs better than UBE, BootDQN [Table]

Original:  Thus, BootDQN-IDS outperforms existing bonus-based exploration methods that perform worse than NoisyNet.
Revised: and NoisyNet. Thus, BootDQN-IDS outperforms existing bonus-based exploration methods that perform worse than NoisyNet.

Original: We re-implement BootDQNIDS with BEBU-based update and observe that OEB3 outperforms BEBU-IDS in both mean and medium scores, thus outperforming existing bonus-based methods that performs worse than NoisyNet.
Revised: We re-implement BootDQN- [Table] IDS with BEBU-based update and observe that OEB3 outperforms BEBU-IDS in both mean and medium scores, thus outperforming the bonus-based methods that performs worse than NoisyNet.

Original: We report the raw scores in Appendix F. Moreover, Appendix G shows that OEB3 outperforms 
Revised: We report the raw scores in Appendix F. Moreover, Appendix G shows that OEB3 outperforms BEBU,

Original: BEBU, BEBU-UCB, and BEBU-IDS in 36, 34, and 35 games out of all 49 games, respectively.
Revised: BEBU-UCB, and BEBU-IDS in 36, 34, and 35 games out of all 49 games, respectively.

Original: [Figure]
Revised: [Figure]

Original: To understand the proposed UCB-bonus, we use a trained OEB agent to interact with the environment for an episode in Breakout and record the UCB-bonuses at each step.
Revised: To understand the proposed UCB-bonus, we use a trained OEB agent to interact with the environment for an episode in Breakout and record the UCB-bonuses at each step.

Original: The curve in Figure 2 shows the UCB-bonuses of the subsampled steps in the episode.
Revised: The curve in Figure 2 shows the UCB-bonuses of the subsampled steps in the episode.

Original: We choose 8 spikes with high UCB-bonuses and visualize the corresponding frames.
Revised: We choose 8 spikes with high UCB-bonuses and visualize the corresponding frames.

Original: The events in spikes correspond to scarcely visited areas or crucial events, which are important for the agent to obtain rewards: digging a tunnel (1), throwing the ball to the top of bricks (2,3), rebounding the ball (4,5,6), eliminating all bricks and starting a new round (7,8).
Revised: The events in spikes correspond to scarcely visited areas or crucial events, which are important for the agent to obtain rewards: digging a tunnel (1), throwing the ball to the top of bricks (2,3), rebounding the ball (4,5,6), eliminating all bricks and starting a new round (7,8).

Original: We provide more examples of visualization in Appendix E. We further record the the mean of UCB-bonus of the training batch in the learning process, which is shown in Figure 3.
Revised: We provide more examples of visualization in Appendix E. We further record the the mean of UCB-bonus of the training batch in the learning process, which is shown in Figure 3.

Original: The UCB-bonus is low at the beginning since the networks are randomly initialized.
Revised: The UCB-bonus is low at the beginning since the networks are randomly initialized.

Original: When the agent starts to explore the environment, the mean UCB-bonus rises rapidly to incentive exploration.
Revised: When the agent starts to explore the environment, the mean UCB-bonus rises rapidly to incentive exploration.

Original: As more experiences of states and actions are gathered, the mean UCB-bonus reduces gradually, which indicates that the bootstrapped value functions concentrate around the optimal value and the epistemic uncertainty decreases.
Revised: As more experiences of states and actions are gathered, the mean UCB-bonus reduces gradually, which indicates that the bootstrapped value functions concentrate around the optimal value and the epistemic uncertainty decreases.

Original: Nevertheless, the UCB-bonuses are relatively high at scarcely visited areas or crucial events, according to Figure 2, which motivates the agent to enhance exploration at the corresponding events.
Revised: Nevertheless, the UCB-bonuses are relatively high at scarcely visited areas or crucial events, according to Figure 2, which motivates the agent to enhance exploration at the corresponding events.

Original: [Table]
Revised: [Table]

Original: We conduct an ablation study to better comprehend the importance of backward update and bonus term in OEB3.
Revised: We conduct an ablation study to better comprehend the importance of backward update and bonus term in OEB3.

Original: We refer to Tablefor the outcomes.
Revised: We refer to Tablefor the outcomes.

Original: We observe that: (1) when we use the ordinary update strategy by sampling transitions instead of episodes in training, OEB3 reduces to BootDQN-UCB with signiﬁcant performance loss.
Revised: We observe that: (1) when we use the ordinary update strategy by sampling transitions instead of episodes in training, OEB3 reduces to BootDQN-UCB with signiﬁcant performance loss.

Original: Hence, the backward update is crucial in OEB3 for sample-efﬁcient training.
Revised: Hence, the backward update is crucial in OEB3 for sample-efﬁcient training.

Original: (2) When the UCB-bonus is set to 0, OEB3 reduces to BEBU.
Revised: (2) When the UCB-bonus is set to 0, OEB3 reduces to BEBU.

Original: We observe that OEB3 outperforms BEBU in 36 out of all 49 games.
Revised: We observe that OEB3 outperforms BEBU in 36 out of all 49 games.

Original: (3) When both the backward update and UCB-bonus are removed, OEB reduces to standard BootDQN, which performs poorly in 20M training frames.
Revised: (3) When both the backward update and UCB-bonus are removed, OEB3 reduces to standard BootDQN, which performs poorly in 20M training frames.

Original: (4) To illustrate the effect of UCB-bonus, we substitute UCB-bonus with the popular RND-bonus (Burda et al., 2019b).
Revised: (4) To illustrate the effect of UCB-bonus, we substitute UCB-bonus with the popular RND-bonus (Burda et al., 2019b).

Original: Speciﬁcally, we use an independent RND network to generate RND-bonus for each state in training.
Revised: Speciﬁcally, we use an independent RND network to generate RND-bonus for each state in training.

Original: The RND-bonus is added to both the immediate reward and next- Q .
Revised: The RND-bonus is added to both the immediate reward and next- Q .

Original: The result shows that UCB-bonus outperforms RND-bonus without introducing additional modules compared to BootDQN.
Revised: The result shows that UCB-bonus outperforms RND-bonus without introducing additional modules compared to BootDQN.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose OEB3, which has theoretical underpinnings from optimistic LSVI.
Revised: In this work, we propose OEB3, which has theoretical underpinnings from optimistic LSVI.

Original: We propose a UCB-bonus to capture the epistemic uncertainty of Q -function and an BEBU algorithm for sample-efﬁcient training.
Revised: We propose a UCB-bonus to capture the epistemic uncertainty of Q -function and an BEBU algorithm for sample-efﬁcient training.

Original: We demonstrate OEB3 empirically by solving Mnist maze and Atari games and show that OEB3 outperforms several strong baselines.
Revised: We demonstrate OEB3 empirically by solving MNIST maze and Atari games and show that OEB3 outperforms several strong baselines.

Original: The visualizations suggest that high
Revised: The visualizations suggest that high

Original: UCB-bonus corresponds to informative experiences for exploration.
Revised: UCB-bonus corresponds to informative experiences for exploration.

Revised: As far as we are concerned, our work seems to establish the ﬁrst empirical attempt of uncertainty propagation in deep RL.

Revised: Moreover, we observe that such a connection between theoretical analysis and practical algorithm provides relatively strong empirical performance in Atari games and outperforms several strong baselines, which hopefully gives useful insights on combining theory and practice to the community.

Original: The future directions include adapting OEB3 to continuous control and integrating OEB3 with more expressive bonus schemes.
Revised: The future directions include adapting OEB3 to continuous control and integrating OEB3 with more expressive bonus schemes.

Original: A UCB B ONUS IN OEB 
Revised: A UCB B ONUS IN OEB

Original: Recall that we consider the following regularized least-square problem,
Revised: Recall that we consider the following regularized least-square problem,

Original: [Equation]
Revised: [Equation]

Original: In the sequel, we consider a Bayesian linear regression perspective of (8) that captures the intuition behind the UCB-bonus in OEB3.
Revised: In the sequel, we consider a Bayesian linear regression perspective of (8) that captures the intuition behind the UCB-bonus in OEB3.

Original: Our objective is to approximate the action-value function Q t via ﬁtting the parameter w , such that
Revised: Our objective is to approximate the action-value function Q t via ﬁtting the parameter w , such that

Original: [Equation]
Revised: [Equation]

Original: where Q t +1 is given.
Revised: where Q t +1 is given.

Original: We assume that we are given a Gaussian prior of the initial parameter w ∼ N (0 , I /λ ) .
Revised: We assume that we are given a Gaussian prior of the initial parameter w ∼ N (0 , I /λ ) .

Original: With a slight abuse of notation, we denote by w t the Bayesian posterior of the parameter w given the set of independent observations D m = { ( s τt , a τt , s τt +1 ) } τ ∈ [0 ,m ] .
Revised: With a slight abuse of notation, we denote by w t the Bayesian posterior of the parameter w given the set of independent observations D m = { ( s τt , a τt , s τt +1 ) } τ ∈ [0 ,m ] .

Original: We further deﬁne the following noise with respect to the least-square problem in (8),
Revised: We further deﬁne the following noise with respect to the least-square problem in (8),

Original: [Equation]
Revised: [Equation]

Original: where ( s t , a t , s t +1 ) follows the distribution of trajectory.
Revised: where ( s t , a t , s t +1 ) follows the distribution of trajectory.

Original: The following theorem justiﬁes the UCBbonus in OEB3 under the Bayesian linear regression perspective.
Revised: The following theorem justiﬁes the UCBbonus in OEB3 under the Bayesian linear regression perspective.

Original: Theorem 2 (Formal Version of Theorem 1) .
Revised: Theorem 2 (Formal Version of Theorem 1) .

Original: We assume that (cid:15) follows the standard Gaussian distribution N (0 , 1) given the state-action pair ( s t , a t ) and the parameter w .
Revised: We assume that (cid:15) follows the standard Gaussian distribution N (0 , 1) given the state-action pair ( s t , a t ) and the parameter w .

Original: Let w follows the
Revised: Let w follows the

Original: Gaussian prior N (0 , I /λ ) .
Revised: Gaussian prior N (0 , I /λ ) .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: It then holds for the posterior of w t given the set of independent observations D m
Revised: It then holds for the posterior of w t given the set of independent observations D m

Original: = { ( s τt , a τt , s τt +1 ) } τ ∈ [0 ,m ] that
Revised: = { ( s τt , a τt , s τt +1 ) } τ ∈ [0 ,m ] that

Original: [Equation]
Revised: [Equation]

Original: Here we denote by ˜ Q t = w (cid:62) t φ the estimated action-value function.
Revised: Here we denote by ˜ Q t = w (cid:62) t φ the estimated action-value function.

Original: Proof.
Revised: Proof.

Original: The proof follows the standard analysis of Bayesian linear regression.
Revised: The proof follows the standard analysis of Bayesian linear regression.

Original: See, e.g., West (1984) for a detailed analysis.
Revised: See, e.g., West (1984) for a detailed analysis.

Original: We denote the target of the linear regression in (8) by
Revised: We denote the target of the linear regression in (8) by

Original: [Equation]
Revised: [Equation]

Original: By the assumption that (cid:15) follows the standard Gaussian distribution, we obtain that
Revised: By the assumption that (cid:15) follows the standard Gaussian distribution, we obtain that

Original: [Equation]
Revised: [Equation]

Original: Recall that we have the prior distribution w ∼ N (0 , I /λ ) .
Revised: Recall that we have the prior distribution w ∼ N (0 , I /λ ) .

Original: Our objective is to compute the posterior density w t = w | D m , where D m = {
Revised: Our objective is to compute the posterior density w t = w | D m , where D m = {

Original: [Equation]
Revised: [Equation]

Original: s τ t , a τ t , s τ t +1 ) }
Revised: s τ t , a τ t , s τ t +1 ) }

Original: τ ∈ [0 ,m ] is the set of observations.
Revised: τ ∈ [0 ,m ] is the set of observations.

Original: It holds from
Revised: It holds from

Original: Bayes rule that
Revised: Bayes rule that

Original: [Equation]
Revised: [Equation]

Original: where p ( · ) denote the probability density function of the respective distributions.
Revised: where p ( · ) denote the probability density function of the respective distributions.

Original: Plugging (11) and the probability density function of Gaussian distribution into (12) yields
Revised: Plugging (11) and the probability density function of Gaussian distribution into (12) yields

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne
Revised: where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Λ

Original: [Equation]
Revised: [Equation]

Original: 1 t [Equation] Thus, by (13), we obtain that w t = w | D m ∼ N ( µ t , Λ − 1 t ) .
Revised: Thus, by (13), we obtain that w t = w | D m ∼ N ( µ t , Λ − 1 t ) .

Original: It then holds for all ( s t , a t ) ∈ S
Revised: It then holds for all ( s t , a t ) ∈ S

Original: × A that
Revised: × A that

Original: Var (cid:0) φ

Original: [Equation]

Original: s t , a t

Original: [Equation]

Original: t

Original: [Equation]

Original: = Var(cid:0) ˜ [Equation] t
Revised: Var

Original: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: = φ

Original: [Equation]

Original: s t [Equation] Λ − 1 t φ [Equation]
Revised: Var

Original: s t

Original: [Equation]
Revised: [Equation]

Original: ) , which concludes the proof of Theorem 2.
Revised: which concludes the proof of Theorem 2.

Original: B A LGORITHMIC D ESCRIPTION
Revised: B A LGORITHMIC D ESCRIPTION

Original: 1: Initialize θ −
Revised: 1: Initialize θ −

Original: 2: Initialize: total
Revised: 2: Initialize: total

Original: 3: while h < H do 4:
Revised: 3: while h < H do 4:

Original: Pick a bootstrapped Q -function to act by sampling k ∼ Unif { 1 , . .
Revised: Pick a bootstrapped Q -function to act by sampling k ∼ Unif { 1 , . .

Original: , K } 5:
Revised: , K } 5:

Original: Reset the environment and receive the initial state s6: for step i = 0 to Terminal do 7:
Revised: Reset the environment and receive the initial state s6: for step i = 0 to Terminal do 7:

Original: With (cid:15) -greedy choose a i = arg max a Q k ( s i , a ) 8:
Revised: With (cid:15) -greedy choose a i = arg max a Q k ( s i , a ) 8:

Original: Take action and observe r i and s i +1 , then save the transition in buffer D 9: if h % training frequency = 

Revised: Take action and observe r i and s i +1 , then save the transition in buffer D 9: if h % training frequency =

Original: then 10:
Revised: then 10:

Original: Sample an episodic experience E = { S , A , R , S (cid:48) } with length T from D 11:
Revised: Sample an episodic experience E = { S , A , R , S (cid:48) } with length T from D 11:

Original: Initialize a Q -table ˜ Q = Q ( S (cid:48) , A ; θ − ) ∈ R K ×|A|× T by the target Q -network 12:
Revised: Initialize a Q -table ˜ Q = Q ( S (cid:48) , A ; θ − ) ∈ R K ×|A|× T by the target Q -network 12:

Original: Compute the UCB-bonus for immediate reward for all steps to construct B ∈ R T 13:
Revised: Compute the UCB-bonus for immediate reward for all steps to construct B ∈ R T 13:

Original: Compute the action matrix ˜A =
Revised: Compute the action matrix ˜A =

Original: arg max a ˜Q [ · , a, · ] ∈ R K × T to gather all a (cid:48) of next- Q 14:
Revised: arg max a ˜Q [ · , a, · ] ∈ R K × T to gather all a (cid:48) of next- Q 14:

Original: Compute the UCB-bonus for next- Q for all heads and all steps to construct ˜B ∈ R K × T 15:
Revised: Compute the UCB-bonus for next- Q for all heads and all steps to construct ˜B ∈ R K × T 15:

Original: Compute the mask matrix M ∈ R K × T where M [ k, t ] = 

Revised: Compute the mask matrix M ∈ R K × T where M [ k, t ] =

Original: ˜A [ k,t ] (cid:54) =
Revised: ˜A [ k,t ] (cid:54) =

Original: A t + 16:
Revised: A t + 16:

Original: Initialize target table y ∈ R K × T to zeros, and set y [ · , T − 1]
Revised: Initialize target table y ∈ R K × T to zeros, and set y [ · , T − 1]

Original: = R T − 1 + α 1 B T −17: for t =
Revised: = R T − 1 + α 1 B T −17: for t =

Original: T − 2 to 0 do 18: ˜ Q [ · , a t +1 , t ] ← β y [ · , t + 1]
Revised: T − 2 to 0 do 18: ˜ Q [ · , a t +1 , t ] ← β y [ · , t + 1]

Original: + (1 − β ) ˜ Q
Revised: + (1 − β ) ˜ Q

Original: [ · , a t +1 , t ] 19:y [ · t ] ← (cid:0) + α B
Revised: [ · , a t +1 , t ] 19:

Original: (cid:1) + γ (cid:0) ˜Q [ ·, a (cid:48) t ] + α2 M [ ·
Revised: (cid:1) + γ (cid:0) ˜Q [ · a (cid:48) t ] + α M [ ·

Original: ] ◦ ˜B [ · a
Revised: ] ◦ ˜B [ · a

Original: ] 20: end for 21:
Revised: ] 20: end for 21:

Original: Compute the Q -value of ( S , A ) for all heads as Q = Q ( S , A ; θ ) ∈ R K × T 22:
Revised: Compute the Q -value of ( S , A ) for all heads as Q = Q ( S , A ; θ ) ∈ R K × T 22:

Original: Perform a gradient descent step on ( y − Q ) 2 with respect to θ 23: end if 24:
Revised: Perform a gradient descent step on ( y − Q ) 2 with respect to θ 23: end if 24:

Original: Every C steps reset θ − ← θ 25: h ←
Revised: Every C steps reset θ − ← θ 25: h ←

Original: 26: end for 27: end while
Revised: 26: end for 27: end while

Revised: 1: Input: Algorithm Type (BEBU, BEBU-UCB, or BEBU-IDS) 2: Initialize: replay buffer D , bootstrapped Q -network Q ( · ; θ ) and target network Q ( · ; θ − ) 3: Initialize: total training frames H = 20M , current frame h =4: while h < H do 5:

Revised: Pick a bootstrapped Q -function to act by sampling k ∼ Unif { 1 , . .

Original: C A DDITIONAL E XPERIMENT : M NIST M AZE
Revised: , K } 6: Reset the environment and receive the initial state s7: for step i = 0 to Terminal do 8: if Algorithm type is BEBU then 9: With (cid:15) -greedy choose a i = arg max a Q k ( s i , a ) 10: else if Algorithm type is BEBU-UCB then 11: With (cid:15) -greedy choose a i = arg max a [ ¯ Q ( s i , a ) + α · σ ( Q ( s i , a ))] , where ¯ Q ( s i , a i )

Revised: =

Revised: 1 K (cid:80)

Revised: Kk =1 Q k ( s i , a i ) and σ ( Q ( s i , a i ))

Revised: = (cid:113) 1 K (cid:80)

Revised: Kk =1 ( Q k ( s i , a i ) − ¯ Q ( s i , a i )) 2 are the mean and standard deviation of the bootstrapped Q-estimates 12: else if Algorithm type is BEBU-IDS then 13:

Revised: With (cid:15) -greedy choose a i = arg min a ˆ∆ i

Original: {+( s i ,a ) 
+}
Revised: {+( s i ,a ) 
+}

Revised: I i ( s i ,a ) by following the regret-information ratio, where ˆ∆ i

Revised: ( s i , a i ) = max a (cid:48)

Revised: ∈A u i ( s i , a (cid:48) )

Revised: − l i ( s i , a i ) is the expected regret, and [ l i ( s i , a i ) , u i ( s i , a i )] is the conﬁdence interval.

Revised: In particular, u i ( s i , a i ) =

Revised: ¯ Q ( s i , a i ) + λ ids · σ ( Q ( s i , a i )) and l i ( s i , a i ) =

Revised: ¯ Q ( s i , a i ) − λ ids · σ ( Q ( s i , a i )) .

Revised: I ( s i , a i ) =

Revised: log(1 + σ ( Q ( s i ,a i ))

Revised: 2 / ρ 2 ) + (cid:15) ids measures the uncertainty, where ρ and (cid:15) ids are constants.

Revised: 14: else 15:

Revised: Algorithm type error.

Revised: 16: end if 17:

Original: {+Take action and observe r i and s i +1 , then save the transition in buffer D 18: if h % training frequency = 
+}
Revised: {+Take action and observe r i and s i +1 , then save the transition in buffer D 18: if h % training frequency = 
+}

Revised: then 19:

Revised: Sample an episodic experience E = { S , A , R , S (cid:48) } with length T from D 20:

Revised: Initialize a Q -table ˜ Q = Q ( S (cid:48) , A ; θ − ) ∈ R K ×|A|× T by the target Q -network 21:

Revised: Compute the action matrix ˜A =

Revised: arg max a ˜Q [ · , a, · ] ∈ R K × T to gather all a (cid:48) of next- Q 22:

Revised: Initialize target table y ∈ R K × T to zeros, and set y [ · , T − 1]

Revised: = R T − 1 + α 1 B T −23: for t =

Revised: T − 2 to 0 do 24: ˜ Q [ · , a t +1 , t ] ← β y [ · , t + 1]

Revised: + (1 − β ) ˜ Q

Revised: [ · , a t +1 , t ] 25: y [ · , t ] ← R t + γ ˜Q

Revised: · , a (cid:48) , t ] where a (cid:48) =

Revised: ˜A [ · , t ] 26: end for 27:

Revised: Compute the Q -value of ( S , A ) for all heads as Q = Q ( S , A ; θ ) ∈ R K × T 28:

Revised: Perform a gradient descent step on ( y − Q ) 2 with respect to θ 29: end if 30:

Revised: Every C steps reset θ − ← θ 31: h ←

Revised: +32: end for 33: end while

Revised: C A DDITIONAL E XPERIMENT : MNIST M AZE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We use 10 × 10 Mnist maze with randomly placed walls to evaluate our method.
Revised: We use 10 × 10 MNIST maze with randomly placed walls to evaluate our method.

Original: The agent starts from the initial position (0 , 0) in the upper-left of the maze and aims to reach the goal position (9 , 9) in the bottom-right.
Revised: The agent starts from the initial position (0 , 0) in the upper-left of the maze and aims to reach the goal position (9 , 9) in the bottom-right.

Original: The state of position ( i, j ) is represented by stacking two randomly sampled images with label i and j from the Mnist dataset.
Revised: The state of position ( i, j ) is represented by stacking two randomly sampled images with label i and j from the MNIST dataset.

Original: When the agent steps to a new position, the state representation is reconstructed by sampling images.
Revised: When the agent steps to a new position, the state representation is reconstructed by sampling images.

Original: Hence the agent gets different states even stepping to the same location twice, which minimizes the correlation among locations.
Revised: Hence the agent gets different states even stepping to the same location twice, which minimizes the correlation among locations.

Original: We further add some stochasticity in the transition probability.
Revised: We further add some stochasticity in the transition probability.

Original: The agent has a probability of 10% to arrive in the adjacent locations when taking an action.
Revised: The agent has a probability of 10% to arrive in the adjacent locations when taking an action.

Original: For example, when taking action ‘left’, the agent has a 10% chance of transiting to ‘up’, and a 10% chance of transiting to ‘down’.
Revised: For example, when taking action ‘left’, the agent has a 10% chance of transiting to ‘up’, and a 10% chance of transiting to ‘down’.

Original: The agent gets a reward of -1 when bumping into a wall, and gets 1000 when reaching the goal.
Revised: The agent gets a reward of -1 when bumping into a wall, and gets 1000 when reaching the goal.

Original: We use the different setup of wall-density in the experiment.
Revised: We use the different setup of wall-density in the experiment.

Original: The wall-density means the proportion of walls in all locations.
Revised: The wall-density means the proportion of walls in all locations.

Original: Figure 5 (left) shows a generated maze with wall-density of 50%.
Revised: Figure 5 (left) shows a generated maze with wall-density of 50%.

Original: The gray positions represent walls.
Revised: The gray positions represent walls.

Original: We train all methods with wall-density of 30%, 40%, and 50%.
Revised: We train all methods with wall-density of 30%, 40%, and 50%.

Original: For each setup, we train 50 independent agents for 50 randomly generated mazes.
Revised: For each setup, we train 50 independent agents for 50 randomly generated mazes.

Original: The relative length deﬁned by l agent /l
Revised: The relative length deﬁned by l agent /l

Original: best is used to evaluate the performance, where l agent is the length of the agent’s path in an episode (maximum steps are 1000), and l best is the length of the shortest path.
Revised: best is used to evaluate the performance, where l agent is the length of the agent’s path in an episode (maximum steps are 1000), and l best is the length of the shortest path.

Original: The performance comparison is shown in Figure 4.
Revised: The performance comparison is shown in Figure 4.

Original: OEB3 performs best in all methods, and BEBU-IDS is a strong baseline.
Revised: OEB3 performs best in all methods, and BEBU-IDS is a strong baseline.

Original: We use a trained OEB3 agent to take action in the maze of Figure 5 (left).
Revised: We use a trained OEB3 agent to take action in the maze of Figure 5 (left).

Original: The
Revised: The

Original: UCB-bonuses of state-action pairs in the agent’s path are computed and visualized in Figure 5 (right).
Revised: UCB-bonuses of state-action pairs in the agent’s path are computed and visualized in Figure 5 (right).

Original: The state-action pairs with high UCB-bonus show the bottleneck positions in the path.
Revised: The state-action pairs with high UCB-bonus show the bottleneck positions in the path.

Original: For example, the state-action pairs in location (3 , 3) and (6 , 7) produce high UCB-bonus to guide the agent to choose the right direction.
Revised: For example, the state-action pairs in location (3 , 3) and (6 , 7) produce high UCB-bonus to guide the agent to choose the right direction.

Original: The UCB-bonus encourages the agent to walk through these bottleneck positions correctly.
Revised: The UCB-bonus encourages the agent to walk through these bottleneck positions correctly.

Original: We give more examples of visualization in Appendix E.
Revised: We give more examples of visualization in Appendix E.

Original: D I MPLEMENTATION D ETAIL D. 
Revised: D I MPLEMENTATION D ETAIL D. MNIST M AZE

Original: M [Equation] AZE

Original: Hyper-parameters of BEBU .
Revised: Hyper-parameters of BEBU .

Original: BEBU is the basic algorithm of BEBU-UCB and BEBU-IDS.
Revised: BEBU is the basic algorithm of BEBU-UCB and BEBU-IDS.

Original: BEBU uses the same network-architecture as Bootstrapped DQN (Osband et al., 2016).
Revised: BEBU uses the same network-architecture as Bootstrapped DQN (Osband et al., 2016).

Original: The diffusion factor and other training parameters are set by following EBU paper (Lee et al., 2019).
Revised: The diffusion factor and other training parameters are set by following EBU paper (Lee et al., 2019).

Original: Details are summarized in Table
Revised: Details are summarized in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Hyper-parameters of BEBU-UCB .
Revised: Hyper-parameters of BEBU-UCB .

Original: BEBU-UCB uses the upper-bound of Q -values to select actions.
Revised: BEBU-UCB uses the upper-bound of Q -values to select actions.

Original: In particular, a = arg max a ∈A [ µ ( s, a ) + λ ucb σ ( s, a )] , where µ ( s, a ) and σ ( s, a ) are the mean and standard deviation of bootstrapped Q -values { Q k ( s, a ) }
Revised: In particular, a = arg max a ∈A [ µ ( s, a ) + λ ucb σ ( s, a )] , where µ ( s, a ) and σ ( s, a ) are the mean and standard deviation of bootstrapped Q -values { Q k ( s, a ) }

Original: Kk
Revised: Kk

Original: =1 .
Revised: =1 .

Original: We use λ ucb = 0 .
Revised: We use λ ucb = 0 .

Original: 1 in our experiment.
Revised: 1 in our experiment.

Original: Hyper-parameters of BEBU-IDS .
Revised: Hyper-parameters of BEBU-IDS .

Original: The action-selection in IDS (Nikolov et al., 2019) follows the regret-information ratio as a t = arg min a ∈A
Revised: The action-selection in IDS (Nikolov et al., 2019) follows the regret-information ratio as a t = arg min a ∈A

Original: ˆ∆ t ( s,a ) 

Revised: ˆ∆ t ( s,a )

Original: I t ( s,a ) , which balances the regret and exploration.
Revised: I t ( s,a ) , which balances the regret and exploration.

Original: ˆ∆ t ( s, a ) is the expected regret that indicates the loss of reward when choosing a suboptimal action a .
Revised: ˆ∆ t ( s, a ) is the expected regret that indicates the loss of reward when choosing a suboptimal action a .

Original: IDS uses a conservative estimate of regret as ˆ∆ t ( s, a ) = max a (cid:48) ∈A u t ( s, a (cid:48) )
Revised: IDS uses a conservative estimate of regret as ˆ∆ t ( s, a ) = max a (cid:48) ∈A u t ( s, a (cid:48) )

Original: − l t ( s, a ) , where [ l t ( s, a ) , u t ( s, a )] is the conﬁdence interval of action-value function.
Revised: − l t ( s, a ) , where [ l t ( s, a ) , u t ( s, a )] is the conﬁdence interval of action-value function.

Original: In particular, u t ( s, a ) = µ ( s, a )+ λ ids σ ( s, a ) and l t ( s, a ) = µ ( s, a ) − λ ids σ ( s, a ) , where µ ( s, a ) and σ ( s, a ) are the mean and standard deviation of bootstrapped Q -values { Q k ( s, a ) } Kk =1 .
Revised: In particular, u t ( s, a ) = µ ( s, a )+ λ ids σ ( s, a ) and l t ( s, a ) = µ ( s, a ) − λ ids σ ( s, a ) , where µ ( s, a ) and σ ( s, a ) are the mean and standard deviation of bootstrapped Q -values { Q k ( s, a ) } Kk =1 .

Original: The information gain I t ( a ) measures the uncertainty of action-values as
Revised: The information gain I t ( a ) measures the uncertainty of action-values as

Original: I ( s, a ) =
Revised: I ( s, a ) =

Original: log(1 + σ ( s,a ) 2 ρ ( s,a ) 2 ) + (cid:15) ids , where ρ ( s, a ) is the variance of the return distribution.
Revised: log(1 + σ ( s,a ) 2 ρ ( s,a ) 2 ) + (cid:15) ids , where ρ ( s, a ) is the variance of the return distribution.

Original: ρ ( s, a ) is measured by C51 (Bellemare et al., 2017) in distributional RL and becomes a constant in ordinary Q -learning.
Revised: ρ ( s, a ) is measured by C51 (Bellemare et al., 2017) in distributional RL and becomes a constant in ordinary Q -learning.

Original: We use λ ids = 0 .
Revised: We use λ ids = 0 .

Original: 1 , ρ ( s, a )
Revised: 1 , ρ ( s, a )

Original: = 1 . 0 , and (cid:15) ids = 10 − 5 in our experiment.
Revised: = 1 . 0 , and (cid:15) ids = 10 − 5 in our experiment.

Original: Hyper-parameters of OEB3 .
Revised: Hyper-parameters of OEB3 .

Original: We set α 1 and α 2 to the same value of 0 .
Revised: We set α 1 and α 2 to the same value of 0 .

Original: We ﬁnd that adding a normalizer to UCB-bonus ˜B of the next Q -value enables us to get more stable performance.
Revised: We ﬁnd that adding a normalizer to UCB-bonus ˜B of the next Q -value enables us to get more stable performance.

Original: ˜B is smoothed by dividing a running estimate of its standard deviation.
Revised: ˜B is smoothed by dividing a running estimate of its standard deviation.

Original: Because the UCB-bonuses for next- Q are different for each Q -head, the normalization is useful in most cases by making the value function have a smooth and stable update.
Revised: Because the UCB-bonuses for next- Q are different for each Q -head, the normalization is useful in most cases by making the value function have a smooth and stable update.

Original: D. A TARI GAMES
Revised: D. A TARI GAMES

Original: Hyper-parameters of BEBU .
Revised: Hyper-parameters of BEBU .

Original: The basic setting of the Atari environment is the same as Nature
Revised: The basic setting of the Atari environment is the same as Nature

Original: DQN (Mnih et al., 2015) and EBU (Lee et al., 2019) papers.
Revised: DQN (Mnih et al., 2015) and EBU (Lee et al., 2019) papers.

Original: We use the different network architecture, environmental setup, exploration factor
Revised: We use the different network architecture, environmental setup, exploration factor

Original: [Table]
Revised: [Table]

Original: and evaluation scheme from Mnist maze experiment.
Revised: and evaluation scheme from MNIST maze experiment.

Original: Details are summarized in Table 4.
Revised: Details are summarized in Table 4.

Original: [Table]
Revised: [Table]

Original: The
Revised: convolution

Original: [Table]
Revised: [Table]

Original: network is updated every
Revised: ﬁrst

Original: [Table]
Revised: [Table]

Original: Adam optimizer is used for training.
Revised: Adam optimizer is used for training.

Original: Detailed parameters:
Revised: Detailed parameters:

Original: [Table]
Revised: [Table]

Original: Hyper-parameters of
Revised: Hyper-parameters of

Original: BEBU-UCB .
Revised: BEBU-UCB .

Original: BEBU-UCB selects actions by a = arg max a ∈A [ µ ( s, a ) + λ ucb σ ( s, a )] .
Revised: BEBU-UCB selects actions by a = arg max a ∈A [ µ ( s, a ) + λ ucb σ ( s, a )] .

Original: The detail is given in Appendix D.1.
Revised: The detail is given in Appendix D.1.

Original: We use λ ucb = 0 .
Revised: We use λ ucb = 0 .

Original: 1 in our experiment by searching coarsely.
Revised: 1 in our experiment by searching coarsely.

Original: Hyper-parameters of BEBU-IDS .
Revised: Hyper-parameters of BEBU-IDS .

Original: The action-selection follows the regret-information ratio as a t = arg min a ∈A
Revised: The action-selection follows the regret-information ratio as a t = arg min a ∈A

Original: ˆ∆ t ( s,a ) 

Revised: ˆ∆ t ( s,a )

Original: I t ( s,a ) .
Revised: I t ( s,a ) .

Original: The details are given in Appendix D.1.
Revised: The details are given in Appendix D.1.

Original: We use λ ids = 0 .
Revised: We use λ ids = 0 .

Original: 1 , ρ ( s, a ) = 1 .and (cid:15) ids = 10 − 5 in our experiment by searching coarsely.
Revised: 1 , ρ ( s, a ) = 1 .and (cid:15) ids = 10 − 5 in our experiment by searching coarsely.

Original: Hyper-parameters of OEB3 .
Revised: Hyper-parameters of OEB3 .

Original: We set α 1 and α 2 to the same value of 0 .
Revised: We set α 1 and α 2 to the same value of 0 .

Original: 5 × 10 − 4 .
Revised: 5 × 10 − 4 .

Original: The UCB-bonus ˜B for the next Q -value is normalized by dividing a running estimate of its standard deviation to have a stable performance.
Revised: The UCB-bonus ˜B for the next Q -value is normalized by dividing a running estimate of its standard deviation to have a stable performance.

Original: E V ISUALIZING OEB 
Revised: E V ISUALIZING OEB

Original: OEB3 uses the UCB-bonus that indicates the disagreement of bootstrapped Q -estimates to measure the uncertainty of Q -functions.
Revised: OEB3 uses the UCB-bonus that indicates the disagreement of bootstrapped Q -estimates to measure the uncertainty of Q -functions.

Original: The state-action pairs with high UCB-bonuses signify the bottleneck positions or meaningful events.
Revised: The state-action pairs with high UCB-bonuses signify the bottleneck positions or meaningful events.

Original: We provide visualization in several tasks to illustrate the effect of UCB-bonuses.
Revised: We provide visualization in several tasks to illustrate the effect of UCB-bonuses.

Original: Speciﬁcally, we choose Mnist-maze and two popular Atari games Breakout and
Revised: Speciﬁcally, we choose Mnist-maze and two popular Atari games Breakout and

Original: Mspacman to analyze.
Revised: Mspacman to analyze.

Original: E. M NIST - MAZE
Revised: E. MNIST- MAZE

Original: Figure 6 illustrates the UCB-bonus in four randomly generated mazes.
Revised: Figure 6 illustrates the UCB-bonus in four randomly generated mazes.

Original: The mazes in Figure 6(a) and 6(b) have a wall-density of 40% , and in Figure 6(c) and 6(d) have a wall-density of 50% .
Revised: The mazes in Figure 6(a) and 6(b) have a wall-density of 40% , and in Figure 6(c) and 6(d) have a wall-density of 50% .

Original: The left of each ﬁgure shows the map of maze, where the black blocks represent the walls.
Revised: The left of each ﬁgure shows the map of maze, where the black blocks represent the walls.

Original: We do not show the Mnist representation of states for simpliﬁcation, and Mnist states are used in training.
Revised: We do not show the MNIST representation of states for simpliﬁcation, and MNIST states are used in training.

Original: A trained OEB3 agent starts at the upper-left, then takes actions to achieve the goal at bottom-right.
Revised: A trained OEB3 agent starts at the upper-left, then takes actions to achieve the goal at bottom-right.

Original: The

Original:  UCB-bonuses of state-action pairs in the agent’s path are computed and shown on the right of each ﬁgure.
Revised: The UCB-bonuses of state-action pairs in the agent’s path are computed and shown on the right of each ﬁgure.

Original: The value is normalized to 0 ∼ 1 for visualization.
Revised: The value is normalized to 0 ∼ 1 for visualization.

Original: We show the maximal value if the agent appears several times in the same location.
Revised: We show the maximal value if the agent appears several times in the same location.

Original: The positions with UCB-bonuses that higher than 0 draw the path of the agent.
Revised: The positions with UCB-bonuses that higher than 0 draw the path of the agent.

Original: The path is usually winding and includes positions beyond the shortest path because the state transition has stochasticity.
Revised: The path is usually winding and includes positions beyond the shortest path because the state transition has stochasticity.

Original: The state-action pairs with high UCB-bonuses show the bottleneck positions in the path.
Revised: The state-action pairs with high UCB-bonuses show the bottleneck positions in the path.

Original: In maze 6(a), the agent slips from the right path in position (4 , 7) to (4 , 9) .
Revised: In maze 6(a), the agent slips from the right path in position (4 , 7) to (4 , 9) .

Original: The state-action in position (4 , 8) produces high bonus to guide the agent back to the right path.
Revised: The state-action in position (4 , 8) produces high bonus to guide the agent back to the right path.

Original: In maze 6(b), the bottleneck state in (3 , 2) has high bonus to avoid the agent from entering into the wrong side of the fork.
Revised: In maze 6(b), the bottleneck state in (3 , 2) has high bonus to avoid the agent from entering into the wrong side of the fork.

Original: The other two mazes also have bottleneck positions, like (3 , 3) in maze 6(c) and (7 , 6) in maze 6(d).
Revised: The other two mazes also have bottleneck positions, like (3 , 3) in maze 6(c) and (7 , 6) in maze 6(d).

Original: Selecting actions incorrectly in these important locations is prone to failure.
Revised: Selecting actions incorrectly in these important locations is prone to failure.

Original: The UCB-bonus encourages the agent to walk through these bottleneck positions correctly.
Revised: The UCB-bonus encourages the agent to walk through these bottleneck positions correctly.

Original: Map 0.
Revised: Map 0.

Original: UCB-bonus 0.
Revised: UCB-bonus 0.

Original: (a) 0 1 2 3 4 5 6 7 80 13 46 

Revised: (a) 0 1 2 3 4 5 6 7 80 13 46

Original: Map 0.
Revised: Map 0.

Original: UCB-bonus 0.
Revised: UCB-bonus 0.

Original: (b) 0 1 2 3 4 5 6 7 80 13 46 

Revised: (b) 0 1 2 3 4 5 6 7 80 13 46

Original: Map 0.
Revised: Map 0.

Original: UCB-bonus 0.
Revised: UCB-bonus 0.

Original: (c) 

Revised: (c)

Original: Map 0.
Revised: Map 0.

Original: UCB-bonus 0.
Revised: UCB-bonus 0.

Original: (d)
Revised: (d)

Original: E. B REAKOUT
Revised: E. B REAKOUT

Original: In Breakout, the agent uses walls and the paddle to rebound the ball against the bricks and eliminate them.
Revised: In Breakout, the agent uses walls and the paddle to rebound the ball against the bricks and eliminate them.

Original: We use a trained OEB3 agent to interact with the environment for an episode in Breakout.
Revised: We use a trained OEB3 agent to interact with the environment for an episode in Breakout.

Original: The whole episode contains 3445 steps, and we subsample them every 4 steps for visualization.
Revised: The whole episode contains 3445 steps, and we subsample them every 4 steps for visualization.

Original: The curve in Figure 7 shows the UCB-bonus in 861 sampled steps.
Revised: The curve in Figure 7 shows the UCB-bonus in 861 sampled steps.

Original: We choose 16 spikes with high UCB-bonuses and visualize the corresponding frames.
Revised: We choose 16 spikes with high UCB-bonuses and visualize the corresponding frames.

Original: The events in spikes usually mean meaningful experiences that are important for the agent to get rewards.
Revised: The events in spikes usually mean meaningful experiences that are important for the agent to get rewards.

Original: In step 1, the agent is hoping to dig a tunnel and get rewards faster.
Revised: In step 1, the agent is hoping to dig a tunnel and get rewards faster.

Original: After digging a tunnel, the balls appear on the top of bricks in steps 2, 3, 4, 5, 6, 9, and 12.
Revised: After digging a tunnel, the balls appear on the top of bricks in steps 2, 3, 4, 5, 6, 9, and 12.

Original: Balls on the top are easier to hit bricks.
Revised: Balls on the top are easier to hit bricks.

Original: The agents rebound the ball and throw it over the bricks in steps 7, 8, 10, and 11.
Revised: The agents rebound the ball and throw it over the bricks in steps 7, 8, 10, and 11.

Original: In state 13, the agent eliminates all bricks and then comes to a new round, which is novel and promising to get more rewards.
Revised: In state 13, the agent eliminates all bricks and then comes to a new round, which is novel and promising to get more rewards.

Original: The agents in steps 14, 15, and 16 rebound the ball and try to dig a tunnel again.
Revised: The agents in steps 14, 15, and 16 rebound the ball and try to dig a tunnel again.

Original: The UCB-bonus encourages the agent to explore the potentially informative and novel state-action pairs to get high rewards.
Revised: The UCB-bonus encourages the agent to explore the potentially informative and novel state-action pairs to get high rewards.

Original: We record 15 frames after each spike for further visualization.
Revised: We record 15 frames after each spike for further visualization.

Original: The video is available at https://youtu.be/VptBkHyMt8g .
Revised: The video is available at https://youtu.be/VptBkHyMt8g .

Original: [Figure]
Revised: [Figure]

Original: E. M S P ACMAN
Revised: E. M S P ACMAN

Original: In MsPacman, the agent earns points by avoiding monsters and eating pellets.
Revised: In MsPacman, the agent earns points by avoiding monsters and eating pellets.

Original: Eating an energizer causes the monsters to turn blue, allowing them to be eaten for extra points.
Revised: Eating an energizer causes the monsters to turn blue, allowing them to be eaten for extra points.

Original: We use a trained
Revised: We use a trained

Original: OEB3 agent to interact with the environment for an episode.
Revised: OEB3 agent to interact with the environment for an episode.

Original: Figure 8 shows the UCB bonus in all 708 steps.
Revised: Figure 8 shows the UCB bonus in all 708 steps.

Original: We choose 16 spikes to visualize the frames.
Revised: We choose 16 spikes to visualize the frames.

Original: The spikes of exploration bonuses correspond to meaningful events for the agent to get rewards: starting a new scenario (1,2,9,10), changing direction (3,4,13,14,16), eating energizer (5,11), eating monsters (7,8,12), and entering the corner (6,15).
Revised: The spikes of exploration bonuses correspond to meaningful events for the agent to get rewards: starting a new scenario (1,2,9,10), changing direction (3,4,13,14,16), eating energizer (5,11), eating monsters (7,8,12), and entering the corner (6,15).

Original: These state-action pairs with high UCB-bonuses make the agent explore the environment efﬁciently.
Revised: These state-action pairs with high UCB-bonuses make the agent explore the environment efﬁciently.

Original: We record 15 frames after each spike, and the video is shown at https: //youtu.be/C_8NHKpBNXM .
Revised: We record 15 frames after each spike, and the video is shown at https: //youtu.be/C_8NHKpBNXM .

Original: [Figure]
Revised: [Figure]

Original: F R AW S CORES OF ALL 49 A TARI G AMES
Revised: F R AW S CORES OF ALL 49 A TARI G AMES

Original: [Table]
Revised: [Table]

Original: Jamesbond
Revised: Jamesbond

Original: [Table]
Revised: [Table]

Original: G P ERFORMANCE C OMPARISON
Revised: G P ERFORMANCE C OMPARISON

Original: We use the relative scores as
Revised: We use the relative scores as

Original: Score Agent − Score Baseline max { Score human , Score baseline } − Score random to compare OEB3 with baselines.
Revised: Score Agent − Score Baseline max { Score human , Score baseline } − Score random to compare OEB3 with baselines.

Original: The results of OEB3 comparing with BEBU, BEBU-UCB
Revised: The results of OEB3 comparing with BEBU, BEBU-UCB

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: BEBU-IDS is shown in Figure 9, Figure
Revised: BEBU-IDS is shown in Figure 9, Figure

Original: [Table]
Revised: [Table]

Original: and Figure 11, respectively.
Revised: and Figure 11, respectively.

Original: [Table]
Revised: [Table]

Original: r o
Revised: r o

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r
Revised: M a s t e r

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: A ss a u l t K r u ll
Revised: A ss a u l t K r u ll

Original: V i
Revised: V i

Original: d e o P i n gb a ll
Revised: d e o P i n gb a ll

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: W i z a r d o f W a r
Revised: W i z a r d o f W a r

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r A ss a u l t G o p h e r
Revised: M a s t e r A ss a u l t G o p h e r

Original: K a n g a r oo
Revised: K a n g a r oo

Original: B a t t l e Z o n e
Revised: B a t t l e Z o n e

Original: R i v e r R a i d
Revised: R i v e r R a i d

Original: A m i d e r A t l a n t
Revised: A m i d e r A t l a n t

Original: i s
Revised: i s

Original: P r i v a t e
Revised: P r i v a t e

Original: E y e
Revised: E y e

Original: S e a q u e s t
Revised: S e a q u e s t

Original: D e m o n
Revised: D e m o n

Original: A tt a c k B
Revised: A tt a c k B

Original: o
Revised: o

Original: w li n g
Revised: w li n g

Original: A s t e r o i d s
Revised: A s t e r o i d s

Original: M o n t e z u m a ' s R e v e n g e A s t e r i x P o n g
Revised: M o n t e z u m a ' s R e v e n g e A s t e r i x P o n g

Original: M s .
Revised: M s .

Original: P a c - M a n
Revised: P a c - M a n

Original: S t a r G unn e r
Revised: S t a r G unn e r

Original: C h
Revised: C h

Original: o pp e r
Revised: o pp e r

Original: C o m m a n d
Revised: C o m m a n d

Original: V e n t u r e
Revised: V e n t u r e

Original: A li e n H .
Revised: A li e n H .

Original: E .
Revised: E .

Original: R .
Revised: R .

Original: O
Revised: O

Original: R o b o t a n k
Revised: R o b o t a n k

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: G r a v i t a r E n d u r o
Revised: G r a v i t a r E n d u r o

Original: T i m e P il o t
Revised: T i m e P il o t

Original: Q * B e r t
Revised: Q * B e r t

Original: R o a d R unn e r T e nn i s
Revised: R o a d R unn e r T e nn i s

Original: U p a n d
Revised: U p a n d

Original: D o w n Z a xx o n
Revised: D o w n Z a xx o n

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: B e a m R i
Revised: B e a m R i

Original: d
Revised: d

Original: e r
Revised: e r

Original: B o x i n g
Revised: B o x i n g

Original: B r e a k o u t
Revised: B r e a k o u t

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t
Revised: i s t

Original: F r o s t b i t e
Revised: F r o s t b i t e

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: J a m e s B o n d
Revised: J a m e s B o n d

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y
Revised: D e r b y

Original: F r ee w a y
Revised: F r ee w a y

Original: N a m e T h i s G a m e
Revised: N a m e T h i s G a m e

Original: [Figure]
Revised: [Figure]

Original: r a z y C li m
Revised: r a z y C li m

Original: [Figure]
Revised: [Figure]

Original: e r
Revised: e r

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: V i
Revised: V i

Original: [Figure]
Revised: [Figure]

Original: e o Pi n
Revised: e o P

Original: [Figure]
Revised: [Figure]

Revised: n

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table]

Original: H F AILURE A NALYSIS
Revised: H F AILURE A NALYSIS

Original: [Table]
Revised: [Table]

Original: Our method does not have a good performance on Montezuma’s Revenge (see Table 6) because the epistemic uncertainty-based methods are not particularly tweaked for this domain.
Revised: Our method does not have a good performance on Montezuma’s Revenge (see Table 6) because the epistemic uncertainty-based methods are not particularly tweaked for this domain.

Original: Bootstrapped DQN, IDS, NoisyNet and BEBU-based methods all fail in this domain and achieves no more than 100 points.
Revised: IDS, NoisyNet and BEBU-based methods all fail in this domain and score zero. Bootstrapped DQN achievespoints, which is also very low and does not indicate successful learning in Montezuma’s revenge.

Original: In contrast, the bonus-based methods achieve signiﬁcantly higher scores in this tasks (e.g., RND achieves 8152 points).
Revised: In contrast, the bonus-based methods achieve signiﬁcantly higher scores in this tasks (e.g., RND achieves 8152 points).

Original: However, according to Taiga et al.
Revised: However, according to Taiga et al.

Original: and Table 1, NoisyNet and IDS signiﬁcantly outperforms all bonus-based methods evaluated by the whole Atari suit.
Revised: and Table 1, NoisyNet and IDS signiﬁcantly outperform several strong bonus-based methods evaluated by the mean and median scores of 49 Atari games.