Original: M AX M IN -N OVELTY : M AXIMIZING N OVELTY VIA M INIMIZING THE S TATE -A CTION V ALUES IN D EEP R EINFORCEMENT L EARNING
Revised: M AX M IN -N OVELTY : M AXIMIZING N OVELTY VIA M INIMIZING THE S TATE -A CTION V ALUES IN D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning research has achieved high acceleration in its progress starting from the initial installation of deep neural networks as function approximators to learn policies that make sequential decisions in high-dimensional state representation MDPs.
Revised: Reinforcement learning research has achieved high acceleration in its progress starting from the initial installation of deep neural networks as function approximators to learn policies that make sequential decisions in high-dimensional state representation MDPs.

Original: While several consecutive barriers have been broken in deep reinforcement learning research (i.e. learning from high-dimensional states, learning purely via self-play), several others still stand.
Revised: While several consecutive barriers have been broken in deep reinforcement learning research (i.e. learning from high-dimensional states, learning purely via self-play), several others still stand.

Original: On this line, the question of how to explore in high-dimensional complex MDPs is a well-understudied and ongoing open problem.
Revised: On this line, the question of how to explore in high-dimensional complex MDPs is a well-understudied and ongoing open problem.

Original: To address this, in our paper we propose a unique exploration technique based on maximization of novelty via minimization of the state-action value function (MaxMin Novelty).
Revised: To address this, in our paper we propose a unique exploration technique based on maximization of novelty via minimization of the state-action value function (MaxMin Novelty).

Original: Our method is theoretically well motivated, and comes with zero computational cost while leading to signiﬁcant sample efﬁciency gains in deep reinforcement learning training.
Revised: Our method is theoretically well motivated, and comes with zero computational cost while leading to signiﬁcant sample efﬁciency gains in deep reinforcement learning training.

Original: We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs.
Revised: We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs.

Original: We show that our technique improves the human normalized median scores of Arcade Learning Environment by 248% in the low-data regime.
Revised: We show that our technique improves the human normalized median scores of Arcade Learning Environment by 248% in the low-data regime.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Utilization of deep neural networks as function approximators enabled learning functioning policies in high-dimensional state representation MDPs (Mnih et al., 2015).
Revised: Utilization of deep neural networks as function approximators enabled learning functioning policies in high-dimensional state representation MDPs (Mnih et al., 2015).

Original: Following this initial work, the current line of work trains deep reinforcement learning policies to solve highly complex problems from game solving (Hasselt et al., 2016; Schrittwieser et al., 2020) to self driving vehicles (Lan et al., 2020).
Revised: Following this initial work, the current line of work trains deep reinforcement learning policies to solve highly complex problems from game solving (Hasselt et al., 2016; Schrittwieser et al., 2020) to self driving vehicles (Lan et al., 2020).

Original: Yet there are still remaining unsolved problems restricting the current capabilities of deep neural policies.
Revised: Yet there are still remaining unsolved problems restricting the current capabilities of deep neural policies.

Original: One of the main intrinsic open problems in deep reinforcement learning research is exploration in high-dimensional state representation MDPs.
Revised: One of the main intrinsic open problems in deep reinforcement learning research is exploration in high-dimensional state representation MDPs.

Original: While prior work extensively studied the exploration problem in bandits and tabular reinforcement learning, and proposed various algorithms and techniques optimal to the tabular form or the bandit setting (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Lu & Roy, 2019; Wang et al., 2020; Karnin et al., 2013; Wagenmaker et al., 2022), exploration in deep reinforcement learning remains an open challenging problem.
Revised: While prior work extensively studied the exploration problem in bandits and tabular reinforcement learning, and proposed various algorithms and techniques optimal to the tabular form or the bandit setting (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Lu & Roy, 2019; Wang et al., 2020; Karnin et al., 2013; Wagenmaker et al., 2022), exploration in deep reinforcement learning remains an open challenging problem.

Original: Despite the provable optimality of these exploration techniques in the tabular or bandit setting, they generally rely strongly on the assumptions of tabular reinforcement learning, and in particular on the ability to record tables of statistical estimates for every state-action pair.
Revised: Despite the provable optimality of these exploration techniques in the tabular or bandit setting, they generally rely strongly on the assumptions of tabular reinforcement learning, and in particular on the ability to record tables of statistical estimates for every state-action pair.

Original: Thus, in high-dimensional complex MDPs, for which deep neural networks are used as function approximators, the efﬁciency and the optimality of exploration methods proposed for tabular settings do not transfer well to deep reinforcement learning exploration.
Revised: Thus, in high-dimensional complex MDPs, for which deep neural networks are used as function approximators, the efﬁciency and the optimality of exploration methods proposed for tabular settings do not transfer well to deep reinforcement learning exploration.

Original: This is primarily due to the increase in the MDP dimensions and the incline in the complexity.
Revised: This is primarily due to the increase in the MDP dimensions and the incline in the complexity.

Original: Hence, in deep reinforcement learning research still, naive and simple exploration techniques (e.g. (cid:15) -greedy) are preferred over the optimal tabular techniques (Mnih et al., 2015;
Revised: Hence, in deep reinforcement learning research still, naive and simple exploration techniques (e.g. (cid:15) -greedy) are preferred over the optimal tabular techniques (Mnih et al., 2015;

Original: Hasselt et al., 2016;
Revised: Hasselt et al., 2016;

Original: Wang et al., 2016; Anschel et al., 2017; Bellemare et al., 2017; Lan et al., 2020).
Revised: Wang et al., 2016; Anschel et al., 2017; Bellemare et al., 2017; Lan et al., 2020).

Original: Sample efﬁciency in deep neural policies is still one of the main challenging problems restricting research progress in reinforcement learning.
Revised: Sample efﬁciency in deep neural policies is still one of the main challenging problems restricting research progress in reinforcement learning.

Original: The magnitude of the number of samples required to learn and adapt continuously is one of the main limiting factors preventing current state-of-the-art deep reinforcement learning algorithms from being deployed in many diverse settings, but most importantly one of the main challenges that needs to be dealt with on the way to building general artiﬁcial intelligence.
Revised: The magnitude of the number of samples required to learn and adapt continuously is one of the main limiting factors preventing current state-of-the-art deep reinforcement learning algorithms from being deployed in many diverse settings, but most importantly one of the main challenges that needs to be dealt with on the way to building general artiﬁcial intelligence.

Original: In our paper we aim to seek answers for the following questions:
Revised: In our paper we aim to seek answers for the following questions:

Original: • Can we explore a high-dimensional state representation MDP more efﬁciently with zero additional computational cost?
Revised: • Can we explore a high-dimensional state representation MDP more efﬁciently with zero additional computational cost?

Original: • Is there a natural theoretical motivation that can be used to design a zero-cost exploration strategy while achieving high sample efﬁciency?
Revised: • Is there a natural theoretical motivation that can be used to design a zero-cost exploration strategy while achieving high sample efﬁciency?

Original: To be able to answer these questions, in our paper we focus on exploration in deep reinforcement learning and make the following contributions:
Revised: To be able to answer these questions, in our paper we focus on exploration in deep reinforcement learning and make the following contributions:

Original: • We propose a novel exploration technique based on minimizing the state-action value function to increase the information gain from each particular experience acquired in the MDP.
Revised: • We propose a novel exploration technique based on minimizing the state-action value function to increase the information gain from each particular experience acquired in the MDP.

Original: • We conduct extensive study in the Arcade Learning Environment 100K benchmark with the state-of-the-art algorithms and demonstrate that our proposed method achieves signiﬁcant performance improvement.
Revised: • We conduct extensive study in the Arcade Learning Environment 100K benchmark with the state-of-the-art algorithms and demonstrate that our proposed method achieves signiﬁcant performance improvement.

Original: • We show the efﬁcacy of our proposed MaxMin Novelty method in terms of sample efﬁciency.
Revised: • We show the efﬁcacy of our proposed MaxMin Novelty method in terms of sample efﬁciency.

Original: Our method based on maximizing novelty via minimizing the state-action value function reaches approximately to the same performance level as model-based deep reinforcement learning algorithms, without building and learning any model of the environment.
Revised: Our method based on maximizing novelty via minimizing the state-action value function reaches approximately to the same performance level as model-based deep reinforcement learning algorithms, without building and learning any model of the environment.

Original:  B ACKGROUND AND P RELIMINARIES 2. D EEP R EINFORCEMENT L EARNING
Revised:  B ACKGROUND AND P RELIMINARIES 2. D EEP R EINFORCEMENT L EARNING

Original: The reinforcement learning problem is formalized as a Markov Decision Process (MDP) M
Revised: The reinforcement learning problem is formalized as a Markov Decision Process (MDP) M

Original: = (cid:104)S , A , r, γ, ρ 0 , P(cid:105) that contains a continous set of states s ∈ S , a set of discrete actions a ∈ A , a probability transition function T ( s, a, s (cid:48) ) on S ×
Revised: = (cid:104)S , A , r, γ, ρ 0 , P(cid:105) that contains a continous set of states s ∈ S , a set of discrete actions a ∈ A , a probability transition function T ( s, a, s (cid:48) ) on S ×

Original: A × S , discount factor γ , a reward function r ( s, a ) : S ×
Revised: A × S , discount factor γ , a reward function r ( s, a ) : S ×

Original: A → R with initial state distribution ρ 0 .
Revised: A → R with initial state distribution ρ 0 .

Original: A policy π ( s, a ) : S → P ( A ) in an MDP is a mapping function between states and actions assigning a probability distribution over actions for each state s ∈ S .
Revised: A policy π ( s, a ) : S → P ( A ) in an MDP is a mapping function between states and actions assigning a probability distribution over actions for each state s ∈ S .

Original: The main goal in reinforcement learning is to learn an optimal policy π that maximizes the discounted expected cumulative discounted rewards.
Revised: The main goal in reinforcement learning is to learn an optimal policy π that maximizes the discounted expected cumulative discounted rewards.

Original: [Equation]
Revised: [Equation]

Original: where a t ∼ π ( s t , · ) .
Revised: where a t ∼ π ( s t , · ) .

Original: In Q -learning the learned policy is parameterized by a state-action value function Q : S ×
Revised: In Q -learning the learned policy is parameterized by a state-action value function Q : S ×

Original: A → R , which represents the value of taking action a in state s .
Revised: A → R , which represents the value of taking action a in state s .

Original: The optimal state-action value function is learnt via iterative Bellman update
Revised: The optimal state-action value function is learnt via iterative Bellman update

Original: [Equation]
Revised: [Equation]

Original: where V ( s t +1 )
Revised: where V ( s t +1 )

Original: = max a Q ( s t +1 , a ) .
Revised: = max a Q ( s t +1 , a ) .

Original: Let a ∗ be the action maximizing the state-action value function, a ∗ ( s )
Revised: Let a ∗ be the action maximizing the state-action value function, a ∗ ( s )

Original: = arg max a Q ( s, a ) , in state s .
Revised: = arg max a Q ( s, a ) , in state s .

Original: Once the Q -function is learnt the policy is determined via taking action a ∗ ( s ) =
Revised: Once the Q -function is learnt the policy is determined via taking action a ∗ ( s ) =

Original: arg max a Q ( s, a ) .
Revised: arg max a Q ( s, a ) .

Original: In deep reinforcement learning, the state space or the action space is large enough that it is not possible to learn and store the state-action values in a tabular form.
Revised: In deep reinforcement learning, the state space or the action space is large enough that it is not possible to learn and store the state-action values in a tabular form.

Original: Thus, the Q -function is approximated via deep neural networks.
Revised: Thus, the Q -function is approximated via deep neural networks.

Original: [Equation]
Revised: [Equation]

Original: In deep double- Q learning, two Q -networks are used to decouple the Q -network deciding which action to take and the Q -network to evaluate the action taken.
Revised: In deep double- Q learning, two Q -networks are used to decouple the Q -network deciding which action to take and the Q -network to evaluate the action taken.

Original: [Equation]
Revised: [Equation]

Original: Q ( s t , a t ; θ t )
Revised: Q ( s t , a t ; θ t )

Original: Current deep reinforcement learning algorithms use (cid:15) -greedy exploration during training (Mnih et al., 2015; Hasselt et al., 2016;
Revised: Current deep reinforcement learning algorithms use (cid:15) -greedy exploration during training (Mnih et al., 2015; Hasselt et al., 2016;

Original: Wang et al., 2016; Hamrick et al., 2020; Flennerhag et al., 2022).
Revised: Wang et al., 2016; Hamrick et al., 2020; Flennerhag et al., 2022).

Original: In particular, the (cid:15) -greedy algorithm takes an action a k ∼ U ( A ) with probability (cid:15) in a given state s , i.e. π ( s, a k )
Revised: In particular, the (cid:15) -greedy algorithm takes an action a k ∼ U ( A ) with probability (cid:15) in a given state s , i.e. π ( s, a k )

Original: = (cid:15) |A| , and takes an action a ∗ = arg max a Q ( s, a ) with probability 1 − (cid:15) , i.e. π ( s, a ∗ ) = 1 − (cid:15) + (cid:15) |A| .
Revised: = (cid:15) |A| , and takes an action a ∗ = arg max a Q ( s, a ) with probability 1 − (cid:15) , i.e. π ( s, a ∗ ) = 1 − (cid:15) + (cid:15) |A| .

Original: 2. E XPLORATION IN R EINFORCEMENT L EARNING
Revised: 2. E XPLORATION IN R EINFORCEMENT L EARNING

Original: In the tabular MDP setting, there has been extensive theoretical work proving optimal regret bounds using the principal of optimism in the face of uncertainty.
Revised: In the tabular MDP setting, there has been extensive theoretical work proving optimal regret bounds using the principal of optimism in the face of uncertainty.

Original: One prominent class of algorithms in this setting utilizes a bonus to value estimates based on the Upper Conﬁdence Bound (UCB) approach (Auer et al., 2008).
Revised: One prominent class of algorithms in this setting utilizes a bonus to value estimates based on the Upper Conﬁdence Bound (UCB) approach (Auer et al., 2008).

Original: In fact, the recent work of Azar et al.
Revised: In fact, the recent work of Azar et al.

Original: (2017) achieves minimax optimal regret using a carefully designed variant of the UCB approach.
Revised: (2017) achieves minimax optimal regret using a carefully designed variant of the UCB approach.

Original: Furthermore, the UCB approach to exploration continues to be an active area of research for deriving new algorithms with provable regret bounds in reinforcement learning (Zanette & Brunskill, 2019; Jin et al., 2020).
Revised: Furthermore, the UCB approach to exploration continues to be an active area of research for deriving new algorithms with provable regret bounds in reinforcement learning (Zanette & Brunskill, 2019; Jin et al., 2020).

Original: The basic idea of
Revised: The basic idea of

Original: UCB algorithms is to explore by adding an optimistic bonus to the state-action values, based on an estimate of the uncertainty in the current value function.
Revised: UCB algorithms is to explore by adding an optimistic bonus to the state-action values, based on an estimate of the uncertainty in the current value function.

Original: The basic UCB approach (Sutton & Barto, 2018) is to use visit-count statistics N t ( s, a ) representing the number of times action a has been taken in state s by time step t in order to estimate the variance of the current state-action values.
Revised: The basic UCB approach (Sutton & Barto, 2018) is to use visit-count statistics N t ( s, a ) representing the number of times action a has been taken in state s by time step t in order to estimate the variance of the current state-action values.

Original: The variance estimate is then used to construct a conﬁdence interval around the current value estimate, usually given by some multiple of the standard deviation c (cid:113) log t N t ( s,a ) .
Revised: The variance estimate is then used to construct a conﬁdence interval around the current value estimate, usually given by some multiple of the standard deviation c (cid:113) log t N t ( s,a ) .

Original: Finally, the action with the highest value for the upper end of its conﬁdence interval is selected.
Revised: Finally, the action with the highest value for the upper end of its conﬁdence interval is selected.

Original: In this sense the UCB algorithm is optimistic, as it chooses an action based on the highest plausible estimate of its value given the previously observed data.
Revised: In this sense the UCB algorithm is optimistic, as it chooses an action based on the highest plausible estimate of its value given the previously observed data.

Original: Note also that as the state action pair ( s, a ) is visited more frequently, the corresponding conﬁdence interval becomes smaller, eventually converging to the ﬁnal estimated value.
Revised: Note also that as the state action pair ( s, a ) is visited more frequently, the corresponding conﬁdence interval becomes smaller, eventually converging to the ﬁnal estimated value.

Original: A second general class of theoretically-justiﬁed algorithms for exploration is based on randomized value functions, where speciﬁcally tuned randomness is added to value estimates in order to encourage exploration.
Revised: A second general class of theoretically-justiﬁed algorithms for exploration is based on randomized value functions, where speciﬁcally tuned randomness is added to value estimates in order to encourage exploration.

Original: Notable examples of algorithms in this category include Thompson sampling (Osband et al., 2013; Agrawal & Jia, 2017), based on sampling from a posterior distribution on actions given past observations, and randomized least-squares value iteration (RLSVI)
Revised: Notable examples of algorithms in this category include Thompson sampling (Osband et al., 2013; Agrawal & Jia, 2017), based on sampling from a posterior distribution on actions given past observations, and randomized least-squares value iteration (RLSVI)

Original: (Osband et al., 2016), based on using tuned Gaussian noise to sample a randomized value function.
Revised: (Osband et al., 2016), based on using tuned Gaussian noise to sample a randomized value function.

Original: Despite the strong theoretical performance of the aforementioned approaches, there are signiﬁcant difﬁculties in effectively extending to the setting of deep reinforcement learning.
Revised: Despite the strong theoretical performance of the aforementioned approaches, there are signiﬁcant difﬁculties in effectively extending to the setting of deep reinforcement learning.

Original: The primary obstacle is that these methods utilize count-based uncertainty estimates (e.g. the state-action visit counts
Revised: The primary obstacle is that these methods utilize count-based uncertainty estimates (e.g. the state-action visit counts

Original: N t ( s, a ) ), which are generally not immediately available in deep reinforcement learning where the state space is modeled as a continuous high-dimensional vector space (e.g. in deep reinforcement learning from pixels).
Revised: N t ( s, a ) ), which are generally not immediately available in deep reinforcement learning where the state space is modeled as a continuous high-dimensional vector space (e.g. in deep reinforcement learning from pixels).

Original: Instead, incorporating count-based methods into deep reinforcement learning requires signiﬁcant complexity including training additional deep neural networks to estimate counts or other uncertainty metrics.
Revised: Instead, incorporating count-based methods into deep reinforcement learning requires signiﬁcant complexity including training additional deep neural networks to estimate counts or other uncertainty metrics.

Original: As a result, many state-of-the-art deep reinforcement learning algorithms use simple, randomized exploration methods such as the (cid:15) -greedy approach of sampling a uniformly random action with probability (cid:15) (Mnih et al., 2015;
Revised: As a result, many state-of-the-art deep reinforcement learning algorithms use simple, randomized exploration methods such as the (cid:15) -greedy approach of sampling a uniformly random action with probability (cid:15) (Mnih et al., 2015;

Original: Hasselt et al., 2016;
Revised: Hasselt et al., 2016;

Original: Wang et al., 2016; Hamrick et al., 2020; Flennerhag et al., 2022), or the injection of random noise via noisynetworks (Hessel et al., 2018).
Revised: Wang et al., 2016; Hamrick et al., 2020; Flennerhag et al., 2022), or the injection of random noise via noisynetworks (Hessel et al., 2018).

Original:  M AXIMIZING N OVELTY
Revised:  M AXIMIZING N OVELTY

Original: In deep reinforcement learning the state-action value function is initialized with random weights (Mnih et al., 2015; 2016;
Revised: In deep reinforcement learning the state-action value function is initialized with random weights (Mnih et al., 2015; 2016;

Original: Hasselt et al., 2016;
Revised: Hasselt et al., 2016;

Original: Wang et al., 2016; Schaul et al., 2016; Oh et al., 2020; Schrittwieser et al., 2020; Hubert et al., 2021).
Revised: Wang et al., 2016; Schaul et al., 2016; Oh et al., 2020; Schrittwieser et al., 2020; Hubert et al., 2021).

Original: Thus, in the early phase of the training the
Revised: Thus, in the early phase of the training the

Original: Q -function will behave more like a random function rather than providing an accurate representation of the optimal state-action values.
Revised: Q -function will behave more like a random function rather than providing an accurate representation of the optimal state-action values.

Original: In particular, early in training the Q -function, on average, will assign approximately similar values to states that are similar, and will have little correlation with the immediate rewards.
Revised: In particular, early in training the Q -function, on average, will assign approximately similar values to states that are similar, and will have little correlation with the immediate rewards.

Original: We ﬁrst formalize this intuition in the following deﬁnitions.
Revised: We ﬁrst formalize this intuition in the following deﬁnitions.

Original: Deﬁnition 3.1 ( η -uninformed Q ) .
Revised: Deﬁnition 3.1 ( η -uninformed Q ) .

Original: Let η > 0 .
Revised: Let η > 0 .

Original: A Q -function parameterized by weights θ ∼ Θ is η -uninformed if for any state s ∈ S with a min = arg min a Q θ ( s, a ) we have
Revised: A Q -function parameterized by weights θ ∼ Θ is η -uninformed if for any state s ∈ S with a min = arg min a Q θ ( s, a ) we have

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3.2 ( δ -smooth Q ) .
Revised: Deﬁnition 3.2 ( δ -smooth Q ) .

Original: Let δ > 0 .
Revised: Let δ > 0 .

Original: A Q -function parameterized by weights θ ∼ Θ is δ -smooth if for any state s ∈ S and action ˆ a ∈ A with s (cid:48) ∼ T ( s, ˆ a, · ) we have
Revised: A Q -function parameterized by weights θ ∼ Θ is δ -smooth if for any state s ∈ S and action ˆ a ∈ A with s (cid:48) ∼ T ( s, ˆ a, · ) we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over both the random initialization of the Q -function weights, and the random transition to state s (cid:48) ∼ T ( s, ˆ a, · ) .
Revised: where the expectation is over both the random initialization of the Q -function weights, and the random transition to state s (cid:48) ∼ T ( s, ˆ a, · ) .

Original: Deﬁnition 3.3 (Disadvantage Gap) .
Revised: Deﬁnition 3.3 (Disadvantage Gap) .

Original: For a state-action value function Q θ the disadvantage gap in a state s ∈ S is given by
Revised: For a state-action value function Q θ the disadvantage gap in a state s ∈ S is given by

Original: [Equation]
Revised: [Equation]

Original: where a min = arg min a Q θ ( s, a ) .
Revised: where a min = arg min a Q θ ( s, a ) .

Original: The following proposition captures the intuition that when the Q -function on average assigns similar maximum values to consecutive states, choosing the action minimizing the state-action value function will achieve an above-average temporal difference loss.
Revised: The following proposition captures the intuition that when the Q -function on average assigns similar maximum values to consecutive states, choosing the action minimizing the state-action value function will achieve an above-average temporal difference loss.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Let η, δ > 0 and suppose that Q θ ( s, a ) is η -uninformed and δ -smooth.
Revised: Let η, δ > 0 and suppose that Q θ ( s, a ) is η -uninformed and δ -smooth.

Original: Let s t ∈ S be a state, and let a min be the action minimizing the state-action value in a given state s t , a min = arg min a Q θ ( s t , a ) .
Revised: Let s t ∈ S be a state, and let a min be the action minimizing the state-action value in a given state s t , a min = arg min a Q θ ( s t , a ) .

Original: Let s min t +1 ∼ T ( s t , a min , · ) .
Revised: Let s min t +1 ∼ T ( s t , a min , · ) .

Original: Then for an action a t ∼ U ( A ) with s t +1 ∼ T ( s t , a t , · ) we have
Revised: Then for an action a t ∼ U ( A ) with s t +1 ∼ T ( s t , a t , · ) we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since Q θ ( s, a ) is δ -smooth
Revised: Since Q θ ( s, a ) is δ -smooth

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last line follows from Deﬁnition 3.3.
Revised: where the last line follows from Deﬁnition 3.3.

Original: Further, because Q θ ( s, a ) is η -uninformed,
Revised: Further, because Q θ ( s, a ) is η -uninformed,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining with the previous inequality completes the proof.
Revised: Combining with the previous inequality completes the proof.

Original: In words, the proposition shows that the temporal difference loss achieved by the minimum-value action is above-average by an amount approximately equal to the disadvantage gap.
Revised: In words, the proposition shows that the temporal difference loss achieved by the minimum-value action is above-average by an amount approximately equal to the disadvantage gap.

Original: The above argument can be extended to the case where action selection and evaluation in the temporal difference loss are computed with two different sets of weights θ and ˆ θ as in double Q -learning.
Revised: The above argument can be extended to the case where action selection and evaluation in the temporal difference loss are computed with two different sets of weights θ and ˆ θ as in double Q -learning.

Original: Deﬁnition 3.4 ( δ -smoothness for Double- Q ) .
Revised: Deﬁnition 3.4 ( δ -smoothness for Double- Q ) .

Original: Let δ > 0 .
Revised: Let δ > 0 .

Original: A pair of Q -functions parameterized by weights θ ∼ Θ and ˆ θ ∼ Θ are δ -smooth if for any state s ∈ S and action ˆ a ∈ A with s (cid:48) ∼ T ( s, ˆ a, · ) we have
Revised: A pair of Q -functions parameterized by weights θ ∼ Θ and ˆ θ ∼ Θ are δ -smooth if for any state s ∈ S and action ˆ a ∈ A with s (cid:48) ∼ T ( s, ˆ a, · ) we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over both the random initialization of the Q -function weights θ and ˆ θ , and the random transition to state s (cid:48) ∼ T ( s, ˆ a, · ) .
Revised: where the expectation is over both the random initialization of the Q -function weights θ and ˆ θ , and the random transition to state s (cid:48) ∼ T ( s, ˆ a, · ) .

Original: With this deﬁnition we can then prove that choosing the minimum valued action will lead to a temporal difference loss that is above-average by approximately D ( s ) .
Revised: With this deﬁnition we can then prove that choosing the minimum valued action will lead to a temporal difference loss that is above-average by approximately D ( s ) .

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Let η, δ > 0 and suppose that Q θ and Q ˆ θ are η -uniformed and δ -smooth.
Revised: Let η, δ > 0 and suppose that Q θ and Q ˆ θ are η -uniformed and δ -smooth.

Original: Let s t ∈ S be a state, and let a min = arg min a Q θ ( s t , a ) .
Revised: Let s t ∈ S be a state, and let a min = arg min a Q θ ( s t , a ) .

Original: Let s min t +1 ∼ T ( s t , a min , · ) .
Revised: Let s min t +1 ∼ T ( s t , a min , · ) .

Original: Then for an action a t ∼ U ( A ) with s t +1 ∼ T ( s t , a t , · ) we have
Revised: Then for an action a t ∼ U ( A ) with s t +1 ∼ T ( s t , a t , · ) we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since Q θ and Q ˆ θ are δ -smooth
Revised: Since Q θ and Q ˆ θ are δ -smooth

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last line follows from Deﬁnition 3.3.
Revised: where the last line follows from Deﬁnition 3.3.

Original: Further, because Q θ and Q ˆ θ are η -uniformed,
Revised: Further, because Q θ and Q ˆ θ are η -uniformed,

Original: [Equation]
Revised: [Equation]

Original: Combining with the previous inequality completes the proof.
Revised: Combining with the previous inequality completes the proof.

Original: At ﬁrst, the results in Proposition 3.1 and 3.2 might appear counterintuitive.
Revised: At ﬁrst, the results in Proposition 3.1 and 3.2 might appear counterintuitive.

Original: The fact that the Qfunction is δ -smooth and η -uninformed seem like properties of a random function.
Revised: The fact that the Qfunction is δ -smooth and η -uninformed seem like properties of a random function.

Original: Thus, taking the minimum Q -value action should be approximately equivalent to taking a uniform random action.
Revised: Thus, taking the minimum Q -value action should be approximately equivalent to taking a uniform random action.

Original: However, Proposition 3.1 and 3.2 show that the temporal difference loss achieved by taking the minimum action is larger than that of a random action by an amount equal to the disadvantage gap
Revised: However, Proposition 3.1 and 3.2 show that the temporal difference loss achieved by taking the minimum action is larger than that of a random action by an amount equal to the disadvantage gap

Original: D ( s ) .
Revised: D ( s ) .

Original: In order to reconcile these two statements it is useful at this point to look at the limiting case of the Q function at initialization.
Revised: In order to reconcile these two statements it is useful at this point to look at the limiting case of the Q function at initialization.

Original: In particular, the following proposition shows that, at initialization, the distribution of the minimum value action in a given state is uniform by itself, but is constant once we condition on the weights θ .
Revised: In particular, the following proposition shows that, at initialization, the distribution of the minimum value action in a given state is uniform by itself, but is constant once we condition on the weights θ .

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Let θ be the random initial weights for the Q -function.
Revised: Let θ be the random initial weights for the Q -function.

Original: For any state s ∈ S let a min
Revised: For any state s ∈ S let a min

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: Q θ
Revised: Q θ

Original: [Equation]
Revised: [Equation]

Original: s, a
Revised: s, a

Original: [Equation]
Revised: [Equation]

Original: Then for any a ∈ A
Revised: Then for any a ∈ A

Original: [Equation]
Revised: [Equation]

Original: i.e. the distribution P θ ∼ Θ [ a min ( s )] is uniform.
Revised: i.e. the distribution P θ ∼ Θ [ a min ( s )] is uniform.

Original: Simultaneously, the conditional distribution P
Revised: Simultaneously, the conditional distribution P

Original: [Equation]
Revised: [Equation]

Original: ) | θ ] is constant.
Revised: ) | θ ] is constant.

Original: Proof.
Revised: Proof.

Original: Since Q θ ( s, · ) is a random function (given the random choice of θ ), each action a ∈ A is equally likely to be assigned the minimum Q -value in state s .
Revised: Since Q θ ( s, · ) is a random function (given the random choice of θ ), each action a ∈ A is equally likely to be assigned the minimum Q -value in state s .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: However, given the value of θ , the value of a min ( s ) is uniquely determined because
Revised: However, given the value of θ , the value of a min ( s ) is uniquely determined because

Original: [Equation]
Revised: [Equation]

Original: Therefore, the distribution of a min ( s ) conditional on θ is constant.
Revised: Therefore, the distribution of a min ( s ) conditional on θ is constant.

Original: Input:
Revised: Input:

Original: In MDP M with γ ∈ (0 , 1] , s ∈ S , a ∈ A with Q ( s, a ) , B experience replay buffer, (cid:15) exploration parameter,
Revised: In MDP M with γ ∈ (0 , 1] , s ∈ S , a ∈ A with Q ( s, a ) , B experience replay buffer, (cid:15) exploration parameter,

Original: N is the training learning steps.
Revised: N is the training learning steps.

Original: Populating Experience Replay Buffer: for s t in e do
Revised: Populating Experience Replay Buffer: for s t in e do

Original: Sample κ ∼ U (0 , 1) if κ < (cid:15) then a min = arg min a Q ( s t , a )
Revised: Sample κ ∼ U (0 , 1) if κ < (cid:15) then a min = arg min a Q ( s t , a )

Original: B ← ( r ( s t , a min ) , s t , s min t +1 , a min ) else a ∗ =
Revised: B ← ( r ( s t , a min ) , s t , s min t +1 , a min ) else a ∗ =

Original: arg max a Q ( s t , a )
Revised: arg max a Q ( s t , a )

Original: B ← ( r ( s t , a ∗ ) , s t , s t +1 , a ∗ ) end if end for
Revised: B ← ( r ( s t , a ∗ ) , s t , s t +1 , a ∗ ) end if end for

Original: Learning: for n in N do
Revised: Learning: for n in N do

Original: Update with probability (cid:15) :
Revised: Update with probability (cid:15) :

Original: T D = r
Revised: T D = r

Original: ( s t , a min ) + γ max a Q ( s min t +1 , a ) − Q ( s t , a min )
Revised: ( s t , a min ) + γ max a Q ( s min t +1 , a ) − Q ( s t , a min )

Original: Update with probability 1 − (cid:15) :
Revised: Update with probability 1 − (cid:15) :

Original: T D = r
Revised: T D = r

Original: ( s t , a ∗ )
Revised: ( s t , a ∗ )

Original: + γ max a Q ( s t +1 , a ) − Q ( s t , a ∗ ) end for return
Revised: + γ max a Q ( s t +1 , a ) − Q ( s t , a ∗ ) end for return

Original: ∇L ( T D )
Revised: ∇L ( T D )

Original: This implies that, in states whose Q -values have not changed much from initialization, taking the minimum action is almost equivalent to taking a random action.
Revised: This implies that, in states whose Q -values have not changed much from initialization, taking the minimum action is almost equivalent to taking a random action.

Original: However, while the action chosen early on in training is almost uniformly random when only considering the current state, it is at the same time completely determined by the current value of the weights θ .
Revised: However, while the action chosen early on in training is almost uniformly random when only considering the current state, it is at the same time completely determined by the current value of the weights θ .

Original: The temporal difference loss is also determined by the weights θ .
Revised: The temporal difference loss is also determined by the weights θ .

Original: Thus while the marginal distribution on actions taken is uniform, the temporal difference loss when taking the minimum action is quite different than from the case where an independently random action is chosen.
Revised: Thus while the marginal distribution on actions taken is uniform, the temporal difference loss when taking the minimum action is quite different than from the case where an independently random action is chosen.

Original: Algorithm 1 summarizes our proposed exploration method MaxMin Novelty based on minimizing the state-action value function as described in detail in Section 3.
Revised: Algorithm 1 summarizes our proposed exploration method MaxMin Novelty based on minimizing the state-action value function as described in detail in Section 3.

Original: Note that populating the experience replay buffer and learning are happening simultaneously with different rates.
Revised: Note that populating the experience replay buffer and learning are happening simultaneously with different rates.

Original:  M OTIVATING E XAMPLE
Revised:  M OTIVATING E XAMPLE

Original: method, (cid:15) -greedy and our proposed method MaxMin Novelty.
Revised: method, (cid:15) -greedy and our proposed method MaxMin Novelty.

Original: As a motivating example we consider the chain MDP which consists of a chain of n states s ∈ S = { 1 , 2 , · · · n } each with two actions.
Revised: As a motivating example we consider the chain MDP which consists of a chain of n states s ∈ S = { 1 , 2 , · · · n } each with two actions.

Original: Each state i has one action that transitions the agent up the chain by one step to state i +1 , and one action which resets the agent to state 1 at the beginning of the chain.
Revised: Each state i has one action that transitions the agent up the chain by one step to state i +1 , and one action which resets the agent to state 1 at the beginning of the chain.

Original: All transitions have reward zero, except for the last transition returning the agent to the beginning from the nth state.
Revised: All transitions have reward zero, except for the last transition returning the agent to the beginning from the nth state.

Original: Thus, when started from the ﬁrst state in the chain, the agent must learn a policy that takes n − 1 consecutive steps up the chain, and then the one ﬁnal step to reset and get the reward.
Revised: Thus, when started from the ﬁrst state in the chain, the agent must learn a policy that takes n − 1 consecutive steps up the chain, and then the one ﬁnal step to reset and get the reward.

Original: For the chain MDP, we compare standard approaches to exploration in tabular Q -learning with our method MaxMin
Revised: For the chain MDP, we compare standard approaches to exploration in tabular Q -learning with our method MaxMin

Original: Novelty based on minimization of the state-action values.
Revised: Novelty based on minimization of the state-action values.

Original: In particular we compare our method MaxMin Novelty with both the (cid:15) -greedy action selection method, and the upper conﬁdence bound (UCB) method.
Revised: In particular we compare our method MaxMin Novelty with both the (cid:15) -greedy action selection method, and the upper conﬁdence bound (UCB) method.

Original: In more detail, in the UCB method the number of training steps t , and the number of times N t ( s, a ) that each action a has been taken in state s by step t are recorded.
Revised: In more detail, in the UCB method the number of training steps t , and the number of times N t ( s, a ) that each action a has been taken in state s by step t are recorded.

Original: Furthermore, the action a ∈
Revised: Furthermore, the action a ∈

Original: A selection is determined as follows:
Revised: A selection is determined as follows:

Original: [Equation]
Revised: [Equation]

Original: In a given state s if N ( s, a ) = 0 for any action a , then an action is sampled uniformly at random from the set of actions a (cid:48) with N ( s, a (cid:48) )
Revised: In a given state s if N ( s, a ) = 0 for any action a , then an action is sampled uniformly at random from the set of actions a (cid:48) with N ( s, a (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: For the experiments reported in our paper the length of the chain is set to n = 10 , and (cid:15) = 0 .
Revised: For the experiments reported in our paper the length of the chain is set to n = 10 , and (cid:15) = 0 .

Original: The Q -function is initialized by independently sampling each state-action value from a normal distribution with µ = 0 and σ = 0 .
Revised: The Q -function is initialized by independently sampling each state-action value from a normal distribution with µ = 0 and σ = 0 .

Original: In each iteration we
Revised: In each iteration we

Original: [Figure]
Revised: [Figure]

Original: train the agent using Q -learning for 100 steps, and then evaluate the reward obtained by the argmax policy using the current Q -function for 100 steps.
Revised: train the agent using Q -learning for 100 steps, and then evaluate the reward obtained by the argmax policy using the current Q -function for 100 steps.

Original: Note that the maximum achievable reward insteps is 10.
Revised: Note that the maximum achievable reward insteps is 10.

Original: The learning curves in Figure 1 demonstrate that our method converges more quickly to the optimal policy than either of the standard approaches.
Revised: The learning curves in Figure 1 demonstrate that our method converges more quickly to the optimal policy than either of the standard approaches.

Original:  L ARGE S CALE E XPERIMENTAL R ESULTS
Revised:  L ARGE S CALE E XPERIMENTAL R ESULTS

Original: The experiments are conducted in the Arcade Learning Environment (ALE)
Revised: The experiments are conducted in the Arcade Learning Environment (ALE)

Original: (Bellemare et al., 2013).
Revised: (Bellemare et al., 2013).

Original: The Double- Q Network (Hasselt et al., 2016) initially proposed by (van Hasselt, 2010) is trained with prioritized experience replay (Schaul et al., 2016) without the dueling architecture with its original version (Hasselt et al., 2016).
Revised: The Double- Q Network (Hasselt et al., 2016) initially proposed by (van Hasselt, 2010) is trained with prioritized experience replay (Schaul et al., 2016) without the dueling architecture with its original version (Hasselt et al., 2016).

Original: The experiments are conducted both in the 100K Arcade
Revised: The experiments are conducted both in the 100K Arcade

Original: Learning Environment benchmark (van Hasselt et al., 2019), and the canonical version withmillion frame training.
Revised: Learning Environment benchmark (van Hasselt et al., 2019), and the canonical version withmillion frame training.

Original: Note that the 100K Arcade Learning Environment benchmark is an established baseline proposed to measure sample efﬁciency in deep reinforcement learning research.
Revised: Note that the 100K Arcade Learning Environment benchmark is an established baseline proposed to measure sample efﬁciency in deep reinforcement learning research.

Original: The
Revised: The

Original: ALE 100K benchmark contains 26 different Arcade Learning Environment games.
Revised: ALE 100K benchmark contains 26 different Arcade Learning Environment games.

Original: The policies are evaluated after 100000 environment interactions.
Revised: The policies are evaluated after 100000 environment interactions.

Original: All of the polices in the experiments are trained over 5 random seeds.
Revised: All of the polices in the experiments are trained over 5 random seeds.

Original: The hyperparameters and the architecture details are reported in the appendix.
Revised: The hyperparameters and the architecture details are reported in the appendix.

Original: All of the results in the paper are reported with the standard error of the mean.
Revised: All of the results in the paper are reported with the standard error of the mean.

Original: The human normalized scores are computed as,
Revised: The human normalized scores are computed as,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: For completeness we also report several results with 200 million frame training (i
Revised: For completeness we also report several results with 200 million frame training (i

Original: [Table]
Revised: [Table]

Original: e. 50 million environment interactions).
Revised: e. 50 million environment interactions).

Original: In particular, Figure 2 demonstrates the learning curves for our proposed algorithm MaxMin Novelty and the original version of the DDQN algorithm with (cid:15) -greedy training (Hasselt et al., 2016).
Revised: In particular, Figure 2 demonstrates the learning curves for our proposed algorithm MaxMin Novelty and the original version of the DDQN algorithm with (cid:15) -greedy training (Hasselt et al., 2016).

Original: In the large data regime we observe that while in some MDPs our proposed method MaxMin Novelty based on exploring with novelty maximization via minimizing the stateaction values converges faster, in other MDPs MaxMin Novelty simply converges to a better policy.
Revised: In the large data regime we observe that while in some MDPs our proposed method MaxMin Novelty based on exploring with novelty maximization via minimizing the stateaction values converges faster, in other MDPs MaxMin Novelty simply converges to a better policy.

Original: More concretely, while the learning curves of StarGunner, FishingDerby, Boxing, Enduro, Hero, and IceHockey games in Figure 2 demonstrate the faster convergence rate of our proposed algorithm MaxMin Novelty, the learning curves of the BankHeist, JamesBond, KungFuMaster, Amidar,
Revised: More concretely, while the learning curves of StarGunner, FishingDerby, Boxing, Enduro, Hero, and IceHockey games in Figure 2 demonstrate the faster convergence rate of our proposed algorithm MaxMin Novelty, the learning curves of the BankHeist, JamesBond, KungFuMaster, Amidar,

Original: Gravitar and Tennis games demonstrate that our exploration technique not only increases the sample efﬁciency in deep reinforcement learning, but also results in learning a policy that is more close to optimal compared to learning a policy with the original method used in the DDQN algorithm.
Revised: Gravitar and Tennis games demonstrate that our exploration technique not only increases the sample efﬁciency in deep reinforcement learning, but also results in learning a policy that is more close to optimal compared to learning a policy with the original method used in the DDQN algorithm.

Original: Additionally, we also compare our proposed MaxMin Novelty algorithm with NoisyNetworks as described in Section 2.2.
Revised: Additionally, we also compare our proposed MaxMin Novelty algorithm with NoisyNetworks as described in Section 2.2.

Original: Table 1 further demonstrates that the MaxMin Novelty algorithm achieves signiﬁcantly better performance results compared to NoisyNetworks.
Revised: Table 1 further demonstrates that the MaxMin Novelty algorithm achieves signiﬁcantly better performance results compared to NoisyNetworks.

Original: Furthermore, note that NoisyNetworks includes adding layers in the Q -network to increase exploration.
Revised: Furthermore, note that NoisyNetworks includes adding layers in the Q -network to increase exploration.

Original: However, this increases the number of parameters that have been added in the training process; thus, introducing additional cost to increase exploration.
Revised: However, this increases the number of parameters that have been added in the training process; thus, introducing additional cost to increase exploration.

Original: Table 1 reports results of human normalized median scores, 20 th percentile, and 80 th percentile for the Arcade Learning Environment 100K benchmark.
Revised: Table 1 reports results of human normalized median scores, 20 th percentile, and 80 th percentile for the Arcade Learning Environment 100K benchmark.

Original: Thus,
Revised: Thus,

Original: Table 1 demonstrates that our proposed MaxMin-Novelty algorithm improves on the performance of the canonical algorithm (cid:15) -greedy by 248% and NoisyNetworks by 204%.
Revised: Table 1 demonstrates that our proposed MaxMin-Novelty algorithm improves on the performance of the canonical algorithm (cid:15) -greedy by 248% and NoisyNetworks by 204%.

Original:  I NVESTIGATING THE T EMPORAL D IFFERENCE L OSS
Revised:  I NVESTIGATING THE T EMPORAL D IFFERENCE L OSS

Original: The original justiﬁcation for exploring with the minimum Q -value action, is that taking this action tends to result in transitions with higher temporal difference loss.
Revised: The original justiﬁcation for exploring with the minimum Q -value action, is that taking this action tends to result in transitions with higher temporal difference loss.

Original: The theoretical analysis from Proposition 3.1 indicates that, when the Q function is δ -smooth and η -uninformed, taking the minimum value action results in an increase in the temporal difference loss proportional to the disadvantage gap.
Revised: The theoretical analysis from Proposition 3.1 indicates that, when the Q function is δ -smooth and η -uninformed, taking the minimum value action results in an increase in the temporal difference loss proportional to the disadvantage gap.

Original: In particular, Proposition 3.1 states that the temporal difference loss achieved when taking the minimum Q -value action in state s exceeds the average loss over a uniform random action by D ( s ) − 

Revised: In particular, Proposition 3.1 states that the temporal difference loss achieved when taking the minimum Q -value action in state s exceeds the average loss over a uniform random action by D ( s ) −

Original: δ − η .
Revised: δ − η .

Original: In order to evaluate how well the theoretical prediction matches reality, in this section we provide empirical measurements of the temporal difference loss in our experiments.
Revised: In order to evaluate how well the theoretical prediction matches reality, in this section we provide empirical measurements of the temporal difference loss in our experiments.

Original: To measure the change in the loss when taking the minimum action versus the average action, we compare the temporal difference loss obtained by MaxMin Novelty exploration with that obtained by (cid:15) -greedy exploration.
Revised: To measure the change in the loss when taking the minimum action versus the average action, we compare the temporal difference loss obtained by MaxMin Novelty exploration with that obtained by (cid:15) -greedy exploration.

Original: In more detail, during training, for each batch Λ of transitions of the form ( s t , a t , s t +1 )
Revised: In more detail, during training, for each batch Λ of transitions of the form ( s t , a t , s t +1 )

Original: we record, the temporal difference loss
Revised: we record, the temporal difference loss

Original: [Equation]
Revised: [Equation]

Original: The results reported in Figure 3 and Figure 5 further conﬁrm the theoretical predictions made via
Revised: The results reported in Figure 3 and Figure 5 further conﬁrm the theoretical predictions made via

Original: Deﬁnition 3.2 and Proposition 3.1.
Revised: Deﬁnition 3.2 and Proposition 3.1.

Original: In addition to the results for individual games reported in Figure 3, we compute a normalized measure of the gain in temporal difference achieved when using
Revised: In addition to the results for individual games reported in Figure 3, we compute a normalized measure of the gain in temporal difference achieved when using

Original: MaxMin Novelty exploration and plot the median across games.
Revised: MaxMin Novelty exploration and plot the median across games.

Original: We deﬁne the normalized T D gain to be
Revised: We deﬁne the normalized T D gain to be

Original: [Equation]
Revised: [Equation]

Original: where T D method and T D (cid:15) -greedy are the temporal difference for any given exploration method and (cid:15) -greedy respectively.
Revised: where T D method and T D (cid:15) -greedy are the temporal difference for any given exploration method and (cid:15) -greedy respectively.

Original: The leftmost and middle plot of Figure 5 report the median across all games of the normalized T D gain results for MaxMin Novelty and NoisyNetworks in the Arcade Learning
Revised: The leftmost and middle plot of Figure 5 report the median across all games of the normalized T D gain results for MaxMin Novelty and NoisyNetworks in the Arcade Learning

Original: Environment 100K benchmark.
Revised: Environment 100K benchmark.

Original: Note that, consistent with the predictions of Proposition 3.1, the median normalized temporal difference gain for MaxMin Novelty is up to 25 percent larger than that of (cid:15) -greedy.
Revised: Note that, consistent with the predictions of Proposition 3.1, the median normalized temporal difference gain for MaxMin Novelty is up to 25 percent larger than that of (cid:15) -greedy.

Original: The results for NoisyNetworks demonstrate that alternate exploration methods lack this positive bias relative to the uniform random action.
Revised: The results for NoisyNetworks demonstrate that alternate exploration methods lack this positive bias relative to the uniform random action.

Original: The fact that, as demonstrated in
Revised: The fact that, as demonstrated in

Original: Table 1, MaxMin Novelty signiﬁcantly outperforms noisy networks in the low-data regime is further evidence of the advantage the positive bias in temporal difference confers.
Revised: Table 1, MaxMin Novelty signiﬁcantly outperforms noisy networks in the low-data regime is further evidence of the advantage the positive bias in temporal difference confers.

Original: The rightmost plot of
Revised: The rightmost plot of

Original: Figure 5 reports T D for the motivating example of the chain MDP.
Revised: Figure 5 reports T D for the motivating example of the chain MDP.

Original: As in the large-scale experiments, prior to convergence MaxMin Novelty exhibits a notably larger temporal difference loss relative to the other exploration methods.
Revised: As in the large-scale experiments, prior to convergence MaxMin Novelty exhibits a notably larger temporal difference loss relative to the other exploration methods.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In our study we focus on the following questions in deep reinforcement learning: (i) Is it possible to increase sample efﬁciency in deep reinforcement learning in a computationally efﬁcient way with conceptually simple choices?
Revised: In our study we focus on the following questions in deep reinforcement learning: (i) Is it possible to increase sample efﬁciency in deep reinforcement learning in a computationally efﬁcient way with conceptually simple choices?

Original: , (ii) What is the theoretical motivation of our proposed perspective, simply minimizing the state-action value function in early training, that results in one of the most computational efﬁcient ways to explore in deep reinforcement learning?
Revised: , (ii) What is the theoretical motivation of our proposed perspective, simply minimizing the state-action value function in early training, that results in one of the most computational efﬁcient ways to explore in deep reinforcement learning?

Original: and, (iii) How would the theoretically motivated simple idea transfer to large scale experiments in high-dimensional state representation MDPs?
Revised: and, (iii) How would the theoretically motivated simple idea transfer to large scale experiments in high-dimensional state representation MDPs?

Original: To be able to answer these questions we propose a novel, theoretically motivated method with zero additional computational cost based on following actions that minimize the state-action value function to explore in deep reinforcement learning.
Revised: To be able to answer these questions we propose a novel, theoretically motivated method with zero additional computational cost based on following actions that minimize the state-action value function to explore in deep reinforcement learning.

Original: We demonstrate theoretically that our method MaxMin Novelty based on minimization of the state-action value results in higher temporal difference loss, and thus creates novel transitions in exploration with more unique experience collection.
Revised: We demonstrate theoretically that our method MaxMin Novelty based on minimization of the state-action value results in higher temporal difference loss, and thus creates novel transitions in exploration with more unique experience collection.

Original: Following the theoretical motivation we initially show in a toy example in the chain MDP setup that our proposed method MaxMin Novelty results in achieving higher sample efﬁciency.
Revised: Following the theoretical motivation we initially show in a toy example in the chain MDP setup that our proposed method MaxMin Novelty results in achieving higher sample efﬁciency.

Original: Then, we expand this intuition and conduct large scale experiments in the Arcade
Revised: Then, we expand this intuition and conduct large scale experiments in the Arcade

Original: Learning Environment, and demonstrate that our proposed method MaxMin Novelty increases the performance on the Arcade Learning Environment 100K benchmark by 248% .
Revised: Learning Environment, and demonstrate that our proposed method MaxMin Novelty increases the performance on the Arcade Learning Environment 100K benchmark by 248% .

Original: A A PPENDIX A. H YPERPARAMETER AND A RCHITECTURE D ETAILS
Revised: A A PPENDIX A. H YPERPARAMETER AND A RCHITECTURE D ETAILS

Original: For reproducibility and completeness in research in Table 2 we report the hyperparameter details for our proposed algorithm MaxMin-Novelty, canonical algorithm (cid:15) -greedy and NoisyNetworks.
Revised: For reproducibility and completeness in research in Table 2 we report the hyperparameter details for our proposed algorithm MaxMin-Novelty, canonical algorithm (cid:15) -greedy and NoisyNetworks.

Original: Furthermore, for all of the algorithms the hyperparameters and the architectures are identical with each other.
Revised: Furthermore, for all of the algorithms the hyperparameters and the architectures are identical with each other.

Original: Note that the architecture parameters are also identical for the 200 million frame training.
Revised: Note that the architecture parameters are also identical for the 200 million frame training.

Original: Note that we did not tune hyperparameters reported below.
Revised: Note that we did not tune hyperparameters reported below.

Original: To increase transparency in research we kept hyperparameters exactly the same with the prior studies.
Revised: To increase transparency in research we kept hyperparameters exactly the same with the prior studies.

Original: We ran our experiments with JAX implementation Bradbury et al.
Revised: We ran our experiments with JAX implementation Bradbury et al.

Original: We used Haiku Hennigan et al.
Revised: We used Haiku Hennigan et al.

Original: (2020) for the neural network library, Optax Hessel et al.
Revised: (2020) for the neural network library, Optax Hessel et al.

Original: (2020) for the optimization library, and RLax for the reinforcement learning library Babuschkin et al.
Revised: (2020) for the optimization library, and RLax for the reinforcement learning library Babuschkin et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Nesterov Momentum
Revised: Nesterov Momentum

Original: True
Revised: True

Original: Hardware
Revised: Hardware

Original: [Table]
Revised: [Table]

Original: NoisyNetwork parameter
Revised: NoisyNetwork parameter

Original: [Table]
Revised: [Table]

Original: Q -Network channels 32,64,
Revised: Q -Network channels 32,64,

Original: Q -Network ﬁlter size 8 × 8 , 4 × 4 , 3 ×
Revised: Q -Network ﬁlter size 8 × 8 , 4 × 4 , 3 ×

Original: Q -Network stride (4 , 4) , (2 , 2) , (1 , 1)
Revised: Q -Network stride (4 , 4) , (2 , 2) , (1 , 1)

Original: Q -Network hidden units
Revised: Q -Network hidden units

Original: A. A RCADE L EARNING E NVIRONMENT R ESULTS
Revised: A. A RCADE L EARNING E NVIRONMENT R ESULTS

Original: Table 3 reports the average scores for human, random, our proposed algorithm MaxMin-Novelty, canonical algorithm (cid:15) -greedy and NoisyNetworks for all of the games in the Arcade Learning Environment 100K benchmark.
Revised: Table 3 reports the average scores for human, random, our proposed algorithm MaxMin-Novelty, canonical algorithm (cid:15) -greedy and NoisyNetworks for all of the games in the Arcade Learning Environment 100K benchmark.

Original: Scores are reported with the mean over 5 random seeds.
Revised: Scores are reported with the mean over 5 random seeds.

Original: The highest score amongst the three algorithms is marked with bold font.
Revised: The highest score amongst the three algorithms is marked with bold font.

Original: We also reported human scores and random scores to provide complete information on the learning curves reported in the main body of the paper.
Revised: We also reported human scores and random scores to provide complete information on the learning curves reported in the main body of the paper.

Original: [Table]
Revised: [Table]

Original: FrostBite 4334.
Revised: FrostBite 4334.

Original: Gopher 2412.
Revised: Gopher 2412.

Original: Hero 30826.
Revised: Hero 30826.

Original: Jamesbond 302.
Revised: Jamesbond 302.

Original: Kangaroo 3035.
Revised: Kangaroo 3035.

Original: Krull 2665.
Revised: Krull 2665.

Original: KungFuMaster 22736.
Revised: KungFuMaster 22736.

Original: Mspacman 6951.
Revised: Mspacman 6951.

Original: Pong 14.
Revised: Pong 14.

Original: PrivateEye 69571.
Revised: PrivateEye 69571.

Original: Qbert 13455.
Revised: Qbert 13455.

Original: RoadRunner 7845.
Revised: RoadRunner 7845.

Original: Seaquest 42054.
Revised: Seaquest 42054.

Original: UpNdDown 11693.
Revised: UpNdDown 11693.

Original: Figure 6 reports learning curves in the Arcade Learning Environment 100K benchmark with our proposed algorithm MaxMin Novelty and the canonical algorithm (cid:15) -greedy.
Revised: Figure 6 reports learning curves in the Arcade Learning Environment 100K benchmark with our proposed algorithm MaxMin Novelty and the canonical algorithm (cid:15) -greedy.

Original: In early training (i.e. up to 4 × 10 4 environment interactions) in half of the games we observe a steeper increase in the performance.
Revised: In early training (i.e. up to 4 × 10 4 environment interactions) in half of the games we observe a steeper increase in the performance.

Original: This is again a result of the MaxMin Novelty algorithm targeting higher temporal difference bias.
Revised: This is again a result of the MaxMin Novelty algorithm targeting higher temporal difference bias.

Original: In particular, in Amidar, CrazyClimber, Hero, JamesBond, Kangaroo, RoadRunner,
Revised: In particular, in Amidar, CrazyClimber, Hero, JamesBond, Kangaroo, RoadRunner,

Original: PrivateEye, Seaquest, UpNDown, Freeway, Breakout and Asterix the gradient of the performance curve for MaxMin Novelty is higher than the canonical algorithm (cid:15) -greedy.
Revised: PrivateEye, Seaquest, UpNDown, Freeway, Breakout and Asterix the gradient of the performance curve for MaxMin Novelty is higher than the canonical algorithm (cid:15) -greedy.

Original: This is again supporting
Revised: This is again supporting

Original: Proposition 3.6 in the main body of the paper.
Revised: Proposition 3.6 in the main body of the paper.

Original: In particular, early in the training the Q -function is η -uninformed
Revised: In particular, early in the training the Q -function is η -uninformed

Original: and δ -smooth as has been described in Deﬁnition 3.1 and 3.2 in the main body of the paper.
Revised: and δ -smooth as has been described in Deﬁnition 3.1 and 3.2 in the main body of the paper.

Original: Thus, the steep increase in early training matches the predictions of Proposition 3.6 where a positive bias in temporal difference yields faster learning.
Revised: Thus, the steep increase in early training matches the predictions of Proposition 3.6 where a positive bias in temporal difference yields faster learning.

Original: Also further note that in 6 games 1 simple double- Q learning already outperforms Rainbow in the low data regime.
Revised: Also further note that in 6 games 1 simple double- Q learning already outperforms Rainbow in the low data regime.

Original: Note that Rainbow has several additional components such as dueling network, multi-step return, distributional reinforcement learning that introduces new parameters as large as the number of bins used in the algorithm, and NoisyNetworks.
Revised: Note that Rainbow has several additional components such as dueling network, multi-step return, distributional reinforcement learning that introduces new parameters as large as the number of bins used in the algorithm, and NoisyNetworks.

Original: Thus, the fact that MaxMin Novelty with simple double-
Revised: Thus, the fact that MaxMin Novelty with simple double-

Original: Q learning achieves a higher score in these games than an algorithm that combines all these various techniques is further evidence that demonstrates the NoisyNetworks can be replaced with the MaxMin Novelty algorithm in Rainbow as a future research direction to obtain better performance.
Revised: Q learning achieves a higher score in these games than an algorithm that combines all these various techniques is further evidence that demonstrates the NoisyNetworks can be replaced with the MaxMin Novelty algorithm in Rainbow as a future research direction to obtain better performance.

Original: Also further note that MaxMin Novelty does not introduce any additional new parameters as NoisyNetworks does; more precisely, NoisyNetworks doubles the number of parameters used in the Q -network.
Revised: Also further note that MaxMin Novelty does not introduce any additional new parameters as NoisyNetworks does; more precisely, NoisyNetworks doubles the number of parameters used in the Q -network.

Original: Hence, the fact that MaxMin Novelty achieves higher performance as also reported in the main body of the paper without any additional computational cost further demonstrates the beneﬁts of the utilization of MaxMin Novelty in a more diversiﬁed portfolio of algorithms as a zero cost exploration technique.
Revised: Hence, the fact that MaxMin Novelty achieves higher performance as also reported in the main body of the paper without any additional computational cost further demonstrates the beneﬁts of the utilization of MaxMin Novelty in a more diversiﬁed portfolio of algorithms as a zero cost exploration technique.

Original: A. M OTIVATING E XAMPLE R ESULTS
Revised: A. M OTIVATING E XAMPLE R ESULTS

Original: In this section we provide more results into the motivating example of the chain-MDP.
Revised: In this section we provide more results into the motivating example of the chain-MDP.

Original: In particular, while the main body of the paper provides results with the baseline chain-MDP in this section we provide more results with the modiﬁed chain-MDP.
Revised: In particular, while the main body of the paper provides results with the baseline chain-MDP in this section we provide more results with the modiﬁed chain-MDP.

Original: In detail, the modiﬁed chain-MDP refers to the chain-MDP with increased action size to obtain more ﬁne-grained observations into the effects of the exploration techniques.
Revised: In detail, the modiﬁed chain-MDP refers to the chain-MDP with increased action size to obtain more ﬁne-grained observations into the effects of the exploration techniques.

Original: Hence, the modiﬁed chain-MDP consists of n states s ∈ S = 1 , 2 , . . . , n each with four actions.
Revised: Hence, the modiﬁed chain-MDP consists of n states s ∈ S = 1 , 2 , . . . , n each with four actions.

Original: In the modiﬁed chain-MDP each state i has one action that transitions the agent up the chain by one step to state i + 1 , one action that transitions the agent to state two, one action that transitions the agent to state three, and one action which resets the agent to state one at the beginning of the chain.
Revised: In the modiﬁed chain-MDP each state i has one action that transitions the agent up the chain by one step to state i + 1 , one action that transitions the agent to state two, one action that transitions the agent to state three, and one action which resets the agent to state one at the beginning of the chain.

Original: The Figure 7 reports results for MaxMin Novelty, canonical (cid:15) -greedy, and the UCB method with varying (cid:15) ∈ [0 . 15 , 0 . 25] with a step size of 0 .
Revised: The Figure 7 reports results for MaxMin Novelty, canonical (cid:15) -greedy, and the UCB method with varying (cid:15) ∈ [0 . 15 , 0 . 25] with a step size of 0 .

Original: The results reported in Figure 7 once more demonstrate that MaxMin Novelty performs signiﬁcantly better compared to prior exploration techniques.
Revised: The results reported in Figure 7 once more demonstrate that MaxMin Novelty performs signiﬁcantly better compared to prior exploration techniques.

Revised: A. E XTENSION OF M AX M IN N OVELTY TO D IFFERENT R EINFORCEMENT L EARNING A LGORITHMS

Revised: While the main paper focuses on Deep Double Q -Network, Figure 8 reports human normalized median and human normalized 80 th percentile scores for MaxMin Novelty and (cid:15) -greedy with the

Revised: QRDQN algorithm 2 across all of the tasks of the Arcade Learning Environment 100K benchmark.

Revised: With MaxMin Novelty QRDQN is able to achieve 0.15582 human normalized median score.

Revised: Note that data-efﬁcient Rainbow can only achieve 0.12 human normalized median scores van Hasselt et al.

Revised: Furthermore, note that Rainbow contains dueling architecture, multi-step return, noisy networks on top of the distributional reinforcement learning.

Revised: Thus, the fact that QRDQN, a baseline distributional reinforcement learning algorithm, can achieve human normalized median score that is already substantially higher than data-efﬁcient Rainbow once more demonstrates the substantial sample efﬁciency gained by the MaxMin Novelty algorithm.

Revised: Furthermore, the signiﬁcantly higher performance gain obtained by MaxMin Novelty over all tasks of the Arcade Learning Environment 100K benchmark as reported via the human normalized scores in Figure 8, once more demonstrates that MaxMin Novelty increases the performance of the baseline algorithms further beyond the performance of much more complicated algorithms.

Revised: [Figure]