Original: O N THE E STIMATION B IAS IN D OUBLE Q-L EARNING
Revised: O N THE E STIMATION B IAS IN D OUBLE Q-L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Double Q-learning is a classical method for reducing overestimation bias, which is caused by taking maximum estimated values in the Bellman operator.
Revised: Double Q-learning is a classical method for reducing overestimation bias, which is caused by taking maximum estimated values in the Bellman operator.

Original: Its variants in the deep Q-learning paradigm have shown great promise in producing reliable value prediction and improving learning performance.
Revised: Its variants in the deep Q-learning paradigm have shown great promise in producing reliable value prediction and improving learning performance.

Original: However, as shown by prior work, double Q-learning is not fully unbiased and still suffers from underestimation bias.
Revised: However, as shown by prior work, double Q-learning is not fully unbiased and still suffers from underestimation bias.

Original: In this paper, we show that such underestimation bias may lead to multiple non-optimal ﬁxed points under an approximated Bellman operation.
Revised: In this paper, we show that such underestimation bias may lead to multiple non-optimal ﬁxed points under an approximated Bellman operation.

Original: To address the concerns of converging to non-optimal stationary solutions, we propose a simple and effective approach as a partial ﬁx for underestimation bias in double Q-learning.
Revised: To address the concerns of converging to non-optimal stationary solutions, we propose a simple and effective approach as a partial ﬁx for underestimation bias in double Q-learning.

Original: This approach leverages real returns to bound the target value.
Revised: This approach leverages real returns to bound the target value.

Original: We extensively evaluate the proposed method in the Atari benchmark tasks and demonstrate its signiﬁcant improvement over baseline algorithms.
Revised: We extensively evaluate the proposed method in the Atari benchmark tasks and demonstrate its signiﬁcant improvement over baseline algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Value-based reinforcement learning with neural networks as function approximators has become a widely-used paradigm and shown great promise in solving complicated decision-making problems in various real-world applications, including robotics control (Lillicrap et al., 2016), molecular structure design (Zhou et al., 2019), and recommendation systems (Chen et al., 2018).
Revised: Value-based reinforcement learning with neural networks as function approximators has become a widely-used paradigm and shown great promise in solving complicated decision-making problems in various real-world applications, including robotics control (Lillicrap et al., 2016), molecular structure design (Zhou et al., 2019), and recommendation systems (Chen et al., 2018).

Original: Towards understanding the foundation of these successes, investigating algorithmic properties of deep-learningbased value function approximation has been seen a growth of attention in recent years (Van Hasselt et al., 2018; Fu et al., 2019; Achiam et al., 2019; Dong et al., 2020).
Revised: Towards understanding the foundation of these successes, investigating algorithmic properties of deep-learningbased value function approximation has been seen a growth of attention in recent years (Van Hasselt et al., 2018; Fu et al., 2019; Achiam et al., 2019; Dong et al., 2020).

Original: One of the phenomena of interest is that Q-learning (Watkins, 1989) is known to suffer from overestimation issues, since it takes a maximum operator over estimated action-values.
Revised: One of the phenomena of interest is that Q-learning (Watkins, 1989) is known to suffer from overestimation issues, since it takes a maximum operator over estimated action-values.

Original: Comparing with underestimated values, overestimation errors are more likely to be propagated through greedy action selections, which leads to an overestimation bias in value prediction (Thrun & Schwartz, 1993).
Revised: Comparing with underestimated values, overestimation errors are more likely to be propagated through greedy action selections, which leads to an overestimation bias in value prediction (Thrun & Schwartz, 1993).

Original: This overoptimistic behavior of decision making has also been investigated in the literature of management science (Smith &
Revised: This overoptimistic behavior of decision making has also been investigated in the literature of management science (Smith &

Original: Winkler, 2006) and economics (Thaler, 1988).
Revised: Winkler, 2006) and economics (Thaler, 1988).

Original: From a statistical perspective, the value estimation error may come from many sources, such as the stochasticity of the environment and the imperfection of the function expressivity.
Revised: From a statistical perspective, the value estimation error may come from many sources, such as the stochasticity of the environment and the imperfection of the function expressivity.

Original: However, for deep
Revised: However, for deep

Original: Q-learning algorithms, even if most benchmark environments are nearly deterministic (Brockman et al., 2016) and millions of samples are collected, the overestimation phenomenon is still dramatic (Hasselt et al., 2016).
Revised: Q-learning algorithms, even if most benchmark environments are nearly deterministic (Brockman et al., 2016) and millions of samples are collected, the overestimation phenomenon is still dramatic (Hasselt et al., 2016).

Original: One cause of this problematic issue is the difﬁculty of optimization.
Revised: One cause of this problematic issue is the difﬁculty of optimization.

Original: Although a deep neural network may have a sufﬁcient expressiveness power to represent an accurate value function, the back-end optimization is hard to solve.
Revised: Although a deep neural network may have a sufﬁcient expressiveness power to represent an accurate value function, the back-end optimization is hard to solve.

Original: As a result of computational considerations, stochastic gradient descent is almost the default choice for deep reinforcement learning algorithms.
Revised: As a result of computational considerations, stochastic gradient descent is almost the default choice for deep reinforcement learning algorithms.

Original: The high variance of such stochastic methods in gradient estimation would lead to an unavoidable approximation error in value prediction.
Revised: The high variance of such stochastic methods in gradient estimation would lead to an unavoidable approximation error in value prediction.

Original: This kind of approximation error cannot be addressed by simply increasing sample size and network capacity, which is a major source of overestimation bias.
Revised: This kind of approximation error cannot be addressed by simply increasing sample size and network capacity, which is a major source of overestimation bias.

Original: Double Q-learning is a classical method to reduce the risk of overestimation, which is a speciﬁc variant of the double estimator (Stone, 1974) in the Q-learning paradigm.
Revised: Double Q-learning is a classical method to reduce the risk of overestimation, which is a speciﬁc variant of the double estimator (Stone, 1974) in the Q-learning paradigm.

Original: It uses a second value function to construct an independent action-value evaluation as cross validation.
Revised: It uses a second value function to construct an independent action-value evaluation as cross validation.

Original: With proper assumptions, double Q-learning was proved to underestimate rather than overestimate the maximum expected values (Van Hasselt, 2010).
Revised: With proper assumptions, double Q-learning was proved to underestimate rather than overestimate the maximum expected values (Van Hasselt, 2010).

Original: However, in practice, obtaining two independent value estimators is usually intractable in large-scale tasks, which makes double Q-learning still suffer from overestimation in some situations.
Revised: However, in practice, obtaining two independent value estimators is usually intractable in large-scale tasks, which makes double Q-learning still suffer from overestimation in some situations.

Original: To address these empirical concerns, Fujimoto et al.
Revised: To address these empirical concerns, Fujimoto et al.

Original: (2018) proposed a variant named clipped double Q-learning, which takes the minimum over two value estimations.
Revised: (2018) proposed a variant named clipped double Q-learning, which takes the minimum over two value estimations.

Original: This approach implicitly penalizes regions with high uncertainty (Fujimoto et al., 2019) and thus signiﬁcantly repress the incentive of overestimation.
Revised: This approach implicitly penalizes regions with high uncertainty (Fujimoto et al., 2019) and thus signiﬁcantly repress the incentive of overestimation.

Original: In this paper, we ﬁrst review an analytical model adopted by prior work (Thrun & Schwartz, 1993;
Revised: In this paper, we ﬁrst review an analytical model adopted by prior work (Thrun & Schwartz, 1993;

Original: Lan et al., 2020) and reveal a fact that, due to the existence of underestimation biases, both double
Revised: Lan et al., 2020) and reveal a fact that, due to the existence of underestimation biases, both double

Original: Q-learning and clipped double Q-learning have multiple approximated ﬁxed points in this model.
Revised: Q-learning and clipped double Q-learning have multiple approximated ﬁxed points in this model.

Original: This result raises a concern that double Q-learning may easily get stuck in some local stationary regions and become inefﬁcient in searching for the optimal policy.
Revised: This result raises a concern that double Q-learning may easily get stuck in some local stationary regions and become inefﬁcient in searching for the optimal policy.

Original: To bootstrap the ability of double
Revised: To bootstrap the ability of double

Original: Q-learning, we propose a simple heuristic that utilizes real return signals as a lower bound estimation to rule out the potential non-optimal ﬁxed points.
Revised: Q-learning, we propose a simple heuristic that utilizes real return signals as a lower bound estimation to rule out the potential non-optimal ﬁxed points.

Original: Beneﬁting from its simplicity, this method is easy to be combined with other existing techniques such as clipped double Q-learning.
Revised: Beneﬁting from its simplicity, this method is easy to be combined with other existing techniques such as clipped double Q-learning.

Original: In the experiments on Atari benchmark tasks, we demonstrate that this simple approach is effective both in improving sample efﬁciency and convergence performance.
Revised: In the experiments on Atari benchmark tasks, we demonstrate that this simple approach is effective both in improving sample efﬁciency and convergence performance.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Markov Decision Process (MDP; Bellman, 1957) is a classical framework to formalize an agentenvironment interaction system which can be deﬁned as a tuple M = (cid:104)S , A , P, R, γ (cid:105) .
Revised: Markov Decision Process (MDP; Bellman, 1957) is a classical framework to formalize an agentenvironment interaction system which can be deﬁned as a tuple M = (cid:104)S , A , P, R, γ (cid:105) .

Original: We use S and A to denote the state and action space, respectively.
Revised: We use S and A to denote the state and action space, respectively.

Original: P ( s (cid:48) | s, a ) and R ( s, a ) denote the transition and reward functions, which are initially unknown to the agent.
Revised: P ( s (cid:48) | s, a ) and R ( s, a ) denote the transition and reward functions, which are initially unknown to the agent.

Original: γ is the discount factor.
Revised: γ is the discount factor.

Original: The goal of reinforcement learning is to construct a policy π : S
Revised: The goal of reinforcement learning is to construct a policy π : S

Original: → A maximizing discounted cumulative rewards,
Revised: → A maximizing discounted cumulative rewards,

Original: [Equation]
Revised: [Equation]

Original: Another quantity of interest in policy learning can be deﬁned through the Bellman equation
Revised: Another quantity of interest in policy learning can be deﬁned through the Bellman equation

Original: Q π ( s, a ) =
Revised: Q π ( s, a ) =

Original: R ( s, a ) +
Revised: R ( s, a ) +

Original: [Equation]
Revised: [Equation]

Original: E s
Revised: E s

Original: [Equation]
Revised: [Equation]

Original: P ( ·| s
Revised: P ( ·| s

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [ V π ( s (cid:48) )] .
Revised: [ V π ( s (cid:48) )] .

Original: The optimal value function Q ∗ corresponds to the unique solution of the Bellman optimality equation,
Revised: The optimal value function Q ∗ corresponds to the unique solution of the Bellman optimality equation,

Original: [Equation]
Revised: [Equation]

Original: Q-learning algorithms are based on the Bellman operator T stated as follows:
Revised: Q-learning algorithms are based on the Bellman operator T stated as follows:

Original: [Equation]
Revised: [Equation]

Original: By iterating this operator, value iteration is proved to converge to the optimal value function Q ∗ .
Revised: By iterating this operator, value iteration is proved to converge to the optimal value function Q ∗ .

Original: To extend Q-learning methods to real-world applications, function approximation is indispensable to deal with a high-dimensional state space.
Revised: To extend Q-learning methods to real-world applications, function approximation is indispensable to deal with a high-dimensional state space.

Original: Deep Q-learning (Mnih et al., 2015) considers a samplebased objective function and constructs an iterative optimization framework:
Revised: Deep Q-learning (Mnih et al., 2015) considers a samplebased objective function and constructs an iterative optimization framework:

Original: [Equation]
Revised: [Equation]

Original: in which Θ denotes the parameter space of the value network, and θ 0 ∈ Θ is initialized by some predetermined method.
Revised: in which Θ denotes the parameter space of the value network, and θ 0 ∈ Θ is initialized by some predetermined method.

Original: D is the data distribution which is changing during exploration.
Revised: D is the data distribution which is changing during exploration.

Original: With inﬁnite samples and a sufﬁciently rich function class, the update rule stated in Eq.
Revised: With inﬁnite samples and a sufﬁciently rich function class, the update rule stated in Eq.

Original: (2) is asymptotically equivalent to applying the Bellman operator T , but the underlying optimization is usually inefﬁcient in practice.
Revised: (2) is asymptotically equivalent to applying the Bellman operator T , but the underlying optimization is usually inefﬁcient in practice.

Original: In deep Q-learning, Eq.
Revised: In deep Q-learning, Eq.

Original: (2) is optimized by mini-batch gradient descent and thus its value estimation suffers from unavoidable approximation errors.
Revised: (2) is optimized by mini-batch gradient descent and thus its value estimation suffers from unavoidable approximation errors.

Original:  E FFECTS OF U NDERESTIMATION B IAS IN D OUBLE Q-L EARNING
Revised:  E FFECTS OF U NDERESTIMATION B IAS IN D OUBLE Q-L EARNING

Original: In this section, we will ﬁrst review a common analytical model used by previous work studying estimation bias (Thrun & Schwartz, 1993; Lan et al., 2020), in which double Q-learning is known to have underestimation bias.
Revised: In this section, we will ﬁrst review a common analytical model used by previous work studying estimation bias (Thrun & Schwartz, 1993; Lan et al., 2020), in which double Q-learning is known to have underestimation bias.

Original: Based on this analytical model, we show that its underestimation bias could make double Q-learning have multiple ﬁxed point solutions under an approximated Bellman operation.
Revised: Based on this analytical model, we show that its underestimation bias could make double Q-learning have multiple ﬁxed point solutions under an approximated Bellman operation.

Original: This result suggests that double Q-learning may have extra non-optimal stationary solutions under the effects of the approximation error.
Revised: This result suggests that double Q-learning may have extra non-optimal stationary solutions under the effects of the approximation error.

Original: 3. M ODELING A PPROXIMATION E RROR IN Q-L EARNING
Revised: 3. M ODELING A PPROXIMATION E RROR IN Q-L EARNING

Original: Following Thrun & Schwartz (1993) and Lan et al.
Revised: Following Thrun & Schwartz (1993) and Lan et al.

Original: (2020), we formalize the underlying approximation error e ( t ) ( s, a ) of target value regression as a set of random noises impacting on the Bellman operation,
Revised: (2020), we formalize the underlying approximation error e ( t ) ( s, a ) of target value regression as a set of random noises impacting on the Bellman operation,

Original: [Equation]
Revised: [Equation]

Original: where T denotes the ground truth Bellman operator using full information of the MDP (see Eq.
Revised: where T denotes the ground truth Bellman operator using full information of the MDP (see Eq.

Original: (1)), and (cid:101) T denotes a stochastic operator with noisy outputs.
Revised: (1)), and (cid:101) T denotes a stochastic operator with noisy outputs.

Original: The main purpose of introducing the explicit noise term e ( t ) ( s, a ) is to emphasize that the approximation error discussed here is different from the sampling error.
Revised: The main purpose of introducing the explicit noise term e ( t ) ( s, a ) is to emphasize that the approximation error discussed here is different from the sampling error.

Original: In an information-theoretic perspective, the sampling error can be reduced asymptotically as the sample size increases.
Revised: In an information-theoretic perspective, the sampling error can be reduced asymptotically as the sample size increases.

Original: However, there is a barrier of optimization difﬁculty to establish a precise estimation in practice, which leads to an unavoidable approximation error in optimization.
Revised: However, there is a barrier of optimization difﬁculty to establish a precise estimation in practice, which leads to an unavoidable approximation error in optimization.

Original: By integrating the noise term e ( t ) ( s, a ) into the Bellman operation, we set up an analytical model to investigate how Q-learning algorithms interact with the inherent approximation error.
Revised: By integrating the noise term e ( t ) ( s, a ) into the Bellman operation, we set up an analytical model to investigate how Q-learning algorithms interact with the inherent approximation error.

Original: In this model, double Q-learning (Van Hasselt, 2010) can be modeled by two estimator instances { Q ( t ) i } i ∈{ 1 , 2 } with separated noise variables { e ( t ) i } i ∈{ 1 , 2 } .
Revised: In this model, double Q-learning (Van Hasselt, 2010) can be modeled by two estimator instances { Q ( t ) i } i ∈{ 1 , 2 } with separated noise variables { e ( t ) i } i ∈{ 1 , 2 } .

Original: For simpliﬁcation, we introduce a policy function π ( t ) ( s ) =
Revised: For simpliﬁcation, we introduce a policy function π ( t ) ( s ) =

Original: arg max a Q ( t ) 1 ( s, a ) to override the state value function as follows:
Revised: arg max a Q ( t ) 1 ( s, a ) to override the state value function as follows:

Original: [Equation]
Revised: [Equation]

Original: The difference of Eq.
Revised: The difference of Eq.

Original: (4) from the deﬁnition of double Q-learning given by Van Hasselt (2010) is using a uniﬁed target value V ( t ) ( s (cid:48) ) for both two estimators.
Revised: (4) from the deﬁnition of double Q-learning given by Van Hasselt (2010) is using a uniﬁed target value V ( t ) ( s (cid:48) ) for both two estimators.

Original: This simpliﬁcation does not affect the derived implications of double Q-learning, and is also implemented by advanced variants of double
Revised: This simpliﬁcation does not affect the derived implications of double Q-learning, and is also implemented by advanced variants of double

Original: Q-learning (Fujimoto et al., 2018; Lan et al., 2020).
Revised: Q-learning (Fujimoto et al., 2018; Lan et al., 2020).

Original: Note that the target value can be constructed only using the state-value function.
Revised: Note that the target value can be constructed only using the state-value function.

Original: Based on this observation, we can deﬁne the ﬁxed point of a stochastic operators on the target value.
Revised: Based on this observation, we can deﬁne the ﬁxed point of a stochastic operators on the target value.

Original: Deﬁnition 1 (Approximated Fixed Points) .
Revised: Deﬁnition 1 (Approximated Fixed Points) .

Original: Let (cid:101) T denote a stochastic Bellman operator, such as what are stated in Eq.
Revised: Let (cid:101) T denote a stochastic Bellman operator, such as what are stated in Eq.

Original: (3) and Eq.
Revised: (3) and Eq.

Original: A state-value function V is regarded as an approximated ﬁxed point under a stochastic Bellman operator (cid:101) T if it satisﬁes E [ (cid:101) T V ]
Revised: A state-value function V is regarded as an approximated ﬁxed point under a stochastic Bellman operator (cid:101) T if it satisﬁes E [ (cid:101) T V ]

Original: = V , where (cid:101) T V denotes the output state-value function while applying the Bellman operator (cid:101) T on V .
Revised: = V , where (cid:101) T V denotes the output state-value function while applying the Bellman operator (cid:101) T on V .

Original: The ﬁxed point deﬁned above is named approximated since they are not truly static, but is invariant under the stochastic Bellman operation in expectation.
Revised: The ﬁxed point deﬁned above is named approximated since they are not truly static, but is invariant under the stochastic Bellman operation in expectation.

Original: In Appendix A.2, we will prove the existence of such ﬁxed points as the following statement.
Revised: In Appendix A.2, we will prove the existence of such ﬁxed points as the following statement.

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume the probability density functions of the noise terms { e ( s, a ) } are continuous.
Revised: Assume the probability density functions of the noise terms { e ( s, a ) } are continuous.

Original: The stochastic Bellman operators deﬁned by Eq.
Revised: The stochastic Bellman operators deﬁned by Eq.

Original: (3) and Eq.
Revised: (3) and Eq.

Original: (4) have approximated ﬁxed points deﬁned as Deﬁnition 1.
Revised: (4) have approximated ﬁxed points deﬁned as Deﬁnition 1.

Original: 3. E XISTENCE OF M ULTIPLE A PPROXIMATED F IXED P OINTS IN D OUBLE Q-L EARNING
Revised: 3. E XISTENCE OF M ULTIPLE A PPROXIMATED F IXED P OINTS IN D OUBLE Q-L EARNING

Original: Given the deﬁnition of the approximated ﬁxed point, a natural question is whether such kind of ﬁxed points are unique or not.
Revised: Given the deﬁnition of the approximated ﬁxed point, a natural question is whether such kind of ﬁxed points are unique or not.

Original: Recall that the optimal value function Q ∗ is the unique solution of the Bellman optimality equation, which is the foundation of Q-learning algorithms.
Revised: Recall that the optimal value function Q ∗ is the unique solution of the Bellman optimality equation, which is the foundation of Q-learning algorithms.

Original: However, in this section, we will show that, under the effects of the approximation error, the approximated ﬁxed points of double Q-learning may not be unique.
Revised: However, in this section, we will show that, under the effects of the approximation error, the approximated ﬁxed points of double Q-learning may not be unique.

Original: Figure 1a illustrates a simple MDP in which double Q-learning stated as Eq.
Revised: Figure 1a illustrates a simple MDP in which double Q-learning stated as Eq.

Original: (4) has multiple approximated ﬁxed points.
Revised: (4) has multiple approximated ﬁxed points.

Original: This MDP is fully deterministic and contains only two states s 0 and s 1 .
Revised: This MDP is fully deterministic and contains only two states s 0 and s 1 .

Original: All actions in state s 1 lead to a self-loop and produce a unit reward signal.
Revised: All actions in state s 1 lead to a self-loop and produce a unit reward signal.

Original: On state s 0 , the result of executing action a 0 is a self-loop with a slightly larger reward signal than choosing action a 1 which leads to state s 1 .
Revised: On state s 0 , the result of executing action a 0 is a self-loop with a slightly larger reward signal than choosing action a 1 which leads to state s 1 .

Original: The only challenge for decision making in this MDP is to distinguish the outcomes of executing action a 0 and a 1 on state s 0 .
Revised: The only challenge for decision making in this MDP is to distinguish the outcomes of executing action a 0 and a 1 on state s 0 .

Original: To make the example more accessible, we assume the approximation errors { e ( t ) ( s, a ) } t,s,a are a set of independent random variables following a uniform
Revised: To make the example more accessible, we assume the approximation errors { e ( t ) ( s, a ) } t,s,a are a set of independent random variables following a uniform

Original: [Figure]
Revised: [Figure]

Original: (cid:47)
Revised: (cid:47)

Original: [Figure]
Revised: [Figure]

Original: (cid:7) (cid:7)
Revised: (cid:7) (cid:7)

Original: [Figure]
Revised: [Figure]

Original: distribution Uniform ( − (cid:15), (cid:15) ) .
Revised: distribution Uniform ( − (cid:15), (cid:15) ) .

Original: This simpliﬁcation is adopted by Thrun & Schwartz (1993) and
Revised: This simpliﬁcation is adopted by Thrun & Schwartz (1993) and

Original: Lan et al.
Revised: Lan et al.

Original: (2020) in case studies.
Revised: (2020) in case studies.

Original: Here, we select the magnitude of noise as (cid:15) = 1 . 

Revised: Here, we select the magnitude of noise as (cid:15) = 1 .

Original: and the discount factor as γ = 0 .
Revised: and the discount factor as γ = 0 .

Original: 99 to balance the scale of involved amounts.
Revised: 99 to balance the scale of involved amounts.

Original: Considering to solve the equation E [ (cid:101) T V ] = V according to the deﬁnition of the approximated ﬁxed point (see Deﬁnition 1), the numerical solutions of such ﬁxed points are presented in Table 1b.
Revised: Considering to solve the equation E [ (cid:101) T V ] = V according to the deﬁnition of the approximated ﬁxed point (see Deﬁnition 1), the numerical solutions of such ﬁxed points are presented in Table 1b.

Original: There are three different ﬁxed point solutions.
Revised: There are three different ﬁxed point solutions.

Original: The ﬁrst thing to notice is that the optimal ﬁxed point V ∗ is retained in this MDP (see the last row of Table 1b), since the noise magnitude (cid:15)
Revised: The ﬁrst thing to notice is that the optimal ﬁxed point V ∗ is retained in this MDP (see the last row of Table 1b), since the noise magnitude (cid:15)

Original: = 1 .is much smaller than the optimality gap Q ∗ ( s 0 , a 0 ) − Q ∗ ( s 0 , a 1 ) = 10 .
Revised: = 1 .is much smaller than the optimality gap Q ∗ ( s 0 , a 0 ) − Q ∗ ( s 0 , a 1 ) = 10 .

Original: The other two ﬁxed points are non-optimal and very close to Q ( s 0 , a 0 )
Revised: The other two ﬁxed points are non-optimal and very close to Q ( s 0 , a 0 )

Original: ≈ Q ( s 0 , a 1 ) = 100 , in which the agent cannot fully distinguish the difference between choosing a 0 and a 1 .
Revised: ≈ Q ( s 0 , a 1 ) = 100 , in which the agent cannot fully distinguish the difference between choosing a 0 and a 1 .

Original: To illustrate this example, we ﬁrst present a sufﬁcient condition for a stochastic Bellman operator to have multiple ﬁxed points, and then we give an intuitive explanation to connect this condition with the constructed MDP.
Revised: To illustrate this example, we ﬁrst present a sufﬁcient condition for a stochastic Bellman operator to have multiple ﬁxed points, and then we give an intuitive explanation to connect this condition with the constructed MDP.

Original: Mathematical Condition.
Revised: Mathematical Condition.

Original: Note that the deﬁnition of the stochastic Bellman operator is a model of an imprecise target value regression.
Revised: Note that the deﬁnition of the stochastic Bellman operator is a model of an imprecise target value regression.

Original: From this perspective, the input of a stochastic Bellman operator can be deﬁned as a set of ground truth target values { ( T Q ( t ) )
Revised: From this perspective, the input of a stochastic Bellman operator can be deﬁned as a set of ground truth target values { ( T Q ( t ) )

Original: ( s, a ) } s,a .
Revised: ( s, a ) } s,a .

Original: Based on this notation, a sufﬁcient condition for the existence of multiple ﬁxed points is stated as follows.
Revised: Based on this notation, a sufﬁcient condition for the existence of multiple ﬁxed points is stated as follows.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let f s ( { ( T Q )( s, a ) } a ∈A ) =
Revised: Let f s ( { ( T Q )( s, a ) } a ∈A ) =

Original: E [( (cid:101) T V )( s )] denote the expected output value of the stochastic Bellman operator (cid:101) T on state s , and assume f s ( · ) is differentiable.
Revised: E [( (cid:101) T V )( s )] denote the expected output value of the stochastic Bellman operator (cid:101) T on state s , and assume f s ( · ) is differentiable.

Original: If a stochastic Bellman operator (cid:101) T satisﬁes Eq.
Revised: If a stochastic Bellman operator (cid:101) T satisﬁes Eq.

Original: (5), there exists an MDP such that (cid:101) T has multiple ﬁxed points.
Revised: (5), there exists an MDP such that (cid:101) T has multiple ﬁxed points.

Original: [Equation]
Revised: [Equation]

Original: 5) where X = { x i } |A| i =1 denotes the input of the function f s .
Revised: 5) where X = { x i } |A| i =1 denotes the input of the function f s .

Original: The proof of Proposition 2 is deferred to Appendix A.4.
Revised: The proof of Proposition 2 is deferred to Appendix A.4.

Original: This proposition suggests that, in order to determine whether a given stochastic Bellman operator (cid:101) T may have multiple ﬁxed points, we need to check whether its expected output values could change dramatically with a slight alter of the input values.
Revised: This proposition suggests that, in order to determine whether a given stochastic Bellman operator (cid:101) T may have multiple ﬁxed points, we need to check whether its expected output values could change dramatically with a slight alter of the input values.

Original: Intuitive Explanation.
Revised: Intuitive Explanation.

Original: To provide the intuition for understanding the constructed MDP in Figure 1, we introduce a simple property of a Bellman operator, named monotonicity, which helps to illustrate how the sufﬁcient condition in Eq.
Revised: To provide the intuition for understanding the constructed MDP in Figure 1, we introduce a simple property of a Bellman operator, named monotonicity, which helps to illustrate how the sufﬁcient condition in Eq.

Original: (5) are met in double Q-learning.
Revised: (5) are met in double Q-learning.

Original: Deﬁnition 2 (Monotonicity) .
Revised: Deﬁnition 2 (Monotonicity) .

Original: A stochastic Bellman operator (cid:101) T is called approximately monotonic if it satisﬁes the following property:
Revised: A stochastic Bellman operator (cid:101) T is called approximately monotonic if it satisﬁes the following property:

Original: [Equation]
Revised: [Equation]

Original: Although this monotonicity property holds for vanilla Q-learning (Watkins, 1989) and a recent variant called maxmin Q-learning (Lan et al., 2020), it absents from double Q-learning (Van Hasselt, 2010) and a famous variant called clipped double Q-learning (Fujimoto et al., 2018).
Revised: Although this monotonicity property holds for vanilla Q-learning (Watkins, 1989) and a recent variant called maxmin Q-learning (Lan et al., 2020), it absents from double Q-learning (Van Hasselt, 2010) and a famous variant called clipped double Q-learning (Fujimoto et al., 2018).

Original: The monotonicity property is not a sufﬁcient condition for Eq.
Revised: The monotonicity property is not a sufﬁcient condition for Eq.

Original: but it is quite important for understanding the constructed example in Figure 1.
Revised: but it is quite important for understanding the constructed example in Figure 1.

Original: Considering the constructed MDP as an example, Figure 1c visualizes the relation between the input state-value V ( t ) ( s 0 ) and the expected output state-value E [ V ( t +1) ( s 0 )] while assuming V ( t ) ( s 1 ) = 100 has converged to its stationary point.
Revised: Considering the constructed MDP as an example, Figure 1c visualizes the relation between the input state-value V ( t ) ( s 0 ) and the expected output state-value E [ V ( t +1) ( s 0 )] while assuming V ( t ) ( s 1 ) = 100 has converged to its stationary point.

Original: The minima point of the output value is located at the situation where V ( t ) ( s 0 ) is slightly smaller than V ( t ) ( s 1 ) , since the expected policy derived by (cid:101) T V ( t ) will have a remarkable probability to choose sub-optimal actions.
Revised: The minima point of the output value is located at the situation where V ( t ) ( s 0 ) is slightly smaller than V ( t ) ( s 1 ) , since the expected policy derived by (cid:101) T V ( t ) will have a remarkable probability to choose sub-optimal actions.

Original: This local minima suffers from the most dramatic underestimation among the whole curve, and the underestimation will eventually vanish as the value of V ( t ) ( s 0 ) increases.
Revised: This local minima suffers from the most dramatic underestimation among the whole curve, and the underestimation will eventually vanish as the value of V ( t ) ( s 0 ) increases.

Original: During this process, a large magnitude of the ﬁrst-order derivative could be found to meet the condition stated in Eq.
Revised: During this process, a large magnitude of the ﬁrst-order derivative could be found to meet the condition stated in Eq.

Original: A heuristics to verify the condition in Eq.
Revised: A heuristics to verify the condition in Eq.

Original: (5) is to check the nearby regions of the local minima and maxima in a non-monotonic function E [ (cid:101) T V ] .
Revised: (5) is to check the nearby regions of the local minima and maxima in a non-monotonic function E [ (cid:101) T V ] .

Original: In Appendix A.5, we also show that clipped double
Revised: In Appendix A.5, we also show that clipped double

Original: Q-learning, another method with non-monotonicity, has multiple ﬁxed points in an MDP slightly modiﬁed from Figure 1a.
Revised: Q-learning, another method with non-monotonicity, has multiple ﬁxed points in an MDP slightly modiﬁed from Figure 1a.

Original: Implications To facilitate discussions, we introduce a concept, named induced policy, to characterize how the agent behaves and evolves around these approximated ﬁxed points.
Revised: Implications To facilitate discussions, we introduce a concept, named induced policy, to characterize how the agent behaves and evolves around these approximated ﬁxed points.

Original: Deﬁnition 3 (Induced Policy) .
Revised: Deﬁnition 3 (Induced Policy) .

Original: Given a target state-value function V , its induced policy ˜ π is deﬁned as a stochastic action selection according to the value estimation produced by a stochastic Bellman operation,
Revised: Given a target state-value function V , its induced policy ˜ π is deﬁned as a stochastic action selection according to the value estimation produced by a stochastic Bellman operation,

Original: [Equation]
Revised: [Equation]

Original: (cid:125)
Revised: (cid:125)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where { e 1 ( s, a ) } s,a are drawing from the same noise distribution as what is used by double Qlearning stated in Eq.
Revised: where { e 1 ( s, a ) } s,a are drawing from the same noise distribution as what is used by double Qlearning stated in Eq.

Original: Proposition 3.
Revised: Proposition 3.

Original: Assume the noise terms e 

Revised: Assume the noise terms e

Original: and e 2 are independently generated in the double estimator stated in Eq.
Revised: and e 2 are independently generated in the double estimator stated in Eq.

Original: Every approximated ﬁxed point V is equal to the ground truth value function
Revised: Every approximated ﬁxed point V is equal to the ground truth value function

Original: V ˜ π with respect to an induced policy ˜ π .
Revised: V ˜ π with respect to an induced policy ˜ π .

Original: The proof of Proposition 3 is deferred to Appendix A.3.
Revised: The proof of Proposition 3 is deferred to Appendix A.3.

Original: As shown by this proposition, the estimated value of a non-optimal ﬁxed point is corresponding to the value of a stochastic policy, which revisits the incentive of double Q-learning to underestimate true maximum values.
Revised: As shown by this proposition, the estimated value of a non-optimal ﬁxed point is corresponding to the value of a stochastic policy, which revisits the incentive of double Q-learning to underestimate true maximum values.

Original: Based on Proposition 3, the concept of induced policy can provide a snapshot to infer the algorithmic properties of approximated ﬁxed points.
Revised: Based on Proposition 3, the concept of induced policy can provide a snapshot to infer the algorithmic properties of approximated ﬁxed points.

Original: Taking the third column of Table 1b as an example, due to the existence of the approximation error, the induced policy ˜ π suffers from a remarkable uncertainty in determining the best action on state s 0 .
Revised: Taking the third column of Table 1b as an example, due to the existence of the approximation error, the induced policy ˜ π suffers from a remarkable uncertainty in determining the best action on state s 0 .

Original: This phenomenon suggests that the approximation error would make the greedy action selection become competitive among actions with near-maximum values.
Revised: This phenomenon suggests that the approximation error would make the greedy action selection become competitive among actions with near-maximum values.

Original: Once the approximation error leads the policy to select a non-optimal action, the learning of the value function would get stuck in non-optimal ﬁxed points.
Revised: Once the approximation error leads the policy to select a non-optimal action, the learning of the value function would get stuck in non-optimal ﬁxed points.

Original: Note that, according to the characterization given by Deﬁnition 3, the stationary property of such kind of ﬁxed points might be similar to that of saddle points in the literature of optimization.
Revised: Note that, according to the characterization given by Deﬁnition 3, the stationary property of such kind of ﬁxed points might be similar to that of saddle points in the literature of optimization.

Original: Whether the agent gets stuck in non-optimal solutions depends on the actual behavior of approximation error.
Revised: Whether the agent gets stuck in non-optimal solutions depends on the actual behavior of approximation error.

Original: In section 4, we will introduce a method to provide incentives for escaping non-optimal ﬁxed points by reducing underestimation bias.
Revised: In section 4, we will introduce a method to provide incentives for escaping non-optimal ﬁxed points by reducing underestimation bias.

Original: 3. A PPROXIMATION E RROR IN D EEP Q-L EARNING
Revised: 3. A PPROXIMATION E RROR IN D EEP Q-L EARNING

Original: [Table]
Revised: [Table]

Original: 10M 0.
Revised: 10M 0.

Original: In the MDP example presented in the previous section, the noise terms { e ( t ) ( s, a ) } t,s,a are modeled by a set of uniform random variables Uniform ( − (cid:15), (cid:15) ) .
Revised: In the MDP example presented in the previous section, the noise terms { e ( t ) ( s, a ) } t,s,a are modeled by a set of uniform random variables Uniform ( − (cid:15), (cid:15) ) .

Original: The noise magnitude (cid:15) = 1 . 0 seems to a bit large when it compares with the scale of one-step reward signal, but it is actually quite small in terms of the scale of the value function.
Revised: The noise magnitude (cid:15) = 1 . 0 seems to a bit large when it compares with the scale of one-step reward signal, but it is actually quite small in terms of the scale of the value function.

Original: Considering an MDP only containing positive rewards, the scale of value functions would be in [0 , R max / (1 − γ )] where
Revised: Considering an MDP only containing positive rewards, the scale of value functions would be in [0 , R max / (1 − γ )] where

Original: R max denotes the upper bound of reward signals.
Revised: R max denotes the upper bound of reward signals.

Original: In this setting, if the relative error of solving the target value regression achieves the order of 1 / (1 − γ ) , the induced approximation error will reach the same scale of reward signals.
Revised: In this setting, if the relative error of solving the target value regression achieves the order of 1 / (1 − γ ) , the induced approximation error will reach the same scale of reward signals.

Original: Revisiting the MDP introduced in the previous section with the discount factor γ = 0 .
Revised: Revisiting the MDP introduced in the previous section with the discount factor γ = 0 .

Original: 99 , the noise magnitude (cid:15) = 1 . 0 is only an 1% relative error comparing to the scale of the value function, which ﬁnally leads to a 10% relative error in value estimation.
Revised: 99 , the noise magnitude (cid:15) = 1 . 0 is only an 1% relative error comparing to the scale of the value function, which ﬁnally leads to a 10% relative error in value estimation.

Original: It is worth noticing that such a small approximation error can cause signiﬁcantly sub-optimal ﬁxed points.
Revised: It is worth noticing that such a small approximation error can cause signiﬁcantly sub-optimal ﬁxed points.

Original: Table 1 presents the scale of the estimation error of DDQN in a deterministic version of one Atari benchmark task.
Revised: Table 1 presents the scale of the estimation error of DDQN in a deterministic version of one Atari benchmark task.

Original: We evaluate the average absolute TD-error E [ | Q − T ˆ Q | ] using a batch of transitions sampling from the replay buffer.
Revised: We evaluate the average absolute TD-error E [ | Q − T ˆ Q | ] using a batch of transitions sampling from the replay buffer.

Original: The evaluation is processed before the target value switches.
Revised: The evaluation is processed before the target value switches.

Original: As shown in Table 1, the value estimation of DDQN suffers from a constant approximation error, which may cause the potential risk of multiple non-optimal ﬁxed points according to our analysis.
Revised: As shown in Table 1, the value estimation of DDQN suffers from a constant approximation error, which may cause the potential risk of multiple non-optimal ﬁxed points according to our analysis.

Original:  L OWER B OUNDED D OUBLE Q-L EARNING
Revised:  L OWER B OUNDED D OUBLE Q-L EARNING

Original: As discussed in the last section, the underestimation bias of double Q-learning may lead to multiple non-optimal ﬁxed points in the analytical model.
Revised: As discussed in the last section, the underestimation bias of double Q-learning may lead to multiple non-optimal ﬁxed points in the analytical model.

Original: A major source of such underestimation is the inherent approximation error caused by the difﬁculty of optimization.
Revised: A major source of such underestimation is the inherent approximation error caused by the difﬁculty of optimization.

Original: In this section, we will introduce a simple heuristic, named lower bounded double Q-learning, which helps to reduce the negative effects of underestimation.
Revised: In this section, we will introduce a simple heuristic, named lower bounded double Q-learning, which helps to reduce the negative effects of underestimation.

Original: The main idea is motivated by a fact that the value prediction produced by double Q-learning does not tend to overestimate the true maximum value.
Revised: The main idea is motivated by a fact that the value prediction produced by double Q-learning does not tend to overestimate the true maximum value.

Original: It is a major difference from the overestimation bias of vanilla Q-learning.
Revised: It is a major difference from the overestimation bias of vanilla Q-learning.

Original: When the environment is deterministic, the return values of collected trajectories can naturally serve as a lower bound for value estimation.
Revised: When the environment is deterministic, the return values of collected trajectories can naturally serve as a lower bound for value estimation.

Original: To utilize this lower bound, we modify the objective function of deep Q-learning as follows,
Revised: To utilize this lower bound, we modify the objective function of deep Q-learning as follows,

Original: [Equation]
Revised: [Equation]

Original: where the target value is computed by taking the maximum over two sources of estimation.
Revised: where the target value is computed by taking the maximum over two sources of estimation.

Original: The ﬁrst term V ˆ θ ( s t +1 ) is the target state-value computed by the frozen parameter ˆ θ .
Revised: The ﬁrst term V ˆ θ ( s t +1 ) is the target state-value computed by the frozen parameter ˆ θ .

Original: The second term
Revised: The second term

Original: V τ ( s t +1 ) is the discounted return value of the corresponding trajectory τ in the replay buffer.
Revised: V τ ( s t +1 ) is the discounted return value of the corresponding trajectory τ in the replay buffer.

Original: Formally, V τ ( s t +1 )
Revised: Formally, V τ ( s t +1 )

Original: = (cid:80) H − t k =0 γ k r t + k +

Revised: = (cid:80) H − t k =0 γ k r t + k +

Original: where H denotes the length of the trajectory.
Revised: where H denotes the length of the trajectory.

Original: This lower bounded objective has two potential impacts on the value estimation:
Revised: This lower bounded objective has three potential impacts on the value estimation:

Original: • Considering the non-optimal ﬁxed points of double Q-learning discussed in the previous section, the values of these ﬁxed points are corresponding to some sub-optimal polices which underestimate the optimal action-values (see Proposition 3).
Revised: • Considering the non-optimal ﬁxed points of double Q-learning discussed in the previous section the values of these ﬁxed points are corresponding to some sub-optimal polices which underestimate the optimal action-values (see Proposition 3).

Original: Under the effects of exploration, some of the collected trajectories may have higher returns than the current policy, which would provide an incentive to get rid of the non-optimal ﬁxed points through the lower bounded objective.
Revised: Under the effects of exploration, some of the collected trajectories may have higher returns than the current policy, which would provide an incentive to get rid of the non-optimal ﬁxed points through the lower bounded objective.

Original: • When the return signal V τ has approached the ground truth maximum value V ∗ , the lower bounded objective will lead to an overestimation bias, since only underestimated values are cut off by the maximum operator.
Revised: • When the return signal V τ has approached the ground truth maximum value V ∗ , the lower bounded objective will lead to an overestimation bias, since only underestimated values are cut off by the maximum operator.

Original: In practice, the effects of such overestimation bias are manageable, because all trajectories are collected by an exploratory policy which can hardly produce the ground truth maximum values.
Revised: In practice, the effects of such overestimation bias are manageable, because all trajectories are collected by an exploratory policy which can hardly produce the ground truth maximum values.

Original: • In addition, when the environment carries some extent of stochasticity, the trajectories with high return values would cause additional risks of overestimation through the proposed lower bounded objective.
Revised: • In addition, when the environment carries some extent of stochasticity, the trajectories with high return values would cause additional risks of overestimation through the proposed lower bounded objective.

Original: In the other hand, our method excels at addressing the issue of non-optimal ﬁxed points, which is exacerbated by the environment stochasticity.
Revised: In the other hand, our method excels at addressing the issue of non-optimal ﬁxed points, which is exacerbated by the environment stochasticity.

Original: This is because the environment stochasticity increases the difﬁculty of optimization in the target regression and leads to larger approximation error that generates non-optimal ﬁxed points.
Revised: This is because the environment stochasticity increases the difﬁculty of optimization in the target regression and leads to larger approximation error that generates non-optimal ﬁxed points.

Original: From this perspective, our method is a trade-off between the underestimation bias of double Q-learning and the overestimation of the trajectory-based lower-bound estimation.
Revised: From this perspective, our method is a trade-off between the underestimation bias of double Q-learning and the overestimation of the trajectory-based lower-bound estimation.

Original: The effectiveness of such trade-off will investigated empirically in section 5.
Revised: The effectiveness of such trade-off will investigated empirically in section 5.

Original: In this paper, we investigate two methods to compute the estimated target value V
Revised: In this paper, we investigate two methods to compute the estimated target value V

Original: ˆ θ ( s t +1 ) .
Revised: ˆ θ ( s t +1 ) .

Original: The ﬁrst choice V DDQN ˆ θ ( s (cid:48) ) corresponds to the implementation proposed by double deep Q-network (DDQN;
Revised: The ﬁrst choice V DDQN ˆ θ ( s (cid:48) ) corresponds to the implementation proposed by double deep Q-network (DDQN;

Original: Hasselt et al., 2016),
Revised: Hasselt et al., 2016),

Original: [Equation]
Revised: [Equation]

Original: It is an adaption of double Q-learning to the framework of deep Q-network (DQN; Mnih et al., 2015), in which the action selection of target values are produced by the online parameter θ .
Revised: It is an adaption of double Q-learning to the framework of deep Q-network (DQN; Mnih et al., 2015), in which the action selection of target values are produced by the online parameter θ .

Original: An imperfection of the combination with DDQN is that, DDQN is known to suffer from overestimation sometimes since value predictions produced by θ and ˆ θ are not fully independent.
Revised: An imperfection of the combination with DDQN is that, DDQN is known to suffer from overestimation sometimes since value predictions produced by θ and ˆ θ are not fully independent.

Original: When the overestimation happens, the real return signal V τ
Revised: When the overestimation happens, the real return signal V τ

Original: ( s t +1 ) cannot provide any beneﬁts to the value prediction.
Revised: ( s t +1 ) cannot provide any beneﬁts to the value prediction.

Original: To address these concerns, we also adopt another advanced technique called clipped double Q-learning (Fujimoto et al., 2018):
Revised: To address these concerns, we also adopt another advanced technique called clipped double Q-learning (Fujimoto et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: This formulation of computing target values is adapted from its actor-critic version, and we name it clipped double deep Q-network (Clipped DDQN).
Revised: This formulation of computing target values is adapted from its actor-critic version, and we name it clipped double deep Q-network (Clipped DDQN).

Original: This method uses a shared objective function to optimize two copies of value networks Q θ 1 and Q θ 2 , in which the target values are computed by taking the minimum over two separated estimations Q ˆ θ 1 and Q ˆ θ 2 .
Revised: This method uses a shared objective function to optimize two copies of value networks Q θ 1 and Q θ 2 , in which the target values are computed by taking the minimum over two separated estimations Q ˆ θ 1 and Q ˆ θ 2 .

Original: This minimum operator will introduces an underestimation bias (Ciosek et al., 2019) which signiﬁcantly repress the incentive of overestimation in all sources.
Revised: This minimum operator will introduces an underestimation bias (Ciosek et al., 2019) which signiﬁcantly repress the incentive of overestimation in all sources.

Original: This underestimated estimator is appropriate to combine with the lower bounded objective since the lower bound provide by trajectory returns can help to prevent the underestimation error from propagation.
Revised: This underestimated estimator is appropriate to combine with the lower bounded objective since the lower bound provide by trajectory returns can help to prevent the underestimation error from propagation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiment environments are based on the Atari benchmark tasks in OpenAI Gym (Brockman et al., 2016).
Revised: Our experiment environments are based on the Atari benchmark tasks in OpenAI Gym (Brockman et al., 2016).

Original: All baselines and our approaches are implemented using the same set of hyperparameters.
Revised: All baselines and our approaches are implemented using the same set of hyperparameters.

Original: A detailed description of experiment settings is deferred to Appendix B.
Revised: A detailed description of experiment settings is deferred to Appendix B.

Original: Overall performance comparison.
Revised: Overall performance comparison.

Original: We investigated six variants of deep Q-learning algorithms, including DQN (Mnih et al., 2015), double DQN (DDQN; Hasselt et al., 2016), dueling DDQN (Wang et al., 2016), averaged DQN (Anschel et al., 2017), maxmin DQN (Lan et al., 2020), and clipped double DQN adapted from Fujimoto et al.
Revised: We investigated six variants of deep Q-learning algorithms, including DQN (Mnih et al., 2015), double DQN (DDQN; Hasselt et al., 2016), dueling DDQN (Wang et al., 2016), averaged DQN (Anschel et al., 2017), maxmin DQN (Lan et al., 2020), and clipped double DQN adapted from Fujimoto et al.

Original: Our proposed lower bounded objective is built upon two variants of double Q-learning which have clear incentive of underestimation.
Revised: Our proposed lower bounded objective is built upon two variants of double Q-learning which have clear incentive of underestimation.

Original: As shown in Figure 2, the proposed lower bounded objective has great promise in bootstrapping the performance of double Q-learning algorithms.
Revised: As shown in Figure 2, the proposed lower bounded objective has great promise in bootstrapping the performance of double Q-learning algorithms.

Original: The improvement can be observed both in terms of sample efﬁciency and ﬁnal performance.
Revised: The improvement can be observed both in terms of sample efﬁciency and ﬁnal performance.

Original: This advance may credit to the removal of potential nonoptimal ﬁxed points.
Revised: This advance may credit to the removal of potential nonoptimal ﬁxed points.

Original: Another notable observation is that, the effects of adopting clipped double
Revised: Another notable observation is that, the effects of adopting clipped double

Original: Q-learning are environment-dependent, since it has an underestimation bias which cannot be fully eliminated by the lower bounded objective.
Revised: Q-learning are environment-dependent, since it has an underestimation bias which cannot be fully eliminated by the lower bounded objective.

Original: In Appendix C, we present more experimental results to reveal the superiority of the proposed method.
Revised: In Appendix C, we present more experimental results to reveal the superiority of the proposed method.

Original: [Figure]
Revised: [Figure]

Original: Comparison with n -step bootstrapping.
Revised: Comparison with n -step bootstrapping.

Original: In addition to our proposed lower bounded objective, n -step bootstrapping (Sutton, 1988) also has potential to attenuate the issue of nonoptimal ﬁxed points by utilizing real returns.
Revised: In addition to our proposed lower bounded objective, n -step bootstrapping (Sutton, 1988) also has potential to attenuate the issue of nonoptimal ﬁxed points by utilizing real returns.

Original: An advantage of our method is that the performance of n -step bootstrapping is sensitive to the choice of the hyper-parameter n .
Revised: An advantage of our method is that the performance of n -step bootstrapping is sensitive to the choice of the hyper-parameter n .

Original: In Figure 3, we evaluate the performance of DDQN with different values of n chosen by Hessel et al.
Revised: In Figure 3, we evaluate the performance of DDQN with different values of n chosen by Hessel et al.

Original: The results indicate that our proposed lower bounded objective is compatible with nstep bootstrapping.
Revised: The results indicate that our proposed lower bounded objective is compatible with nstep bootstrapping.

Original: The performance can be further improved by combing two approaches.
Revised: The performance can be further improved by combing two approaches.

Original: More experimental results on the comparison with n -step bootstrapping are provided in Appendix D.
Revised: More experimental results on the comparison with n -step bootstrapping are provided in Appendix D.

Original: [Figure]
Revised: [Figure]

Original: comparison more clear, the curves are smoothed by averaging 10 most recent checkpoints.
Revised: comparison more clear, the curves are smoothed by averaging 10 most recent checkpoints.

Original: More experiment results are included in Appendix C.
Revised: More experiment results are included in Appendix C.

Original: [Figure]
Revised: [Figure]

Original: Performance comparison in stochastic environments with sticky actions The experiments presented in Figure 2 and Figure 3 are based on the standard environment setting used by Mnih et al.
Revised: Performance comparison in stochastic environments with sticky actions The experiments presented in Figure 2 and Figure 3 are based on the standard environment setting used by Mnih et al.

Original: (2015), in which the environment stochasticity only comes from the partial observability of visual observations.
Revised: (2015), in which the environment stochasticity only comes from the partial observability of visual observations.

Original: When the environment is stochastic, the trajectories with high return values would cause additional risks of overestimation through the proposed lower bounded objective.
Revised: When the environment is stochastic, the trajectories with high return values would cause additional risks of overestimation through the proposed lower bounded objective.

Original: To investigate the performance of our method in stochastic environments, we adopt a stochastic variant of Atari environment setting with sticky actions Machado et al.
Revised: To investigate the performance of our method in stochastic environments, we adopt a stochastic variant of Atari environment setting with sticky actions Machado et al.

Original: (2018), in which every environ- ment step will execute the agent’s previous action with probability 0.25.
Revised: (2018), in which every environ- ment step will execute the agent’s previous action with probability 0.25.

Original: In this experiment, we investigate two state-of-the-art baselines, Rainbow (Hessel et al., 2018) and C51 (Bellemare et al., 2017), which use distributional representation to address the optimization difﬁculty of environment stochasticity.
Revised: In this experiment, we investigate two state-of-the-art baselines, Rainbow (Hessel et al., 2018) and C51 (Bellemare et al., 2017), which use distributional representation to address the optimization difﬁculty of environment stochasticity.

Original: To fairly compare with these state-of-the-art baselines that integrates many standard techniques, we also combine our proposed lower bounded objective with clipped double Q-learning and multi-step bootstrapping.
Revised: To fairly compare with these state-of-the-art baselines that integrates many standard techniques, we also combine our proposed lower bounded objective with clipped double Q-learning and multi-step bootstrapping.

Original: As shown in Figure 4, our proposed method still outperform baselines in stochastic environments.
Revised: As shown in Figure 4, our proposed method still outperform baselines in stochastic environments.

Original: This result suggests that, comparing to the overestimation bias introduced by the lower bounded objective, stuck in non-optimal ﬁxed points is a more critical issue for deep Q-learning algorithms.
Revised: This result suggests that, comparing to the overestimation bias introduced by the lower bounded objective, stuck in non-optimal ﬁxed points is a more critical issue for deep Q-learning algorithms.

Original: Another explanation is that the environment stochasticity may increase the difﬁculty of optimization and lead to larger approximation error for generating non-optimal ﬁxed points.
Revised: Another explanation is that the environment stochasticity may increase the difﬁculty of optimization and lead to larger approximation error for generating non-optimal ﬁxed points.

Original: Our proposed method is an effective trade-off between the risks of overestimation and nonoptimal ﬁxed points.
Revised: Our proposed method is an effective trade-off between the risks of overestimation and nonoptimal ﬁxed points.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Correcting the estimation bias in double Q-learning is an active topic which induces a series of approaches.
Revised: Correcting the estimation bias in double Q-learning is an active topic which induces a series of approaches.

Original: Weighted double Q-learning (Zhang et al., 2017) considers an importance weight parameter to integrate the overestimated and underestimated estimators.
Revised: Weighted double Q-learning (Zhang et al., 2017) considers an importance weight parameter to integrate the overestimated and underestimated estimators.

Original: Clipped double Q-learning (Fujimoto et al., 2018), which uses a minimum operator in target values, has become the default implementation of most advanced actor-critic algorithms (Haarnoja et al., 2018).
Revised: Clipped double Q-learning (Fujimoto et al., 2018), which uses a minimum operator in target values, has become the default implementation of most advanced actor-critic algorithms (Haarnoja et al., 2018).

Original: Based on clipped double Q-learning, several methods have been investigated to reduce the its underestimation and achieve promising performance Ciosek et al.
Revised: Based on clipped double Q-learning, several methods have been investigated to reduce the its underestimation and achieve promising performance Ciosek et al.

Original: ; Li & Hou (2019).
Revised: ; Li & Hou (2019).

Original: Besides the variants of double Q-learning, there are lots of other techniques proposed regarding the bias-variance trade-off in Q-learning algorithms.
Revised: Besides the variants of double Q-learning, there are lots of other techniques proposed regarding the bias-variance trade-off in Q-learning algorithms.

Original: Similar to our proposed approach, most of existing methods focus on the construction of target values.
Revised: Similar to our proposed approach, most of existing methods focus on the construction of target values.

Original: Averaged DQN (Anschel et al., 2017) uses multiple historical copies of target networks to reduce the variance of estimation.
Revised: Averaged DQN (Anschel et al., 2017) uses multiple historical copies of target networks to reduce the variance of estimation.

Original: The target used by truncated quantile critics (TQC; Kuznetsov et al., 2020) is based on a distributional representation of value functions, which contributes to reduce the overestimation of actor-critic methods.
Revised: The target used by truncated quantile critics (TQC; Kuznetsov et al., 2020) is based on a distributional representation of value functions, which contributes to reduce the overestimation of actor-critic methods.

Original: Using the softmax operator in Bellman operation is also considered as an effective approach to reduce the effects of approximation error (Fox et al., 2016; Asadi & Littman, 2017; Song et al., 2019; Kim et al., 2019).
Revised: Using the softmax operator in Bellman operation is also considered as an effective approach to reduce the effects of approximation error (Fox et al., 2016; Asadi & Littman, 2017; Song et al., 2019; Kim et al., 2019).

Original: The characteristic of our approach is the usage of real return signals, which uses the environment prior to break statistical barriers.
Revised: The characteristic of our approach is the usage of real return signals, which uses the environment prior to break statistical barriers.

Original: Our method is a variant of self-imitation (Oh et al., 2018) and is integrated with the Bellman operation.
Revised: Our method is a variant of self-imitation (Oh et al., 2018) and is integrated with the Bellman operation.

Original: The effectiveness of such a ground truth lower bound is also observed by a recent work (Fujita et al., 2020).
Revised: The effectiveness of such a ground truth lower bound is also observed by a recent work (Fujita et al., 2020).

Original: When the environment carries stochasticity, taking the average return among similar states is a plausible approach to reducing the bias of the trajectorybased estimation.
Revised: When the environment carries stochasticity, taking the average return among similar states is a plausible approach to reducing the bias of the trajectorybased estimation.

Original: From this perspective, the proposed method is also related to the memory-based approaches for nearly deterministic environments, such as episodic control (Blundell et al., 2016;
Revised: From this perspective, the proposed method is also related to the memory-based approaches for nearly deterministic environments, such as episodic control (Blundell et al., 2016;

Original: Pritzel et al., 2017) and graph-based planning (Huang et al., 2019; Eysenbach et al., 2019).
Revised: Pritzel et al., 2017) and graph-based planning (Huang et al., 2019; Eysenbach et al., 2019).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we reveal an interesting fact that, under the effects of approximation error, double
Revised: In this paper, we reveal an interesting fact that, under the effects of approximation error, double

Original: Q-learning may have multiple non-optimal ﬁxed points.
Revised: Q-learning may have multiple non-optimal ﬁxed points.

Original: The main cause of such non-optimal ﬁxed points is the underestimation bias of double Q-learning.
Revised: The main cause of such non-optimal ﬁxed points is the underestimation bias of double Q-learning.

Original: Regarding this issue, we provide some analysis to characterize what kind of Bellman operators may suffer from the same problem, and how the hard cases are constructed.
Revised: Regarding this issue, we provide some analysis to characterize what kind of Bellman operators may suffer from the same problem, and how the hard cases are constructed.

Original: To address the potential risk of converging to non-optimal solutions, we propose an alternative objective function to reduce the underestimation in double Qlearning.
Revised: To address the potential risk of converging to non-optimal solutions, we propose an alternative objective function to reduce the underestimation in double Qlearning.

Original: The main idea of this approach is to leverage real environment returns as a lower bound for value estimation.
Revised: The main idea of this approach is to leverage real environment returns as a lower bound for value estimation.

Original: Our proposed method is simple and easy to be combined with other advanced techniques in deep Q-learning.
Revised: Our proposed method is simple and easy to be combined with other advanced techniques in deep Q-learning.

Original: The experiments show that the proposed method has shown great promise in improving both sample efﬁciency and convergence performance.
Revised: The experiments show that the proposed method has shown great promise in improving both sample efﬁciency and convergence performance.

Original: It achieves a signiﬁcant improvement over baselines algorithms
Revised: It achieves a signiﬁcant improvement over baselines algorithms

Original: A O MITTED S TATEMENTS AND P ROOFS A. T HE R ELATION BETWEEN E STIMATION B IAS AND A PPROXIMATED F IXED P OINTS
Revised: A O MITTED S TATEMENTS AND P ROOFS A. T HE R ELATION BETWEEN E STIMATION B IAS AND A PPROXIMATED F IXED P OINTS

Original: An intuitive characterization of such ﬁxed point solutions is considering one-step estimation bias with respect to the maximum expected value, which is deﬁned as
Revised: An intuitive characterization of such ﬁxed point solutions is considering one-step estimation bias with respect to the maximum expected value, which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ( T V )( s ) corresponds to the precise state value after applying the ground truth Bellman operation.
Revised: where ( T V )( s ) corresponds to the precise state value after applying the ground truth Bellman operation.

Original: The amount of estimation bias E characterizes the deviation from the standard Bellman operator T , which can be regarded as imaginary rewards in ﬁxed point solutions.
Revised: The amount of estimation bias E characterizes the deviation from the standard Bellman operator T , which can be regarded as imaginary rewards in ﬁxed point solutions.

Original: Every approximated ﬁxed point solution under a stochastic Bellman operator can be characterized as the optimal value function in a modiﬁed MDP where only the reward function is changed.
Revised: Every approximated ﬁxed point solution under a stochastic Bellman operator can be characterized as the optimal value function in a modiﬁed MDP where only the reward function is changed.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let (cid:101) V denote an approximation ﬁxed point under a stochastic Bellman operator (cid:101) T .
Revised: Let (cid:101) V denote an approximation ﬁxed point under a stochastic Bellman operator (cid:101) T .

Original: Deﬁne
Revised: Deﬁne

Original: a modiﬁed MDP (cid:102) M = (cid:104)S , A , P, R + (cid:101) R, γ (cid:105) based on M , where the extra reward term is deﬁned as
Revised: a modiﬁed MDP (cid:102) M = (cid:104)S , A , P, R + (cid:101) R, γ (cid:105) based on M , where the extra reward term is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where E is the one-step estimation bias deﬁned in Eq.
Revised: where E is the one-step estimation bias deﬁned in Eq.

Original: Then (cid:101) V is the optimal state-value function of the modiﬁed MDP (cid:102) M .
Revised: Then (cid:101) V is the optimal state-value function of the modiﬁed MDP (cid:102) M .

Original: Proof.
Revised: Proof.

Original: Deﬁne a value function (cid:101) Q based on (cid:101) V , ∀ ( s, a ) ∈ S × A ,
Revised: Deﬁne a value function (cid:101) Q based on (cid:101) V , ∀ ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: We can verify (cid:101) Q is consistent with (cid:101) V , ∀ s ∈ S ,
Revised: We can verify (cid:101) Q is consistent with (cid:101) V , ∀ s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: Let T (cid:102)
Revised: Let T (cid:102)

Original: M denote the Bellman operator of (cid:102) M .
Revised: M denote the Bellman operator of (cid:102) M .

Original: We can verify (cid:101) Q satisﬁes Bellman optimality equation to prove the given statement, ∀ ( s, a ) ∈ S × A ,
Revised: We can verify (cid:101) Q satisﬁes Bellman optimality equation to prove the given statement, ∀ ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: Thus we can see (cid:101) V is the solution of Bellman optimality equation in (cid:102) M .
Revised: Thus we can see (cid:101) V is the solution of Bellman optimality equation in (cid:102) M .

Original: A. T HE E XISTENCE OF A PPROXIMATED F IXED P OINTS
Revised: A. T HE E XISTENCE OF A PPROXIMATED F IXED P OINTS

Original: The key technique for proving the existence of approximated ﬁxed points is Brouwer’s ﬁxed point theorem.
Revised: The key technique for proving the existence of approximated ﬁxed points is Brouwer’s ﬁxed point theorem.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let B =
Revised: Let B =

Original: [ − L, − L ] d denote a d -dimensional bounding box.
Revised: [ − L, − L ] d denote a d -dimensional bounding box.

Original: For any continuous function f : B → B , there exists a ﬁxed point x such that f ( x ) = x ∈ B .
Revised: For any continuous function f : B → B , there exists a ﬁxed point x such that f ( x ) = x ∈ B .

Original: Proof.
Revised: Proof.

Original: It refers to a special case of Brouwer’s ﬁxed point theorem (Brouwer, 1911).
Revised: It refers to a special case of Brouwer’s ﬁxed point theorem (Brouwer, 1911).

Original: Lemma 2.
Revised: Lemma 2.

Original: Let (cid:101) T denote the stochastic Bellman operator deﬁned by Eq.
Revised: Let (cid:101) T denote the stochastic Bellman operator deﬁned by Eq.

Original: There exists a real range L , ∀ V ∈
Revised: There exists a real range L , ∀ V ∈

Original: [ L, − L ] |S| , E [ (cid:101) T V ] ∈
Revised: [ L, − L ] |S| , E [ (cid:101) T V ] ∈

Original: [ L, − L ] |S| .
Revised: [ L, − L ] |S| .

Original: Proof.
Revised: Proof.

Original: Let R max denote the range of the reward function for MDP M .
Revised: Let R max denote the range of the reward function for MDP M .

Original: Let R e denote the range of the noisy term.
Revised: Let R e denote the range of the noisy term.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Note that the L ∞ -norm of state value functions satisﬁes ∀ V ∈ R |S|
Revised: Note that the L ∞ -norm of state value functions satisﬁes ∀ V ∈ R |S|

Original: [Equation]
Revised: [Equation]

Original: We can construct the range L =
Revised: We can construct the range L =

Original: ( R max + R e ) / (1 − γ ) to prove the given statement.
Revised: ( R max + R e ) / (1 − γ ) to prove the given statement.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let (cid:101) T denote the stochastic Bellman operator deﬁned by Eq.
Revised: Let (cid:101) T denote the stochastic Bellman operator deﬁned by Eq.

Original: There exists a real range L , ∀ V ∈
Revised: There exists a real range L , ∀ V ∈

Original: [ L, − L ] |S| , E [ (cid:101) T V ] ∈
Revised: [ L, − L ] |S| , E [ (cid:101) T V ] ∈

Original: [ L, − L ] |S| .
Revised: [ L, − L ] |S| .

Original: Proof.
Revised: Proof.

Original: Let R max denote the range of the reward function for MDP M .
Revised: Let R max denote the range of the reward function for MDP M .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Note that the L ∞ -norm of state value functions satisﬁes
Revised: Note that the L ∞ -norm of state value functions satisﬁes

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: We can construct the range L = R max / (1 − γ ) to prove the given statement.
Revised: We can construct the range L = R max / (1 − γ ) to prove the given statement.

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume the probability density functions of the noise terms { e ( s, a ) } are continuous.
Revised: Assume the probability density functions of the noise terms { e ( s, a ) } are continuous.

Original: The stochastic Bellman operators deﬁned by Eq.
Revised: The stochastic Bellman operators deﬁned by Eq.

Original: (3) and Eq.
Revised: (3) and Eq.

Original: (4) have approximated ﬁxed points deﬁned as Deﬁnition 1.
Revised: (4) have approximated ﬁxed points deﬁned as Deﬁnition 1.

Original: Proof.
Revised: Proof.

Original: Let f ( V ) =
Revised: Let f ( V ) =

Original: E [ (cid:101) T V ] denote the expected return of a stochastic Bellman operation.
Revised: E [ (cid:101) T V ] denote the expected return of a stochastic Bellman operation.

Original: This function is continuous because all involved formulas only contain elementary functions.
Revised: This function is continuous because all involved formulas only contain elementary functions.

Original: The given statement is proved by combining Lemma 1, 2, and 3.
Revised: The given statement is proved by combining Lemma 1, 2, and 3.

Original: A. T HE I NDUCED P OLICY OF D OUBLE Q-L EARNING
Revised: A. T HE I NDUCED P OLICY OF D OUBLE Q-L EARNING

Original: Proposition 3.
Revised: Proposition 3.

Original: Assume the noise terms e 

Revised: Assume the noise terms e

Original: and e 2 are independently generated in the double estimator stated in Eq.
Revised: and e 2 are independently generated in the double estimator stated in Eq.

Original: Every approximated ﬁxed point V is equal to the ground truth value function
Revised: Every approximated ﬁxed point V is equal to the ground truth value function

Original: V ˜ π with respect to an induced policy ˜ π .
Revised: V ˜ π with respect to an induced policy ˜ π .

Original: Proof.
Revised: Proof.

Original: Let V denote an approximated ﬁxed point under the stochastic Bellman operator (cid:101) T deﬁned by Eq.
Revised: Let V denote an approximated ﬁxed point under the stochastic Bellman operator (cid:101) T deﬁned by Eq.

Original: By plugging the deﬁnition of the induced policy into the stochastic operator of double
Revised: By plugging the deﬁnition of the induced policy into the stochastic operator of double

Original: Q-learning, we can get
Revised: Q-learning, we can get

Original: [Equation]
Revised: [Equation]

Original: which matches the Bellman expectation equation.
Revised: which matches the Bellman expectation equation.

Original: As shown by this proposition, the estimated value of a non-optimal ﬁxed point is corresponding to the value of a stochastic policy, which revisits the incentive of double Q-learning to underestimate true maximum values.
Revised: As shown by this proposition, the estimated value of a non-optimal ﬁxed point is corresponding to the value of a stochastic policy, which revisits the incentive of double Q-learning to underestimate true maximum values.

Original: A. A S UFFICIENT C ONDITION FOR M ULTIPLE F IXED P OINTS
Revised: A. A S UFFICIENT C ONDITION FOR M ULTIPLE F IXED P OINTS

Original: Proposition 2.
Revised: Proposition 2.

Original: Let f s ( { ( T Q )( s, a ) } a ∈A ) =
Revised: Let f s ( { ( T Q )( s, a ) } a ∈A ) =

Original: E [( (cid:101) T V )( s )] denote the expected output value of the stochastic Bellman operator (cid:101) T on state s , and assume f s ( · ) is differentiable.
Revised: E [( (cid:101) T V )( s )] denote the expected output value of the stochastic Bellman operator (cid:101) T on state s , and assume f s ( · ) is differentiable.

Original: If a stochastic Bellman operator (cid:101) T satisﬁes Eq.
Revised: If a stochastic Bellman operator (cid:101) T satisﬁes Eq.

Original: (5), there exists an MDP such that (cid:101) T has multiple ﬁxed points.
Revised: (5), there exists an MDP such that (cid:101) T has multiple ﬁxed points.

Original: [Equation]
Revised: [Equation]

Original: where X = { x i } |A| i =1 denotes the input of the function f s .
Revised: where X = { x i } |A| i =1 denotes the input of the function f s .

Original: Proof.
Revised: Proof.

Original: Suppose f s is a function satisfying the given condition, and x i = ¯ x and X denote the corresponding point satisfying Eq.
Revised: Suppose f s is a function satisfying the given condition, and x i = ¯ x and X denote the corresponding point satisfying Eq.

Original: Let g ( x ) denote the value of f s while only changes the input value of x i to x , so that g (cid:48) (¯ x ) > 1 .
Revised: Let g ( x ) denote the value of f s while only changes the input value of x i to x , so that g (cid:48) (¯ x ) > 1 .

Original: Since f s is differentiable, we can ﬁnd a small region ¯ x L < ¯ x < ¯ x R around ¯ x such that ∀ x ∈ [¯ x L , ¯ x R ] , g (cid:48)
Revised: Since f s is differentiable, we can ﬁnd a small region ¯ x L < ¯ x < ¯ x R around ¯ x such that ∀ x ∈ [¯ x L , ¯ x R ] , g (cid:48)

Original: ( x ) > 1 .
Revised: ( x ) > 1 .

Original: And then, we have g (¯ x R ) − g
Revised: And then, we have g (¯ x R ) − g

Original: (¯ x L ) >
Revised: (¯ x L ) >

Original: ¯ x R − ¯ x L .
Revised: ¯ x R − ¯ x L .

Original: Consider to construct an MDP with only one state.
Revised: Consider to construct an MDP with only one state.

Original: We can use the action corresponding to x i to construct a self-loop transition with reward r .
Revised: We can use the action corresponding to x i to construct a self-loop transition with reward r .

Original: All other actions lead to a termination signal and an immediate reward where the immediate rewards correspond to other components of X .
Revised: All other actions lead to a termination signal and an immediate reward where the immediate rewards correspond to other components of X .

Original: By setting the discount factor as γ =
Revised: By setting the discount factor as γ =

Original: [Equation]
Revised: [Equation]

Original: and the reward as r = ¯ x L − γg (¯ x L ) =
Revised: and the reward as r = ¯ x L − γg (¯ x L ) =

Original: ¯ x R − γg (¯ x R ) , we can ﬁnd both ¯ x L and ¯ x R are solutions of the equation x
Revised: ¯ x R − γg (¯ x R ) , we can ﬁnd both ¯ x L and ¯ x R are solutions of the equation x

Original: = r + γg ( x ) , in which g (¯ x L ) and g (¯ x R ) correspond to two ﬁxed points of the constructed MDP.
Revised: = r + γg ( x ) , in which g (¯ x L ) and g (¯ x R ) correspond to two ﬁxed points of the constructed MDP.

Original: A. A B AD C ASE FOR C LIPPED D OUBLE Q-L EARNING
Revised: A. A B AD C ASE FOR C LIPPED D OUBLE Q-L EARNING

Original: The stochastic Bellman operator corresponding to clipped double Q-learning is stated as follows.
Revised: The stochastic Bellman operator corresponding to clipped double Q-learning is stated as follows.

Original: [Equation]
Revised: [Equation]

Original: An MDP where clipped double Q-learning has multiple ﬁxed points is illustrated as Figure 5.
Revised: An MDP where clipped double Q-learning has multiple ﬁxed points is illustrated as Figure 5.

Original: [Figure]
Revised: [Figure]

Original: B E XPERIMENT S ETTINGS B. E VALUATION S ETTINGS
Revised: B E XPERIMENT S ETTINGS B. E VALUATION S ETTINGS

Original: All curves presented in this paper are plotted from the median performance of 5 runs with random initialization.
Revised: All curves presented in this paper are plotted from the median performance of 5 runs with random initialization.

Original: The shaded region indicates 60% population around median.
Revised: The shaded region indicates 60% population around median.

Original: The evaluation is processed in every 50000 timesteps.
Revised: The evaluation is processed in every 50000 timesteps.

Original: Every evaluation point is averaged from 5 trajectories.
Revised: Every evaluation point is averaged from 5 trajectories.

Original: The evaluated policy is combined with a 0.1% random execution.
Revised: The evaluated policy is combined with a 0.1% random execution.

Original: B. H YPER -P ARAMETERS
Revised: B. H YPER -P ARAMETERS

Original: All algorithm investigated in this paper use the same set of hyper-parameters.
Revised: All algorithm investigated in this paper use the same set of hyper-parameters.

Original: • Number of noop actions while starting a new episode: 30; • Number of stacked frames in observations: 4; • Scale of rewards: clipping to [ − 1 , 1] ; • Buffer size: 10 6 ; • Batch size: 32; • Start training: after collecting 20000 transitions; • Training frequency: 4 timesteps; • Target updating frequency: 8000 timesteps; • (cid:15) decaying: from 1 . 0 to 0 .
Revised: • Number of noop actions while starting a new episode: 30; • Number of stacked frames in observations: 4; • Scale of rewards: clipping to [ − 1 , 1] ; • Buffer size: 10 6 ; • Batch size: 32; • Start training: after collecting 20000 transitions; • Training frequency: 4 timesteps; • Target updating frequency: 8000 timesteps; • (cid:15) decaying: from 1 . 0 to 0 .

Original: 01 in the ﬁrst 250000 timesteps; • Optimizer: Adam with ε = 1 . 5 · 10 − 4 ; • Learning rate: 0 .
Revised: 01 in the ﬁrst 250000 timesteps; • Optimizer: Adam with ε = 1 . 5 · 10 − 4 ; • Learning rate: 0 .

Original: · 10 − 4 .
Revised: · 10 − 4 .

Original: C A DDITIONAL E XPERIMENTS ON S TANDARD A TARI B ENCHMARK T ASKS
Revised: C A DDITIONAL E XPERIMENTS ON S TANDARD A TARI B ENCHMARK T ASKS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ''4
Revised: ''4

Original: D A BLATION S TUDIES ON M ULTI -S TEP B OOTSTRAPPING
Revised: D A BLATION S TUDIES ON M ULTI -S TEP B OOTSTRAPPING

Original: [Table]
Revised: [Table]