Original: Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning
Revised: Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Accurate value estimates are important for off-policy reinforcement learning.
Revised: Accurate value estimates are important for off-policy reinforcement learning.

Original: Algorithms based on temporal difference learning typically are prone to an over- orunderestimation bias building up over time.
Revised: Algorithms based on temporal difference learning typically are prone to an over- or underestimation bias building up over time.

Original: In this paper, we propose a generalmethod called Adaptively Calibrated Critics (ACC) that uses the most recent highvariance but unbiased on-policy rollouts to alleviate the bias of the low variancetemporal difference targets.
Revised: In this paper, we propose a general method called Adaptively Calibrated Critics (ACC) that uses the most recent high variance but unbiased on-policy rollouts to alleviate the bias of the low variance temporal difference targets.

Original: We apply ACC to Truncated Quantile Critics [22],which is an algorithm for continuous control that allows regulation of the bias with ahyperparameter tuned per environment.
Revised: We apply ACC to Truncated Quantile Critics [22], which is an algorithm for continuous control that allows regulation of the bias with a hyperparameter tuned per environment.

Original: The resulting algorithm adaptively adjuststhe parameter during training rendering hyperparameter search unnecessary andsets a new state of the art on the OpenAI gym continuous control benchmark amongall algorithms that do not tune hyperparameters for each environment.
Revised: The resulting algorithm adaptively adjusts the parameter during training rendering hyperparameter search unnecessary and sets a new state of the art on the OpenAI gym continuous control benchmark among all algorithms that do not tune hyperparameters for each environment.

Original: Additionally,we demonstrate that ACC is quite general by further applying it to TD3 [11] andshowing an improved performance also in this setting.
Revised: Additionally, we demonstrate that ACC is quite general by further applying it to TD3 [11] and showing an improved performance also in this setting.

Original:  Introduction
Revised:  Introduction

Original: Off-policy reinforcement learning is an important research direction as the reuse of old experiencepromises to make these methods more sample ef ﬁ cient than their on-policy counterparts.
Revised: Off-policy reinforcement learning is an important research direction as the reuse of old experience promises to make these methods more sample efﬁcient than their on-policy counterparts.

Original: This is animportant property for many applications such as robotics where interactions with the environmentare very time- and cost-intensive.
Revised: This is an important property for many applications such as robotics where interactions with the environment are very time- and cost-intensive.

Original: Many successful off-policy methods make use of a learned Q-valuefunction [11, 14, 18, 27].
Revised: Many successful off-policy methods make use of a learned Q-value function [11, 14, 18, 27].

Original: If the action space is discrete the Q-function can be directly used to generateactions while for continuous action spaces it is usually used in an actor-critic setting where the policyis trained to choose actions that maximize the Q-function.
Revised: If the action space is discrete the Q-function can be directly used to generate actions while for continuous action spaces it is usually used in an actor-critic setting where the policy is trained to choose actions that maximize the Q-function.

Original: In both cases accurate estimates of the
Revised: In both cases accurate estimates of the

Original: Q-values are of crucial importance.
Revised: Q-values are of crucial importance.

Original: Unfortunately, learning the Q-function off-policy can lead to an overestimation bias [33].
Revised: Unfortunately learning the Q-function off-policy can lead to an overestimation bias [33].

Original: Especiallywhen a nonlinear function approximator is used to model the Q-function, there are many potentialsources of bias.
Revised: Especially when a nonlinear function approximator is used to model the Q-function, there are many potential sources of bias.

Original: Different heuristics were proposed for their mitigation, such as the double estimatorin the case of discrete action spaces [35] or taking the minimum of two estimates in the case ofcontinuous actions [11].
Revised: Different heuristics were proposed for their mitigation, such as the double estimator in the case of discrete action spaces [35] or taking the minimum of two estimates in the case of continuous actions [11].

Original: While these methods successfully prevent extreme overestimation due totheir coarse nature, they can still induce under- or overestimation bias to a varying degree dependingon the environment [23].
Revised: While these methods successfully prevent extreme overestimation due to their coarse nature, they can still induce under- or overestimation bias to a varying degree depending on the environment [23].

Original: To overcome these problems we propose a principled and general method to alleviate the biascalled Adaptively Calibrated Critics (ACC).
Revised: To overcome these problems we propose a principled and general method to alleviate the bias called Adaptively Calibrated Critics (ACC).

Original: Our algorithm uses the most recent on-policy rollouts to determine the current bias of the Q-estimates and adjusts a bias controlling parameter accordingly.
Revised: Our algorithm uses the most recent on-policy rollouts to determine the current bias of the Q-estimates and adjusts a bias controlling parameter accordingly.

Original: This parameter adapts the size of the temporal difference (TD) targets such that the bias can becorrected in the subsequent updates.
Revised: This parameter adapts the size of the temporal difference (TD) targets such that the bias can be corrected in the subsequent updates.

Original: As the parameter changes slower than the rollout returns, our method still bene ﬁ ts from stable and low-variance temporal difference targets, while it incorporates the information from unbiased but high variance samples from the recent policy to reduce the bias
Revised: As the parameter changes slower than the rollout returns, our method still beneﬁts from stable and low-variance temporal difference targets, while it incorporates the information from unbiased but high variance samples from the recent policy to reduce the bias.

Original: We apply ACC to Truncated Quantile Critics (TQC) [22], which is a recent off-policy actor-criticalgorithm for continuous control showing strong performance on various tasks.
Revised: We apply ACC to Truncated Quantile Critics (TQC) [22], which is a recent off-policy actor-critic algorithm for continuous control showing strong performance on various tasks.

Original: In TQC the bias canbe controlled in a ﬁ negrained way with the help of a hyperparameter that has to be tuned for everyenvironment.
Revised: In TQC the bias can be controlled in a ﬁnegrained way with the help of a hyperparameter that has to be tuned for every environment.

Original: ACC allows to automatically adjusts this parameter online during the training in theenvironment.
Revised: ACC allows to automatically adjusts this parameter online during the training in the environment.

Original: As a result, it eliminates the need to tune this hyperparameter in a new environment,which is very expensive or even infeasible for many applications.
Revised: As a result, it eliminates the need to tune this hyperparameter in a new environment, which is very expensive or even infeasible for many applications.

Original: We evaluate our algorithm on a range of continuous control tasks from OpenAI gym [4] and exceedthe current state of the art results among all algorithms that do not need tuning of environment speci ﬁ chyperparameters.
Revised: We evaluate our algorithm on a range of continuous control tasks from OpenAI gym [4] and exceed the current state of the art results among all algorithms that do not need tuning of environment speciﬁc hyperparameters.

Original: For each environment ACC matches the performance of TQC with the optimalhyperparameter for that environment.
Revised: For each environment ACC matches the performance of TQC with the optimal hyperparameter for that environment.

Original: Further, we show that the automatic bias correction allowsto increase the number of value function updates performed per environment step, which results ineven larger performance gains in the sample-ef ﬁ cient regime.
Revised: Further, we show that the automatic bias correction allows to increase the number of value function updates performed per environment step, which results in even larger performance gains in the sample-efﬁcient regime.

Original: We additionally apply ACC to the TD3algorithm [11] where it also leads to notably improved performance, underscoring the generality ofour proposed method.
Revised: We additionally apply ACC to the TD algorithm [11] where it also leads to notably improved performance, underscoring the generality of our proposed method.

Original: To summarize, the main contributions of this work are:
Revised: To summarize, the main contributions of this work are:

Original: We propose Adaptively Calibrated Critics, a new general algorithm that reduces the bias of valueestimates in a principled fashion with the help of the most recent unbiased on-policy rollouts.2.
Revised: We propose Adaptively Calibrated Critics, a new general algorithm that reduces the bias of value estimates in a principled fashion with the help of the most recent unbiased on-policy rollouts.

Original: As a practical implementation we describe how ACC can be applied to learn a bias controllinghyperparameter of the TQC algorithm and show that the resulting algorithm sets a new state of the art on the OpenAI continuous control benchmark suite.
Revised: As a practical implementation we describe how ACC can be applied to learn a bias controlling hyperparameter of the TQC algorithm and show that the resulting algorithm sets a new state of the art on the OpenAI continuous control benchmark suite.

Original: We demonstrate that ACC is a general algorithm by additionally applying it to TD3.
Revised: We demonstrate that ACC is a general algorithm by additionally applying it to TD3.

Original: To allow for reproducibility of our results we describe our algorithm in detail, report all hyperparameters, use a large number of random seeds for evaluation, and will publish the source code.
Revised: To allow for reproducibility of our results we describe our algorithm in detail, report all hyperparameters, use a large number of random seeds for evaluation, and open sourced the code 1 .

Original:  Background
Revised:  Background

Original: We consider model-free reinforcement learning for episodic tasks with continuous state and actionspaces S and A .
Revised: We consider model-free reinforcement learning for episodic tasks with continuous state and action spaces S and A .

Original: An agent interacts with its environment by selecting an action a t ∈ A in state62 s t ∈ S for every discrete time step t .
Revised: An agent interacts with its environment by selecting an action a t ∈ A in state s t ∈ S for every discrete time step t .

Original: The agent receives a scalar reward r t and observes the newstate s t +1 .
Revised: The agent receives a scalar reward r t and observes the new state s t +1 .

Original: To model this in a mathematical framework we use a Markov decision process, de ﬁ ned bythe tuple ( S , A , P , R , γ ) .
Revised: To model this in a mathematical framework we use a Markov decision process, deﬁned by the tuple ( S , A , P , R , γ ) .

Original: Given an action a ∈ A in state s ∈ S the unknown state transition density
Revised: Given an action a ∈ A in state s ∈ S the unknown state transition density

Original: P de ﬁ nes a distribution over the next state.
Revised: P deﬁnes a distribution over the next state.

Original: Rewards are given to the agent according to the rewardfunction R and future rewards are discounted via the discount factor γ ∈ [0 , 1] .
Revised: Rewards are given to the agent according to the reward function R and future rewards are discounted via the discount factor γ ∈ [0 , 1] .

Original: The goal is to learn a policy π that maps a state s to a distribution over actions such that the sum offuture discounted rewards R t = �
Revised: The goal is to learn a policy π that maps a state s to a distribution over actions such that the sum of future discounted rewards R t = (cid:80)

Original: Ti = t γ i − t r i is maximized.
Revised: Ti = t γ i − t r i is maximized.

Original: We use the term π φ for the policy withparameters φ , that is trained to maximize the expected return J ( φ ) =
Revised: We use the term π φ for the policy with parameters φ , that is trained to maximize the expected return J ( φ ) =

Original: E s i ∼ P ,a i ∼ π
Revised: E s i ∼P ,a i ∼ π

Original: [ R 0 ] .
Revised: [ R 0 ] .

Original: For a given70 state-action pair ( s, a ) the value function is de ﬁ ned as Q π ( s, a ) =
Revised: For a given state-action pair ( s, a ) the value function is deﬁned as Q π ( s, a ) =

Original: E s i ∼ P ,a i ∼ π
Revised: E s i ∼P ,a i ∼ π

Original: [ R t | s, a ] , which is the71 expected return when executing action a in state s and following π afterwards.
Revised: [ R t | s, a ] , which is the expected return when executing action a in state s and following π afterwards.

Original: 2.1 Soft Actor Critic
Revised: 2.1 Soft Actor Critic

Original: TQC extends Soft Actor-Critic (SAC) [14], which is a strong off-policy algorithm for continuouscontrol that uses entropy regularization.
Revised: TQC extends Soft Actor-Critic (SAC) [14], which is a strong off-policy algorithm for continuous control that uses entropy regularization.

Original: This means that while in the end we are interested inmaximizing the performance with respect to the total amount of reward collected in the environment,
Revised: This means that while in the end we are interested in maximizing the performance with respect to the total amount of reward collected in the environment,

Original: SAC maximizes for an auxiliary objective that augments the original reward with the entropy of thepolicy J ( φ ) =
Revised: SAC maximizes for an auxiliary objective that augments the original reward with the entropy of the policy J ( φ ) =

Original: E s t ∼ P ,a t ∼ π [ � t γ t ( r t + α H
Revised: E s t ∼P ,a t ∼ π [ (cid:80) t γ t ( r t + α H

Original: ( π ( ·| s t )))] , where H denotes the entropy.
Revised: ( π ( ·| s t )))] , where H denotes the entropy.

Original: A critic is learned that evaluates the policy π in terms of its Q-value of the entropy augmented reward.
Revised: A critic is learned that evaluates the policy π in terms of its Q-value of the entropy augmented reward.

Original: The policy—called actor—is trained to choose actions such that the Q-function is maximized with anadditional entropy regularization
Revised: The policy—called actor—is trained to choose actions such that the Q-function is maximized with an additional entropy regularization

Original: [Equation]
Revised: [Equation]

Original: The weighting parameter α of the entropy term can be automatically adjusted during the training [15].
Revised: The weighting parameter α of the entropy term can be automatically adjusted during the training [15].

Original: Both the training of actor and critic happen off-policy with transitions sampled from a replay buffer.
Revised: Both the training of actor and critic happen off-policy with transitions sampled from a replay buffer.

Original: 2.2 Truncated Quantile Critics
Revised: 2.2 Truncated Quantile Critics

Original: The TQC algorithm uses distributional reinforcement learning [2] to learn a distribution over thefuture augmented reward instead of a Q-function which is a point estimate for the expectation ofthis quantity.
Revised: The TQC algorithm uses distributional reinforcement learning [2] to learn a distribution over the future augmented reward instead of a Q-function which is a point estimate for the expectation of this quantity.

Original: In TQC this is done with quantile regression [6] which approximates the distributionwith Dirac delta functions Z θ ( s t , a t )
Revised: In TQC this is done with quantile regression [6] which approximates the distribution with Dirac delta functions Z θ ( s t , a t )

Original: = 1 M � Mm =1 δ ( θ m ( s t , a t )) .
Revised: = 1 M (cid:80) Mm =1 δ ( θ m ( s t , a t )) .

Original: The Diracs are located at thequantile locations for fractions
Revised: The Diracs are located at the quantile locations for fractions

Original: τ m = 2 m − 1 m , m ∈ { 1 , . .
Revised: τ m = 2 m − 1 m , m ∈ { 1 , . .

Original: , M } .
Revised: , M } .

Original: The network is trained to learnthe quantile locations θ m ( s, a ) by regressing the predictions θ m ( s t , a t ) onto the Bellman targetsy m ( s t , a t ) =
Revised: The network is trained to learn the quantile locations θ m ( s, a ) by regressing the predictions θ m ( s t , a t ) onto the Bellman targets y m ( s t , a t ) =

Original: r t + γ ( θ m ( s t +1 , a t +1 )
Revised: r t + γ ( θ m ( s t +1 , a t +1 )

Original: − α log π φ ( a t +1 | s t +1 )) via the Huber quantile loss.
Revised: − α log π φ ( a t +1 | s t +1 )) via the Huber quantile loss.

Original: TQC uses an ensemble of N networks ( θ 1 , · · · , θ N ) where each network θ n predicts the distribution
Revised: TQC uses an ensemble of N networks ( θ 1 , · · · , θ N ) where each network θ n predicts the distribution

Original: Z θ n ( s t , a t ) =1 M �
Revised: Z θ n ( s t , a t ) =

Original:  Mm =1 δ ( θ mn ( s t , a t )) .
Revised: 1 M (cid:80) Mm =1 δ ( θ mn ( s t , a t )) .

Original: A single Bellman target distribution is computed for allnetworks.
Revised: A single Bellman target distribution is computed for all networks.

Original: This happens by ﬁ rst computing all targets for all networks, pooling all targets togetherin one set and sorting them in ascending order.
Revised: This happens by ﬁrst computing all targets for all networks, pooling all targets together in one set and sorting them in ascending order.

Original: Let k ∈ { 1 , . .
Revised: Let k ∈ { 1 , . .

Original: , M } , then the kN smallest of thesetargets
Revised: , M } , then the kN smallest of these targets

Original: y i are used to de ﬁ ne the target distribution Y ( s t , a t ) = 1 kN [Equation]
Revised: y i are used to deﬁne the target distribution Y ( s t , a t ) = 1 kN (cid:80) kNi =1 δ ( y i ( s t , a t )) .

Original: δ

Original: [Equation]

Original: i ( s t , a t )) .

Original: The networksare trained by minimizing the quantile Huber loss which in this case is given by
Revised: The networks are trained by minimizing the quantile Huber loss which in this case is given by

Original: [Equation]
Revised: [Equation]

Original: where ρ H τ ( u ) = | τ −  ( u < 0) |L 1 H ( u ) and L 1 H ( u ) is the Huber loss with parameter 1 .
Revised: where ρ Hτ ( u ) = | τ − 1 ( u < 0) |L 1 H ( u ) and L 1 H ( u ) is the Huber loss with parameter 1 .

Original: The rationale behind truncating some quantiles from the target distribution is to prevent overestimationbias.
Revised: The rationale behind truncating some quantiles from the target distribution is to prevent overestimation bias.

Original: In TQC the number of dropped targets per network d = M − k is a hyperparameter that has tobe tuned per environment but allows for a ﬁ negrained control of the bias.
Revised: In TQC the number of dropped targets per network d = M − k is a hyperparameter that has to be tuned per environment but allows for a ﬁnegrained control of the bias.

Original: The policy is trained as in SAC by maximizing the entropy penalized estimate of the Q-value whichis the expectation over the distribution obtained from the critic
Revised: The policy is trained as in SAC by maximizing the entropy penalized estimate of the Q-value which is the expectation over the distribution obtained from the critic

Original: [Equation]
Revised: [Equation]

Original:  Adaptively Calibrated Critics
Revised:  Adaptively Calibrated Critics

Original: In this section, we will ﬁ rst introduce the problem of estimation bias in TD learning.
Revised: In this section, we will ﬁrst introduce the problem of estimation bias in TD learning.

Original: Then we willpresent our method ACC followed by an explanation how it can be applied to TQC.
Revised: Then we will present our method ACC followed by an explanation how it can be applied to TQC.

Original: 3.1 Over- and Underestimation Bias
Revised: 3.1 Over- and Underestimation Bias

Original: The problem of overestimation bias in temporal difference learning with function approximation hasbeen known for a long time [33].
Revised: The problem of overestimation bias in temporal difference learning with function approximation has been known for a long time [33].

Original: In Q-learning [37] the predicted Q-value Q ( s t , a t ) is regressedonto the target given by y =
Revised: In Q-learning [37] the predicted Q-value Q ( s t , a t ) is regressed onto the target given by y =

Original: r t + γ max a Q
Revised: r t + γ max a Q

Original: ( s t +1 , a ) .
Revised: ( s t +1 , a ) .

Original: In the tabular case and under mild assumptionsthe Q-values converge to that of the optimal policy [37] with this update rule.
Revised: In the tabular case and under mild assumptions the Q-values converge to that of the optimal policy [37] with this update rule.

Original: However, using afunction approximator to generate the Q-value introduces an approximation error.
Revised: However, using a function approximator to generate the Q-value introduces an approximation error.

Original: Even under theassumption of zero mean noise corruption of the Q-value E [ � a ] = 0 , an overestimation bias occurs inthe computation of the target value because of Jensen’s inequality
Revised: Even under the assumption of zero mean noise corruption of the Q-value E [ (cid:15) a ] = 0 , an overestimation bias occurs in the computation of the target value because of Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: In continuous action spaces it is not possible to take the maximum over all actions.
Revised: In continuous action spaces it is not possible to take the maximum over all actions.

Original: The mostsuccessful algorithms rely on an actor-critic structure where the actor is trained to choose actions thatmaximize the Q-value [11, 14, 24].
Revised: The most successful algorithms rely on an actor-critic structure where the actor is trained to choose actions that maximize the Q-value [11, 14, 24].

Original: So the actor can be interpreted an approximation to the argmaxof the Q-value.
Revised: So the actor can be interpreted an approximation to the argmax of the Q-value.

Original: With deep neural networks as function approximators other problems such as over-generalization[8, 27] can occur where the updates to Q ( s t , a t ) also increases the target through Q ( s t +1 , a ) for all awhich could lead to divergence.
Revised: With deep neural networks as function approximators other problems such as over-generalization [8, 27] can occur where the updates to Q ( s t , a t ) also increases the target through Q ( s t +1 , a ) for all a which could lead to divergence.

Original: There are many other potential sources for overestimation bias suchas stochasticity of the environment [16] or computing the Q-target from actions that lie outside of thecurrent training data distribution [21].
Revised: There are many other potential sources for overestimation bias such as stochasticity of the environment [16] or computing the Q-target from actions that lie outside of the current training data distribution [21].

Original: While for discrete action spaces the overestimation can be controlled with the double estimator[16, 35], it was shown that this estimator does not prevent overestimation when the action space iscontinuous [11].
Revised: While for discrete action spaces the overestimation can be controlled with the double estimator [16, 35], it was shown that this estimator does not prevent overestimation when the action space is continuous [11].

Original: As a solution the TD3 algorithm [11] uses the minimum of two separate estimatorsto compute the critic target.
Revised: As a solution the TD3 algorithm [11] uses the minimum of two separate estimators to compute the critic target.

Original: This approach was shown to prevent overestimation but can introducean underestimation bias.
Revised: This approach was shown to prevent overestimation but can introduce an underestimation bias.

Original: In TQC [22] the problem is handled by dropping some targets from thepooled set of all targets of an ensemble of distributional critics.
Revised: In TQC [22] the problem is handled by dropping some targets from the pooled set of all targets of an ensemble of distributional critics.

Original: This allows for more ﬁ negrainedcontrol of over- or underestimation by choosing how many targets are dropped.
Revised: This allows for more ﬁnegrained control of over- or underestimation by choosing how many targets are dropped.

Original: TQC is able toachieve an impressive performance but the parameter d determining the number of dropped targetshas to be set for each environment individually.
Revised: TQC is able to achieve an impressive performance but the parameter d determining the number of dropped targets has to be set for each environment individually.

Original: This is highly undesirable for many applicationssince the hyperparameter sweep to determine a good choice of the parameter increases the actualnumber of environment interactions proportional to the number of hyperparameters tested.
Revised: This is highly undesirable for many applications since the hyperparameter sweep to determine a good choice of the parameter increases the actual number of environment interactions proportional to the number of hyperparameters tested.

Original: For manyapplications like robotics this makes the training prohibitively expensive.
Revised: For many applications like robotics this makes the training prohibitively expensive.

Original: 3.2 Dynamically Adjusting the Bias
Revised: 3.2 Dynamically Adjusting the Bias

Original: In the following we present a new general approach to adaptively control bias emerging in TD targetsregardless of the source of the bias.
Revised: In the following we present a new general approach to adaptively control bias emerging in TD targets regardless of the source of the bias.

Original: Let R π ( s, a ) be the random variable denoting the sum of futurediscounted rewards when the agent starts in state s , executes action a and follows policy π afterwards.
Revised: Let R π ( s, a ) be the random variable denoting the sum of future discounted rewards when the agent starts in state s , executes action a and follows policy π afterwards.

Original: This means that the Q-value is de ﬁ ned as its expectation Q π ( s, a ) =
Revised: This means that the Q-value is deﬁned as its expectation Q π ( s, a ) =

Original: E [ R π ( s, a )] .
Revised: E [ R π ( s, a )] .

Original: For notationalconvenience we will drop the dependency on the policy π in the following.
Revised: For notational convenience we will drop the dependency on the policy π in the following.

Original: We start with the tabularcase.
Revised: We start with the tabular case.

Original: Suppose for each state-action pair ( s, a ) we have a family { ˆ Q β ( s, a ) } β ∈ [ β min , β max ] ⊂ R ofestimators for Q ( s, a ) with the property that ˆ Q β min ( s, a ) ≤ Q ( s, a ) ≤ ˆ Q β max ( s, a ) , where Q ( s, a )is the true Q-value of the policy π and Q β a continuous monotone increasing function in β .
Revised: Suppose for each state-action pair ( s, a ) we have a family { ˆ Q β ( s, a ) } β ∈ [ β min ,β max ] ⊂ R of estimators for Q ( s, a ) with the property that ˆ Q β min ( s, a ) ≤ Q ( s, a ) ≤ ˆ Q β max ( s, a ) , where Q ( s, a ) is the true Q-value of the policy π and Q β a continuous monotone increasing function in β .

Original: If we have samples from R i ∼ R ( s, a ) an unbiased estimator for Q ( s, a ) is given by the average ofthe R i .
Revised: If we have samples from R i ∼ R ( s, a ) an unbiased estimator for Q ( s, a ) is given by the average of the R i .

Original: This is also called Monte Carlo estimation [32].
Revised: This is also called Monte Carlo estimation [32].

Original: We further de ﬁ ne the estimator ˆ Q β ∗ ( s, a ) ,where β ∗ is given by
Revised: We further deﬁne the estimator ˆ Q β ∗ ( s, a ) , where β ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: In the following Theorem we show that the estimator is unbiased under some assumptions.
Revised: In the following Theorem we show that the estimator is unbiased under some assumptions.

Original: Theorem 1 Let Q β ( s, a ) be a continuous monotone increasing function in β and assume that forall ( s, a ) it holds ˆ Q β min ( s, a ) ≤ Q ( s, a )
Revised: Theorem 1 Let Q β ( s, a ) be a continuous monotone increasing function in β and assume that for all ( s, a ) it holds ˆ Q β min ( s, a ) ≤ Q ( s, a )

Original: ≤ ˆ Q β max ( s, a ) , the returns R ( s, a ) follow a symmetricprobability distribution and that ˆ Q β min ( s, a ) and ˆ Q β max ( s, a ) have the same distance to Q ( s, a ) .
Revised: ≤ ˆ Q β max ( s, a ) , the returns R ( s, a ) follow a symmetric probability distribution and that ˆ Q β min ( s, a ) and ˆ Q β max ( s, a ) have the same distance to Q ( s, a ) .

Original: Then Q β ∗ from Equation 5 is an unbiased estimator for the true value Q for all ( s, a ) .
Revised: Then Q β ∗ from Equation 5 is an unbiased estimator for the true value Q for all ( s, a ) .

Original: The proof is provided in the appendix.
Revised: The proof is provided in the appendix.

Original: The symmetry and same distance assumption can be replacedby assuming that ˆ Q β min ( s, a ) ≤ R i ≤ ˆ Q β max ( s, a ) with probability one.
Revised: The symmetry and same distance assumption can be replaced by assuming that ˆ Q β min ( s, a ) ≤ R i ≤ ˆ Q β max ( s, a ) with probability one.

Original: In this case the proof isstraightforward since Q β can take any value for which R i has positive mass.
Revised: In this case the proof is straightforward since Q β can take any value for which R i has positive mass.

Original: We are interested in the case where ˆ Q is given by a function approximator such that there isgeneralization between state-action pairs and that it is possible to generate estimates for pairs forwhich there are no samples of the return available.
Revised: We are interested in the case where ˆ Q is given by a function approximator such that there is generalization between state-action pairs and that it is possible to generate estimates for pairs for which there are no samples of the return available.

Original: Consider off-policy TD learning where thesamples for updates of the Q-function are sampled from a replay buffer of past experience.
Revised: Consider off-policy TD learning where the samples for updates of the Q-function are sampled from a replay buffer of past experience.

Original: Whilethe above assumptions might not hold anymore in this case, we have an estimator for all state-actionpairs and not just the ones for which we have samples of the return.
Revised: While the above assumptions might not hold anymore in this case, we have an estimator for all state-action pairs and not just the ones for which we have samples of the return.

Original: Also in practice rolling outthe policy several times from each state action pair is undesirable and so we set N = 1 whichallows the use of the actual exploration rollouts.
Revised: Also in practice rolling out the policy several times from each state action pair is undesirable and so we set N = 1 which allows the use of the actual exploration rollouts.

Original: Our proposed algorithm starts by initializing thebias-controlling parameter β to some value.
Revised: Our proposed algorithm starts by initializing the bias-controlling parameter β to some value.

Original: After a number of environment steps and when the nextepisode is ﬁ nished, the Q-value estimates and actual observed returns are compared.
Revised: After a number of environment steps and when the next episode is ﬁnished, the Q-value estimates and actual observed returns are compared.

Original: Depending onthe difference β is adjusted according to
Revised: Depending on the difference β is adjusted according to

Original: [Equation]
Revised: [Equation]

Original:  where α is a step size parameter and ( s t , a t ) T β t =1 are the T β ∈ N most recent state-action pairs.
Revised: (6) where α is a step size parameter and ( s t , a t ) T β t =1 are the T β ∈ N most recent state-action pairs.

Original: As aresult β is decreased in the case of overestimation, where the Q-estimates are larger than the actual observed returns, and increased in the case of underestimation.
Revised: As a result β is decreased in the case of overestimation, where the Q-estimates are larger than the actual observed returns, and increased in the case of underestimation.

Original: We assumed that Q β is continuousand monotonically increasing in β .
Revised: We assumed that Q β is continuous and monotonically increasing in β .

Original: Hence, increasing β increases Q β and vice versa.
Revised: Hence, increasing β increases Q β and vice versa.

Original: For updatingthe Q-function the target will be computed from Q β .
Revised: For updating the Q-function the target will be computed from Q β .

Original: Only performing one update step and not the complete minimization from Equation 5 has theadvantage that β is changing relatively slow which means the targets are more stable.
Revised: Only performing one update step and not the complete minimization from Equation 5 has the advantage that β is changing relatively slow which means the targets are more stable.

Original: Through thismechanism our method can incorporate the high variance on-policy samples to correct for under- oroverestimation bias.
Revised: Through this mechanism our method can incorporate the high variance on-policy samples to correct for under- or overestimation bias.

Original: At the same time our method can bene ﬁ t from the low variance TD targets.
Revised: At the same time our method can beneﬁt from the low variance TD targets.

Original: ACCin its general form is summarized in Algorithm 1 in the appendix.
Revised: ACC in its general form is summarized in Algorithm 1 in the appendix.

Original: Other algorithms that attempt to control the bias arising in TD learning with non-linear functionapproximators usually use some kind of heuristic that includes more than one estimator.
Revised: Other algorithms that attempt to control the bias arising in TD learning with non-linear function approximators usually use some kind of heuristic that includes more than one estimator.

Original: Someapproaches use them to decouple the choice of the maximizing action and the evaluation of themaximum in the computation of the TD targets [35].
Revised: Some approaches use them to decouple the choice of the maximizing action and the evaluation of the maximum in the computation of the TD targets [35].

Original: Alternative approaches take the minimum,maximum or a combination of both over the different estimators [1, 11, 12, 23].
Revised: Alternative approaches take the minimum, maximum or a combination of both over the different estimators [1, 11, 12, 23].

Original: All of these have incommon that the same level of bias correction is done for every environment and for all time stepsduring training.
Revised: All of these have in common that the same level of bias correction is done for every environment and for all time steps during training.

Original: In the deep case there are many different sources that can in ﬂ uence the tendency of
Revised: In the deep case there are many different sources that can inﬂuence the tendency of

Original: TD learning building up bias in non-trivial ways.
Revised: TD learning building up bias in non-trivial ways.

Original: ACC is more principled in the regard that it allowsto dynamically adjust the magnitude and direction of bias correction during training.
Revised: ACC is more principled in the regard that it allows to dynamically adjust the magnitude and direction of bias correction during training.

Original: Regardless ofthe source and amount of bias ACC provides a way to alleviate it.
Revised: Regardless of the source and amount of bias ACC provides a way to alleviate it.

Original: This makes ACC promising towork robustly on a wide range of different environments.
Revised: This makes ACC promising to work robustly on a wide range of different environments.

Original: One assumption of ACC is that there is a way to adjust the estimated Q-value with a parameter βsuch that ˆ Q β is continuous and monotonically increasing in β .
Revised: One assumption of ACC is that there is a way to adjust the estimated Q-value with a parameter β such that ˆ Q β is continuous and monotonically increasing in β .

Original: There are many different functionsthat are in accordance with this assumption.
Revised: There are many different functions that are in accordance with this assumption.

Original: We give one general example of how such a ˆ Q β can beeasily constructed for any algorithm that learns a Q-value.
Revised: We give one general example of how such a ˆ Q β can be easily constructed for any algorithm that learns a Q-value.

Original: Let ˆ Q be the current estimate.
Revised: Let ˆ Q be the current estimate.

Original: Then de ﬁ neˆ Q β = β | ˆ Q | /K
Revised: Then deﬁne ˆ Q β = β | ˆ Q | /K

Original: + ˆ Q , where K is a constant (e.g. 100 ) and [ β min , β max ] is some interval around 0 .
Revised: + ˆ Q , where K is a constant (e.g. 100 ) and [ β min , β max ] is some interval around 0 .

Original: In the following section we will present an application of ACC in a more sophisticated way.
Revised: In the following section we will present an application of ACC in a more sophisticated way.

Original: 3. Applying ACC to TQC
Revised: 3. Applying ACC to TQC

Original: As a practical instantiation of the general ACC algorithm we apply it to adjust the number of targetsdropped from the set of all targets in TQC.
Revised: As a practical instantiation of the general ACC algorithm we apply it to adjust the number of targets dropped from the set of all targets in TQC.

Original: Denote with d max ∈ { 0 , . .
Revised: Denote with d max ∈ { 0 , . .

Original: , M } some upper limit oftargets to drop per network.
Revised: , M } some upper limit of targets to drop per network.

Original: De ﬁ ne β min = 0 , β max = d max and let d = d max − β be the currentnumber of targets dropped for each network.
Revised: Deﬁne β min = 0 , β max = d max and let d = d max − β be the current number of targets dropped for each network.

Original: Further, we write Q β for the TQC estimate with dNtargets dropped from the pooled set of all targets.
Revised: Further, we write Q β for the TQC estimate with dN targets dropped from the pooled set of all targets.

Original: If d max is set high enough the TQC estimatewithout dropped targets Q β max induces overestimation while the TQC estimate with d max droppedtargets per net Q β min induces underestimation.
Revised: If d max is set high enough the TQC estimate without dropped targets Q β max induces overestimation while the TQC estimate with d max dropped targets per net Q β min induces underestimation.

Original: In general, β ∈ [0 , d max ] is continuous and hence also d is a continuous value.
Revised: In general, β ∈ [0 , d max ] is continuous and hence also d is a continuous value.

Original: However, the numberof dropped targets from the pooled set of all targets has to be a discrete number in { 0 , . . . , NM } .
Revised: However, the number of dropped targets from the pooled set of all targets has to be a discrete number in { 0 , . . . , NM } .

Original: Thus, in the computation of the TD target the total number of dropped targets dN is rounded to thenearest integer.
Revised: Thus, in the computation of the TD target the total number of dropped targets dN is rounded to the nearest integer.

Original: We use a normalization when updating β .
Revised: We use a normalization when updating β .

Original: A moving average of the absolute value ofthe difference between returns and estimated Q-values is stored.
Revised: A moving average of the absolute value of the difference between returns and estimated Q-values is stored.

Original: When updating β with Equation 6,the expectation is divided by the moving average.
Revised: When updating β with Equation 6, the expectation is divided by the moving average.

Original:  Experiments
Revised:  Experiments

Original: We evaluate our algorithm on a range of continuous control tasks from OpenAI Gym [4] that makesuse of the physics engine MuJoCo [34] (version 1.5).
Revised: We evaluate our algorithm on a range of continuous control tasks from OpenAI Gym [4] that makes use of the physics engine MuJoCo [34] (version 1.5).

Original: First, we benchmark ACC against strongmethods that do not use environment speci ﬁ c hyerparameters.
Revised: First, we benchmark ACC against strong methods that do not use environment speciﬁc hyerparameters.

Original: Then we compare the performanceof TQC with a ﬁ xed number of dropped targets per network with that of ACC.
Revised: Then we compare the performance of TQC with a ﬁxed number of dropped targets per network with that of ACC.

Original: Next, we evaluatethe effect of more critic updates for ACC and show results in the sample ef ﬁ cient regime.
Revised: Next, we evaluate the effect of more critic updates for ACC and show results in the sample efﬁcient regime.

Original: Further,we study the effect of ACC on the accuracy of the value estimate, and investigate the generality of
Revised: Further, we study the effect of ACC on the accuracy of the value estimate, and investigate the generality of

Original: ACC by applying it to TD3.
Revised: ACC by applying it to TD3.

Original: To give more insight into the learning dynamics we also analyze how thecalibration parameter develops during training in the appendix.
Revised: To give more insight into the learning dynamics we also analyze how the calibration parameter develops during training in the appendix.

Original: We implemented ACC on top of the PyTorch code published by the authors 1 to ensure a faircomparison.
Revised: We implemented ACC on top of the PyTorch code published by the authors 2 to ensure a fair comparison.

Original: Because in TQC the tuned optimal number of dropped targets per network is for every
Revised: Because in TQC the tuned optimal number of dropped targets per network is for every

Original: [Figure]
Revised: [Figure]

Original: v
Revised: r a g

Original: [Figure]
Revised: [Figure]

Original: r a g ee v a l 
Revised: a l u a t i o n

Original: [Figure]
Revised: [Figure]

Revised: v e r a g ee v a l u a t

Original: a t i o 
Revised: i o n r

Original: [Figure]
Revised: [Figure]

Revised: t

Original:  environment set in the interval [0 , 5] , we set d max = 5 .
Revised: [Figure] r n [Figure] environment set in the interval [0 , 5] , we set d max = 5 .

Original: At the beginning of the training we initializeβ = 2 . 5 and set the step size parameter to α = 0 .
Revised: At the beginning of the training we initialize β = 2 . 5 and set the step size parameter to α = 0 .

Original: We spend only a very limited amount ofcomputation time into into the tuning of the previously mentioned hyperparameters and describein the appendix in detail the process of choosing the selected hyperparameters.
Revised: We spend only a very limited amount of computation time into into the tuning of the previously mentioned hyperparameters and describe in the appendix in detail the process of choosing the selected hyperparameters.

Original: We also presentpseudocode for our method as well as the complete list of all hyperparameters in the appendix.
Revised: We also present pseudocode for our method as well as the complete list of all hyperparameters in the appendix.

Original: Compared to TQC the additional computational overhead caused by ACC is minimal because there isonly one update to β that is very cheap compared to one training step of the actor-critic and there areat least T β = 1000 training steps in between one update to β .
Revised: Compared to TQC the additional computational overhead caused by ACC is minimal because there is only one update to β that is very cheap compared to one training step of the actor-critic and there are at least T β = 1000 training steps in between one update to β .

Original: During training, the policy is evaluated every 1,000 environment steps by taking the average over theepisode reward obtained by rolling out the current policy without sampling noise 10 times.
Revised: During training, the policy is evaluated every 1,000 environment steps by taking the average over the episode reward obtained by rolling out the current policy without sampling noise 10 times.

Original: For eachtask and algorithm we average the results of 10 trials each with a different random seed.
Revised: For each task and algorithm we average the results of 10 trials each with a different random seed.

Original: 4.1 Comparative Evaluation
Revised: 4.1 Comparative Evaluation

Original: We compare ACC to the state of the art continuous control methods SAC [14] (with learned temperature parameter [15]) and TD3 [11].
Revised: We compare ACC to the state of the art continuous control methods SAC [14] (with learned temperature parameter [15]) and TD3 [11].

Original: The learning curves are shown in Figure 1a.
Revised: The learning curves are shown in Figure 1a.

Original: On all sixenvironments ACC achieves considerably better results setting a new state of the art among allalgorithms without environment speci ﬁ c hyperparameters.
Revised: On all six environments ACC achieves considerably better results setting a new state of the art among all algorithms without environment speciﬁc hyperparameters.

Original: 4.2 Fixing the Number of Dropped Targets
Revised: 4.2 Fixing the Number of Dropped Targets

Original: In this experiment we evaluate how well ACC performs when compared to TQC where we the numberof dropped targets per network d is ﬁ xed to some value.
Revised: In this experiment we evaluate how well ACC performs when compared to TQC where we the number of dropped targets per network d is ﬁxed to some value.

Original: Since in the original publication for eachenvironment the optimal value was one of the three values 0 , 2 , and 5 , we evaluated TQC with d ﬁ xedto one of these values for each environment.
Revised: Since in the original publication for each environment the optimal value was one of the three values 0 , 2 , and 5 , we evaluated TQC with d ﬁxed to one of these values for each environment.

Original: To ensure comparability we used the same codebaseas for ACC.
Revised: To ensure comparability we used the same codebase as for ACC.

Original: The results are shown in Figure 1b.
Revised: The results are shown in Figure 1b.

Original: It can be seen that ACC matches the performancesof TQC with the best hyperparameter in every environment.
Revised: It can be seen that ACC matches the performances of TQC with the best hyperparameter in every environment.

Original: Furthermore, there is no single choice
Revised: Furthermore, there is no single choice

Original: ��� ��� ��� ��� ��� ������ � ���� ���� ���� ���� ����� ����� ����� ����� � � � � � � � � � � � � � � � � � � � � � � � � � ����������� ���

Original: [Figure]

Original: ���� ���� ���� ���� ���� ���� ��������

Original: [Figure]

Original: ���� ���� ���� ���� ��� ��� ��� ��� ��� ��� ��� ����������������� ��� � ���� ���� ���� ���� ���� ���� ���� � � � � � � � � � � � � � � � � � � � � � � � � � �������� ������������������������������������������������������ � � � � � � ����������������� ��� � �� �� �� ��� ��� ��� ������� � � � � �

Original: [Figure]

Original: �����������������

Original: [Figure]

Original: �

Original: [Figure]
Revised: [Figure]

Original: that can compete with ACC on all environments.
Revised: that can compete with ACC on all environments.

Original: With d = 0 , TQC is substantially worse on threeenvironments and unstable on the Ant environment.
Revised: With d = 0 , TQC is substantially worse on three environments and unstable on the Ant environment.

Original: Setting d = 2 is overall the best choice but stillperforms clearly worse for two environments and is also slightly worse for Humanoid .
Revised: Setting d = 2 is overall the best choice but still performs clearly worse for two environments and is also slightly worse for Humanoid .

Original: Droppingd = 5 targets per network leads to an algorithm that can compete with ACC only on two of the sixenvironments.
Revised: Dropping d = 5 targets per network leads to an algorithm that can compete with ACC only on two of the six environments.

Original: The experiments showed that it is not possible to ﬁ nd one value for d that performswell on all environments.
Revised: The experiments showed that it is not possible to ﬁnd one value for d that performs well on all environments.

Original: Furthermore, even if there would be one tuned parameter that performsequally well as ACC on a given set of environments we hypothesize there are likely very differentenvironments for which the speci ﬁ c parameter choice will not perform well.
Revised: Furthermore, even if there would be one tuned parameter that performs equally well as ACC on a given set of environments we hypothesize there are likely very different environments for which the speciﬁc parameter choice will not perform well.

Original: The principled nature of
Revised: The principled nature of

Original: ACC on the other hand provides reason to believe that it can perform robustly on a wide range ofdifferent environments.
Revised: ACC on the other hand provides reason to believe that it can perform robustly on a wide range of different environments.

Original: This is supported by the robust performance on all considered environments.
Revised: This is supported by the robust performance on all considered environments.

Original: 4.3 Evaluation of Sample Ef ﬁ cient Variant
Revised: 4.3 Evaluation of Sample Efﬁcient Variant

Original: In principle more critic updates per environment step should make learning faster.
Revised: In principle more critic updates per environment step should make learning faster.

Original: However, becauseof the bootstrapping in the target computation this can easily become unstable.
Revised: However, because of the bootstrapping in the target computation this can easily become unstable.

Original: The problem is thatas targets are changing faster, bias can build up easier and divergence becomes more likely.
Revised: The problem is that as targets are changing faster, bias can build up easier and divergence becomes more likely.

Original: ACCprovides a way to detect upbuilding bias in the TD targets and to correct the bias accordingly.
Revised: ACC provides a way to detect upbuilding bias in the TD targets and to correct the bias accordingly.

Original: Thismotivates to increase the number of gradient updates of the critic.
Revised: This motivates to increase the number of gradient updates of the critic.

Original: In TD3, SAC and TQC onecritic update is performed per environment step.
Revised: In TD3, SAC and TQC one critic update is performed per environment step.

Original: We conducted an experiment to study the effect ofincreasing this rate up to 4 .
Revised: We conducted an experiment to study the effect of increasing this rate up to 4 .

Original: ACC using 4 , 2 and 1 updates are denoted with ACC_4q, ACC_2q and
Revised: ACC using 4 , 2 and 1 updates are denoted with ACC_4q, ACC_2q and

Original: ACC_1q respectively.
Revised: ACC_1q respectively.

Original: ACC_1q is equal to ACC from the previous experiments.
Revised: ACC_1q is equal to ACC from the previous experiments.

Original: We use the samenotation also for TD3 and SAC.
Revised: We use the same notation also for TD3 and SAC.

Original: Scaling the number of critic updates by a factor of 4 increases the computation time by a factor of4 .
Revised: Scaling the number of critic updates by a factor of 4 increases the computation time by a factor of 4 .

Original: But this can be worthwhile in the sample ef ﬁ cient regime, where a huge number of environmentinteractions is not possible or the interaction cost dominate the computational costs.
Revised: But this can be worthwhile in the sample efﬁcient regime, where a huge number of environment interactions is not possible or the interaction cost dominate the computational costs.

Original: This for examplethe case if the aim is to train robots in the real world.
Revised: This for example the case if the aim is to train robots in the real world.

Original: The results of our experiment are shown in
Revised: The results of our experiment are shown in

Original: Figure 2a.
Revised: Figure 2a.

Original: It can be seen that in the sample ef ﬁ cient regime ACC4q further increases over plain ACC.
Revised: It can be seen that in the sample efﬁcient regime ACC4q further increases over plain ACC.

Original: ACC4q reaches the performance that TD3 and SAC achieve at the end of the training in less than athird of the amount of steps for ﬁ ve environments.
Revised: ACC4q reaches the performance that TD3 and SAC achieve at the end of the training in less than a third of the amount of steps for ﬁve environments.

Original: On Humanoid it needs roughly half the numberof steps for that.
Revised: On Humanoid it needs roughly half the number of steps for that.

Original: Increasing the number of critic updates for TD3 and SAC shows mixed results,increasing performance on some environment while decreasing performance on others.
Revised: Increasing the number of critic updates for TD3 and SAC shows mixed results, increasing performance on some environment while decreasing performance on others.

Original: Only ACCbene ﬁ ts from more critic updates on all environments, which supports the hypothesis that ACC issuccessful at calibrating the critic estimate.
Revised: Only ACC beneﬁts from more critic updates on all environments, which supports the hypothesis that ACC is successful at calibrating the critic estimate.

Original: 4.4 Analysis of the Value Estimate
Revised: 4.4 Analysis of the Value Estimate

Original: To better understand the effect of ACC on the bias of the value estimate, we analyze the differencebetween the value estimate and the corresponding observed return when ACC is applied to TQC. 
Revised: To better understand the effect of ACC on the bias of the value estimate, we analyze the difference between the value estimate and the corresponding observed return when ACC is applied to TQC. For

Original: For each state-action pair encountered during exploration, we compute its value estimate at that timeand at the end of the episode compare it with the actual discounted return from that state onwards.
Revised: [Figure] HalfCheetah [Figure] each state-action pair encountered during exploration, we compute its value estimate at that time and at the end of the episode compare it with the actual discounted return from that state onwards.

Original: Hence, the state-action pair was not used to update the value function at the point when the valueestimate has been computed.
Revised: Hence, the state-action pair was not used to update the value function at the point when the value estimate has been computed.

Original: If an episode ends because the maximum number of episode time-stepshas been reached, which is 1,000 for the considered environments, we ignore the last 100 state-actionpairs.
Revised: If an episode ends because the maximum number of episode time-steps has been reached, which is 1,000 for the considered environments, we ignore the last 100 state-action pairs.

Original: The reason is that in TQC the value estimator is trained to ignore the episode timeout anduses a bootstrapped target also at the end of the episode.
Revised: The reason is that in TQC the value estimator is trained to ignore the episode timeout and uses a bootstrapped target also at the end of the episode.

Original: We normalize for different value scalesby computing the absolute error between the value estimate and the observed discounted return anddivide that by the absolute value of the discounted return.
Revised: We normalize for different value scales by computing the absolute error between the value estimate and the observed discounted return and divide that by the absolute value of the discounted return.

Original: Every 1,000 steps, the average over theerrors of the last 1,000 state-action pairs is computed.
Revised: Every 1,000 steps, the average over the errors of the last 1,000 state-action pairs is computed.

Original: The results are plotted in Figure 3 and showthat ACC indeed achieves a low value error compared to TQC with ﬁ xed hyperparameter.
Revised: The results are plotted in Figure 3 and show that ACC indeed achieves a low value error compared to TQC with ﬁxed hyperparameter.

Original: Thissupports our hypothesis that the strong performance of ACC applied to TQC indeed stems from bettervalues estimates.
Revised: This supports our hypothesis that the strong performance of ACC applied to TQC indeed stems from better values estimates.

Original: 4.5 Beyond TQC: Improving TD3 with ACC
Revised: 4.5 Beyond TQC: Improving TD3 with ACC

Original: To demonstrate the generality of ACC, we additionally applied it to TD3 algorithm [11], whichuses a actor-critic framework with two critics.
Revised: To demonstrate the generality of ACC, we additionally applied it to TD3 algorithm [11], which uses a actor-critic framework with two critics.

Original: These are initialized differently but are trained withthe same target value, which is the minimum over the two targets computed from the two critics.
Revised: These are initialized differently but are trained with the same target value, which is the minimum over the two targets computed from the two critics.

Original: This is done to prevent overestimation in the value estimates.
Revised: This is done to prevent overestimation in the value estimates.

Original: While successfully preventing theoverestimation, using the minimum of the two target estimates is very coarse and can instead lead toan underestimation bias.
Revised: While successfully preventing the overestimation, using the minimum of the two target estimates is very coarse and can instead lead to an underestimation bias.

Original: We applied ACC to TD3 by de ﬁ ning the target for each critic network to bea convex combination between its own target and the target estimate coming from the minimum ofboth:
Revised: We applied ACC to TD3 by deﬁning the target for each critic network to be a convex combination between its own target and the target estimate coming from the minimum of both:

Original: [Equation]
Revised: [Equation]

Original: where β ∈ [0 , 1] is the ACC parameter that is adjusted to balance between under- and overestimation.
Revised: where β ∈ [0 , 1] is the ACC parameter that is adjusted to balance between under- and overestimation.

Original: The results are displayed in Figure 2b and show that ACC also improves the performance of TD3.
Revised: The results are displayed in Figure 2b and show that ACC also improves the performance of TD3.

Original:  Related Work 
Revised:  Related Work 5.1 Overestimation in Reinforcement Learning

Original: 5.1 Overestimation in Reinforcement Learning

Original: The problem of overestimation in Q-learning with function approximation was introduced by [33].
Revised: The problem of overestimation in Q-learning with function approximation was introduced by [33].

Original: For discrete actions the double estimator has been proposed [16] where two Q-functions are learnedand one is used to determine the maximizing action, while the other evaluates the Q-function forthat action.
Revised: For discrete actions the double estimator has been proposed [16] where two Q-functions are learned and one is used to determine the maximizing action, while the other evaluates the Q-function for that action.

Original: The Double DQN algorithm extended this to neural networks [35].
Revised: The Double DQN algorithm extended this to neural networks [35].

Original: However, Zhanget al.
Revised: However, Zhang et al.

Original: [40] observed that the double estimator sometimes underestimates the Q-value and proposeto use a weighted average of the single and the double estimator as target.
Revised: [40] observed that the double estimator sometimes underestimates the Q-value and propose to use a weighted average of the single and the double estimator as target.

Original: This work is similar toours in the regard that depending on the parameter over- or underestimation could be corrected.
Revised: This work is similar to ours in the regard that depending on the parameter over- or underestimation could be corrected.

Original: Amajor difference to our algorithm is that the weighting parameter is computed from the maximumand minimum of the estimated Q-value and does not use unbiased rollouts.
Revised: A major difference to our algorithm is that the weighting parameter is computed from the maximum and minimum of the estimated Q-value and does not use unbiased rollouts.

Original: Lv et al.
Revised: Lv et al.

Original: [25] proposea similar weighting but suggest a stochastic selection of either the single or double estimator.
Revised: [25] propose a similar weighting but suggest a stochastic selection of either the single or double estimator.

Original: Theprobability of choosing one or the other follows a prede ﬁ ned schedule.
Revised: The probability of choosing one or the other follows a predeﬁned schedule.

Original: Other approaches computethe weighted average of the minimum and maximum over different estimates for the Q-value [12, 21].
Revised: Other approaches compute the weighted average of the minimum and maximum over different estimates for the Q-value [12, 21].

Original: However, the weighting parameter is a ﬁ xed hyperparameter.
Revised: However, the weighting parameter is a ﬁxed hyperparameter.

Original: The TD3 algorithm [11] proposed to use the minimum over two Q-value estimates as the target.
Revised: The TD3 algorithm [11] proposed to use the minimum over two Q-value estimates as the target.

Original: Maxmin Q-learning is another approachfor discrete action spaces that uses an ensemble of Q-functions.
Revised: Maxmin Q-learning is another approach for discrete action spaces that uses an ensemble of Q-functions.

Original: In the target computation, ﬁ rstthe minimum of over all Q-functions is computed followed by maximization with respect to theaction [23].
Revised: In the target computation, ﬁrst the minimum of over all Q-functions is computed followed by maximization with respect to the action [23].

Original: Decreasing the ensemble size increases the estimated targets while increasing the sizedecreases the targets.
Revised: Decreasing the ensemble size increases the estimated targets while increasing the size decreases the targets.

Original: Similarly to TQC this provides a way to control the bias in a more ﬁ ne-grainedway; the respective hyperparameter has to be set before the start of the training for each environment,however.
Revised: Similarly to TQC this provides a way to control the bias in a more ﬁne-grained way; the respective hyperparameter has to be set before the start of the training for each environment, however.

Original: Other approaches have identi ﬁ ed overgeneralization as a potential source of wrong or divergend TDtargets.
Revised: Other approaches have identiﬁed overgeneralization as a potential source of wrong or divergend TD targets.

Original: The idea is that, in the case of function approximation, updates to Q ( s t , a t ) also change theestimate of Q ( s t +1 , a t +1 ) which again changes Q ( s t , a t ) through the TD target [8].
Revised: The idea is that, in the case of function approximation, updates to Q ( s t , a t ) also change the estimate of Q ( s t +1 , a t +1 ) which again changes Q ( s t , a t ) through the TD target [8].

Original: In a recent work[31], it is proposed to use a regularization term to account for that.
Revised: In a recent work [31], it is proposed to use a regularization term to account for that.

Original: Furthermore, this work uses acombination of minimum and maximum over different Q-value estimates to compute the TD target.
Revised: Furthermore, this work uses a combination of minimum and maximum over different Q-value estimates to compute the TD target.

Original: However, the method has several environment speci ﬁ c hyperparameters.
Revised: However, the method has several environment speciﬁc hyperparameters.

Original: What sets ACC apart from the previously mentioned works is that unbiased on-policy rollouts areused to adjust a term that controls the bias correction instead of using some prede ﬁ ned heuristic.
Revised: What sets ACC apart from the previously mentioned works is that unbiased on-policy rollouts are used to adjust a term that controls the bias correction instead of using some predeﬁned heuristic.

Original: 5.2 Combining On- and Off-Policy Learning
Revised: 5.2 Combining On- and Off-Policy Learning

Original: There are many approaches that combine on- and off-policy learning by combining policy gradientswith off-policy samples [7, 9, 20, 29, 36].
Revised: There are many approaches that combine on- and off-policy learning by combining policy gradients with off-policy samples [7, 9, 20, 29, 36].

Original: In Gu et al.
Revised: In Gu et al.

Original: [13] an actor-critic is used where the critic isupdated off-policy and the actor is updated with a mixture of policy gradient and Q-gradient.
Revised: [13] an actor-critic is used where the critic is updated off-policy and the actor is updated with a mixture of policy gradient and Q-gradient.

Original: Thisdiffers from our work in that we are interested only in better critic estimates through the informationof on-policy samples.
Revised: This differs from our work in that we are interested only in better critic estimates through the information of on-policy samples.

Original: To learn better value estimates by combining on- and off-policy data manyworks propose the use of some form of importance sampling [26, 28, 30].
Revised: To learn better value estimates by combining on- and off-policy data many works propose the use of some form of importance sampling [26, 28, 30].

Original: In Hausknecht and Stone[17] the TD target is computed by mixing Monte Carlo samples with the bootstrap estimator.
Revised: In Hausknecht and Stone [17] the TD target is computed by mixing Monte Carlo samples with the bootstrap estimator.

Original: Thesemethods provide a tradeoff between variance and bias.
Revised: These methods provide a tradeoff between variance and bias.

Original: They differ from our work in using the actualreturns directly in the TD targets while we incorporate the returns indirectly via another parameter.
Revised: They differ from our work in using the actual returns directly in the TD targets while we incorporate the returns indirectly via another parameter.

Original: Bhatt et al.
Revised: Bhatt et al.

Original: [3] propose the use of a mixture of on- and off-policy transitions to generate a featurenormalization that can be used in off-policy TD learning.
Revised: [3] propose the use of a mixture of on- and off-policy transitions to generate a feature normalization that can be used in off-policy TD learning.

Original: Applied to TD3, learning becomes morestable eliminating the need to use a delayed target network.
Revised: Applied to TD3, learning becomes more stable eliminating the need to use a delayed target network.

Original: 5.3 Hyperparameter Tuning for Reinforcement Learning
Revised: 5.3 Hyperparameter Tuning for Reinforcement Learning

Original: Most algorithms that tune hyperparameters of RL algorithms use many different instances of theenvironment to ﬁ nd a good setting [5, 10, 19, 39].
Revised: Most algorithms that tune hyperparameters of RL algorithms use many different instances of the environment to ﬁnd a good setting [5, 10, 19, 39].

Original: There is, however, also work that adjusts ahyperparameter online during training [38].
Revised: There is, however, also work that adjusts a hyperparameter online during training [38].

Original: In this work the meta-gradient (i.e., the gradient of theupdate rule) is used to adjust the discount factor and the length of bootstrapping intervals.
Revised: In this work the meta-gradient (i.e., the gradient of the update rule) is used to adjust the discount factor and the length of bootstrapping intervals.

Original: However,it would not be straightforward to apply this method to control the bias of the value estimate.
Revised: However, it would not be straightforward to apply this method to control the bias of the value estimate.

Original: Theirmethod also differs from ours in that they do not use a combination of on- and off-policy data.
Revised: Their method also differs from ours in that they do not use a combination of on- and off-policy data.

Original:  Conclusion
Revised:  Conclusion

Original: We present Adaptively Calibrated Critics (ACC), a general off-policy algorithm that learns a Q-valuefunction with bias calibrated TD targets.
Revised: We present Adaptively Calibrated Critics (ACC), a general off-policy algorithm that learns a Q-value function with bias calibrated TD targets.

Original: The bias correction in the targets is determined via aparameter that is adjusted by comparing the current value estimates with the most recently observedon-policy returns.
Revised: The bias correction in the targets is determined via a parameter that is adjusted by comparing the current value estimates with the most recently observed on-policy returns.

Original: Our method allows to incorporate information from the unbiased sample returnsinto the TD targets while keeping the high variance of the samples out.
Revised: Our method allows to incorporate information from the unbiased sample returns into the TD targets while keeping the high variance of the samples out.

Original: We apply ACC to Truncated
Revised: We apply ACC to Truncated

Original: Quantile Critics, a recent off-policy continuous control algorithm that allows ﬁ ne-grained controlof the TD target scale through a hyperparameter tuned per environment.
Revised: Quantile Critics, a recent off-policy continuous control algorithm that allows ﬁne-grained control of the TD target scale through a hyperparameter tuned per environment.

Original: With ACC, this parametercan automatically be adjusted during training, obviating the need for extensive tuning.
Revised: With ACC, this parameter can automatically be adjusted during training, obviating the need for extensive tuning.

Original: The strongexperimental results suggest that our method provides an ef ﬁ cient and general way to control the biasoccurring in TD learning.
Revised: The strong experimental results suggest that our method provides an efﬁcient and general way to control the bias occurring in TD learning.

Original: Interesting directions for future work are to evaluate the effectiveness of ACC applied to algorithms that work with discrete action spaces and when learning on a real robot where tuning ofhyperparameters is very costly.Checklist For all authors...
Revised: Interesting directions for future work are to evaluate the effectiveness of ACC applied to algorithms that work with discrete action spaces and when learning on a real robot where tuning of hyperparameters is very costly.

Original: (a) Do the main claims made in the abstract and introduction accurately re ﬂ ect the paper’s

Original: contributions and scope? [Yes
Revised: Acknowledgements

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Appendix

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See

Original: Appendix

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results? [Yes] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: The authors would like to thank the following people for their help throughout the process of writing this paper, in alphabetical order: Daniel Honerkamp, Johannes Meyer, Niklas Wetzel and

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they were chosen)? [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: Tim Welschehold.

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [No] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: This work was supported by the European Union’s Horizon 2020 research and innovation program under grant agreement No 871449-OpenDR and by the Freiburg Graduate School of Robotics.

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identi ﬁ able

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board

Original: ) approvals, if applicable?

Original: [N/A]

Original: [Table]

Original: ) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation

Original: [N/A]

Original: [Table]