Original: L EARNING V ALUE F UNCTIONS IN D EEP P OLICY G RADIENTS USING R ESIDUAL V ARIANCE
Revised: L EARNING V ALUE F UNCTIONS IN D EEP P OLICY G RADIENTS USING R ESIDUAL V ARIANCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Policy gradient algorithms have proven to be successful in diverse decision making and control tasks.
Revised: Policy gradient algorithms have proven to be successful in diverse decision making and control tasks.

Original: However, these methods suffer from high sample complexity and instability issues.
Revised: However, these methods suffer from high sample complexity and instability issues.

Original: In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework.
Revised: In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework.

Original: Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in ﬁtting the true value function, calling for the need to identify a better objective for the critic.
Revised: Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in ﬁtting the true value function, calling for the need to identify a better objective for the critic.

Original: In our method, the critic uses a new state-value (resp.
Revised: In our method, the critic uses a new state-value (resp.

Original: state-action-value) function approximation that learns therelative value of the states (resp.
Revised: state-actionvalue) function approximation that learns the value of the states (resp.

Original: state-action pairs)  rather than their absolute value as in conventional actor-critic.
Revised: state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic.

Original: We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms.
Revised: We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms.

Original: Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights
Revised: Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Model-free deep reinforcement learning (RL) has been successfully used in a wide range of problem domains, ranging from teaching computers to control robots to playing sophisticated strategy games (Silver et al., 2014; Schulman et al., 2016;
Revised: Model-free deep reinforcement learning (RL) has been successfully used in a wide range of problem domains, ranging from teaching computers to control robots to playing sophisticated strategy games (Silver et al., 2014; Schulman et al., 2016;

Original: Lillicrap et al., 2016; Mnih et al., 2016).
Revised: Lillicrap et al., 2016; Mnih et al., 2016).

Original: Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework (Sutton et al., 2000;
Revised: Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework (Sutton et al., 2000;

Original: Schulman et al., 2017; Haarnoja et al., 2018).
Revised: Schulman et al., 2017; Haarnoja et al., 2018).

Original: While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.
Revised: While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.

Original: For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose a critic formulation that allows for a more stable policy gradient.
Revised: For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose a critic formulation that allows for a more stable policy gradient.

Original: However, new studies (Tucker et al., 2018; Ilyas et al., 2020) indicate that state-of-the-art policy gradient methods (Schulman et al., 2015; 2017) fail to ﬁt the true value function and that recently proposed state-action-dependent baselines (Gu et al., 2016; Liu et al., 2018; Wu et al., 2018) do not reduce gradient variance more than state-dependent ones.
Revised: However, new studies (Tucker et al., 2018; Ilyas et al., 2020) indicate that state-of-the-art policy gradient methods (Schulman et al., 2015; 2017) fail to ﬁt the true value function and that recently proposed state-action-dependent baselines (Gu et al., 2016; Liu et al., 2018; Wu et al., 2018) do not reduce gradient variance more than state-dependent ones.

Original: These ﬁndings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning.
Revised: These ﬁndings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning.

Original: Consequently, attempting to ﬁnd a better baseline is questionable, as critics would typically fail to ﬁt it (Ilyas et al., 2020).
Revised: Consequently, attempting to ﬁnd a better baseline is questionable, as critics would typically fail to ﬁt it (Ilyas et al., 2020).

Original: In Tucker et al.
Revised: In Tucker et al.

Original: , the authors argue that “much larger gains could be achieved by instead improving the accuracy of the value function”.
Revised: , the authors argue that “much larger gains could be achieved by instead improving the accuracy of the value function”.

Original: Following this line of thought, we are interested in ways to better approximate the value function.
Revised: Following this line of thought, we are interested in ways to better approximate the value function.

Revised: One approach addressing this issue is to put more focus on relative state-action values, an idea introduced in the literature on advantage reinforcement learning (Harmon & Baird III) followed by works on dueling (Wang et al., 2016) neural networks.

Original: Recent evidence (Lin & Zhou, 2020)  suggests that considering the relative action values  leads to better policies; the leading argument behind this intuition is that it sufﬁces to identify the optimal actions to solve a task.
Revised: More recent work (Lin & Zhou, 2020) also suggests that considering the relative action values , or more precisely the ranking of actions in a state leads to better policies. The main argument behind this intuition is that it sufﬁces to identify the optimal actions to solve a task.

Original: We extend this principle of relative action  to include both state and state-action-value functions with a new objective for the critic:  the  residual error variance.
Revised: We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic: minimizing the variance of residual errors.

Revised: In essence, this modiﬁed loss function puts more focus on the values of states (resp.

Revised: state-actions) relative to their mean value rather than their absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state.

Original: In summary, this paper:
Revised: In summary, this paper:

Original: • Introduces A ctor with V ariance E stimated C ritic ( AVEC ), an actor-critic method providing a new training objective for the critic based on the residual variance.
Revised: • Introduces A ctor with V ariance E stimated C ritic ( AVEC ), an actor-critic method providing a new training objective for the critic based on the residual variance.

Original: • Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modiﬁed gradient estimator.
Revised: • Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modiﬁed gradient estimator.

Original: • Demonstrates experimentally that AVEC , when coupled with state-of-the-art policy gradient algorithms, yields a signiﬁcant performance boost on a set of challenging tasks, including environments with sparse rewards.
Revised: • Demonstrates experimentally that AVEC , when coupled with state-of-the-art policy gradient algorithms, yields a signiﬁcant performance boost on a set of challenging tasks, including environments with sparse rewards.

Original: • Provides empirical evidence supporting a better ﬁt of the true value function and a substantial stabilization of the gradient.
Revised: • Provides empirical evidence supporting a better ﬁt of the true value function and a substantial stabilization of the gradient.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RL.
Revised: Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RL.

Original: Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator.
Revised: Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator.

Original: This was originally formulated as the REINFORCE algorithm (Williams, 1992).
Revised: This was originally formulated as the REINFORCE algorithm (Williams, 1992).

Original: Kakade & Langford (2002) later created conservative policy iteration and provided lower bounds for the minimum objective improvement.
Revised: Kakade & Langford (2002) later created conservative policy iteration and provided lower bounds for the minimum objective improvement.

Original: Peters et al.
Revised: Peters et al.

Original: (2010) replaced regularization by a trust region constraint to stabilize training.
Revised: (2010) replaced regularization by a trust region constraint to stabilize training.

Original: In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high.
Revised: In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high.

Original: To improve stability, Weaver & Tao (2001) show that subtracting a baseline (Williams, 1992) from the value function in the policy gradient can be very beneﬁcial in reducing variance without damaging the bias.
Revised: To improve stability, Weaver & Tao (2001) show that subtracting a baseline (Williams, 1992) from the value function in the policy gradient can be very beneﬁcial in reducing variance without damaging the bias.

Original: However, in practice, these modiﬁcations on the actor-critic framework usually result in improved performance without a signiﬁcant variance reduction (Tucker et al., 2018; Ilyas et al., 2020).
Revised: However, in practice, these modiﬁcations on the actor-critic framework usually result in improved performance without a signiﬁcant variance reduction (Tucker et al., 2018; Ilyas et al., 2020).

Original: Currently, one of the most dominant on-policy methods are proximal policy optimization (PPO)
Revised: Currently, one of the most dominant on-policy methods are proximal policy optimization (PPO)

Original: (Schulman et al., 2017) and trust region policy optimization (TRPO) (Schulman et al., 2015), both of which require new samples to be collected for each gradient step.
Revised: (Schulman et al., 2017) and trust region policy optimization (TRPO) (Schulman et al., 2015), both of which require new samples to be collected for each gradient step.

Original: Another direction of research that overcomes this limitation is off-policy algorithms, which therefore beneﬁt from all sample transitions; soft actor-critic (SAC) (Haarnoja et al., 2018) is one such approach achieving state-of-the-art performance.
Revised: Another direction of research that overcomes this limitation is off-policy algorithms, which therefore beneﬁt from all sample transitions; soft actor-critic (SAC) (Haarnoja et al., 2018) is one such approach achieving state-of-the-art performance.

Original: Several works also investigate regularization effects on the policy gradient (Jaderberg et al., 2016;
Revised: Several works also investigate regularization effects on the policy gradient (Jaderberg et al., 2016;

Original: Namkoong & Duchi, 2017; Flet-Berliac & Preux, 2019; Kartal et al., 2019); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias.
Revised: Namkoong & Duchi, 2017; Flet-Berliac & Preux, 2019; Kartal et al., 2019); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias.

Original: In RL, regularization is often used to encourage exploration and takes the form of an entropy term (Williams & Peng, 1991; Schulman et al., 2017).
Revised: In RL, regularization is often used to encourage exploration and takes the form of an entropy term (Williams & Peng, 1991; Schulman et al., 2017).

Original: Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting, Thodoroff et al.
Revised: Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting, Thodoroff et al.

Original: (2018) show that it is possible to smooth over the temporal dimension as well.
Revised: (2018) show that it is possible to smooth over the temporal dimension as well.

Original: Furthermore, Zhao et al.
Revised: Furthermore, Zhao et al.

Original: (2016) analyze the effects of a regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent (Johnson & Zhang, 2013)) which proves to provide more consistent policy improvements at the expense of reduced performance.
Revised: (2016) analyze the effects of a regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent (Johnson & Zhang, 2013)) which proves to provide more consistent policy improvements at the expense of reduced performance.

Original: In contrast, as we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.
Revised: In contrast, as we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.

Original: Exploration has been studied under different angles in RL, one common strategy is (cid:15) -greedy, where the agent explores with probability (cid:15) by taking a random action.
Revised: Exploration has been studied under different angles in RL, one common strategy is (cid:15) -greedy, where the agent explores with probability (cid:15) by taking a random action.

Original: This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing environments (Mnih et al., 2013; Van Hasselt et al., 2015; Mnih et al., 2016).
Revised: This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing environments (Mnih et al., 2013; Van Hasselt et al., 2015; Mnih et al., 2016).

Original: On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability.
Revised: On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability.

Original: Furthermore, policy gradient literature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993;
Revised: Furthermore, policy gradient literature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993;

Original: Tokic, 2010), and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration (Schmidhuber, 2006; Bellemare et al., 2016).
Revised: Tokic, 2010), and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration (Schmidhuber, 2006; Bellemare et al., 2016).

Original: While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function.
Revised: While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function.

Original: In the context of linear regression,
Revised: In the context of linear regression,

Original: Brown (1947) considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function (Pham-Gia & Hung, 2001)
Revised: Brown (1947) considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function (Pham-Gia & Hung, 2001)

Original: (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours.
Revised: (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours.

Original: Indeed, they seek to be robust to outliers whereas, when considering noiseless RL problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.
Revised: Indeed, they seek to be robust to outliers whereas, when considering noiseless RL problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.

Original:  P RELIMINARIES 3. B ACKGROUND AND N OTATIONS
Revised:  P RELIMINARIES 3. B ACKGROUND AND N OTATIONS

Original: We consider an inﬁnite-horizon Markov Decision Problem (MDP) with continuous states s ∈ S , continuous actions a ∈ A , transition distribution s t +1 ∼ P ( s t , a t ) and reward function r t ∼
Revised: We consider an inﬁnite-horizon Markov Decision Problem (MDP) with continuous states s ∈ S , continuous actions a ∈ A , transition distribution s t +1 ∼ P ( s t , a t ) and reward function r t ∼

Original: R ( s t , a t ) .
Revised: R ( s t , a t ) .

Original: Let π θ ( a | s ) denote a stochastic policy with parameter θ , we restrict policies to being
Revised: Let π θ ( a | s ) denote a stochastic policy with parameter θ , we restrict policies to being

Original: Gaussian distributions.
Revised: Gaussian distributions.

Original: In the following, π and π θ denote the same object.
Revised: In the following, π and π θ denote the same object.

Original: The agent repeatedly interacts with the environment by sampling action a t ∼ π ( .
Revised: The agent repeatedly interacts with the environment by sampling action a t ∼ π ( .

Original: | s t ) , receives reward r t and transitions to a new state s t +1 .
Revised: | s t ) , receives reward r t and transitions to a new state s t +1 .

Original: The objective is to maximize the expected sum of discounted rewards:
Revised: The objective is to maximize the expected sum of discounted rewards:

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ [0 , 1) is a discount factor (Puterman, 1994), and τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . .
Revised: where γ ∈ [0 , 1) is a discount factor (Puterman, 1994), and τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . .

Original: is a trajectory sampled from the environment using policy π .
Revised: is a trajectory sampled from the environment using policy π .

Original: We denote the value of a state s in the MDP framework while following a policy π by V π ( s ) (cid:44) E τ ∼ π
Revised: We denote the value of a state s in the MDP framework while following a policy π by V π ( s ) (cid:44) E τ ∼ π

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t r ( s t , a t ) | s 0 = s ] and the value of a state-action pair of performing action a in state s and then following policy π by Q π ( s, a ) (cid:44) Eτ ∼ π
Revised: γ t r ( s t , a t ) | s 0 = s ] and the value of a state-action pair of performing action a in state s and then following policy π by Q π ( s, a ) (cid:44) E

Original: [-[ (cid:80) ∞ t =
-]
Revised: [-[ (cid:80) ∞ t =
-][Equation]

Original: γ t r ( s t , a t ) | s 0 = s, a 0 = a ] .
Revised: t r ( s t , a t ) | s 0 = s, a 0 = a ] .

Original: Finally, the advantage function which quantiﬁes how an action a is better than the average action in state s is denoted A π ( s, a )
Revised: Finally, the advantage function which quantiﬁes how an action a is better than the average action in state s is denoted A π ( s, a )

Original: (cid:44) Q π ( s, a ) − V π ( s ) .
Revised: (cid:44) Q π ( s, a ) − V π ( s ) .

Original: 3. C RITICS IN D EEP P OLICY G RADIENTS
Revised: 3. C RITICS IN D EEP P OLICY G RADIENTS

Original: In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient.
Revised: In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient.

Original: Without loss of generality, we consider the algorithms that approximate the state-value function V .
Revised: Without loss of generality, we consider the algorithms that approximate the state-value function V .

Original: The analysis holds for algorithms that approximate the state-action-value function Q .
Revised: The analysis holds for algorithms that approximate the state-action-value function Q .

Original: Let f φ : S → R be an estimator of ˆ V π with φ its parameter.
Revised: Let f φ : S → R be an estimator of ˆ V π with φ its parameter.

Original: f φ is traditionally learned through minimizing the mean squared error (MSE) against ˆ V π .
Revised: f φ is traditionally learned through minimizing the mean squared error (MSE) against ˆ V π .

Original: At iteration k , the critic minimizes:
Revised: At iteration k , the critic minimizes:

Original: [Equation]
Revised: [Equation]

Original: where the states s are collected under policy π θ k , and ˆ V π θk ( s ) is an empirical estimate of V (see
Revised: where the states s are collected under policy π θ k , and ˆ V π θk ( s ) is an empirical estimate of V (see

Original: Section 4.3 for details).
Revised: Section 4.3 for details).

Original: Similarly, using f φ : S × A → R instead, one can ﬁt an empirical target ˆ Q π .
Revised: Similarly, using f φ : S × A → R instead, one can ﬁt an empirical target ˆ Q π .

Original:  M ETHOD : A CTOR WITH V ARIANCE E STIMATED C RITIC
Revised:  M ETHOD : A CTOR WITH V ARIANCE E STIMATED C RITIC

Original: In this section, we introduce AVEC and discuss its correctness, motivations and implementation.
Revised: In this section, we introduce AVEC and discuss its correctness, motivations and implementation.

Original: 4. D EFINING AN A LTERNATIVE C RITIC
Revised: 4. D EFINING AN A LTERNATIVE C RITIC

Original: Recent work (Ilyas et al., 2020) empirically demonstrates that while the value network succeeds in the supervised learning task of ﬁtting ˆ V π (resp.
Revised: Recent work (Ilyas et al., 2020) empirically demonstrates that while the value network succeeds in the supervised learning task of ﬁtting ˆ V π (resp.

Original: ˆ Q π )
Revised: ˆ Q π )

Original: , it does not ﬁt V π (resp.
Revised: , it does not ﬁt V π (resp.

Original: Q π ).
Revised: Q π ).

Original: We address this deﬁciency in the estimation of the critic by introducing an alternative value network loss.
Revised: We address this deﬁciency in the estimation of the critic by introducing an alternative value network loss.

Original: Following empirical evidence indicating that the problem is the approximation error and not the estimator per se , AVEC adopts a loss that can provide a better approximation error, and yields better estimators of the value function (as will be shown in Section 5.3).
Revised: Following empirical evidence indicating that the problem is the approximation error and not the estimator per se , AVEC adopts a loss that can provide a better approximation error, and yields better estimators of the value function (as will be shown in Section 5.3).

Original: At update k :
Revised: At update k :

Original: [Equation]
Revised: [Equation]

Original: with states s collected using π θ k .
Revised: with states s collected using π θ k .

Revised: Note that the gradient ﬂows in f φ twice using Eq. 3.

Original: Then we deﬁne our bias-corrected estimator: g φ : S → R such that g φ ( s ) =
Revised: Then, we deﬁne our bias-corrected estimator: g φ : S → R such that g φ ( s ) =

Original: f φ ( s )+ E s [ ˆ V π θk ( s ) − f φ ( s )] .
Revised: f φ ( s ) + E s [ ˆ V π θk ( s ) − f φ ( s )] .

Original: Analogously, we deﬁne an alternative critic for the estimation of Q π by replacing ˆ V π by ˆ Q π and f φ ( s ) by f φ ( s, a ) in Eq. 3.
Revised: Analogously to Eq. 3, we deﬁne an alternative critic for the estimation of Q π by replacing ˆ V π by ˆ Q π and f φ ( s ) by f φ ( s, a ) .

Original: Proposition ( AVEC Policy Gradient) .
Revised: Proposition ( AVEC Policy Gradient) .

Original: If f φ : S ×
Revised: If f φ : S ×

Original: A → R satisﬁes the parameterization assumption (Sutton et al., 2000) then g φ provides an unbiased policy gradient:
Revised: A → R satisﬁes the parameterization assumption (Sutton et al., 2000) then g φ provides an unbiased policy gradient:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: This result also holds for the estimation of V π θ with f φ : S → R .
Revised: This result also holds for the estimation of V π θ with f φ : S → R .

Original: 4. B UILDING M OTIVATION
Revised: 4. B UILDING M OTIVATION

Original: Here, we present the intuition behind using AVEC for actor-critic algorithms.
Revised: Here, we present the intuition behind using AVEC for actor-critic algorithms.

Original: State-value function estimation.

Original: Tucker et al.
Revised: Tucker et al.

Original: and Ilyas et al.
Revised: and Ilyas et al.

Original: (2020) indicate that the approximation error (cid:107) ˆ V π − V π (cid:107) is problematic, suggesting that the variance of the empirical targets ˆ V π
Revised: (2020) indicate that the approximation error (cid:107) ˆ V π − V π (cid:107) is problematic, suggesting that the variance of the empirical targets ˆ V π

Original: ( s t ) is high.
Revised: ( s t ) is high.

Revised: Using L AVEC , our approach reduces the variance term of the MSE (or distance to V π ) but mechanistically also increases the bias.

Revised: Our intuition is that since the bias is already quite substantial (Ilyas et al., 2020), it may be possible to reduce the variance enough so that even though the bias increases, the total MSE reduces.

Revised: State-value function estimation.

Original: Optimizing the critic with L AVEC can be interpreted as ﬁtting ˆ V (cid:48)
Revised: In this case, optimizing the critic with L AVEC can be interpreted as ﬁtting ˆ V (cid:48)

Original: π ( s ) =
Revised: π ( s ) =

Original: ˆ V π ( s ) − E s (cid:48)
Revised: ˆ V π ( s ) − E s (cid:48)

Original: [ ˆ V π ( s (cid:48) )] using the MSE.
Revised: [ ˆ V π ( s (cid:48) )] using the MSE.

Original: We show that the targets ˆ V (cid:48)
Revised: We show that the targets ˆ V (cid:48)

Original: π are better estimations of V (cid:48) π ( s ) =
Revised: π are better estimations of V (cid:48) π ( s ) =

Original: V π ( s ) − E s (cid:48)
Revised: V π ( s ) − E s (cid:48)

Original: [ V π ( s (cid:48) )] than ˆ V π are of V π .
Revised: [ V π ( s (cid:48) )] than ˆ V π are of V π .

Original: To illustrate this, consider T independent random variables ( X i ) i ∈{ 1 ,...,T } .
Revised: To illustrate this, consider T independent random variables ( X i ) i ∈{ 1 ,...,T } .

Original: We denote X (cid:48)
Revised: We denote X (cid:48)

Original: i
Revised: i

Original: =
Revised: =

Original: X i − 1 T (cid:80)
Revised: X i − 1 T (cid:80)

Original: Tj =1 X j and V ( X ) the variance of X .
Revised: Tj =1 X j and V ( X ) the variance of X .

Original: Then,
Revised: Then,

Original: V ( X (cid:48) i ) =
Revised: [Equation]

Original: V ( X i ) − 2 T V ( X i ) + 1 T 2 (cid:80) Tj =1 V ( X j ) andV ( X (cid:48) i ) <
Revised: and

Original: V ( X i ) as long as ∀ i 1 T (cid:80) Tj =1 V ( X j ) <
Revised: [Equation] as long as ∀ i 1 T (cid:80) Tj =1 V ( X j ) <

Original: 2 V ( X i ) , or more generally when state-values are not strongly negatively correlated 1 and not very discordant.
Revised: 2 V ( X i ) , or more generally when state-values are not strongly negatively correlated 1 and not very discordant.

Original: This entails that ˆ V (cid:48) π has a more compact span, and is consequently easier to ﬁt.
Revised: This entails that ˆ V (cid:48) π has a more compact span, and is consequently easier to ﬁt.

Revised: This analysis shows that the variance term of the MSE is reduced compared to traditional actor-critic algorithms, but does not guarantee it counterbalances the bias increase.

Revised: Nevertheless, in practice, the bias is so high that the difference due to learning with AVEC is only marginal and the total MSE decreases.

Revised: We empirically demonstrate this claim in Section 5.3.

Original: State-action-value function estimation.
Revised: State-action-value function estimation.

Original: In this case, Eq.
Revised: In this case, Eq.

Original: 3 translates into replacing ˆ V π ( s ) by ˆ Q π ( s, a ) and f φ ( s ) by f φ ( s, a ) and the rationale for optimizing the residual variance of the value function instead of the full MSE becomes more straightforward: the practical use of the
Revised: 3 translates into replacing ˆ V π ( s ) by ˆ Q π ( s, a ) and f φ ( s ) by f φ ( s, a ) and the rationale for optimizing the residual variance of the value function instead of the full MSE becomes more straightforward: the practical use of the

Original: Q-function is to disentangle the relative values of actions for each state (Sutton et al., 2000).
Revised: Q-function is to disentangle the relative values of actions for each state (Sutton et al., 2000).

Original: [Figure]
Revised: [Figure]

Original: Var [Figure] AVEC ’s effect on relative values is illustrated in a didactic regression with one variable example in Fig.where grey markers are observations and the blue line is our current estimation.
Revised: AVEC ’s effect on relative values is illustrated in a didactic regression with one variable example in Fig.where grey markers are observations and the blue line is our current estimation.

Original: Minimizing the MSE, the line is expected to move towards the orange one in order to reduce errors uniformly.
Revised: Minimizing the MSE, the line is expected to move towards the orange one in order to reduce errors uniformly.

Original: Minimizing the residual variance, it is expected to move near the red one.
Revised: Minimizing the residual variance, it is expected to move near the red one.

Original: In fact, L AVEC tends to further penalize observations that are far away from the mean, implying that
Revised: In fact, L AVEC tends to further penalize observations that are far away from the mean, implying that

Original: AVEC allows a better recovery of the “shape” of the target near extrema.
Revised: AVEC allows a better recovery of the “shape” of the target near extrema.

Original: In particular, we see in the ﬁgure that the maximum and minimum observation values are quickly identiﬁed.
Revised: In particular, we see in the ﬁgure that the maximum and minimum observation values are quickly identiﬁed.

Original: Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation.
Revised: Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation.

Original: It should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic.
Revised: It should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic.

Original: As such, there are no outliers and extreme state-action values correspond to learning signals.
Revised: As such, there are no outliers and extreme state-action values correspond to learning signals.

Original: In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved.
Revised: In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved.

Original: More generally, this added term is used to put more focus on the relative values of states (resp. stateactions) rather than absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state.

Original: 4. I MPLEMENTATION
Revised: 4. I MPLEMENTATION

Original: We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function.
Revised: We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function.

Original: A better estimation of the value function implies better policy improvements.
Revised: A better estimation of the value function implies better policy improvements.

Original: We now describe how AVEC incorporates its residual variance objective into the critics of PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and
Revised: We now describe how AVEC incorporates its residual variance objective into the critics of PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and

Original: SAC (Haarnoja et al., 2018).
Revised: SAC (Haarnoja et al., 2018).

Original: Let B be a batch of transitions.
Revised: Let B be a batch of transitions.

Original: In PPO and TRPO, AVEC modiﬁes the learning of V φ (line 12 of Algorithm 1) using:
Revised: In PPO and TRPO, AVEC modiﬁes the learning of V φ (line 12 of Algorithm 1) using:

Original: [Equation]
Revised: [Equation]

Original: then V φ = f φ ( s ) + E s ∼B [ ˆ V π ( s ) − f φ ( s )] , where ˆ V π ( s t ) =
Revised: then V φ = f φ ( s ) + E s ∼B [ ˆ V π ( s ) − f φ ( s )] , where ˆ V π ( s t ) =

Original: f φ old ( s t ) + A t such that f φ old ( s t ) are the estimates given by the last value function and A t is the advantage of the policy, i.e. the returns minus the expected values ( A t is often estimated using generalized advantage estimation (Schulman et al., 2016).
Revised: f φ old ( s t ) + A t such that f φ old ( s t ) are the estimates given by the last value function and A t is the advantage of the policy, i.e. the returns minus the expected values ( A t is often estimated using generalized advantage estimation (Schulman et al., 2016).

Original: In SAC, AVEC modiﬁes the objective function of ( Q φ i ) i =1 , 2 (line 13 of Algorithm 2 in
Revised: In SAC, AVEC modiﬁes the objective function of ( Q φ i ) i =1 , 2 (line 13 of Algorithm 2 in

Original: Appendix C) using:
Revised: Appendix C) using:

Original: [Equation]
Revised: [Equation]

Original: then Qφ i =
Revised: then Q

Original:  f φ i ( s, a ) + E ( s,a ) ∼B 
Revised: [Equation] = f [Equation] ∼B [ ˆ

Original: [ ˆ Q π ( s, a ) − f φ i ( s, a )] , where ˆ Q π ( s, a ) is estimated using temporal difference (see Haarnoja et al.
Revised: [Equation] ] , where ˆ Q π ( s, a ) is estimated using temporal difference (see Haarnoja et al.

Original: (2018)): ˆ Q π ( s t , a t ) =
Revised: (2018)): ˆ Q π ( s t , a t ) =

Original: r ( s t , a t ) + γ E s t +

Revised: r ( s t , a t ) + γ E s t +

Original: ∼ π
Revised: ∼ π

Original: [ V ¯ ψ ( s t +1 )] with ¯ ψ the value function parameter (see Algorithm 2).
Revised: [ V ¯ ψ ( s t +1 )] with ¯ ψ the value function parameter (see Algorithm 2).

Original: The reader may have noticed that L 1 AVEC and L 2 AVEC slightly differ from Eq.
Revised: The reader may have noticed that L 1 AVEC and L 2 AVEC slightly differ from Eq.

Original: The residual variance of the value function ( L AVEC ) is not tractable since a priori state-values are dependent and their joint law is unknown.
Revised: The residual variance of the value function ( L AVEC ) is not tractable since a priori state-values are dependent and their joint law is unknown.

Original: Consequently, in practice, we use the empirical variance proxy assuming independence ( cf.
Revised: Consequently, in practice, we use the empirical variance proxy assuming independence ( cf.

Original: Appendix D).
Revised: Appendix D).

Original: Greensmith et al.
Revised: Greensmith et al.

Original: (2004) provide some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones.
Revised: (2004) provide some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones.

Original: Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which makes its implementation straightforward and keeps the same computational complexity.
Revised: Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which makes its implementation straightforward and keeps the same computational complexity.

Original:  E XPERIMENTAL S TUDY
Revised:  E XPERIMENTAL S TUDY

Original: Algorithm 
Revised: Algorithm 1 AVEC coupled with PPO or TRPO.

Original: J ALGO denotes the policy loss of either algorithm (described in Schulman et al.
Revised: J ALGO denotes the policy loss of either algorithm (described in Schulman et al.

Original: 1: Input parameters: λ π ≥ 0 , λ V ≥2: - tion 3: for  do 4: batch B ← ∅ 5:  do 6: a t ∼ π θ ( s t ) 7: s t +1 ∼ P ( s t , a t ) 8:
Revised: 1: Input parameters: λ π ≥ 0 , λ V ≥2: Initialize policy parameter θ and value function parameter φ 3: for each update step do 4: batch B ← ∅ 5: for each environment step do 6: a t ∼ π θ ( s t ) 7: s t +1 ∼ P ( s t , a t ) 8:

Original: B ← B ∪ { ( s t , a t , r t , s t +1 ) } 9: end for 10: for each gradient step do 11: θ ← θ − λ π ˆ ∇ θ J ALGO ( π θ ) 12: φ ← φ − λ V ˆ ∇ φ L 1 AVEC ( φ ) 13: end for 14: end for
Revised: B ← B ∪ { ( s t , a t , r t , s t +1 ) } 9: end for 10: for each gradient step do 11: θ ← θ − λ π ˆ ∇ θ J ALGO ( π θ ) 12: φ ← φ − λ V ˆ ∇ φ L 1 AVEC ( φ ) 13: end for 14: end for

Original: In this section, we conduct experiments along four orthogonal directions.
Revised: In this section, we conduct experiments along four orthogonal directions.

Original: (a) We validate the superiority of AVEC compared to the traditional actor-critic training.
Revised: (a) We validate the superiority of AVEC compared to the traditional actor-critic training.

Original: (b) We evaluate AVEC in environments with sparse rewards.
Revised: (b) We evaluate AVEC in environments with sparse rewards.

Original: (c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient.
Revised: (c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient d)

Original: (d) We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks.
Revised: We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks.

Original: We point out that a comparison to variancereduction methods is not considered in this paper: Tucker et al.
Revised: We point out that a comparison to variancereduction methods is not considered in this paper: Tucker et al.

Original: (2018) demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either.
Revised: (2018) demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either.

Original: Note that in all experiments we choose the hyper-parameters providing the best performance for the considered methods which can only penalyze AVEC ( cf.
Revised: Note that in all experiments we choose the hyper-parameters providing the best performance for the considered methods which can only penalyze AVEC ( cf.

Original: Appendix E).
Revised: Appendix E).

Original: In all the ﬁgures hereafter (except Fig.
Revised: In all the ﬁgures hereafter (except Fig.

Original: 3c and 3d), lines are average performances and shaded areas represent one standard deviation.
Revised: 3c and 3d) lines are [Table] [Figure]

Original: 5. C ONTINUOUS C ONTROL
Revised: 5. C ONTINUOUS C ONTROL

Original: For ease of comparison with other methods, we evaluate AVEC on the MuJoCo (Todorov et al., 2012) and the PyBullet (Coumans & Bai, 2016) continuous control benchmarks (see Appendix G for details) using OpenAI Gym (Brockman et al., 2016).
Revised: For ease of comparison with other methods, we evaluate AVEC on the MuJoCo (Todorov et al., 2012) and the PyBullet (Coumans & Bai, 2016) continuous control benchmarks (see Appendix G for details) using OpenAI Gym (Brockman et al., 2016).

Original: Note that the PyBullet versions of the locomotion tasks are harder than the MuJoCo equivalents 2 .
Revised: Note that the PyBullet versions of the locomotion tasks are harder than the MuJoCo equivalents 2 .

Original: We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Appendix H.
Revised: We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Appendix H.

Original: We assess the beneﬁts of AVEC when coupled with the most prominent policy gradient algorithms, currently state-of-the-art methods: PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015), both on-policy methods, and SAC (Haarnoja et al., 2018), an off-policy maximum entropy deep RL[Table] [Figure] algorithm.
Revised: We assess the beneﬁts of AVEC when coupled with the most prominent policy gradient algorithms, currently state-of-the-art methods: PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015), both on-policy methods, and SAC (Haarnoja et al., 2018), an off-policy maximum entropy deep RL algorithm.

Original: The code for the method is included in the Supplementary Material.
Revised: The code for the method is included in the Supplementary Material.

Original: We provide the list of hyper-parameters and further implementation details in Appendix D and E.
Revised: We provide the list of hyper-parameters and further implementation details in Appendix D and E.

Original: Table 1 reports the results while Fig. 2 and 8 show the total average return for SAC and PPO.
Revised: Table 1 reports the results while Fig. 2 and 8 show the total average return for SAC and PPO.

Original: TRPO results are provided in Appendix F for readability.
Revised: TRPO results are provided in Appendix F for readability.

Original: When coupled with SAC and PPO, AVEC brings very signiﬁcant improvement (on average +26% for SAC and +39% for PPO) in the performance of the policy gradient algorithms, improvement which is consistent across tasks.
Revised: When coupled with SAC and PPO, AVEC brings very signiﬁcant improvement (on average +26% for SAC and +39% for PPO) in the performance of the policy gradient algorithms, improvement which is consistent across tasks.

Original: As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efﬁcient in terms of sampling in all tasks.
Revised: As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efﬁcient in terms of sampling in all tasks.

Original: Overall, AVEC improves TRPO, PPO and SAC in terms of performance and efﬁciency.
Revised: Overall, AVEC improves TRPO, PPO and SAC in terms of performance and efﬁciency.

Original: This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on- and off-policy algorithms, we believe that these experiments are sufﬁcient to prove the relevance of AVEC .
Revised: This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on- and off-policy algorithms, we believe that these experiments are sufﬁcient to prove the relevance of AVEC .

Original: Notice that since no additional calculations are needed in AVEC ’s implementation, computational complexity remains unchanged.
Revised: Notice that since no additional calculations are needed in AVEC ’s implementation, computational complexity remains unchanged.

Original: 5. S PARSE R EWARD S IGNALS
Revised: 5. S PARSE R EWARD S IGNALS

Original: Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards.
Revised: Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards.

Original: In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration.
Revised: In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration.

Original: The relative value estimate of an unseen state is more accurate: in Section 4.2, AVEC identiﬁes extreme state-values ( e.g. , non-zero rewards in tasks with sparse rewards) faster.
Revised: The relative value estimate of an unseen state is more accurate: in Section 4.2, AVEC identiﬁes extreme state-values ( e.g. , non-zero rewards in tasks with sparse rewards) faster.

Original: In Fig.
Revised: In Fig.

Original: 3a and 3b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards.
Revised: 3a and 3b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards.

Original: AVEC enhances TRPO and PPO in both experiments.
Revised: AVEC enhances TRPO and PPO in both experiments.

Original: When PPO and AVEC -PPO both reach the best possible performance, AVEC -PPO exhibits better sample efﬁciency.
Revised: When PPO and AVEC -PPO both reach the best possible performance, AVEC -PPO exhibits better sample efﬁciency.

Original: Fig.
Revised: Fig.

Original: 3c and 3d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of , the graph suggest that AVEC -PPO learns the difﬁcult locomotion principles in the absence of rewards and visits a much larger part of the state space 
Revised: 3c and 3d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of the hill (red), the graph suggest that AVEC -PPO learns the difﬁcult locomotion principles in the absence of rewards and visits a much larger part of the state space (green).

Original: This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up on experienced positive reward more easily.
Revised: This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up on experienced positive reward more easily.

Original: Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values.
Revised: Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values.

Original: [Figure]
Revised: [Figure]

Original: 5. A NALYSIS OF THE V ARIANCE E STIMATED C RITIC
Revised: 5. A NALYSIS OF THE V ARIANCE E STIMATED C RITIC

Original: In order to further validate AVEC , we evaluate the performance of the value network in more detail: we examine (a) the estimation error (distance to the empirical target), (b) the approximation error (distance to the true target) and (c) the empirical variance of the gradient.
Revised: In order to further validate AVEC , we evaluate the performance of the value network in more detail: we examine (a) the estimation error (distance to the empirical target), (b) the approximation error (distance to the true target) and (c) the empirical variance of the gradient.

Original: (a,b) should be put into perspective with the conclusions of Ilyas et al.
Revised: (a,b) should be put into perspective with the conclusions of Ilyas et al.

Original: (2020) where it is found that the critic only ﬁts the empirical value function but not the true one.
Revised: (2020) where it is found that the critic only ﬁts the empirical value function but not the true one.

Original: (c) should be placed in light of Tucker et al.
Revised: (c) should be placed in light of Tucker et al.

Original: (2018) highlighting a failure of recently proposed state-action-dependent baselines to reduce the variance.[-1.0 Timesteps 1e
 D i s t an c e t o e m p-]
Revised: (2018) highlighting a failure of recently proposed state-action-dependent baselines to reduce the variance.[-1.0 Timesteps 1e
 D i s t an c e t o e m p-]

Original: i r i c a l V - f un c t i on

Original: AntBullet AVEC-PPOPPO 0.

Original: 1.0 Timesteps 1e

Original: [Figure]

Original: [Figure]

Original: D i s t an c e t

Original: o e m p

Original: i r i c a l V

Original: [Figure]

Original: f un c t

Original: i

Original: [Figure]
Revised: [Figure]

Original: Learning the Empirical Target.
Revised: Learning the Empirical Target.

Original: In Fig.
Revised: In Fig.

Original: 4, we report the quality of ﬁt (MSE) of ˆ V π by PPO and AVEC -PPO in the AntBullet and HalfCheetahBullet tasks.
Revised: 4, we report the quality of ﬁt (MSE) of ˆ V π by PPO and AVEC -PPO in the AntBullet and HalfCheetahBullet tasks.

Original: We observe that PPO better ﬁts the empirical target than when equipped with
Revised: We observe that PPO better ﬁts the empirical target than when equipped with

Original: AVEC , which is to be expected since vanilla PPO optimizes the MSE directly.
Revised: AVEC , which is to be expected since vanilla PPO optimizes the MSE directly.

Original: This result put aside the remarkable improvement in the performance of AVEC -PPO (Fig. 2) suggests that AVEC might be a better estimator of the true value function.
Revised: This result put aside the remarkable improvement in the performance of AVEC -PPO (Fig. 2) suggests that AVEC might be a better estimator of the true value function.

Original: We examine this claim below because if true, it would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method.[Figure]
Revised: We examine this claim below because if true, it would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method.

Original: and ∀ t ∈ { 1 , 2 , 4 , 6 , 9 } · 10 5 we stop training, use the current policy to collect 3 · 10transitions to estimate V π .

Original: Learning the True Target.
Revised: Learning the True Target.

Original: A fundamental premise of policy gradient methods is that optimizing the objective based on an empirical estimation of the value function leads to a better policy.
Revised: A fundamental premise of policy gradient methods is that optimizing the objective based on an empirical estimation of the value function leads to a better policy.

Original: Which is why we investigate the quality of ﬁt of the true target.
Revised: Which is why we investigate the quality of ﬁt of the true target.

Original: To approximate the true value function, we ﬁt the returns sampled from the current policy using a large number of transitions ( 3 · 10 5 ). 
Revised: To approximate the true value function, we ﬁt the returns sampled from the current policy using a large number of transitions ( 3 · 10 5 ). [Figure]

Original: Fig.
Revised: Fig.

Original: 5 shows that g φ is far closer to the true value function half of the time (horizon is 10 6 ) than the estimator obtained with MSE, then as close to it.
Revised: 5 shows that g φ is far closer to the true value function half of the time (horizon is 10 6 ) than the estimator obtained with MSE, then as close to it.

Original: Comparing Fig.
Revised: Comparing Fig.

Original: 5 with Fig.
Revised: 5 with Fig.

Original: 4, we see that the distance to the true target is close to the estimation error for AVEC -PPO, while for PPO, it is at least two orders of magnitude higher at all times.
Revised: 4, we see that the distance to the true target is close to the estimation error for AVEC -PPO, while for PPO, it is at least two orders of magnitude higher at all times.

Original: A similar analysis for the Q-function estimator for SAC and  AVEC -SAC in AntBullet and
Revised: We further investigate these results in Fig. 9 in Appendix B.2 where we study the variation of the squared bias and variance components of the MSE to the true target (MSE = Var+Bias 2 ). We ﬁnd, as expected, that using AVEC reduces the variance term signiﬁcantly while slightly increasing the bias term, which Fig.

Original: HalfCheetahBullet is given in Appendix B.2, with similar results and interpretation.
Revised: 5 conﬁrms is negligible since the total MSE is substantially reduced. For completeness, we also analyze the distance to the true target for the Qfunction estimator in SAC and AVEC -SAC in AntBullet and HalfCheetahBullet in Appendix B.3, with similar results and interpretation.

Original: We conclude that AVEC improves the value function approximation and we expect that the gradient is more stable.
Revised: We conclude that AVEC improves the value function approximation and we expect that the gradient is more stable.

Original: [Figure]
Revised: [Figure]

Original: 1e 0.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: A v e r age pa i r w i s e c o s s im
Revised: r w i s e c o s s i

Original: AntBullet AVEC-PPOPPO 0. 1.0 Timesteps 1e6 0.
Revised: [Figure]

Original: A v e r age pa i r w i s e c o s s
Revised: A v e [Figure] i r w i s e c o s [Figure]

Original: im
Revised: i

Original: HalfCheetahBullet

Original: [Figure]
Revised: [Figure]

Original: Empirical Variance Reduction.
Revised: Empirical Variance Reduction.

Original: We choose to study the gradient variance using the average pairwise cosine similarity metric as it allows a comparison with Ilyas et al.
Revised: We choose to study the gradient variance using the average pairwise cosine similarity metric as it allows a comparison with Ilyas et al.

Original: (2020), with which we share the same experimental setup and scales.
Revised: (2020), with which we share the same experimental setup and scales.

Original: Fig.
Revised: Fig.

Original: 6 shows that AVEC yields a higher average (10 batches per iteration) pairwise cosine similarity, which means closer batch-estimates of the gradient and, in turn, indicates smaller gradient variance.
Revised: 6 shows that AVEC yields a higher average (10 batches per iteration) pairwise cosine similarity, which means closer batch-estimates of the gradient and, in turn, indicates smaller gradient variance.

Original: Further analysis with additional tasks is included in Appendix B.3.
Revised: Further analysis with additional tasks is included in Appendix B.4.

Original: The variance reduction effect observed in several environments suggests that AVEC is the ﬁrst method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.
Revised: The variance reduction effect observed in several environments suggests that AVEC is the ﬁrst method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: [Figure]
Revised: [Figure]

Original: In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning.
Revised: In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning.

Original: For this study, we choose difﬁcult tasks of PyBullet and use PPO because it is more efﬁcient than TRPO and requires less computations than SAC.
Revised: For this study, we choose difﬁcult tasks of PyBullet and use PPO because it is more efﬁcient than TRPO and requires less computations than SAC.

Original: For an estimator ˆ y n of ( y i ) i ∈{ 1 ,...,n } , we write Bias = 1 n (cid:80) ni
Revised: For an estimator ˆ y n of ( y i ) i ∈{ 1 ,...,n } , we write Bias = 1 n (cid:80) ni

Original: =

Revised: =

Original: (ˆ y i − y i ) and Var =n − 1 (cid:80) ni
Revised: (ˆ y i − y i ) and Var =n − 1 (cid:80) ni

Original: =

Revised: =

Original: (ˆ y i − y
Revised: (ˆ y i − y

Original: i − Bias) 2 .
Revised: i − Bias) 2 .

Original: Consequently:
Revised: Consequently:

Original: MSE =
Revised: MSE =

Original: Var + Bias 2 .
Revised: Var + Bias 2 .

Original: We denote L α =
Revised: We denote L α =

Original: Var + α Bias 2 , with α ∈ R .
Revised: Var + α Bias 2 , with α ∈ R .

Original: In Fig.
Revised: In Fig.

Original: 7, Bias- α means that we use L α and Var- α means that we use L 1 α .
Revised: 7, Bias- α means that we use L α and Var- α means that we use L 1 α .

Original: We observe that while no consistent order on the choices of α is identiﬁed, AVEC seems to outperform all other weightings.
Revised: We observe that while no consistent order on the choices of α is identiﬁed, AVEC seems to outperform all other weightings.

Original: Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Fig.
Revised: Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Fig.

Original: 7a and 7c, and in Fig.
Revised: 7a and 7c, and in Fig.

Original: 7b and 7d.
Revised: 7b and 7d.

Original: A more extensive hyper-parameter study with more α values might provide even higher performances, nevertheless we believe that the stability of an algorithm is crucial for a reliable performance.
Revised: A more extensive hyper-parameter study with more α values might provide even higher performances, nevertheless we believe that the stability of an algorithm is crucial for a reliable performance.

Original: As such, the tuning of hyper-parameters to achieve good results should remain mild.
Revised: As such, the tuning of hyper-parameters to achieve good results should remain mild.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we introduce a new training objective for the critic in actor-critic algorithms to better approximate the true value function.
Revised: In this work, we introduce a new training objective for the critic in actor-critic algorithms to better approximate the true value function.

Original: In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modiﬁcation is both theoretically sound and intuitively supported by the need to improve the approximation error of the critic.
Revised: In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modiﬁcation is both theoretically sound and intuitively supported by the need to improve the approximation error of the critic.

Original: The application of A ctor with V ariance E stimated C ritic ( AVEC ) to state-of-the-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyper-parameter tuning.
Revised: The application of A ctor with V ariance E stimated C ritic ( AVEC ) to state-of-the-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyper-parameter tuning.

Original: First, for SAC-like algorithms where the critic learns a state-action-value function, our results strongly suggest that state-actions with extreme values are identiﬁed more quickly.
Revised: First, for SAC-like algorithms where the critic learns a state-action-value function, our results strongly suggest that state-actions with extreme values are identiﬁed more quickly.

Original: Second, for PPO-like methods where the critic learns the state-values, we show that the variance of the gradient is reduced and empirically demonstrate that this is due to a better approximation of the state-values.
Revised: Second, for PPO-like methods where the critic learns the state-values, we show that the variance of the gradient is reduced and empirically demonstrate that this is due to a better approximation of the state-values.

Original: In sparse reward environments, the theoretical intuition behind a variance estimated critic is more explicit and is also supported by empirical evidence.
Revised: In sparse reward environments, the theoretical intuition behind a variance estimated critic is more explicit and is also supported by empirical evidence.

Original: In addition to corroborating the results in Ilyas et al.
Revised: In addition to corroborating the results in Ilyas et al.

Original: (2020) proving that the value estimator fails to ﬁt V π , we propose a method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms.
Revised: (2020) proving that the value estimator fails to ﬁt V π , we propose a method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms.

Original: Furthermore,
Revised: Furthermore,

Original: AVEC beneﬁts from its simplicity of implementation since no further assumptions are required (such as horizon awareness Tucker et al.
Revised: AVEC beneﬁts from its simplicity of implementation since no further assumptions are required (such as horizon awareness Tucker et al.

Original: (2018) to remedy the deﬁciency of existing variance-reduction methods) and the modiﬁcation of current algorithms represents only a few lines of code.
Revised: (2018) to remedy the deﬁciency of existing variance-reduction methods) and the modiﬁcation of current algorithms represents only a few lines of code.

Original: In this paper we have demonstrated the beneﬁts of a more thorough analysis of the critic objective in policy gradient methods.
Revised: In this paper, we have demonstrated the beneﬁts of a more thorough analysis of the critic objective in policy gradient methods.

Original: Despite our strongly favourable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study.
Revised: Despite our strongly favourable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study.

Original: In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments is anticipated; we consider the problem of noise separation in the latter, as this is the ﬁrst obstacle to accessing the variance and distinguishing extreme values from outliers.
Revised: In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments is anticipated; we consider the problem of noise separation in the latter, as this is the ﬁrst obstacle to accessing the variance and distinguishing extreme values from outliers.

Original: A U NBIASED AVEC P OLICY G RADIENT
Revised: A U NBIASED AVEC P OLICY G RADIENT

Original: In this section, we consider the case in which the state-action-value function of a policy π θ is approximated.
Revised: In this section, we consider the case in which the state-action-value function of a policy π θ is approximated.

Original: We prove that given some assumptions on this estimator function, we can use it to yield a valid gradient direction,
Revised: We prove that given some assumptions on this estimator function, we can use it to yield a valid gradient direction,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. , we are able to prove policy improvement when following this direction.
Revised: e. , we are able to prove policy improvement when following this direction.

Original: In this setting
Revised: In this setting

Original: [Equation]
Revised: [Equation]

Original: the critic minimizes the following loss:
Revised: the critic minimizes the following loss:

Original: [Equation]
Revised: [Equation]

Original: When a local optimum is reached
Revised: When a local optimum is reached

Original: [Equation]
Revised: [Equation]

Original: the gradient of the latter expression is zero:
Revised: the gradient of the latter expression is zero:

Original: [Equation]
Revised: [Equation]

Original: In the expression above
Revised: In the expression above

Original: [Equation]
Revised: [Equation]

Original: the expected value of the partial derivative disappears because the term in the ﬁrst bracket is centered:
Revised: the expected value of the partial derivative disappears because the term in the ﬁrst bracket is centered:

Original: [Equation]
Revised: [Equation]

Original: Simplifying the gradient at the local optimum becomes:
Revised: Simplifying the gradient at the local optimum becomes:

Original: [Equation]
Revised: [Equation]

Original: Then, if we denote g φ = f φ ( s, a )+ E ( s,a ) ∼ π
Revised: Then, if we denote g φ = f φ ( s, a )+ E ( s,a ) ∼ π

Original: [ ˆ Q π ( s, a ) − f φ ( s, a )]
Revised: [ ˆ Q π ( s, a ) − f φ ( s, a )]

Original: [Equation]
Revised: [Equation]

Original: and use the policy parameterization assumption:
Revised: and use the policy parameterization assumption:

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By combining the parameterization assumption in Eq.
Revised: By combining the parameterization assumption in Eq.

Original: 5 with Eq.
Revised: 5 with Eq.

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Since the expression above is null
Revised: Since the expression above is null

Original: [Equation]
Revised: [Equation]

Original: we have the following:
Revised: we have the following:

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: While the proof seems more or less generic, the assumption in Eq.
Revised: While the proof seems more or less generic, the assumption in Eq.

Original: 5 is extremely constraining to the possible approximators.
Revised: 5 is extremely constraining to the possible approximators.

Original: Sutton et al.
Revised: Sutton et al.

Original: (2000) quotes J. Tsitsiklis who believes that a linear g φ in the features of the policy may be the only feasible solution for this condition.
Revised: (2000) quotes J. Tsitsiklis who believes that a linear g φ in the features of the policy may be the only feasible solution for this condition.

Original: Concretely, such an assumption cannot hold since neural networks are the standard approximators used in practice.
Revised: Concretely, such an assumption cannot hold since neural networks are the standard approximators used in practice.

Original: Moreover, empirical analysis (Ilyas et al., 2020) indicates that commonly used algorithms fail to ﬁt the true value function.
Revised: Moreover, empirical analysis (Ilyas et al., 2020) indicates that commonly used algorithms fail to ﬁt the true value function.

Original: However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines.
Revised: However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines.

Original: B A DDITIONAL E XPERIMENTS B. C ONTINUOUS C ONTROL
Revised: B A DDITIONAL E XPERIMENTS B. C ONTINUOUS C ONTROL

Original: Fig.
Revised: Fig.

Original: 8 shows the total average return for AVEC coupled with SAC and PPO on the Walker2d task.
Revised: 8 shows the total average return for AVEC coupled with SAC and PPO on the Walker2d task.

Original: Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings a signiﬁcant performance improvement (+26% for SAC and +33% for PPO), conﬁrming the generality of our approach. 
Revised: Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings a signiﬁcant performance improvement (+26% for SAC and +33% for PPO), conﬁrming the generality of our approach. [Figure] B. V ARIATION OF THE B IAS AND V ARIANCE TERMS : PPO In Fig.

Original: [Figure]
Revised: [Figure]

Revised: we show the variation of the bias and variance terms in the MSE between the estimators (of

Revised: AVEC-PPO and PPO) and the true target.

Revised: We observe that the variance reduction is more substantial than that of the bias.

Revised: Using those results and Fig.

Revised: 5 showing that the distance of the estimator to V π is lower when using AVEC conﬁrms that the variance reduction effect counterbalances the bias increase.

Revised: Note that the % Variation of the Var term is always negative in our experiments, and that the shaded areas that suggest otherwise are merely due to a false assumption of symmetrical deviations, itself due to the assumption of Gaussianity needed to construct conﬁdence intervals.

Revised: [Figure]

Revised: [Figure]

Original: B. L EARNING THE T RUE T ARGET : SAC
Revised: B. L EARNING THE T RUE T ARGET : SAC

Original: In Fig.
Revised: In Fig.

Original: [Figure]
Revised: [Figure]

Original: we compare the error between the Q-function estimator and the true Q-function for SAC and AVEC -SAC in AntBullet and HalfCheetahBullet.
Revised: we compare the error between the Q-function estimator and the true Q-function for SAC and AVEC -SAC in AntBullet and HalfCheetahBullet.

Original: We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the signiﬁcant performance gains in Fig.[Figure] t i
Revised: We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the signiﬁcant performance gains in Fig.

Original: [Figure]
Revised: [Figure]

Original: B. V ARIANCE R EDUCTION
Revised: B. V ARIANCE R EDUCTION

Original: In Fig.
Revised: In Fig.

Original: [Figure]
Revised: [Figure]

Original: we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks:
Revised: we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks:

Original: HopperBullet and Walker2DBullet.
Revised: HopperBullet and Walker2DBullet.

Original: We also vary the trajectory size used in the estimation of the gradient.
Revised: We also vary the trajectory size used in the estimation of the gradient.

Original: [Figure]

Original: age

Original: [Figure]
Revised: [Figure]

Original: C I MPLEMENTATION OF AVEC COUPLED WITH SAC
Revised: C I MPLEMENTATION OF AVEC COUPLED WITH SAC

Original: In Algorithm 2, J V is the squared residual error objective to train the soft value function.
Revised: In Algorithm 2, J V is the squared residual error objective to train the soft value function.

Original: See Haarnoja et al.
Revised: See Haarnoja et al.

Original: (2018) for further details and notations about SAC, not directly relevant here.
Revised: (2018) for further details and notations about SAC, not directly relevant here.

Original: Algorithm 2 AVEC coupled with SAC.
Revised: Algorithm 2 AVEC coupled with SAC.

Original: 1: Input
Revised: 1: Input

Original: 2: Initialize policy θ
Revised: 2: Initialize policy θ

Original: Q φ3: D ← ∅ 4: for each iteration do 5: for each step do 6: a t ∼ π θ ( a t | s t ) 7: s t +1 ∼ P ( s t , a t ) 8:
Revised: Q φ3: D ← ∅ 4: for each iteration do 5: for each step do 6: a t ∼ π θ ( a t | s t ) 7: s t +1 ∼ P ( s t , a t ) 8:

Original: D ← D ∪ { ( s t , a t , r t , s t +1 ) } 9: end for 10: for each gradient step do 11: sample batch B from D 12: ψ ← ψ − λ V ˆ ∇ ψ J V ψ ) 13: φ i ← φ i − λ Q
Revised: D ← D ∪ { ( s t , a t , r t , s t +1 ) } 9: end for 10: for each gradient step do 11: sample batch B from D 12: ψ ← ψ − λ V ˆ ∇ ψ J V ψ ) 13: φ i ← φ i − λ Q

Original: ∇ φ
Revised: ∇ φ

Original: L
Revised: L

Original: AVEC φ i for i ∈ { 1 , 2 } 14: θ ← θ − λ π ˆ ∇ θ J ( π θ ) 15: ¯ ψ ← βψ + (1 − β ) ¯ ψ 16: end for 17: end for
Revised: AVEC φ i for i ∈ { 1 , 2 } 14: θ ← θ − λ π ˆ ∇ θ J ( π θ ) 15: ¯ ψ ← βψ + (1 − β ) ¯ ψ 16: end for 17: end for

Original: D I MPLEMENTATION D ETAILS
Revised: D I MPLEMENTATION D ETAILS

Original: Theoretically, L AVEC is deﬁned as the residual variance of the value function ( cf.
Revised: Theoretically, L AVEC is deﬁned as the residual variance of the value function ( cf.

Original: Eq. 3).
Revised: Eq. 3).

Original: However, state-values for a non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values.
Revised: However, state-values for a non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values.

Original: Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:
Revised: Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:

Original: [Equation]
Revised: [Equation]

Original: T (cid:88) t
Revised: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: where T is the size of the sampled trajectory.
Revised: where T is the size of the sampled trajectory.

Original: E E XPERIMENT D ETAILS
Revised: E E XPERIMENT D ETAILS

Original: In all experiments we choose to use the same hyper-parameter values for all tasks as the bestperforming ones reported in the literature or in their respective open source implementation documentation.
Revised: In all experiments we choose to use the same hyper-parameter values for all tasks as the bestperforming ones reported in the literature or in their respective open source implementation documentation.

Original: We thus ensure the best performance for the conventional actor-critic framework.
Revised: We thus ensure the best performance for the conventional actor-critic framework.

Original: In other words, since we are interested in evaluating the impact of this new critic, everything else is kept as is.
Revised: In other words, since we are interested in evaluating the impact of this new critic, everything else is kept as is.

Original: This experimental protocol may not beneﬁt AVEC .
Revised: This experimental protocol may not beneﬁt AVEC .

Original: In Table 2, 3 and
Revised: In Table 2, 3 and

Original: [Table]
Revised: [Table]

Original: we report the list of hyper-parameters common to all continuous control experiments.
Revised: we report the list of hyper-parameters common to all continuous control experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: F C OMPARATIVE E VALUATION OF AVEC WITH TRPO
Revised: F C OMPARATIVE E VALUATION OF AVEC WITH TRPO

Original: In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO (Schulman et al., 2015) algorithm.
Revised: In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO (Schulman et al., 2015) algorithm.

Original: Fig.shows the learning curves while Table 5 reports the results.
Revised: Fig.shows the learning curves while Table 5 reports the results.

Original: [Figure]
Revised: [Figure]

Original: Task
Revised: Task

Original: TRPO
Revised: TRPO

Original: AVEC -TRPO
Revised: AVEC -TRPO

Original: Ant −
Revised: Ant −

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: AntBullet 970 ± 70 (
Revised: AntBullet 970 ± 70 (

Original: [Table]
Revised: [Table]

Original: HCheetah 466 ± 56 (+ 35 %)
Revised: HCheetah 466 ± 56 (+ 35 %)

Original: HCBullet 1281 ± 94 (+ 11 %)
Revised: HCBullet 1281 ± 94 (+ 11 %)

Original: Humanoid
Revised: Humanoid

Original: [Table]
Revised: [Table]

Original: G E NVIRONMENTS D ETAILS
Revised: G E NVIRONMENTS D ETAILS

Original: [Table]
Revised: [Table]

Original: H D IMENSIONS OF S TUDIED T ASKS
Revised: H D IMENSIONS OF S TUDIED T ASKS

Original: [Table]
Revised: [Table]