Original: CASA: B RIDGING THE G AP BETWEEN P OLICY I MPROVEMENT AND P OLICY E VALUATION WITH C ONFLICT A VERSE P OLICY I TERATION
Revised: CASA: B RIDGING THE G AP BETWEEN P OLICY I MPROVEMENT AND P OLICY E VALUATION WITH C ONFLICT A VERSE P OLICY I TERATION

Original: Anonymous authors
Revised: Changnan Xiao xiaochangnan@bytedance.com Haosen Shi shihaosen98@gmail.com Jiajun Fan fanjj21@mails.tsinghua.edu.cn Shihong Deng dengshihong@bytedance.com

Original: Paper under double-blind review
Revised: Haiyan Yin yinhaiyan@outlook.com

Original: ABSTRACT
Revised: A

Revised: BSTRACT

Original: We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI).
Revised: We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI).

Original: While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods with function approximation assume the independence of GPI steps, despite of the inherent connections between them.
Revised: While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods with function approximation assume the independence of GPI steps, despite of the inherent connections between them.

Original: In this paper, we present a method that attempts to eliminate the inconsistency between policy evaluation step and policy improvement step, leading to a conflict averse GPI solution with gradient-based functional approximation.
Revised: In this paper, we present a method that attempts to eliminate the inconsistency between policy evaluation step and policy improvement step, leading to a conflict averse GPI solution with gradient-based functional approximation.

Original: Our method is capital to balancing exploitation and exploration between policy-based and value-based methods and is applicable to existing policy-based and value-based methods.
Revised: Our method is capital to balancing exploitation and exploration between policy-based and value-based methods and is applicable to existing policy-based and value-based methods.

Original: We conduct extensive experiments to study theoretical properties of our method and demonstrate the effectiveness of our method on Atari 200M benchmark
Revised: We conduct extensive experiments to study theoretical properties of our method and demonstrate the effectiveness of our method on Atari 200M benchmark.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Model-free reinforcement learning has made many impressive breakthroughs in a wide range of Markov
Revised: Model-free reinforcement learning has made many impressive breakthroughs in a wide range of Markov

Original: Decision Processes (MDP) (Vinyals et al., 2019; Pedersen, 2019; Badia et al., 2020).
Revised: Decision Processes (MDP) (Vinyals et al., 2019; Pedersen, 2019; Badia et al., 2020).

Original: Overall, the methods could be cast into two categories, value-based methods such as DQN (Mnih et al., 2015) and Rainbow (Hessel et al., 2017), and policy-based methods such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017) and IMPALA (Espeholt et al., 2018).
Revised: Overall, the methods could be cast into two categories, value-based methods such as DQN (Mnih et al., 2015) and Rainbow (Hessel et al., 2017) and policy-based methods such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017) and IMPALA (Espeholt et al., 2018).

Original: Value-based methods learn state-action values and select the action according to their values.
Revised: Value-based methods learn state-action values and select the action according to their values.

Original: The main target of value-based methods is to approximate the fixed point of the Bellman equation through the generalized policy iteration (GPI) (Sutton & Barto, 2018), which generally consists of policy evaluation and policy improvement.
Revised: The main target of value-based methods is to approximate the fixed point of the Bellman equation through the generalized policy iteration (GPI) (Sutton & Barto, 2018), which generally consists of policy evaluation and policy improvement.

Original: One characteristic of the value-based methods is that unless a more accurate state-action value is estimated by iterations of the policy evaluation, the policy will not be improved.
Revised: One characteristic of the value-based methods is that unless a more accurate state-action value is estimated by iterations of the policy evaluation, the policy will not be improved.

Original: Previous works equip value-based methods with many carefully designed structures to achieve more promising reward learning and 
Revised: Previous works equip value-based methods with many carefully designed structures to achieve more promising reward learning and sample efficiency (Wang et al., 2016;

Revised: Schaul et al., 2015; Kapturowski et al., 2018).

Original: Policy-based methods learn a parameterized policy directly without consulting state-action values.
Revised: Policy-based methods learn a parameterized policy directly without consulting state-action values.

Original: One characteristic of policy-based methods is that they incorporate a policy improvement phase in every training step, while in contrast, the value-based methods only change the policy after the action corresponding to the highest state-action values is changed.
Revised: One characteristic of policy-based methods is that they incorporate a policy improvement phase in every training step, while in contrast, the value-based methods only change the policy after the action corresponding to the highest state-action values is changed.

Original: In principle, policy-based methods perform policy improvement more frequently than value-based methods.
Revised: In principle, policy-based methods perform policy improvement more frequently than value-based methods.

Original: We notice that value-based and policy-based methods locate at the two extremes of GPI, where value-based methods won’t improve the policy until a more accurate policy evaluation is achieved, while policy-based methods improve the policy for every training step even when the policy evaluation hasn’t converged.
Revised: We notice that value-based and policy-based methods locate at the two extremes of GPI, where value-based methods won’t improve the policy until a more accurate policy evaluation is achieved, while policy-based methods improve the policy for every training step even when the policy evaluation hasn’t converged.

Original: To mitigate the defect of each, we pursuit a technique that is capable of balancing between the two extremes flexibly.
Revised: To mitigate the defect of each, we pursuit a technique that is capable of balancing between the two extremes flexibly.

Original: We first study the gradients between policy improvement and policy evaluation and notice that they show a positive correlation statistically during the entire training process.
Revised: We first study the gradients between policy improvement and policy evaluation and notice that they show a positive correlation statistically during the entire training process.

Original: To find out if there exists a way that the gradients of the policy improvement and the policy evaluation are parallel, we propose CASA,
Revised: To find out if there exists a way that the gradients of the policy improvement and the policy evaluation are parallel, we propose CASA,

Original: C ritic AS an A ctor, which satisfies a weaker compatible condition (Sutton et al., 1999) and enhances gradient consistency between policy improvement and policy evaluation.
Revised: C ritic AS an A ctor, which satisfies a weaker compatible condition (Sutton et al., 1999) and enhances gradient consistency between policy improvement and policy evaluation.

Original: With further delving into the properties of CASA, we find CASA is an innovative combination of value-based and policy-based methods.
Revised: With further delving into the properties of CASA, we find CASA is an innovative combination of value-based and policy-based methods.

Original: When the policy-based methods are equipped with CASA, the collapse to the sub-optimal solution as the entropy goes to zero is prevented by the evaluation of the state-action values, which encourages exploration.
Revised: When the policy-based methods are equipped with CASA, the collapse to the sub-optimal solution as the entropy goes to zero is prevented by the evaluation of the state-action values, which encourages exploration.

Original: When the value-based methods are equipped with CASA, the policy improvement via policy gradient is equivalent to the evaluation of the state-action values and a self-bootstrapped policy improvement, which enhances exploitation.
Revised: When the value-based methods are equipped with CASA, the policy improvement via policy gradient is equivalent to the evaluation of the state-action values and a self-bootstrapped policy improvement, which enhances exploitation.

Original: To enable CASA for a large scale off-policy learning, we introduce Doubly-Robust Trace (DR-Trace), which exploits doubly-robust estimator (Jiang & Li, 2016) and guarantees the synchronous convergence of the state-action values and the state values.
Revised: To enable CASA for a large scale off-policy learning, we introduce Doubly-Robust Trace (DR-Trace), which exploits doubly-robust estimator (Jiang & Li, 2016) and guarantees the synchronous convergence of the state-action values and the state values.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: (i)
Revised: (i)

Original: We present a novel method CASA which enhances gradient consistency between policy evaluation and policy improvement and present extensive studies on the behavior of the gradients.
Revised: We present a novel method CASA which enhances gradient consistency between policy evaluation and policy improvement and present extensive studies on the behavior of the gradients.

Original: (ii) We demonstrate CASA could be freely applied to both policy-based and value-based algorithms with motivating examples.
Revised: (ii) We demonstrate CASA could be freely applied to both policy-based and value-based algorithms with motivating examples.

Original: (iii)
Revised: (iii)

Original: We present extensive empirical study on Atari benchmark , where our conflict averse algorithm brings substantial improvements over the baseline methods.
Revised: We present extensive empirical study on Atari benchmark , where our conflict averse algorithm brings substantial improvements over the baseline methods.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: Consider an infinite-horizon MDP, defined by a tuple ( S , A , p, r, γ ) , where S is the state space, A is the action space, p : S ×
Revised: Consider an infinite-horizon MDP, defined by a tuple ( S , A , p, r, γ ) , where S is the state space, A is the action space, p : S ×

Original: A × S → [0 , 1] is the state transition probability function, r : S ×
Revised: A × S → [0 , 1] is the state transition probability function, r : S ×

Original: A → R is the reward function, and γ is the discounted factor.
Revised: A → R is the reward function, and γ is the discounted factor.

Original: The policy is a mapping π : S × A → [0 , 1] which assigns a distribution over the action space given a state.
Revised: The policy is a mapping π : S × A → [0 , 1] which assigns a distribution over the action space given a state.

Original: The objective of reinforcement learning is to maximize the return , or cumulative discounted rewards,
Revised: The objective of reinforcement learning is to maximize the return , or cumulative discounted rewards,

Original: [Equation]
Revised: [Equation]

Original: where traj = { s 0 , a 0 , r 0 , . . .
Revised: where traj = { s 0 , a 0 , r 0 , . . .

Original: } is a trajectory sampled by π with policy-environment interaction.
Revised: } is a trajectory sampled by π with policy-environment interaction.

Original: Value-based methods maximize J by estimating various type of value functions: the state value function is defined as V π ( s ) =
Revised: Value-based methods maximize J by estimating various type of value functions: the state value function is defined as V π ( s ) =

Original: E π [ (cid:80) t γ t r t | s 0 =
Revised: E π [ (cid:80) t γ t r t | s 0 =

Original: s ] , the state-action value function is defined as Q π ( s, a ) =
Revised: s ] , the state-action value function is defined as Q π ( s, a ) =

Original: E π [ (cid:80) t γ t r t | s 

Revised: E π [ (cid:80) t γ t r t | s

Original: =
Revised: =

Original: s, a 0 = a ] ; the advantage function is defined as A π ( s, a ) =
Revised: s, a 0 = a ] ; the advantage function is defined as A π ( s, a ) =

Original: Q π ( s, a ) − V π ( s ) .
Revised: Q π ( s, a ) − V π ( s ) .

Original: The objective of maximizing the value functions in value-based methods can be improved through GPI until converging to the optimal policy.
Revised: The objective of maximizing the value functions in value-based methods can be improved through GPI until converging to the optimal policy.

Original: For the approximated state-value function Q θ that estimates Q π , the policy evaluation is conducted by:
Revised: For the approximated state-value function Q θ that estimates Q π , the policy evaluation is conducted by:

Original: [Equation]
Revised: [Equation]

Original: where Q π is estimated by various methods, e.g., λ -return (Sutton, 1988) and ReTrace (Munos et al., 2016).
Revised: where Q π is estimated by various methods, e.g., λ -return (Sutton, 1988) and ReTrace (Munos et al., 2016).

Original: The policy improvement is usually achieved by greedily selecting actions with the highest state-action values.
Revised: The policy improvement is usually achieved by greedily selecting actions with the highest state-action values.

Revised: π *, Q *

Revised: Q = Q π π

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: π θ , Q θ Q θ = Q π π *, Q * π θ =

Original:  Policy-based methods maximize J by optimizing some parameterized policy π θ according to the policy gradient theorem (Sutton & Barto, 2018),
Revised: p g ( Q θ ) Policy-based methods maximize J by optimizing some parameterized policy π θ according to the policy gradient theorem (Sutton & Barto, 2018),

Original: [Equation]
Revised: [Equation]

Original: The vanilla policy gradient uses Ψ
Revised: The vanilla policy gradient uses Ψ

Original: = (cid:80) ∞ t =0 γ t r t .
Revised: = (cid:80) ∞ t =0 γ t r t .

Original: Actor-critic algorithms approximate Ψ( s, a ) in the form of baseline, e g., IMPALA (Espeholt et al., 2018) adopts Ψ( s, a ) =
Revised: Actor-critic algorithms approximate Ψ( s, a ) in the form of baseline, e.g., IMPALA (Espeholt et al., 2018) adopts Ψ( s, a ) =

Original: r + γV ˜ π
Revised: r + γV ˜ π

Original: ( s ′ )
Revised: ( s ′ )

Original: − V θ ( s ) and uses V-Trace to estimate V ˜ π .
Revised: − V θ ( s ) and uses V-Trace to estimate V ˜ π .

Original:  M ETHODOLOGY 3. M OTIVATION
Revised:  M ETHODOLOGY 3. M OTIVATION

Original: We use V θ to estimate V π , Q θ to estimate Q π and π θ to represent the policy, where θ represents all parameters to be optimized.
Revised: We use V θ to estimate V π , Q θ to estimate Q π and π θ to represent the policy, where θ represents all parameters to be optimized.

Original: In this work, there is one backbone and two individual heads after the backbone.
Revised: In this work, there is one backbone and two individual heads after the backbone.

Original: The advantage function and the policy share one head, and state value function is the other head.
Revised: The advantage function and the policy share one head, and state value function is the other head.

Original: Hence the policy reuses all parameters of value functions except that temperature τ is only for the policy.
Revised: Hence the policy reuses all parameters of value functions except that temperature τ is only for the policy.

Original: We keep τ static in this work.
Revised: We keep τ static in this work.

Original: We use E to represent the policy evaluation, which gives the ascent direction of the gradient by θ ← θ + η E π [( Q π − Q θ ) ∇ θ Q θ ] .
Revised: We use E to represent the policy evaluation, which gives the ascent direction of the gradient by θ ← θ + η E π [( Q π − Q θ ) ∇ θ Q θ ] .

Original: We use I to represent the policy improvement, which gives θ ← θ + η E π [( Q π − V θ ) ∇ θ log π θ ] .
Revised: We use I to represent the policy improvement, which gives θ ← θ + η E π [( Q π − V θ ) ∇ θ log π θ ] .

Original: Let’s recap the GPI process as shown in Figure 1.
Revised: Let’s recap the GPI process as shown in Figure 1.

Original: To get rid of the function approximation error, we first assume the approximation function enjoys infinite capacity.
Revised: To get rid of the function approximation error, we first assume the approximation function enjoys infinite capacity.

Original: We use < x, y > to denote the angle between two vectors, where < x, y > = arccos( x · y || x ||·|| y || ) with arccos : [ − 1 , 1] → [0 , π ] .
Revised: We use < x, y > to denote the angle between two vectors, where < x, y > = arccos( x · y || x ||·|| y || ) with arccos : [ − 1 , 1] → [0 , π ] .

Original: We define an important notion β , which represents the angle between the gradient ascent directions of I and E , as follows,
Revised: We define an important notion β , which represents the angle between the gradient ascent directions of I and E , as follows,

Original: [Equation]
Revised: [Equation]

Original: When β = 0 i.e. cos( β ) = 1 , I and E become parallel to each other, which is the blue arrow in Figure 1, and there is no conflict between the gradient ascent directions of I and E anymore.
Revised: When β = 0 i.e. cos( β ) = 1 , I and E become parallel to each other, which is the blue arrow in Figure 1, and there is no conflict between the gradient ascent directions of I and E anymore.

Original: When β = π/ 2 i.e. cos( β ) = 0 ,
Revised: When β = π/ 2 i.e. cos( β ) = 0 ,

Original: I and E are perpendicular.
Revised: I and E are perpendicular.

Original: When β = π i.e. cos( β ) = − 1 , I and E are toward exactly opposite directions.
Revised: When β = π i.e. cos( β ) = − 1 , I and E are toward exactly opposite directions.

Original: Next, we assume the representation capacity of the approximation function is limited.
Revised: Next, we assume the representation capacity of the approximation function is limited.

Original: When the function approximation is involved, i.e. Q π is estimated by Q θ and π is approximated by π θ , from the view of operators (Ghosh et al., 2020), each of I and E can be further decomposed into two operators, as shown in Figure 2.
Revised: When the function approximation is involved, i.e. Q π is estimated by Q θ and π is approximated by π θ , from the view of operators (Ghosh et al., 2020), each of I and E can be further decomposed into two operators, as shown in Figure 2.

Original: One is to do the policy improvement and the policy evaluation, the other is to project into the restricted function space.
Revised: One is to do the policy improvement and the policy evaluation, the other is to project into the restricted function space.

Original: When β > 0 , GPI with function approximation would involve two projection
Revised: When β > 0 , GPI with function approximation would involve two projection

Original: [Figure]
Revised: [Figure]

Original: operators in each iteration [Figure] which introduces inevitable approximation error.
Revised: operators in each iteration, which introduces inevitable approximation error.

Original: When β = 0 , if the function approximation error is not considered, we find that the gradient conflict between I and E would be totally eliminated.
Revised: When β = 0 , if the function approximation error is not considered, we find that the gradient conflict between I and E would be totally eliminated.

Original: If we consider the limitation of the approximation function, similar to the blue arrow in Figure 1, one iteration (represented by two black arrows and two dotted arrows) can be united into one arrow and one dotted arrow (not shown in Figure 2 but analogy to the blue arrow in Figure 1), where the gradient conflict is eliminated and the two projection operators are reduced to one correspondingly.
Revised: If we consider the limitation of the approximation function, similar to the blue arrow in Figure 1, one iteration (represented by two black arrows and two dotted arrows) can be united into one arrow and one dotted arrow (not shown in Figure 2 but analogy to the blue arrow in Figure 1), where the gradient conflict is eliminated and the two projection operators are reduced to one correspondingly.

Original: As stated above, if β = 0 holds, we can expect that the gradient conflict between the policy improvement and the policy evaluation is eliminated and the function approximation error could be reduced.
Revised: As stated above, if β = 0 holds, we can expect that the gradient conflict between the policy improvement and the policy evaluation is eliminated and the function approximation error could be reduced.

Original: However, β is usually estimated by sampling with stochasticity.
Revised: However, β is usually estimated by sampling with stochasticity.

Original: It’s difficult to let β = 0 by optimizing θ .
Revised: It’s difficult to let β = 0 by optimizing θ .

Original: Instead, we consider another notion χ by removing step sizes and taking expectation outside, where the angle of each state is fully controllable by θ .
Revised: Instead, we consider another notion χ by removing step sizes and taking expectation outside, where the angle of each state is fully controllable by θ .

Original: [Equation]
Revised: [Equation]

Original: In fact, χ is highly correlated to compatible value function (Sutton et al., 1999), and Theorem 3 shows that χ
Revised: In fact, χ is highly correlated to compatible value function (Sutton et al., 1999), and Theorem 3 shows that χ

Original: = 1 is the necessary condition for the compatible condition ∇ θ Q θ = ∇ θ log π θ , which is a weaker compatible condition.
Revised: = 1 is the necessary condition for the compatible condition ∇ θ Q θ = ∇ θ log π θ , which is a weaker compatible condition.

Original: More details about compatible value function are in Appendix A.
Revised: More details about compatible value function are in Appendix A.

Original: To further understand the behavior of β and χ , we track cos( β ) and χ of two algorithms PPO and R2D as representatives for policy-based and value-based methods, respectively.
Revised: To further understand the behavior of β and χ , we track cos( β ) and χ of two algorithms PPO and R2D as representatives for policy-based and value-based methods, respectively.

Original: We show an important fact in
Revised: We show an important fact in

Original: Figure 3 that both χ and cos( β ) are statistically positive for both original version and adjusted versions, which means that arccos( χ ) and β are likely to be less then π/ 2 with neural network approximated functions.
Revised: Figure 3 that both χ and cos( β ) are statistically positive for both original version and adjusted versions, which means that arccos( χ ) and β are likely to be less then π/ 2 with neural network approximated functions.

Original: The aforementioned conceptual and empirical findings inspire us to raise the following question on GPI: whether we can guarantee χ = 1 , so that cos( β ) is also closer to 1 .
Revised: The aforementioned conceptual and empirical findings inspire us to raise the following question on GPI: whether we can guarantee χ = 1 , so that cos( β ) is also closer to 1 .

Original: 3. F ORMULATION
Revised: 3. F ORMULATION

Original: Denote τ ∈ R + to be a positive temperature and sg to be a stop gradient operator.
Revised: Denote τ ∈ R + to be a positive temperature and sg to be a stop gradient operator.

Original: CASA can estimate V θ and A θ by any function parameterized by θ , where π θ and Q θ are derived as follows:
Revised: CASA can estimate V θ and A θ by any function parameterized by θ , where π θ and Q θ are derived as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that there exist two sg operators in equation 6.
Revised: Note that there exist two sg operators in equation 6.

Original: The first sg operator is used for computing advantage as ¯ A θ =
Revised: The first sg operator is used for computing advantage as ¯ A θ =

Original: A θ − E π
Revised: A θ − E π

Original: [ A θ ]
Revised: [ A θ ]

Original: =
Revised: =

Original: A θ − sg ( π θ ) ·
Revised: A θ − sg ( π θ ) ·

Original: A θ , where the sg operator here guarantees the gradients between policy improvement and policy evaluation are parallel, which we elaborate later.
Revised: A θ , where the sg operator here guarantees the gradients between policy improvement and policy evaluation are parallel, which we elaborate later.

Original: Intuitively, this sg operator also means that we keep π θ unchanged when evaluating the policy π θ .
Revised: Intuitively, this sg operator also means that we keep π θ unchanged when evaluating the policy π θ .

Original: The second sg operator exists in
Revised: The second sg operator exists in

Original: Q θ =
Revised: Q θ =

Original: ¯
Revised: ¯

Original: A θ + sg ( V θ ) .
Revised: A θ + sg ( V θ ) .

Original: As (Chen & He, 2020) regards sg in siamese representation learning as a case of
Revised: As (Chen & He, 2020) regards sg in siamese representation learning as a case of

Original: EM-algorithm (Dempster et al., 1977), a similar interpretation exists here.
Revised: EM-algorithm (Dempster et al., 1977), a similar interpretation exists here.

Original: Q θ =
Revised: Q θ =

Original: ¯ A θ + sg ( V θ ) decomposes the estimation of Q θ into a two stage problem, where the first is to estimate the advantage of each action without changing the expectation, the second is to estimate the expectation.
Revised: ¯ A θ + sg ( V θ ) decomposes the estimation of Q θ into a two stage problem, where the first is to estimate the advantage of each action without changing the expectation, the second is to estimate the expectation.

Original: The equation 6 includes a straightforward refinement of dueling-DQN.
Revised: The equation 6 includes a straightforward refinement of dueling-DQN.

Original: We know dueling-DQN estimates Q π by Q θ =
Revised: We know dueling-DQN estimates Q π by Q θ =

Original: A θ + V θ , but it cannot guarantee E π
Revised: A θ + V θ , but it cannot guarantee E π

Original: [ A θ ] = 

Revised: [ A θ ] =

Original: i.e. E π [ Q θ ] =
Revised: i.e. E π [ Q θ ] =

Original: V θ due to the function approximation error.
Revised: V θ due to the function approximation error.

Original: But if we estimate Q π by Q θ =
Revised: But if we estimate Q π by Q θ =

Original: A θ − E π
Revised: A θ − E π

Original: [ A θ ] + V θ , it satisfies the necessary condition
Revised: [ A θ ] + V θ , it satisfies the necessary condition

Original: E π [ Q θ ]
Revised: E π [ Q θ ]

Original: = V θ without loss of generality.
Revised: = V θ without loss of generality.

Original: 3. P ATH C ONSISTENCY B ETWEEN P OLICY E VALUATION AND P OLICY I MPROVEMENT
Revised: 3. P ATH C ONSISTENCY B ETWEEN P OLICY E VALUATION AND P OLICY I MPROVEMENT

Original: For brevity, we omit θ and V, Q, A, π are all approximated functions.
Revised: For brevity, we omit θ and V, Q, A, π are all approximated functions.

Original: Denote the estimations of V and Q as
Revised: Denote the estimations of V and Q as

Original: V ˜ π and Q ˜ π respectively.
Revised: V ˜ π and Q ˜ π respectively.

Original: For instance, one choice is to calculate V ˜ π and Q ˜ π by V-Trace (Espeholt et al., 2018) and ReTrace (Munos et al., 2016) respectively.
Revised: For instance, one choice is to calculate V ˜ π and Q ˜ π by V-Trace (Espeholt et al., 2018) and ReTrace (Munos et al., 2016) respectively.

Original: At training time, the policy evaluation is achieved by updating θ to minimize,
Revised: At training time, the policy evaluation is achieved by updating θ to minimize,

Original: [Equation]
Revised: [Equation]

Original:  which gives the ascent direction of θ by:
Revised: , which gives the ascent direction of θ by:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: And we make the policy improvement by policy gradient, which gives the ascent direction of θ by:
Revised: And we make the policy improvement by policy gradient, which gives the ascent direction of θ by:

Original: [Equation]
Revised: [Equation]

Original: where J ( τ, θ ) =
Revised: where J ( τ, θ ) =

Original: τ E π [ (cid:80) γ t r t ] .
Revised: τ E π [ (cid:80) γ t r t ] .

Original: It takes an additional τ , which frees the scale of gradient from τ .
Revised: It takes an additional τ , which frees the scale of gradient from τ .

Original: The final gradient ascent direction of θ is given by:
Revised: The final gradient ascent direction of θ is given by:

Original: [Equation]
Revised: [Equation]

Original: With ( V, Q, π ) defined in equation 6, by Lemma E.1, we have,
Revised: With ( V, Q, π ) defined in equation 6, by Lemma E.1, we have,

Original: [Equation]
Revised: [Equation]

Original: For brevity, denote the shared gradient path as g = ( 1 − π ) ∇ θ A.
Revised: For brevity, denote the shared gradient path as g = ( 1 − π ) ∇ θ A.

Original: Plugging equation 10 into equation 7 equation 8, we have,
Revised: Plugging equation 10 into equation 7 equation 8, we have,

Original: [Equation]
Revised: [Equation]

Original: By equation 11, ∇ θ L Q and ∇ θ J walk along the same vector direction of gradient path g for each state.
Revised: By equation 11, ∇ θ L Q and ∇ θ J walk along the same vector direction of gradient path g for each state.

Original: By equation 10, this is exactly the case χ = 1 .
Revised: By equation 10, this is exactly the case χ = 1 .

Original: Since all parameters to estimate Q and π are shared except for τ , we call it C ritic AS an A ctor.
Revised: Since all parameters to estimate Q and π are shared except for τ , we call it C ritic AS an A ctor.

Original: If we make a subtraction between ∇ θ L Q and ∇ θ J , we have,
Revised: If we make a subtraction between ∇ θ L Q and ∇ θ J , we have,

Original: [Equation]
Revised: [Equation]

Original: We know E π
Revised: We know E π

Original: [( Q − V ) g ] is a self-bootstrapped policy gradient with function approximated Q .
Revised: [( Q − V ) g ] is a self-bootstrapped policy gradient with function approximated Q .

Original: Recalling the fact that the value-based methods improves the policy by greedily selecting actions according to Q , if we apply ∇ θ J on θ , it additionally utilizes Q to do policy improvement.
Revised: Recalling the fact that the value-based methods improves the policy by greedily selecting actions according to Q , if we apply ∇ θ J on θ , it additionally utilizes Q to do policy improvement.

Original: This is a more greedy usage of Q to improve policy than its usual usage.
Revised: This is a more greedy usage of Q to improve policy than its usual usage.

Original: If we exploit the structural information as ( V, Q, π ) defined by equation 6, by Lemma E.2,
Revised: If we exploit the structural information as ( V, Q, π ) defined by equation 6, by Lemma E.2,

Original: [Equation]
Revised: [Equation]

Original: then we have,
Revised: then we have,

Original: [Equation]
Revised: [Equation]

Original: The equation 13 shows ∇ θ L Q is a policy gradient with an entropy regularization.
Revised: The equation 13 shows ∇ θ L Q is a policy gradient with an entropy regularization.

Original: If we apply ∇ θ L Q on θ for policy-based methods, an entropy regularization works implicitly by α 2 ∇ θ L Q in equation 9, which prevents the policy collapse to a sub-optimal solution.
Revised: If we apply ∇ θ L Q on θ for policy-based methods, an entropy regularization works implicitly by α 2 ∇ θ L Q in equation 9, which prevents the policy collapse to a sub-optimal solution.

Original: 3. DR-T RACE AND O FF -P OLICY T RAINING
Revised: 3. DR-T RACE AND O FF -P OLICY T RAINING

Original: [Table]
Revised: [Table]

Original: [Equation] V ˜ π
Revised: Trace

Original: [Equation]
Revised: [Equation]

Original: To enable off-policy training with behavior policy µ , one choice is to estimate V ˜ π and Q ˜ π in equationand equation 8 by V-Trace and ReTrace.
Revised: To enable off-policy training with behavior policy µ , one choice is to estimate V ˜ π and Q ˜ π in equationand equation 8 by V-Trace and ReTrace.

Original: As CASA estimates ( V, Q, π ) , applying Doubly Robust (Jiang &
Revised: As CASA estimates ( V, Q, π ) , applying Doubly Robust (Jiang &

Original: Li, 2016) is feasible and suitable.
Revised: Li, 2016) is feasible and suitable.

Original: We propose DR-Trace and find the convergence rate and the fixed point of DR-Trace are the same as V-Trace’s
Revised: We propose DR-Trace and find the convergence rate and the fixed point of DR-Trace are the same as V-Trace’s

Original: according to its convergence proof.
Revised: according to its convergence proof.

Original: For completeness, we provide
Revised: For completeness, we provide

Original: DR-Trace and its comparison with V-Trace/ReTrace in Table 1.
Revised: DR-Trace and its comparison with V-Trace/ReTrace in Table 1.

Original: More details are in Appendix D.
Revised: More details are in Appendix D.

Original:  E XPERIMENTS 4. B ASIC S ETUP
Revised:  E XPERIMENTS 4. B ASIC S ETUP

Original: We employ a Learner-Actor pipeline (Espeholt et al., 2018) for large-scale training.
Revised: We employ a Learner-Actor pipeline (Espeholt et al., 2018) for large-scale training.

Original: Motivation and ablation experiments on PPO and R2D2 don’t use LSTM, only experiments on CASA+DR-Trace use LSTM (Schmidhuber, 1997), which is for comparison with other algorithms.
Revised: Motivation and ablation experiments on PPO and R2D2 don’t use LSTM, only experiments on CASA+DR-Trace use LSTM (Schmidhuber, 1997), which is for comparison with other algorithms.

Original: We use burn-in (Kapturowski et al., 2018)
Revised: We use burn-in (Kapturowski et al., 2018)

Original: when
Revised: when

Original: LSTM is used.
Revised: LSTM is used.

Original: All estimated values share the same backbone, which is followed by two fully connected layers for each individual head.
Revised: All estimated values share the same backbone, which is followed by two fully connected layers for each individual head.

Original: We use no intrinsic reward and no entropy regularization in any experiment.
Revised: We use no intrinsic reward and no entropy regularization in any experiment.

Original: We find that using life information can greatly increase the performance of some games.
Revised: We find that using life information can greatly increase the performance of some games.

Original: However, to be general, we will not end the episode if life is lost.
Revised: However, to be general, we will not end the episode if life is lost.

Original: All hyperparameters are in Appendix F.
Revised: All hyperparameters are in Appendix F.

Original: For brevity, we denote ∇ L V =
Revised: For brevity, we denote ∇ L V =

Original: E π
Revised: E π

Original: [( V π − V θ ) ∇ V θ ] , ∇ L Q =
Revised: [( V π − V θ ) ∇ V θ ] , ∇ L Q =

Original: E π
Revised: E π

Original: [( Q π − Q θ ) ∇ Q θ ] and ∇J = E π
Revised: [( Q π − Q θ ) ∇ Q θ ] and ∇J = E π

Original: [( Q π −
Revised: [( Q π −

Original: V θ ) ∇ log π θ ] , where expectation is batch-wise average in our implementation.
Revised: V θ ) ∇ log π θ ] , where expectation is batch-wise average in our implementation.

Original: When we write < a, b > with a, b ∈ {∇ L V , ∇ L Q , ∇J } , we firstly calculate batch-wise averaged gradient of a and b , then we calculate the angle in-between.
Revised: When we write < a, b > with a, b ∈ {∇ L V , ∇ L Q , ∇J } , we firstly calculate batch-wise averaged gradient of a and b , then we calculate the angle in-between.

Original: When we write cos < ∇ Q, ∇ log π > or χ , we mean E π [cos < ∇ θ Q θ , ∇ θ log π θ > ] , which firstly calculates element-wise cosines and then takes batch-wise average.
Revised: When we write cos < ∇ Q, ∇ log π > or χ , we mean E π [cos < ∇ θ Q θ , ∇ θ log π θ > ] , which firstly calculates element-wise cosines and then takes batch-wise average.

Original: To avoid numerical problem, we calculate [Equation]
Revised: To avoid numerical problem, we calculate x · y || x ||·||

Original: by
Revised: y

Original: [Equation]
Revised: || by x · y max( || x || , 10 − 8 ) · max( || y || , 10 − 8 ) .

Original: 4. A PPLICATION OF CASA ON R EPRESENTATIVE A LGORITHMS
Revised: 4. A PPLICATION OF CASA ON R EPRESENTATIVE A LGORITHMS

Original: CASA is applicable to existing algorithms.
Revised: CASA is applicable to existing algorithms.

Original: We take PPO and R2D2 for demonstration.
Revised: We take PPO and R2D2 for demonstration.

Original: The application of CASA on PPO is straightforward.
Revised: The application of CASA on PPO is straightforward.

Original: Applying CASA on R2D2 is impossible as either ϵ -greedy policy
Revised: Applying CASA on R2D2 is impossible as either ϵ -greedy policy

Original: [Table]
Revised: PPO PPO+CASA R2D R2D2+CASA ⇒ ( V,A ) =

Original:  V [Table]
Revised: ( V θ ,

Original: A  )[Table]
Revised: A θ ) ⇒ ( V,A )

Original: π
Revised: =

Original: [Table]
Revised: ( V θ ,A θ ) Func.

Original: [Table]
Revised: ( V,logit ) = ( V θ ,logit θ ) π = softmax ( A/τ ) ( V,A ) = ( V θ ,A θ ) π = softmax ( A/τ )

Original: Approx
Revised: Approx.

Original: [Table]
Revised: π = softmax ( logit ) ¯ A = A − sg ( π ) · A

Original:  =
Revised: Q =

Original: [Table]
Revised: A + V ¯ A = A − sg ( π ) · A Q =

Original:  A 
Revised: ¯ A + sg ( V ) Q = ¯ A + sg ( V )

Original: [Table]
Revised: [Table]

Original: or arg max Q policy breaks the gradient.
Revised: or arg max Q policy breaks the gradient.

Original: This problem is the same as calculating the gradients of policy improvement for value-based methods.
Revised: This problem is the same as calculating the gradients of policy improvement for value-based methods.

Original: We use a surrogate policy π surrogate = softmax ( A/τ ) , which is discussed in Appendix B. Table 2 summarizes adjustments of function approximations and training gradients.
Revised: We use a surrogate policy π surrogate = softmax ( A/τ ) , which is discussed in Appendix B. Table 2 summarizes adjustments of function approximations and training gradients.

Original: Since PPO+CASA and R2D2+CASA have the same function approximation, recalling the fact that valuebased methods improve the policy when a more accurate evaluation is achieved and policy-based methods improve the policy for every step, we can balance the two flexibly with χ = 1 by α 1 , α 2 , α 3 in equation 9.
Revised: Since PPO+CASA and R2D2+CASA have the same function approximation, recalling the fact that valuebased methods improve the policy when a more accurate evaluation is achieved and policy-based methods improve the policy for every step, we can balance the two flexibly with χ = 1 by α 1 , α 2 , α 3 in equation 9.

Original: In Figure 3, algorithms with CASA show much higher cos( β ) and χ .
Revised: In Figure 3, algorithms with CASA show much higher cos( β ) and χ .

Original: PPO+CASA does more exploration than the original PPO, as the entropy of π doesn’t easily drop to zero.
Revised: PPO+CASA does more exploration than the original PPO, as the entropy of π doesn’t easily drop to zero.

Original: R2D2+CASA tends to distinct the state-action values, where we use the entropy of Q to measure how greedy the current state-action values are.
Revised: R2D2+CASA tends to distinct the state-action values, where we use the entropy of Q to measure how greedy the current state-action values are.

Original: 4. B EHAVIOR OF G RADIENTS ON DIFFERENT STRUCTURES
Revised: 4. B EHAVIOR OF G RADIENTS ON DIFFERENT STRUCTURES

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: Though we show that CASA satisfies ∇ Q ∝ ∇ log π , which means χ = 1 , it’s unknown if the structure of
Revised: Though we show that CASA satisfies ∇ Q ∝ ∇ log π , which means χ = 1 , it’s unknown if the structure of

Original: CASA is unique.
Revised: CASA is unique.

Original: As Q =
Revised: As Q =

Original: A − E π
Revised: A − E π

Original: [ A ] + sg ( V ) is a direct refinement of dueling-DQN, we try several different structures of PPO+CASA.
Revised: [ A ] + sg ( V ) is a direct refinement of dueling-DQN, we try several different structures of PPO+CASA.

Original: All settings of estimating state-action values are shown in Table 3.
Revised: All settings of estimating state-action values are shown in Table 3.

Original: We always use 0 . 5 · ∇ L V + ∇ L Q + ∇J as the training gradient.
Revised: We always use 0 . 5 · ∇ L V + ∇ L Q + ∇J as the training gradient.

Original: We present Breakout and Qbert in Figure 4.
Revised: We present Breakout and Qbert in Figure 4.

Original: For the sake of clarity, we group PPO+CASA and type 3 as sg - V group, type 2 and type 4 as no - sg - V group.
Revised: For the sake of clarity, we group PPO+CASA and type 3 as sg - V group, type 2 and type 4 as no - sg - V group.

Original: The sg - V group has higher χ and higher cos( β ) , which is closer to the compatible condition and the consistency between two GPI steps, and no - sg - V group is always worst than its contrast in sg - V group.
Revised: The sg - V group has higher χ and higher cos( β ) , which is closer to the compatible condition and the consistency between two GPI steps, and no - sg - V group is always worst than its contrast in sg - V group.

Original: PPO+CASA has χ = 1 and the highest cos( β ) .
Revised: PPO+CASA has χ = 1 and the highest cos( β ) .

Original: Type 1 has less returns than PPO+CASA.
Revised: Type 1 has less returns than PPO+CASA.

Original: Hence, when applying a CASA-like structure, stopping the gradient of π is always preferred.
Revised: Hence, when applying a CASA-like structure, stopping the gradient of π is always preferred.

Original: Type 5 uses an individual head to estimate Q π , which performs the worst.
Revised: Type 5 uses an individual head to estimate Q π , which performs the worst.

Original: Hence, a well-designed CASA-like or dueling-like structure is always preferred.
Revised: Hence, a well-designed CASA-like or dueling-like structure is always preferred.

Original: By scatter plot and box plot in Figure 4, χ and cos( β ) are positive correlated depending on different structures.
Revised: By scatter plot and box plot in Figure 4, χ and cos( β ) are positive correlated depending on different structures.

Original: This phenomenon answers part of the last question of Section 
Revised: This phenomenon answers part of the last question of Section 3.1: for these specific designed structures, χ and cos( β ) show positive correlation. [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: c o s <
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Original: L Q , J >
Revised: [Table] [Figure] [Table] [Figure]

Original:  E VALUATION OF CASA ON A TARI G AMES
Revised: 4. E VALUATION OF CASA ON A TARI G AMES

Original: We present an extensive evaluation on CASA, where we train CASA + DR-Trace on 57 Atari games and report the results in terms of two metrics.
Revised: We present an extensive evaluation on CASA, where we train CASA + DR-Trace on 57 Atari games and report the results in terms of two metrics.

Original: The first is Human Normalized Score (HNS), which normalizes the reward by random policy and human expert policies.
Revised: The first is Human Normalized Score (HNS), which normalizes the reward by random policy and human expert policies.

Original: The other is Standardized Atari BEnchmark for RL (SABER), which normalizes the reward by random policy and human world records, where the normalized score is capped by 200 % .
Revised: The other is Standardized Atari BEnchmark for RL (SABER), which normalizes the reward by random policy and human world records, where the normalized score is capped by 200 % .

Original: SABER is considered because recent studies show that the median HNS could easily get hacked by the algorithm since it is sensitive to improvement on a small subset of games.
Revised: SABER is considered because recent studies show that the median HNS could easily get hacked by the algorithm since it is sensitive to improvement on a small subset of games.

Original: Tablesummarizes the results.
Revised: Tablesummarizes the results.

Original: [Table]
Revised: [Table]

Original: Note that CASA is a variant of IMPALA with DR-Trace, and it achieves substantially better records than
Revised: Note that CASA is a variant of IMPALA with DR-Trace, and it achieves substantially better records than

Original: IMPALA across all the evaluation metrics.
Revised: IMPALA across all the evaluation metrics.

Original: It also scores substantially better than all the methods in terms of mean HNS and median SABER scores.
Revised: It also scores substantially better than all the methods in terms of mean HNS and median SABER scores.

Original: Though off-policy methods are known as privileged for HNS evaluation due to replay, CASA outperforms strong off-policy baseline Rainbow.
Revised: Though off-policy methods are known as privileged for HNS evaluation due to replay, CASA outperforms strong off-policy baseline Rainbow.

Original: Though LASER outperforms
Revised: Though LASER outperforms

Original: CASA in Median HNS and Mean SABER, CASA outperforms it in median SABER and mean HNS.
Revised: CASA in Median HNS and Mean SABER, CASA outperforms it in median SABER and mean HNS.

Original: Overall, the aforementioned results demonstrate the conflict-averse strategy efficiently boosts the performance in largescale training scenarios and outperform strong on/off-policy algorithms.
Revised: Overall, the aforementioned results demonstrate the conflict-averse strategy efficiently boosts the performance in largescale training scenarios and outperform strong on/off-policy algorithms.

Original: Hyperparameters and individual games are presented in Appendix F and Appendix G, respectively.
Revised: Hyperparameters and individual games are presented in Appendix F and Appendix G, respectively.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Both value-based or policy-based approaches comply with the principle of GPI, but two GPI steps are coarsely related to each other such that jointly optimizing both functions might potentially bring conflicts.
Revised: Both value-based or policy-based approaches comply with the principle of GPI, but two GPI steps are coarsely related to each other such that jointly optimizing both functions might potentially bring conflicts.

Original: Despite of such crucial issue in GPI with function approximation, most decent model-free algorithms adopt a standard policy improvement/evaluation regime without considering conflict diminishing properties.
Revised: Despite of such crucial issue in GPI with function approximation, most decent model-free algorithms adopt a standard policy improvement/evaluation regime without considering conflict diminishing properties.

Original: The issue of reducing conflicts among multiple models trained simultaneously was considered in earlier machine learning literature, such as for robust parameter estimation for multiple estimators under incomplete data (Robins &
Revised: The issue of reducing conflicts among multiple models trained simultaneously was considered in earlier machine learning literature, such as for robust parameter estimation for multiple estimators under incomplete data (Robins &

Original: Rotnitzky, 1995; Lunceford & Davidian, 2004; Kang & Schafer, 2007) and multitask learning with gradient similarity measure (Chen et al., 2020; Yu et al., 2020; Javaloy & Valera, 2022).
Revised: Rotnitzky, 1995; Lunceford & Davidian, 2004; Kang & Schafer, 2007) and multitask learning with gradient similarity measure (Chen et al., 2020; Yu et al., 2020; Javaloy & Valera, 2022).

Original: When the idea is introduced to reinforcement learning, earliest attempts tackle conservative and safe policy iteration problems (Kakade & Langford, 2002; Hazan & Kale, 2011; Pirotta et al., 2013).
Revised: When the idea is introduced to reinforcement learning, earliest attempts tackle conservative and safe policy iteration problems (Kakade & Langford, 2002; Hazan & Kale, 2011; Pirotta et al., 2013).

Original: Recently, more works have emerged to study GPI in a fine-grained manner.
Revised: Recently, more works have emerged to study GPI in a fine-grained manner.

Original: In (Ghosh et al., 2020), a new Bellman operator is introduced which implements GPI with a policy improvement operator and a projection operator, where the projection attempts to find the best approximation of policy among realizable policies.
Revised: In (Ghosh et al., 2020), a new Bellman operator is introduced which implements GPI with a policy improvement operator and a projection operator, where the projection attempts to find the best approximation of policy among realizable policies.

Original: In (Raileanu &
Revised: In (Raileanu &

Original: Fergus, 2021), the policy and value updates are decoupled by approximating two networks with representation regularization.
Revised: Fergus, 2021), the policy and value updates are decoupled by approximating two networks with representation regularization.

Original: In (Cobbe et al., 2021), GPI is separated into a policy improvement and a feature distillation step.
Revised: In (Cobbe et al., 2021), GPI is separated into a policy improvement and a feature distillation step.

Original: On contrast to the aforementioned works, we tackle the conflicts in GPI at the gradient-level, with theoretical analysis.
Revised: On contrast to the aforementioned works, we tackle the conflicts in GPI at the gradient-level, with theoretical analysis.

Original: Our work is related to (Nachum et al., 2017), which utilizes both the unbiasedness and stability of on-policy training and the data efficiency of off-policy training to form a soft consistency error.
Revised: Our work is related to (Nachum et al., 2017), which utilizes both the unbiasedness and stability of on-policy training and the data efficiency of off-policy training to form a soft consistency error.

Original: Our work bridges the gap between the two GPI steps from an alternative angle of establishing a closer relationship between policy and value functions in their forms, without the focus on off-policy correction.
Revised: Our work bridges the gap between the two GPI steps from an alternative angle of establishing a closer relationship between policy and value functions in their forms, without the focus on off-policy correction.

Original: Due to the difficulty of controlling the gap between GPI steps, we instead consider χ .
Revised: Due to the difficulty of controlling the gap between GPI steps, we instead consider χ .

Original: The condition χ = 1 is close to compatible value function (Sutton et al., 1999; Kakade, 2001), shown in Section 3.1 and Appendix A.
Revised: The condition χ = 1 is close to compatible value function (Sutton et al., 1999; Kakade, 2001), shown in Section 3.1 and Appendix A.

Original:  L IMITATION
Revised:  L IMITATION

Original: It’s noticeable that CASA is only applied on discrete action space for now.
Revised: It’s noticeable that CASA is only applied on discrete action space for now.

Original: We further find CASA applicable to any function approximation that is able to estimate advantage functions of all actions.
Revised: We further find CASA applicable to any function approximation that is able to estimate advantage functions of all actions.

Original: We provide additional discussion on continuous action space in Appendix C.
Revised: We provide additional discussion on continuous action space in Appendix C.

Original: Since π shares all parameters of value functions, it brings χ = 1 but sacrifices the freedom of π to be parameterized by other parameters.
Revised: Since π shares all parameters of value functions, it brings χ = 1 but sacrifices the freedom of π to be parameterized by other parameters.

Original: We conjecture that CASA is one endpoint of a trade-off curve between χ and the freedom of π , where the other endpoint is that π shares no parameter with value functions.
Revised: We conjecture that CASA is one endpoint of a trade-off curve between χ and the freedom of π , where the other endpoint is that π shares no parameter with value functions.

Original:  E THICS AND R EPRODUCIBILITY S TATEMENT
Revised:  E THICS AND R EPRODUCIBILITY S TATEMENT

Original: This paper is aimed at academic issues in deep reinforcement learning, and the experiment used is also in the early stage, but it may provide opportunities for malicious applications of reinforcement learning in the future.
Revised: This paper is aimed at academic issues in deep reinforcement learning, and the experiment used is also in the early stage, but it may provide opportunities for malicious applications of reinforcement learning in the future.

Original: We describe all details to reproduce the main experimental results in Appendix F.
Revised: We describe all details to reproduce the main experimental results in Appendix F.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper attempts to eliminate gradient inconsistency between policy improvement and policy evaluation.
Revised: This paper attempts to eliminate gradient inconsistency between policy improvement and policy evaluation.

Original: The proposed innovative actor-critic design C ritic AS an A ctor (CASA) enhances consistency of two GPI steps by satisfying a weaker compatible condition.
Revised: The proposed innovative actor-critic design C ritic AS an A ctor (CASA) enhances consistency of two GPI steps by satisfying a weaker compatible condition.

Original: We present both theoretical proof and empirical evaluation for CASA.
Revised: We present both theoretical proof and empirical evaluation for CASA.

Original: The results show that our proposed method achieves state-of-the-art performance standards with noticeable performance gain over several strong baselines when evaluated on ALE 200 million (200M) benchmark.
Revised: The results show that our proposed method achieves state-of-the-art performance standards with noticeable performance gain over several strong baselines when evaluated on ALE 200 million (200M) benchmark.

Original: We also present several ablation studies, which demonstrates the effectiveness of the proposed method’s theoretical properties.
Revised: We also present several ablation studies, which demonstrates the effectiveness of the proposed method’s theoretical properties.

Original: Future work includes studying the connection between the compatible condition and the gradient consistency between policy improvement and policy evalution. 
Revised: Future work includes studying the connection between the compatible condition and the gradient consistency between policy improvement and policy evalution. R

Original: A C OMPATIBLE V ALUE F UNCTION
Revised: A C OMPATIBLE V ALUE F UNCTION

Original: The original policy gradient with compatible value function is stated as follow.
Revised: The original policy gradient with compatible value function is stated as follow.

Original: Theorem 1 (Sutton et al.
Revised: Theorem 1 (Sutton et al.

Original: Let Q w be a state-action function with parameter w and π θ be a policy function with parameter θ .
Revised: Let Q w be a state-action function with parameter w and π θ be a policy function with parameter θ .

Original: If Q w satisfies E π [( Q π − Q w ) ∇ w Q w ] = 0 and ∇ w Q w = ∇ θ log π θ , then 
Revised: If Q w satisfies E π [( Q π − Q w ) ∇ w Q w ] = 0 and ∇ w Q w = ∇ θ log π θ , then ∇ [Equation] log π

Original: [Equation]
Revised: [Equation]

Original: If we let w = θ in Theorem 1, where Q w and π θ share parameters, we have the following theorem.
Revised: If we let w = θ in Theorem 1, where Q w and π θ share parameters, we have the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q θ be a state-action function with parameter θ and π θ be a policy function with parameter θ .
Revised: Let Q θ be a state-action function with parameter θ and π θ be a policy function with parameter θ .

Original: If Q θ satisfies E π [( Q π − Q θ ) ∇ θ Q θ ] = 0 and ∇ θ Q θ = ∇ θ log π θ , then
Revised: If Q θ satisfies E π [( Q π − Q θ ) ∇ θ Q θ ] = 0 and ∇ θ Q θ = ∇ θ log π θ , then

Original: [Equation]
Revised: [Equation]

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: We show that χ
Revised: We show that χ

Original: = 1 is the necessary condition for the compatible condition ∇ θ Q θ = ∇ θ log π θ .
Revised: = 1 is the necessary condition for the compatible condition ∇ θ Q θ = ∇ θ log π θ .

Original: Theorem 3.
Revised: Theorem 3.

Original: i)
Revised: i)

Original: If ∇ θ Q θ ∝ ∇ θ log π θ for all states, then χ = 1 . ii) If χ = 1 , then ∇ θ Q θ ∝ ∇ θ log π θ for all states.
Revised: If ∇ θ Q θ ∝ ∇ θ log π θ for all states, then χ = 1 . ii) If χ = 1 , then ∇ θ Q θ ∝ ∇ θ log π θ for all states.

Original: By Theorem 3, χ = 1 is equivalent to ∇ θ Q θ ∝ ∇ θ log π θ , and ∇ θ Q θ ∝ ∇ θ log π θ is the necessary condition for ∇ θ Q θ = ∇ θ log π θ , hence χ
Revised: By Theorem 3, χ = 1 is equivalent to ∇ θ Q θ ∝ ∇ θ log π θ , and ∇ θ Q θ ∝ ∇ θ log π θ is the necessary condition for ∇ θ Q θ = ∇ θ log π θ , hence χ

Original: = 1 is the necessary condition for ∇ θ Q θ = ∇ θ log π θ .
Revised: = 1 is the necessary condition for ∇ θ Q θ = ∇ θ log π θ .

Original: Proof.
Revised: Proof.

Original: i)
Revised: i)

Original: Since ∇ θ Q θ ∝ ∇ θ log π θ , we have < ∇ θ Q θ , ∇ θ log π θ > = 0 .
Revised: Since ∇ θ Q θ ∝ ∇ θ log π θ , we have < ∇ θ Q θ , ∇ θ log π θ > = 0 .

Original: By definition of χ , we haveχ = E π
Revised: By definition of χ , we have

Original: [cos < ∇ θ Q θ , ∇ θ log π θ > ] = E π [1] = 1 . ii)
Revised: [Equation]

Original:  Since χ ≤ 1 and cos( x ) is monotonic decreasing as x goes from 0 to π , the equality χ = 1 only holds when all states satisfy cos < ∇ θ Q θ , ∇ θ log π θ > = 0 , which means ∇ θ Q θ ∝ ∇ θ log π θ .
Revised: ii) Since χ ≤ 1 and cos( x ) is monotonic decreasing as x goes from 0 to π , the equality χ = 1 only holds when all states satisfy cos < ∇ θ Q θ , ∇ θ log π θ > = 0 , which means ∇ θ Q θ ∝ ∇ θ log π θ .

Original: B G RADIENTS B ETWEEN P OLICY I MPROVEMENT AND P OLICY E VALUATION
Revised: B G RADIENTS B ETWEEN P OLICY I MPROVEMENT AND P OLICY E VALUATION

Original: Function Approximation
Revised: Function Approximation

Original: Train Gradients
Revised: Train Gradients

Original: Cosine of Interested Angles
Revised: Cosine of Interested Angles

Original: PPO( V [Equation] logit [Equation]
Revised: PPO

Original: θ ) 0 .

Original: [Equation]

Original: L

Original: [Equation]

Original: π

Original: [Equation]
Revised: [Equation]

Original: PPO ver.
Revised: PPO+CASA

Original: ( Q,logit )

Original: [Equation]

Original: logit

Original: [Equation]

Original: [Equation]

Original: L

Original: [Equation]

Original: cos < ∇ L Q ,

Original: [Equation]

Original: >

Original: π = softmax

Original: [Equation]
Revised: [Equation]

Original: ) cos < ∇ Q, ∇ log π > V =

Original: [Equation]

Original: π ) · Q

Original: PPO ver.

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]

Original: θ ,logit θ ) , 0 .

Original: 5 ∇ L V + ∇ L Q + ∇J cos < ∇ L Q , ∇J > pi = softmax ( logit ) cos < ∇ Q, ∇ log π > V = sg ( π ) · Q

Original: PPO+CASA ( V,A ) =

Original: ( V θ ,A θ ) , 0 .

Original: 5 ∇ L V + ∇ L Q + ∇J cos < ∇ L Q , ∇J > π = softmax ( A/τ ) , cos < ∇ Q, ∇ log π > ¯ A =

Original: A − sg ( π ) · A

Original: Q =

Original: ¯ A + sg ( V ) is

Original: [Table]
Revised: [Table]

Original: Angles
Revised: Angles

Original: R2D
Revised: R2D

Original: [Equation]

Original: V π

Original: [Equation]
Revised: [Equation]

Original: R2D2 ver.
Revised: R2D2 ver.

Original: [Equation]
Revised: [Equation]

Original: [Equation] Q cos [Equation]
Revised: R2D2+CASA ( V,A ) =

Original: >
Revised: ( V

Original: [Equation]
Revised: [Equation]

Original: V π = softmax

Original: ( A/τ )
Revised: A θ )

Original: R2D2+CASA (

Original: [Equation]
Revised: [Equation]

Original: To understand the behavior of
Revised: To understand the behavior of

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: in reinforcement learning algorithms, we choose PPO as a representative for policy-based methods and R2D as a representative for value-based algorithms.
Revised: in reinforcement learning algorithms, we choose PPO as a representative for policy-based methods and R2D as a representative for value-based algorithms.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We usually have above three kinds of loss functions in reinforcement learning, which aim to estimate the state values, state-action values and the policy.
Revised: We usually have above three kinds of loss functions in reinforcement learning, which aim to estimate the state values, state-action values and the policy.

Original: We do not talk about the estimations of V π and Q π as they are estimated as their usual way of PPO’s and R2D2’s.
Revised: We do not talk about the estimations of V π and Q π as they are estimated as their usual way of PPO’s and R2D2’s.

Original: All hyperparameters are listed in Appendix F.
Revised: All hyperparameters are listed in Appendix F.

Original: For brevity, we write
Revised: For brevity, we write

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The fact that PPO only has ∇ θ L V and ∇ θ J and R2D2 only has ∇ θ L Q is the main difficulty to track cos( β ) and χ .
Revised: The fact that PPO only has ∇ θ L V and ∇ θ J and R2D2 only has ∇ θ L Q is the main difficulty to track cos( β ) and χ .

Original: To solve the problem, we adjust PPO and R2D2 with different versions.
Revised: To solve the problem, we adjust PPO and R2D2 with different versions.

Original: For PPO, we displace the estimation of V θ by sg ( π ) · Q θ , where Q θ is estimated by function approximation and V θ is estimated by taking the expectation of Q θ .
Revised: For PPO, we displace the estimation of V θ by sg ( π ) · Q θ , where Q θ is estimated by function approximation and V θ is estimated by taking the expectation of Q θ .

Original: All versions of PPO are listed in Table 5.
Revised: All versions of PPO are listed in Table 5.

Original: For R2D2, we point out that though we apply ϵ -greedy to interact with environments, ϵ is only used for exploration and the final target policy of value-based methods is simply arg max Q θ .
Revised: For R2D2, we point out that though we apply ϵ -greedy to interact with environments, ϵ is only used for exploration and the final target policy of value-based methods is simply arg max Q θ .

Original: Because arg max Q θ breaks the gradient, we use a surrogate policy to approximate the gradient of policy improvement.
Revised: Because arg max Q θ breaks the gradient, we use a surrogate policy to approximate the gradient of policy improvement.

Original: Since
Revised: Since

Original: R2D2 uses dueling structure and softmax ( A θ /τ )
Revised: R2D2 uses dueling structure and softmax ( A θ /τ )

Original: =
Revised: =

Original: softmax ( Q θ /τ )
Revised: softmax ( Q θ /τ )

Original: τ → 0+ −→ arg max Q θ , we use π surrogate = softmax ( A θ /τ ) to calculate the policy gradient.
Revised: τ → 0+ −→ arg max Q θ , we use π surrogate = softmax ( A θ /τ ) to calculate the policy gradient.

Original: We only use π surrogate on learner to calculate the gradient, where the policy that interacts with environments is still ϵ -greedy.
Revised: We only use π surrogate on learner to calculate the gradient, where the policy that interacts with environments is still ϵ -greedy.

Original: All versions of R2D2 are listed in Table 6.
Revised: All versions of R2D2 are listed in Table 6.

Original: C O N D ISCUSSING A PPLICATION OF CASA ON C ONTINUOUS A CTION S PACE
Revised: C O N D ISCUSSING A PPLICATION OF CASA ON C ONTINUOUS A CTION S PACE

Original: As we can see CASA is only applied to discrete action space in the main context, we make a discussion on whether CASA is applicable on continuous action space.
Revised: As we can see CASA is only applied to discrete action space in the main context, we make a discussion on whether CASA is applicable on continuous action space.

Original: For brevity, we let τ = 1 and write equation 6 as:
Revised: For brevity, we let τ = 1 and write equation 6 as:

Original: [Equation]
Revised: [Equation]

Original: The difficulty comes from estimating two quantities, one is softmax ( A ) , the other is E π
Revised: The difficulty comes from estimating two quantities, one is softmax ( A ) , the other is E π

Original: [ A ] .
Revised: [ A ] .

Original: This comes from the fact that discrete action space is countable so these two quantities are expressed in a closed-form, while continuous action space is uncountable so an accurate estimation of these two quantities is intractable.
Revised: This comes from the fact that discrete action space is countable so these two quantities are expressed in a closed-form, while continuous action space is uncountable so an accurate estimation of these two quantities is intractable.

Original: We can surely apply Monte Carlo methods to approximate, but a more elegant close-form expression may be preferred.
Revised: We can surely apply Monte Carlo methods to approximate, but a more elegant close-form expression may be preferred.

Original: Then this becomes another problem: how to estimate (state-action values / advantages / policy probabilities) of all actions in a continuous action space efficiently without loss of generality?
Revised: Then this becomes another problem: how to estimate (state-action values / advantages / policy probabilities) of all actions in a continuous action space efficiently without loss of generality?

Original: This is another representational design problem, which is out of scope of this paper, so we don’t touch much about it.
Revised: This is another representational design problem, which is out of scope of this paper, so we don’t touch much about it.

Original: But with the hope of inspiring a better solution to this problem, we provide one practical way of applying CASA on continuous action space based on kernel-based machine learning.
Revised: But with the hope of inspiring a better solution to this problem, we provide one practical way of applying CASA on continuous action space based on kernel-based machine learning.

Original: Let a 0 , . .
Revised: Let a 0 , . .

Original: . , a k to be basis actions in the action space.
Revised: . , a k to be basis actions in the action space.

Original: Let A ( s, a 0 ) , . . .
Revised: Let A ( s, a 0 ) , . . .

Original: , A ( s, a k ) to be advantage functions for tuples of states and basis actions.
Revised: , A ( s, a k ) to be advantage functions for tuples of states and basis actions.

Original: They can either share parameters or be isolated.
Revised: They can either share parameters or be isolated.

Original: Let K ( · , · ) be a kernel function defined on the product of two action spaces.
Revised: Let K ( · , · ) be a kernel function defined on the product of two action spaces.

Original: For any a in the action space, we can estimate A ( s, a ) by a decomposition such like
Revised: For any a in the action space, we can estimate A ( s, a ) by a decomposition such like

Original: [Equation]
Revised: [Equation]

Original: where Z a = (cid:80) ki =

Revised: where Z a = (cid:80) ki =

Original: K ( a i , a ) is a normalization constant.
Revised: K ( a i , a ) is a normalization constant.

Original: Since K ( · , a ) is a closed-form function of a , and |{ A ( s, a 0 ) , . .
Revised: Since K ( · , a ) is a closed-form function of a , and |{ A ( s, a 0 ) , . .

Original: A ( s, a k ) }| is finite, we can make a closedform expression of both softmax ( A ) and E π
Revised: A ( s, a k ) }| is finite, we can make a closedform expression of both softmax ( A ) and E π

Original: [ A ] .
Revised: [ A ] .

Original: Then we can apply CASA directly on this expression, with one function estimates V and the other function estimates advantages of all actions in a closed-form with only state as input.
Revised: Then we can apply CASA directly on this expression, with one function estimates V and the other function estimates advantages of all actions in a closed-form with only state as input.

Original: The policy is defined directly by softmax of all advantages.
Revised: The policy is defined directly by softmax of all advantages.

Original: In details, we define
Revised: In details, we define

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then it satisfies the consistency of CASA on continuous action space.
Revised: Then it satisfies the consistency of CASA on continuous action space.

Original: [Equation]
Revised: [Equation]

Original: D DR-T RACE
Revised: D DR-T RACE

Original: As CASA estimates ( V, Q, π ) , we would ask i) how to guarantee that ˜ π V Trace =
Revised: As CASA estimates ( V, Q, π ) , we would ask i) how to guarantee that ˜ π V Trace =

Original: ˜
Revised: ˜

Original: π ReTrace , ii) how to exploit ( V, Q, π ) to make a better estimation.
Revised: π ReTrace , ii) how to exploit ( V, Q, π ) to make a better estimation.

Original: Though we can apply V-Trace to estimate V and ReTrace to estimate Q with proper hyperparameters to guarantee ˜ π V Trace =
Revised: Though we can apply V-Trace to estimate V and ReTrace to estimate Q with proper hyperparameters to guarantee ˜ π V Trace =

Original: ˜
Revised: ˜

Original: π ReTrace , it’s more reasonable to estimate ( V, Q ) together.
Revised: π ReTrace , it’s more reasonable to estimate ( V, Q ) together.

Original: Inspired by Doubly Robust, which is shown to maximally reduce the variance, we introduce
Revised: Inspired by Doubly Robust, which is shown to maximally reduce the variance, we introduce

Original: DR-Trace, which estimates V by
Revised: DR-Trace, which estimates V by

Original: [Equation]
Revised: [Equation]

Original: where µ is the behavior policy, δ DRt def = r t + γV ( s t +1 ) − Q ( s t , a t ) is one-step Doubly Robust error, ρ t def = min { π t µ t , ¯ ρ } and c t def
Revised: where µ is the behavior policy, δ DRt def = r t + γV ( s t +1 ) − Q ( s t , a t ) is one-step Doubly Robust error, ρ t def = min { π t µ t , ¯ ρ } and c t def

Original: = min { π t µ t , ¯ c } are clipped per-step importance sampling, c
Revised: = min { π t µ t , ¯ c } are clipped per-step importance sampling, c

Original: [ t : t + k ] def = (cid:81) ki
Revised: [ t : t + k ] def = (cid:81) ki

Original: =

Revised: =

Original: c t + i .
Revised: c t + i .

Original: With one step Bellman equation, we estimate Q by
Revised: With one step Bellman equation, we estimate Q by

Original: [Equation]
Revised: [Equation]

Original: where ˜ ρ t,k = 1 { k =0 } + 1 { k> 0 } ρ t + k .
Revised: where ˜ ρ t,k = 1 { k =0 } + 1 { k> 0 } ρ t + k .

Original: Theorem 4.
Revised: Theorem 4.

Original: Define ¯ A =
Revised: Define ¯ A =

Original: A − E π
Revised: A − E π

Original: [ A ] , Q =
Revised: [ A ] , Q =

Original: ¯ A + sg ( V ) ,
Revised: ¯ A + sg ( V ) ,

Original: [Equation]
Revised: [Equation]

Original: k − 1] ˜ ρ t,k δ DRt + k ] , 
Revised: + k ] , [Equation]

Original: S ( V ) def = E µ [ V ( s t ) + (cid:88)
Revised: def [Equation] µ [ [Equation]

Original: k ≥γ k c
Revised: k ≥

Original:  [ t : t + k − 1] ρ t,k δ DRt + k ] ,
Revised: [Equation] [ t : [Equation]

Original: U ( Q, V ) = ( T ( Q ) − E π
Revised: U ( n − [Equation] Q

Original: [ Q ] + S ( V ) , S ( V )) ,
Revised: [Equation] V )) [Equation]

Original: U ( n ) ( Q, V ) = U ( U ( n − 1) ( Q, V )) , then U ( n ) ( Q, V ) → ( Q ˜ π , V ˜ π ) that corresponds to
Revised: then U ( n ) ( Q, V ) → ( Q ˜ π , V ˜ π ) that corresponds to

Original: [Equation]
Revised: [Equation]

Original: as n → + ∞ .
Revised: as n → + ∞ .

Original: Proof.
Revised: Proof.

Original: See Appendix E, Theorem E.1.
Revised: See Appendix E, Theorem E.1.

Original: Theorem 4 shows that DR-Trace is a contraction mapping and ( V, Q ) converges to ( V ˜ π , Q ˜ π ) that corresponds to
Revised: Theorem 4 shows that DR-Trace is a contraction mapping and ( V, Q ) converges to ( V ˜ π , Q ˜ π ) that corresponds to

Original: [Equation]
Revised: [Equation]

Original: According to our proof, DR-Trace should work similar to V-Trace and ReTrace, as the convergence rate and the limitation are same.
Revised: According to our proof, DR-Trace should work similar to V-Trace and ReTrace, as the convergence rate and the limitation are same.

Original: We compare DR-Trace with V-Trace+ReTrace in Figure 5, where we replace estimation of state values by V-Trace and estimation of state-action values by ReTrace.
Revised: We compare DR-Trace with V-Trace+ReTrace in Figure 5, where we replace estimation of state values by V-Trace and estimation of state-action values by ReTrace.

Original: We call V-Trace+ReTrace as
Revised: We call V-Trace+ReTrace as

Original: No-DR-Trace for brevity.
Revised: No-DR-Trace for brevity.

Original: No-DR-Trace performs better on Breakout and ChopperCommand, but fails to make a breakthrough on Krull.
Revised: No-DR-Trace performs better on Breakout and ChopperCommand, but fails to make a breakthrough on Krull.

Original: Recalling the fact that Doubly Robust can maximally reduce the variance of Bellman error, No-DR-Trace is less stable but also potential to achieve a better performance.
Revised: Recalling the fact that Doubly Robust can maximally reduce the variance of Bellman error, No-DR-Trace is less stable but also potential to achieve a better performance.

Original: A conclusion cannot be made about No-DR-Trace, as this phenomenon means that No-DR-Trace is less stable than DR-Trace, but it also holds the potential to achieve a better performance.
Revised: A conclusion cannot be made about No-DR-Trace, as this phenomenon means that No-DR-Trace is less stable than DR-Trace, but it also holds the potential to achieve a better performance.

Original: [Figure]
Revised: [Figure]

Original: E P ROOFS
Revised: E P ROOFS

Original: Lemma E.1.
Revised: Lemma E.1.

Original: (i)
Revised: (i)

Original: Define π = softmax
Revised: Define π = softmax

Original: ( A/τ ) , then ∇ log π = ( 1 − π ) ∇ Aτ .
Revised: ( A/τ ) , then ∇ log π = ( 1 − π ) ∇ Aτ .

Original: (ii)
Revised: (ii)

Original: Denote sg to be stop gradient and define ¯ A
Revised: Denote sg to be stop gradient and define ¯ A

Original: =
Revised: =

Original: A − E π
Revised: A − E π

Original: [ A ] , Q =
Revised: [ A ] , Q =

Original: ¯ A + sg ( V ) , then ∇ Q = ( 1 − π ) ∇ A .
Revised: ¯ A + sg ( V ) , then ∇ Q = ( 1 − π ) ∇ A .

Original: Proof.
Revised: Proof.

Original: As Q =
Revised: As Q =

Original: ¯ A + sg ( V ) =
Revised: ¯ A + sg ( V ) =

Original: A − sg ( π ) ·
Revised: A − sg ( π ) ·

Original: A + sg ( V ) , it’s obvious that ∇ Q = ( 1 − π ) ∇ A .
Revised: A + sg ( V ) , it’s obvious that ∇ Q = ( 1 − π ) ∇ A .

Original: For log π , it’s a standard derivative of cross entropy, so we have ∇ log π
Revised: For log π , it’s a standard derivative of cross entropy, so we have ∇ log π

Original: =
Revised: =

Original: ( 1 − π ) ∇
Revised: ( 1 − π ) ∇

Original: ( A/τ ) =
Revised: ( A/τ ) =

Original: ( 1 − π ) ∇ Aτ .
Revised: ( 1 − π ) ∇ Aτ .

Original: Lemma E.2.
Revised: Lemma E.2.

Original: Define ¯ A =
Revised: Define ¯ A =

Original: A − E π
Revised: A − E π

Original: [ A ] , Q =
Revised: [ A ] , Q =

Original: ¯ A + sg ( V ) , π = softmax ( A/τ ) , then
Revised: ¯ A + sg ( V ) , π = softmax ( A/τ ) , then

Original: E 
Revised: E π [

Original: [Equation]
Revised: [Equation]

Revised: Q −

Original: [Equation]
Revised: [Equation]

Original:  log
Revised: ∇ log

Original: [Equation]
Revised: [Equation]

Original: = − τ ∇ H [ π ] .
Revised: [Equation] − τ ∇ H [ π ] .

Original: Proof.
Revised: Proof.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Based on the observation that E π [ f ( s ) ∇ log π ( ·| s )] = 0 , we haveE
Revised: Based on the observation that E π [ f ( s ) ∇ log π ( ·| s )] = 0 , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: On the one hand,
Revised: On the one hand,

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original:  Hence, E π [( Q − V ) ∇ log π ] = − τ ∇ H
Revised: − [Equation] Hence, E π [( Q − V ) ∇ log π ] = − τ ∇ H

Original: [ π ] .
Revised: [ π ] .

Original: Theorem E.1.
Revised: Theorem E.1.

Original: Define ¯ A =
Revised: Define ¯ A =

Original: A − E π
Revised: A − E π

Original: [ A ] , Q =
Revised: [ A ] , Q =

Original: ¯ A + sg ( V ) .
Revised: ¯ A + sg ( V ) .

Original: Define
Revised: Define

Original: T [Equation] Q ) def = [Equation]
Revised: T ( Q ) def = E µ [ Q ( s t , a t ) + (cid:88)

Original: µ [
Revised: k ≥γ k c

Original: [Equation]
Revised: [Equation]

Original: s t , a t) +(cid:88) k ≥γ [Equation]
Revised: +1: t + k − 1] ˜ ρ t,k δ DRt + k ] ,

Original: − [Equation] [Equation]
Revised: S ( V ) def = E µ [ V ( s t ) + (cid:88)

Original: ρ
Revised: k ≥

Original: [Equation]
Revised: [Equation]

Original: ) def =
Revised: + k −

Original: [Equation]
Revised: [Equation]

Original: µ [
Revised: ] ρ

Original: [Equation]
Revised: [Equation]

Original: ( s t ) +(cid:88) k ≥ [Equation] −
Revised: δ DRt + k ]

Original: [Equation]
Revised: [Equation]

Original: U ( Q, V ) =
Revised: U ( Q, V ) =

Original: ( T ( Q ) − E π
Revised: ( T ( Q ) − E π

Original: [ Q ] + S ( V ) , S ( V )) ,
Revised: [ Q ] + S ( V ) , S ( V )) ,

Original: U ( n ) ( Q, V ) =
Revised: U ( n ) ( Q, V ) =

Original: U ( U ( n − 1) ( Q, V )) , then U ( n ) ( Q, V ) → ( Q ˜ π , V ˜ π ) that corresponds to ˜ π ( a | s ) =
Revised: U ( U ( n − 1) ( Q, V )) , then U ( n ) ( Q, V ) → ( Q ˜ π , V ˜ π ) that corresponds to ˜ π ( a | s ) =

Original: min { ¯ ρµ ( a | s ) , π ( a | s ) } (cid:80) b ∈A min { ¯ ρµ ( b | s ) , π ( b | s ) } .
Revised: min { ¯ ρµ ( a | s ) , π ( a | s ) } (cid:80) b ∈A min { ¯ ρµ ( b | s ) , π ( b | s ) } .

Original: as n → + ∞ .
Revised: as n → + ∞ .

Original: Remark.
Revised: Remark.

Original: T ( Q ) − E π [ Q ]+ S ( V ) is exactly how Q is updated at training time.
Revised: T ( Q ) − E π [ Q ]+ S ( V ) is exactly how Q is updated at training time.

Original: Since Q = ¯ A + sg ( V ) , if we apply gradient ascent on Q and V in directions ∇ L Q ( θ ) and ∇ L V ( θ ) respectively, change of Q comes from two aspects.
Revised: Since Q = ¯ A + sg ( V ) , if we apply gradient ascent on Q and V in directions ∇ L Q ( θ ) and ∇ L V ( θ ) respectively, change of Q comes from two aspects.

Original: One comes from ∇ L Q ( θ ) , which changes A , the other comes from ∇ L V ( θ ) , which changes V .
Revised: One comes from ∇ L Q ( θ ) , which changes A , the other comes from ∇ L V ( θ ) , which changes V .

Original: Because the gradient of V is stopped when estimating Q , the latter is captured by "minus old baseline, add new baseline", which is − E π [ Q ] + S ( V ) in Theorem E.1.
Revised: Because the gradient of V is stopped when estimating Q , the latter is captured by "minus old baseline, add new baseline", which is − E π [ Q ] + S ( V ) in Theorem E.1.

Original: Proof.
Revised: Proof.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: By Lemma E.3, (cid:102) T ( n ) ( Q ) converges to some A ∗ as n → ∞ .
Revised: By Lemma E.3, (cid:102) T ( n ) ( Q ) converges to some A ∗ as n → ∞ .

Original: This process will not influence the estimation of
Revised: This process will not influence the estimation of

Original: V as the gradient of V is stopped when estimating Q .
Revised: V as the gradient of V is stopped when estimating Q .

Original: According to the proof, A ∗ does not depend on V .
Revised: According to the proof, A ∗ does not depend on V .

Original: By Lemma E.4, S ( n ) ( V ) converges to some V ∗ as n → ∞ .
Revised: By Lemma E.4, S ( n ) ( V ) converges to some V ∗ as n → ∞ .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: we can regard (cid:102) T ( Q ) + S ( V ) as Q and regard S ( V ) as V , then
Revised: we can regard (cid:102) T ( Q ) + S ( V ) as Q and regard S ( V ) as V , then

Original: [Equation]
Revised: [Equation]

Original: By induction,
Revised: By induction,

Original: [Equation]
Revised: [Equation]

Original: Same as (Espeholt et al., 2018),
Revised: Same as (Espeholt et al., 2018),

Original: [Equation]
Revised: [Equation]

Original: is the policy s.t.
Revised: is the policy s.t.

Original: the Bellman equation holds, which is
Revised: the Bellman equation holds, which is

Original: [Equation]
Revised: [Equation]

Original: and U ( Q ˜ π , V ˜ π ) =
Revised: and U ( Q ˜ π , V ˜ π ) =

Original: ( Q ˜ π , V ˜ π ) .
Revised: ( Q ˜ π , V ˜ π ) .

Original: So we have ( A ∗ + V ∗ , V ∗ ) =
Revised: So we have ( A ∗ + V ∗ , V ∗ ) =

Original: ( Q ˜ π , V ˜ π ) .
Revised: ( Q ˜ π , V ˜ π ) .

Original: Lemma E.3.
Revised: Lemma E.3.

Original: Define ¯ A =
Revised: Define ¯ A =

Original: A − E π
Revised: A − E π

Original: [ A ] , Q =
Revised: [ A ] , Q =

Original: ¯ A + sg ( V ) , then operator
Revised: ¯ A + sg ( V ) , then operator

Original: [Equation]

Original: def

Original: [Equation]

Original: µ [ Q

Original: [Equation]

Original: [Equation]

Original: +

Original: [Equation]

Original: DRt

Original: [Equation]
Revised: [Equation]

Original: ] is a contraction mapping w.r.t.
Revised: is a contraction mapping w.r.t.

Original: Q .
Revised: Q .

Original: Remark.
Revised: Remark.

Original: Note that T ( Q ) is exactly equation D.
Revised: Note that T ( Q ) is exactly equation D.

Original: Since Q =
Revised: Since Q =

Original: A + sg ( V ) , the gradient of V is stopped when estimating Q , updating Q will not change V , which is equivalent to updating A .
Revised: A + sg ( V ) , the gradient of V is stopped when estimating Q , updating Q will not change V , which is equivalent to updating A .

Original: Without loss of generality, we assume V is fixed as V ∗ in the proof.
Revised: Without loss of generality, we assume V is fixed as V ∗ in the proof.

Original: Proof.
Revised: Proof.

Original: ¯
Revised: ¯

Original: A =
Revised: A =

Original: A − E π
Revised: A − E π

Original: [ A ] shows E π
Revised: [ A ] shows E π

Original: [ ¯ A ] = 0 , which guarantees that no matter how we update A , we always have E π [ Q ] = V ∗ .
Revised: [ ¯ A ] = 0 , which guarantees that no matter how we update A , we always have E π [ Q ] = V ∗ .

Original: Based on above observations, define
Revised: Based on above observations, define

Original: [Equation]
Revised: [Equation]

Original: It’s obvious that we only need to prove (cid:102) T ( Q ) is a contraction mapping.
Revised: It’s obvious that we only need to prove (cid:102) T ( Q ) is a contraction mapping.

Original: For brevity, we denote
Revised: For brevity, we denote

Original: [Equation]
Revised: [Equation]

Original: Noticing that ˜ ρ t, 0 = 1 , let F represent filtration, we can rewrite (cid:102) T as
Revised: Noticing that ˜ ρ t, 0 = 1 , let F represent filtration, we can rewrite (cid:102) T as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By definition of Q
Revised: By definition of Q

Original: [Equation]
Revised: [Equation]

Original: we can rewrite equation 17 as
Revised: we can rewrite equation 17 as

Original: [Equation]
Revised: [Equation]

Original: For any Q 1 =
Revised: For any Q 1 =

Original: A 1 + sg ( V ∗ ) , Q 2 =
Revised: A 1 + sg ( V ∗ ) , Q 2 =

Original: A 2 + sg ( V ∗ ) , since
Revised: A 2 + sg ( V ∗ ) , since

Original: [Equation]
Revised: [Equation]

Original: by equation 16 equation
Revised: by equation 16 equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Hence, (cid:102) T ( Q ) is a contraction mapping and converges to some fixed function, which we denote as A ∗ .
Revised: Hence, (cid:102) T ( Q ) is a contraction mapping and converges to some fixed function, which we denote as A ∗ .

Original: So T ( Q ) is also a contraction mapping and converges to A ∗ + V ∗ .
Revised: So T ( Q ) is also a contraction mapping and converges to A ∗ + V ∗ .

Original: Lemma E.4.
Revised: Lemma E.4.

Original: Define Q =
Revised: Define Q =

Original: A + sg ( V ) with E π
Revised: A + sg ( V ) with E π

Original: [ A ] = 0 , then operator
Revised: [ A ] = 0 , then operator

Original: [Equation]
Revised: [Equation]

Original: is a contraction mapping w.r.t.
Revised: is a contraction mapping w.r.t.

Original: V .
Revised: V .

Original: Remark.
Revised: Remark.

Original: Note that S ( V ) is exactly equation D.
Revised: Note that S ( V ) is exactly equation D.

Original: Proof.
Revised: Proof.

Original: Same as Lemma E.3, we can get
Revised: Same as Lemma E.3, we can get

Original: [Equation]
Revised: [Equation]

Original: so we have
Revised: so we have

Original: [Equation]
Revised: [Equation]

Original: The remaining proof is identical to (Espeholt et al., 2018)’s.
Revised: The remaining proof is identical to (Espeholt et al., 2018)’s.

Original: F H YPERPARAMETERS
Revised: F H YPERPARAMETERS

Original: Our python packages are shown in Table 7.
Revised: Our python packages are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: All experiments follow the shared hyperparameters as in Table 8.
Revised: All experiments follow the shared hyperparameters as in Table 8.

Original: The specific hyperparameters for PPO,
Revised: The specific hyperparameters for PPO,

Original: R2D2 and CASA+DR-Trace are shown in Table 9, Table 10 and Table 11.
Revised: R2D2 and CASA+DR-Trace are shown in Table 9, Table 10 and Table 11.

Original: The only exceptions are V -loss scaling, Q -loss scaling and π -loss scaling, which may be zero depending on some specific ablation settings.
Revised: The only exceptions are V -loss scaling, Q -loss scaling and π -loss scaling, which may be zero depending on some specific ablation settings.

Original: We will state these three hyperparameters every time in all experiments.
Revised: We will state these three hyperparameters every time in all experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G E VALUATION OF CASA ON A TARI G AMES
Revised: G E VALUATION OF CASA ON A TARI G AMES

Original: Random scores and average human’s scores are from (Badia et al., 2020).
Revised: Random scores and average human’s scores are from (Badia et al., 2020).

Original: Human World Records (HWR) are from (Toromanoff et al., 2019).
Revised: Human World Records (HWR) are from (Toromanoff et al., 2019).

Original: Rainbow’s scores are from (Hessel et al., 2017).
Revised: Rainbow’s scores are from (Hessel et al., 2017).

Original: IMPALA’s scores are from (Espeholt et al., 2018).
Revised: IMPALA’s scores are from (Espeholt et al., 2018).

Original: LASER’s scores are from (Schmitt et al., 2020), no sweep at 200M.
Revised: LASER’s scores are from (Schmitt et al., 2020), no sweep at 200M.

Original: [Table]
Revised: [Table]

Original: 3208.64 robotank 2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 95 seaquest 68.
Revised: 95 seaquest 68.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 28.09 skiing - -
Revised: 09 skiing - -

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 23 solaris1236.
Revised: 23 solaris

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 76 space invaders
Revised: 76 space invaders

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 58 star 
Revised: 58 star gunner

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3510.18 surround -
Revised: .18 surround -

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 00 tennis
Revised: 00 tennis

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: pilot

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 62 tutankham
Revised: 4862.62 tutankham

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] 62 up n down
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3723.06 venture 1187.
Revised: [Table] [Table] 236.62 up n down

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: montezumarevenge
Revised: 06 venture

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure]

Revised: spaceinvaders

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]{+-5e+3 0 5e+ 1e+4 1.5e+4 2e+4 2.5e+ 3e+4 3.5e+ -20k 0 20k 40k 60k 80k100k120k140k160k
 Alien - -20k 0 20k 40k 60k 80k 100k120k140k16 (2.)+}
Revised: {+-5e+3 0 5e+ 1e+4 1.5e+4 2e+4 2.5e+ 3e+4 3.5e+ -20k 0 20k 40k 60k 80k100k120k140k160k
 Alien - -20k 0 20k 40k 60k 80k 100k120k140k16 (2.)+}

Original: [Figure] [Table] [Figure]
Revised: Amidar -2e+302e+34e+36e+ 8e+31e+41.2e+41.4e+41.6e+41.8e+ 2e+ -20k 0 20k 40k 60k 80k100k120k140k16 (3.)

Original: [Table] [Figure] [Table]
Revised: Assault -1e+501e+52e+ 3e+54e+55e+56e+ 7e+58e+ -20k 0 20k 40k 60k 80k 100k120k140k16 (4.)

Original: [Figure] [Table] [Figure]
Revised: Asterix -1e+401e+42e+43e+44e+ 5e+46e+47e+48e+49e+41e+ 1.1e+51.2e+ -20k 0 20k 40k 60k 80k100k120k140k160k18 (5.)

Original: [Table] [Figure]
Revised: Asteroids -5e+505e+51e+ 1.5e+62e+62.5e+63e+ 3.5e+64e+ -10k0 10k20k30k40k50k60k70k80k90k100k11 (6.) Atlantis -20010001002003004005006007008009001e

Original: [Table] [Figure] [Table] [Figure]
Revised: +31.1e+31.2e+ -20k 0 20k 40k 60k 80k100k120k140k16 (7.)

Original: [Table]
Revised: BankHeist -2e+4 0 2e+

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]