Rethinking Deep Policy Gradients via State-Wise Policy Improvement
Abstract
Deep policy gradient is one of the major frameworks in reinforcement learning, and it has been shown to improve parameterized policies across various tasks and environments.
However, recent studies show that the key components of the deep policy gradient methods, such as gradient estimation, value prediction, and optimization landscapes, fail to reﬂect the conceptual framework.
This paper aims to investigate the mechanism behind the deep policy gradient methods through the lens of state-wise policy improvement.
Based on the fundamental properties of policy improvement, we propose an alternative theoretical framework to reinterpret the deep policy gradient update as training a binary classiﬁer, with labels provided by the advantage function.
This framework obviates the statistical difﬁculties in the gradient estimates and predicted values of the deep policy gradient update.
Experimental results are included to corroborate the proposed framework.
Introduction
Reinforcement learning (RL) is a branch of machine learning that achieves optimal sequential decision making by interacting with the environment and learning from the underlying random process.
RL is often considered to be an optimization problem, and the major approaches that follow this principle are the policy gradient methods [8, 22, 15, 23].
The foundation of the policy gradient methods is built upon the theory of REINFORCE [30] or its equivalent representation called the policy gradient theorem [26].
The existing RL algorithms built upon the policy gradient theorem were often unable to evaluate the policy gradient by its original formula, and therefore an alternative statistical representation was introduced by leveraging the idea of discounted state visitation distribution.
This statistical representation of the policy gradient is often considered as a variant of stochastic optimization, which is typically appraised by the update noise in the measurements regarding the objective.
In the context of policy gradient, this noise corresponds to the joint effect of the error in the predicted value function and the variance of the gradient estimate.
Despite the great success of the policy gradient methods, their theoretical foundation is currently being challenged.
For example, Thomas [27] pointed out that there is a missing term in the standard update rule of the value function approximator, and this term is required to reﬂect the discounted state visitation distribution in the statistical representation of the policy gradient.
The fact that the default practice is missing this term leads to additional bias in the gradient estimates and could affect the convergence guarantees.
Subsequently, Nota and Thomas [17] showed that many of the existing estimates of the policy gradient do not follow the gradient direction of any function due to this missing term, and therefore these methods are not guaranteed to converge to a reasonable stationary point.
On the other hand, Ilyas et al.
[7] empirically studied the value prediction error and the quality of the gradient estimates for the benchmark algorithms TRPO [22] and its simpliﬁed variant PPO with a clipped objective [23], and the results showed that the value prediction accuracy was poor and the gradient estimates suffered from rather high variance.
Surprisingly, both algorithms TRPO and
PPO can still improve the policy during the training process by operating under such restrictions.
This mismatch suggests a deeper look into the theoretical justiﬁcation of the deep policy gradients.
This paper is meant to take the ﬁrst step toward understanding the reason behind this mismatch between theory and practice in the deep policy gradient methods.
To address this phenomenon, we propose an alternative theoretical framework to reinterpret the deep policy gradient methods through the lens of state-wise policy improvement .
Speciﬁcally, the main contributions of this paper are:
• We identify two fundamental results of state-wise policy improvement, which provide useful sufﬁcient conditions for ensuring policy improvement.
Based on these conditions, we reinterpret the deep policy gradient update as training a binary classiﬁer via empirical risk minimization.
Through the lens of state-wise policy improvement, the sample distributions in the policy gradient are thereby not required in improving a policy, and hence the inherent variance problem vanishes.
• We proceed to propose a generic classiﬁcation-based policy update scheme, in which we use the sign of the advantage as the label and view the change in the action probability as the classiﬁer.
As an example, we present a practical training algorithm by leveraging a large margin classiﬁer and the hinge loss.
Under this design, the classiﬁer attempts to meet the minimum requirement of the sufﬁcient conditions for state-wise policy improvement by skipping those data samples that satisfy the sufﬁcient conditions by a large margin.
Through a proper rearrangement of the objective, the resulting scheme is equivalent to a variant of the PPO algorithm with a clipped objective.
• Finally, we empirically validate the proposed theoretical framework through an experiment of the classic optimal time control problem.
Through a comparison with the baseline method, we demonstrate that the proposed classiﬁcation-based scheme with data skipping is indeed effective in achieving steady policy improvement.
Preliminaries
In this section, we describe the problem setup and formally introduce the notations used in this paper.
For ease of notation, we use ∆ Z to denote the set of all probability distributions over a ﬁnite support set Z .
All vectors are column vectors by default, unless stated otherwise.
We consider a discounted Markov Decision Process (MDP) deﬁned by a tuple ( S , A , P, r, γ ) [20], where S is the ﬁnite state space, A is the ﬁnite action space, P : S ×
A × S → [0 , 1] is the Markov transition kernel, r : S ×
A → R is the reward function, and γ ∈ (0 , 1) is the discount factor.
Given an MDP described above, a policy π ∈ ∆ |S|A speciﬁes the action distributions at all states s ∈ S .
With a slight abuse of notation, we also use π to denote the | S ||A| -dimensional vectorized version of the policy π .
The goal of an RL algorithm is to ﬁnd a policy that maximizes the expected accumulated discounted rewards.
For any policy π , the state value function V π and the action value function Q π , in terms of expected discounted accumulated rewards, are deﬁned by [Equation] Q [Equation]
(cid:34)
[Equation]
t
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
Under this formulation, the standard deﬁnition of state-wise policy improvement is formalized by the following partial ordering relation.
Deﬁnition 2.1 (Partial ordering over policies) .
Let π and µ be two policies.
We say that π improves µ (denoted by π ≥ µ ) if and only if V π ( s ) ≥ V µ ( s ) , ∀ s ∈ S .
A well-known result is that there exists a policy π ∗ that maximizes V π ( s ) over all policies, for all state s ∈ S .
In order to apply the numerical optimization techniques, such as gradient ascent, a natural alternative performance metric is introduced by taking a weighted average of the values at all states to construct a total ordering over policies, i.e.
[Equation]
where ν ∈ ∆ S is some static distribution over S .
We also deﬁne the corresponding discounted state visitation distribution under a policy π as
[Equation]
where 1 − γ serves as a normalization factor.
The theoretical foundation of the policy gradient methods, namely, the policy gradient theorem [26], is formulated as follows (in terms of statistical representation with advantage function):
[Equation]
where b ( s ) is called a baseline function and π ( a | s ) in the denominator comes from the importance weight.
One typical choice of the baseline function is V π ( s ) .
In this case, the baseline function is called the advantage function deﬁned as A π ( s, a ) :
= Q π ( s, a ) − V π ( s ) , for a policy π .
To apply the policy gradient deﬁned in (5), the existing deep policy gradient algorithms estimate ∇ θ η ( π ) by following two major design principles as the default practice:
• The value functions, also referred to as the critic in an actor-critic architecture, can be estimated accurately by TD learning [24, 21] or its variants, such as n -step TD, TD( λ ), GAE, and Retrace( λ ), all of which is built on the weighted average approach [16, 29, 2].
• The policy, also referred to as the actor in an actor-critic architecture, can be trained by estimating the policy gradient in (5) via a sample average.
Some variants of the sample average are: (i) [19, 1] incorporate probability distribution differences using importance sampling; (ii) [8, 22] leverage geometrical direction correction, such as Fisher information matrix and the second order approximation of Kullback-Leibler (KL) divergence; (iii) [23] introduces external penalties such as entropy or KL divergence.
The default practice presumes that the performance of an RL algorithm is mainly determined by the underlying accuracy of the estimated gradient and the estimated value functions.
However, as mentioned in Section 1, these presumptions do not seem to hold.
This motivates us to rethink the default practice and develop an alternative theoretical framework to better understand the underlying rationale of the deep policy gradient methods.
An Alternative Theoretical Framework for Deep Policy Gradients
In this section, we present a theoretical framework based on state-wise policy improvement to reinterpret the deep policy gradient methods.
We start by summarizing the fundamental results of state-wise policy improvement.
Based on these results, we describe the connection between statewise policy improvement and the policy gradient methods and then present the proposed training algorithms.
We close this section by presenting an experiment to justify our theory.
The proofs of all the theoretical results are provided in Appendix A.
3.1 Fundamental Theorems of Policy Improvement
To begin with, we state two fundamental results that provide useful sufﬁcient conditions for state-wise policy improvement.
Theorem 3.1.
Given any arbitrary policy µ , a policy π improves µ if π satisﬁes either one of the following conditions:
[Equation]
It is easy to verify that both (6) and (7) automatically hold if π and µ are identical.
Note that (6) can be viewed as the state-wise version of the classic performance difference lemma in [9, 22], which characterizes the difference in value function under some state distribution ν between any two policies π and µ as
[Equation]
Speciﬁcally, for any state s , V π ( s ) − V µ ( s ) can be recovered from (8) by setting ν to be a Dirac measure with the atom located in state s .
By interchanging the roles of π and µ in (8), the condition in (7) can be obtained by a similar argument (with its proof provided in Appendix A for completeness).
Note that (8) provides a sufﬁcient condition for policy improvement in terms of the weighted average value η ν ( · ) .
By contrast, (6)-(7) serve as sufﬁcient conditions for state-wise policy improvement in the sense of Deﬁnition 2.1.
In the sequel, we will show that this seemingly subtle change of perspective obviates the inherent statistical difﬁculty in the deep policy gradient update.
Remark 1.
Various popular policy update schemes can be justiﬁed by Theorem 3.1.
For example: (i) Given any policy µ , the classic one-step greedy policy improvement constructs an improved deterministic policy π by choosing ¯ µ ( s ) =
arg max a ∈A Q µ ( s, a ) , for each state s .
It is easy to verify that this design satisﬁes (6); (ii) Moreover, by applying Theorem 3.1, one can show that the convex mixture of µ and ¯ µ considered in [9, 28], i.e., α ¯ µ + (1 − α ) µ , is also an improved policy over µ , for any α ∈ [0 , 1] ; (iii) By a similar argument, the one-step (cid:15) -greedy policy improvement also follows directly from Theorem 3.1 [25].
(iv) In the regularized policy optimization using a quadratic penalty function [5], given a policy µ , an updated policy π reg is constructed by solving π reg = arg max π (cid:80) a Q µ ( s, a ) π ( a | s ) − C (cid:107) µ − π (cid:107) 2 , where C is a positive constant.
Again, by verifying the conditions in Theorem 3.1, one can show that π reg is an improved policy over µ .
Based on the above discussion, we thereby view Theorem 3.1 as one fundamental theorem of policy improvement.
By combining Theorem 3.1 and the fact that (cid:80) a ∈A π ( a | s )
A π ( s, a ) =
for any policy π and for any state s , we present alternative sufﬁcient conditions that are checked on a per state-action-pair basis.
Theorem 3.2.
Given an arbitrary policy µ , a policy π improves µ if it satisﬁes either one of the following conditions: (i) ( π ( a | s ) − µ ( a | s ))
A π ( s, a ) ≥ 0 , ∀ s ∈ S , ∀ a ∈ A ; (ii) ( π ( a | s ) − µ ( a | s ))
A µ ( s, a ) ≥ 0 , ∀ s ∈ S , ∀ a ∈ A .
Theorem 3.2 indicates that policy improvement can be achieved by determining the sign of the advantage of each state-action pair (regardless of its magnitude) and adjusting the action probabilities accordingly.
In this way, no additional constraints, such as the KL divergence constraint used in
TRPO [22], are needed to ensure policy improvement.
This also naturally motivates the design of using the signs of the advantage function as labels in determining the direction of the policy update.
More speciﬁcally, we can draw an analogy between the conditions in Theorem 3.2 and the training of a linear classiﬁer: (i) The state-action pair serves as the feature vector of a training sample; (ii)
The sign of A π ( s, a ) or A µ ( s, a ) plays the role of a binary label; (iii) π ( a | s ) − µ ( a | s ) resembles the prediction of a linear classiﬁer.
In Section 3.3, we discuss the practical issues in achieving policy improvement via classiﬁcation and present an exemplary training algorithm.
3.2 Connecting Deep Policy Gradients With State-Wise Policy Improvement
Before stating the proposed training algorithms, we ﬁrst highlight the statistical issue inherent in the deep policy gradient methods and then connect Theorem 3.2 with the deep policy gradients.
Difﬁculty in estimating deep policy gradients.
Here we consider parameterized policies π θ , where θ denotes the parameters of the underlying parametric model (e.g. a neural network).
Recall from (5) that the policy gradient of a parameterized policy π θ can be explicitly characterized by
[Equation]
where 1 /π θ ( a | s ) is the importance weight and B denotes a mini-batch of samples drawn under π θ .
While most of the deep policy gradient methods are built on the mini-batch estimate of the policy gradient, the gradient ∇ θ η ν ( π θ ) is fundamentally difﬁcult to estimate via sample average, as shown by the gradient estimate quality in [7].
Furthermore, [7] also found that the deep policy gradient methods can work under a small number of training samples (roughly 2 × 10 3 ), in which most of state-action pairs in the training set are sampled only at most once.
Therefore, the importance weight of each state-action pair can hardly take effect in correcting the distribution mismatch, which is inherent in the default practice of the deep policy gradient methods.
Remark 2.
The theory of TRPO considers a gradient update scheme that is slightly different from (10) by leveraging a surrogate objective adapted from (8).
To be more speciﬁc, TRPO introduces a surrogate objective by approximating ρ πν ( s ) by ρ µν ( s ) in (8), under the condition that π is sufﬁciently close to µ .
However, the gradient of the surrogate objective is still fundamentally difﬁcult to estimate via sample average and suffers from the same variance issue as (10), as shown by the surrogate objective landscape in [7].
Interpreting deep policy gradients via state-wise policy improvement.
Given the above gap between theory and the default practice, we propose to reinterpret the deep policy gradient methods through the lens of state-wise policy improvement.
We start by leveraging the fact that (cid:80) a [Equation]
µ [Equation] s, a ) = 0 and rewriting (8) as [Equation]
[Equation]
Note that (11) holds without any approximation.
Moreover, a policy π that satisﬁes the sufﬁcient conditions in Theorem 3.2 for state-wise policy improvement shall also improve µ in terms of the weighted-average value η ν ( · ) , as shown in (11).
More importantly, we can reinterpret the sample average in (10) with the help of Theorem 3.2 as follows:
This sample average can be viewed as a weighted sum of ∇ θ π θ ( a | s )
A π θ
( s, a ) across the data samples in the mini-batch.
For each sampled state-action pair ( s, a ) , ∇ θ π θ ( a | s )
A π θ
( s, a ) serves as an improvement direction for this speciﬁc pair ( s, a ) , and this improvement direction depends on the sign of A π θ
( s, a ) but not on the magnitude of the advantage.
Therefore, the deep policy gradient approach using sample average can be viewed as the construct of state-wise policy improvement.
We thereby draw an analogy between the deep policy gradient in (10) and the training of classic empirical risk minimization problems for classiﬁcation: (i)
The state-action pair ( s, a ) plays the role of a feature vector; (ii) Each ∇ θ π θ ( a | s )
A π θ
( s, a ) is analogous to the gradient of the loss incurred by a training sample; (iii)
A π θ
( s, a ) plays the role of the label of a training sample.
Hence, similar to the empirical risk minimization, what matters the most in the deep policy gradient update is the training data.
This interpretation suggests that training a classiﬁer based on Theorem 3.2 is a viable way to improve a policy in terms of η ν ( · ) .
3.3 Deep Policy Gradient as a Binary Classiﬁcation Problem
In this section, we expand on the insight provided by Theorem 3.2, which suggests that policy improvement can be achieved by training a classiﬁer based on the advantage.
We use ˆ A µ ( s, a ) to denote the estimated advantage of a policy µ for the state-action pair ( s, a ) .
A prototypical linear classiﬁcation method for policy update.
One prototypical method is to use the sign of ˆ A µ ( s, a ) as the binary label, π ( a | s ) − µ ( a | s ) as the prediction of the linear classiﬁer, and − ( π ( a | s ) − µ ( a | s ))
ˆ A µ ( s, a ) as the loss incurred by each training sample.
At each training step, given a batch of samples D , the policy is updated by minimizing the loss (cid:80) ( s,a ) ∈D ( µ ( a | s ) − π ( a | s ))
ˆ A µ ( s, a ) .
Compared to the typical sample-average-based deep policy gradient update in (10), one major beneﬁt of this classiﬁcation-based update scheme is that the error of this scheme only comes from the estimation error of the advantage function, and does not involve the sample distribution of the state-action pair.
Hence, the classiﬁcation-based approach completely obviates the statistical difﬁculty resulting from the sample average in the default practice of deep policy gradients.
Compared to the conventional classiﬁcation problem, one salient feature of this classiﬁcation-based policy update is that it needs to produce valid action distributions, i.e. an increase in the probability of a certain action implies the simultaneous decrease in the probabilities of the others.
If there is a training sample with a particularly large ( π ( a | s ) − µ ( a | s ))
ˆ A µ ( s, a ) , then it is relatively easy to construct a counterexample with degraded performance.
Moreover, this issue can be quite severe as the limited amount of collected data from the environment in each training step will most likely be unrepresentative of the problem space (e.g., only a single action per visited state).
For the above reasons, a more careful design of the training algorithm is needed.
An exemplary training algorithm with data skipping.
To address the above issue, we propose to consider a different loss function for the linear classiﬁer.
Inspired by [4], we formulate the policy improvement objective based on the large margin classiﬁcation and the hinge loss, i.e.,
[Equation]
where (cid:15) > 0 denotes the margin.
By deﬁnition, the hinge loss does not take into consideration the samples that satisfy
[Equation]
In other words, the minimizer of the objective in (12) attempts to meet the minimum requirement of
Theorem 3.2 for the training set D by skipping those data samples that satisfy the sufﬁcient condition in Theorem 3.2 by a large margin.
Connecting PPO and the proposed training algorithm.
It is worth noting that if the classiﬁer is set to be πµ − 1 and the margin set to be (cid:15) (cid:55)→ (cid:15) | ˆ A ( s i , a i ) | , the resulting update scheme is equivalent to the PPO-clip objective and was proved by Pi et al.
[18], which is independent of the sampling distribution and the choice of regularizer.
This property also provides a theoretical basis that can explain the algorithms proposed by Han and Sung [6], Luo et al.
[14], where they combined IMPALA [2] with PPO-clip.
Theoretically, the sufﬁcient conditions in Theorem 3.2 can be applied on top of an arbitrary smooth monotone increasing function F to construct an objective function as
[Equation]
The empirical performance of the hinge-loss-based objective has been studied by Schulman et al. [23], Han and Sung [6], Luo et al.
[14], Pi et al.
The theoretical basis is now provided by
Theorem 3.2.
Remark 3.
The idea of casting RL as a classiﬁcation problem has been investigated by [11, 12], which view the one-step greedy policy update (e.g. in Q-learning) as a binary classiﬁcation problem.
Despite the high-level resemblance, this paper is fundamentally different from the prior works [11, 12] as this paper is meant to study the theoretical foundation of the deep policy gradient methods, from the perspective of state-wise policy improvement.
3.4 Labeling Quality
In this section, we discuss the effect of the estimated advantage function on the labelling quality of the proposed classiﬁcation-based scheme.
One fundamental challenge is that the quantiﬁcation of the accuracy of the estimated value functions with non-linear function approximation remains an open problem.
Despite this, since (12) is a binary classiﬁer, the component that matters the most is the sign of estimated advantage for training this classiﬁer.
This feature allows us to further study the interplay between the noisy estimation of the advantage function and its corresponding labels for the classiﬁer.
To begin with, we present a necessary and sufﬁcient condition for the correctness of the labels in the following theorem.
Here we consider a more general baseline function denoted by φ ( s, a ) .
As the correctness of the label is a state-wise property, to make the notations uncluttered, we drop the inputs s and a from the value functions.
Let ˆ Q be the estimated action value function.
With a slight abuse of notation, we deﬁne ˆ A :
= ˆ Q − φ to be the advantage function under the more general baseline φ .
Under a policy π , we also deﬁne ∆ Q :
=
ˆ Q − Q π , ∆ V = φ − V π , and ∆ A := ∆ Q − ∆ V = [Equation] ˆ Q
[Equation]
Q
[Equation]
Theorem 3.3.
Given a policy π , for any baseline function φ and any state-action pair, the label is correct if and only if
[Equation]
As the error term is caused by ∆ Q − ∆ V , the worst-case error is max | ∆ Q | + max | ∆ V | .
Note that if ˆ Q and the baseline φ are correlated, the error bound may be further reduced.
The following corollary is an example.
Corollary 3.4.
For a deterministic transition environment, assume that V π
and ˆ V are locally Lipchitz continuous, i.e.,
[Equation]
and that the function approximation error is bounded by (cid:15) .
Then, if ˆ A is estimated by TD
[Equation]
Remark 4.
Given the way TD-learning is formulated, an interesting question is why TD updates work despite the inaccurate value function approximation.
Corollary 3.4 answers this question as follows.
We ﬁrst postulate that given a sufﬁciently small state change, max | ∆ Q | + max | ∆ V | = (1 + γ ) (cid:15) (derivation provided in Appendix A).
Since the value function error is represented by the term (cid:15) , we can see from (17) that with γ close to 1, | ∆ A | is dominated by L , ˆ L , and (cid:107) s
t +1 − s t (cid:107) , where
L is determined by the reward design, ˆ L is determined by the value function approximator, and (cid:107) s t +1 − s t (cid:107) is determined by the feature design.
In other words, these three factors dictate how successfully the TD update performs, rather than the accuracy of the value function approximators.
3.5 Experimental Justiﬁcation
In this section, we empirically investigate the rationale of the proposed classiﬁcation-based scheme.
In the experiment, we use IMPALA [2] as the baseline.
For comparison, to illustrate the proposed scheme, we update the policy based on the minimum requirement of Theorem 3.2, under which we only need to determine the sign of the advantage.
We ignore the magnitude of the advantage and incorporate the data-skipping mechanism into the ﬁnial objective.
In the standard off-policy setting, the policy gradient is estimated by
[Equation]
where µ is the behavior policy used for collecting data.
We use incorrect weights for importance sampling by replacing the behavior policy µ with the current policy π .
The resulting objective is
[Equation]
Keep in mind that according to Theorem 3.2, the only difference between (19) and IMPALA in this experiment is that we ignore speciﬁc cases in the training data while the baseline IMPALA algorithm does not.
Furthermore, it is not possible for the objective (19) to approximate the policy gradient theorem due to the incorrect importance weights and the incorrect advantage estimation.
The details of the used algorithms as well as the environment conﬁgurations are provided in Appendix B.
In order to avoid the problem of hyper-parameter dependence, the target environment needs to be kept as simple as passable.
Therefore, we use a sample problem proposed by Evans [3] involving a railroad car powered by rocket engines on either side.
Our goal is to move the car from an arbitrary initial position and ultimately have the car stop in a target position.
If we attempt to complete this task in minimal time, intuitively, the strategy would be to choose maximum output in the target direction to get maximum acceleration, then at some critical point in time switch to maximum deceleration so that the car stops right at the target.
This problem is also referred to as the optimal time control problem, whose model follows the following equation of motion:
[Equation]
where x ( t ) is the position of the car at time t , m is the mass of the car, and a ( t ) ∈ [ − 1 , 1] is the total thrust generated by the rocket engines.
To apply the learning-based approach to this control problem, the dynamical system can be approximated by the forward Euler method [13] [Equation]
[Equation]
where s =
( x, v ) T , a n ∈ [ − 1 , 1] is the action output at step n .
We deﬁne the reward (or in this case, the cost) to be the negative distance to the target position, r ( s n , a n ) =
−| x n +1 | , which allows us to apply RL to the problem.
The support of the Gaussian policy is the entire real line, while the feasible action belongs to the interval [ − 1 , 1] , i.e., the sampled action is truncated.
To avoid over-representation of the sampled actions at the interval boundaries, the mean ˆ a and σ of the Gaussian policy must be controlled.
Additional modiﬁcations are summarized as follows
[Equation]
where NN ( s ; θ ) is a neural network parameterized by θ .
Since function approximation is used, the training set needs to be distributed over the entire range of the states of interest.
Therefore, in our experiment the initial state is randomly initialized over this range.
The result in Figure 1 shows that the scheme of the standard policy gradient obtains a low total cost at early training stages, and thereafter the distribution of accumulated costs becomes widely varied.
In contrast, the scheme with data skipping has a high accumulated cost initially, but its cost monotonically drops to a low value that shows stable performance improvement.
Since the only difference between the two methods is data skipping, this experiment clearly shows that the performance of the baseline is poor due to training with undesired training data instances and that by skipping these instances, the performance can be steadily improved.
Conclusion
This paper re-investigates deep policy gradient methods via state-wise policy improvement, and attempts to bridge the gap between theory and practice.
We present two fundamental policy improvement theorems and based on these theorems we show the policy gradient methods are equivalent to training a binary classiﬁer.
Moreover, we propose a generic classiﬁcation-based policy update scheme with data skipping and identify that the PPO-clip objective is equivalent to training a perceptron classiﬁer.
Finally, we conduct an experiment to validate the proposed theoretical framework.
A Proof of Theorems
The proofs for the theorems in Section 4 are given in this Appendix.
Theorem A.1.
Given any arbitrary policy µ , a policy π improves µ if π satisﬁes either one of the following conditions:
[Equation]
Proof.
For any given policy π , the expected reward provided by a state s is
[Equation]
and the total discounted expected reward provided by s can be represented by
[Equation]
where P ( s t = s ) is the probability that visits the state s at time step t .
From (26),
[Equation]
if following policy π for P ( s t = s ) .
Following this idea and consider a new reward deﬁned by
[Equation]
Obviously
[Equation]
The expected reward r (cid:48) provided by s is
[Equation]
By deﬁnition
[Equation]
Hence,
[Equation]
This formula holds for all states s ∈ S .
Since all are non-negative from (23), the above value in (32) is non-negative too.
Thus, this implies that the values of (32) are all non-negative for all states.
Thus, we obtain that the policy π is improved from µ .
In the case that the formula (24) is satisﬁed, we can prove that the policy π is improved from µ in a similar way, which is omitted here.
Property A.2 shows the existence of some actions with non-negative values for all states, namely, there exist a action a do not worst then current policy.
This is useful in the proof of Theorem 3.2.
Property A.2.
For arbitrary policy π , the advantage function
[Equation]
has non-negative value support.
Formally,
[Equation]
Proof.
Suppose not,
[Equation]
then
[Equation]
which contradiction to (cid:80) a ∈A π ( a | s )
A
[Equation]
Theorem A.3 (Second Fundamental Theorem of Policy Improvement) .
Given a policy µ , if a new policy π is selected according to
[Equation]
where ∗ can be either π or µ , then π ≥ µ .
Proof.
Since
[Equation]
we obtain the following equation
[Equation]
According to Property A.2 there exists non-negative advantages of actions for all states.
Thus, there exists a new policy π that can be selected to increase the probability of selecting the actions with non-negative advantages of the origin policy µ and decrease those for others.
Thus, the new policy π satisﬁes Theorem 3.1, and this theorem when ∗ = µ .
When ∗ = π , the proof is similar to above by exchanging π with µ .
Theorem 3.2 above turns the policy gradient method into a classiﬁcation approach, and the classiﬁer is set to the formula
[Equation]
[Equation]
In the other words, if the new policy is selected according Theorem 3.2, then whether an action is expected to be good or not can be simply predicted by the sign of classiﬁer (40).
Theorem A.4 (Labelling Quality of Stochastic Policy) .
Given a policy π , for arbitrary state-action value estimation ˆ Q ( s, a ) and arbitrary baseline function φ ( s, a ) , denote
[Equation]
and
[Equation]
Then, the label is correct if
[Equation]
and incorrect otherwise.
Proof.
The label is correct if the true advantage has the same sign of the estimated advantage, namely
[Equation]
Rewrite the left item as
[Equation]
Thus, (43) is satisﬁed from (44) and (45).
Corollary A.5.
For a deterministic transition environment, assume that V π
and ˆ V are locally
Lipchitz continuous, i
[Equation]
e.,
[Equation]
and that the function approximation error is uniform bounded, | V π ( s ) − ˆ V ( s ) | < (cid:15).
Then, if ˆ A is estimated by TD
[Equation]
Proof.
Since the transition is deterministic
[Equation]
the Q value error is determined by
[Equation]
Therefore
[Equation]
Its absolute value are
[Equation]
From above,
[Equation]
and
[Equation]
Thus [Equation] in the worse case,
[Equation]
Result:
Learned policy π ˆ
V ( s ; θ v ) : random initialized value network ; ˆ a ( s θ random initialized policy network ; σ = 0 . 5 ;
; while True do
Sample N newest trajectory τ new = { ( s i , a i , r i , µ i , s i +1 ) ∼ π, i = 1 , 2 , · · · , T } ;
Sample M trajectory τ old ∈ D ;
B = { τ new } ∪ { τ old } ; for τ ∈ B do tmp = 0 ; for i
= T to 1 do ˆ Q i = min { r t + γ ˆ V
( s t +1 ) , 11 − γ } ;
A i =
ˆ Q
i − V
( s i ) ;
A retracei =
A i + γ × tmp ; tmp = min( π i µ i , 1)
A retracei ;
V retracei =
V i + tmp ; end (
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
; θ
[Table]
[Equation]
[Table]
; σ ← clip ( σ, 0 . 1 , 0 . 5) end
D
= D ∪ { τ new } ;
Remove oldest N trajectory from D end
Algorithm 2:
Result:
Learned policy π ˆ
V ( s ; θ v ) : random initialized value network ; ˆ a ( s ; θ π ) : random initialized policy network ; σ = 0 . 5 ;
D : the replay buffer fulled by policy π deﬁned in (22); while True do
Sample N newest trajectory τ new = { ( s i , a i , r i , µ i , s i +1 ) ∼ π, i = 1 , 2 , · · · , T } ;
Sample M trajectory τ old ∈ D ;
B = { τ new } ∪ { τ old } ; for τ ∈ B do tmp = 0 ; for i
= T to 1 do ˆ Q i = min { r t + γ ˆ V
( s t +1 ) , 11 − γ } ;
A i =
ˆ Q
i − V
( s i ) ;
A retracei =
A i + γ × tmp ; tmp = min( π i µ i , 1)
A retracei ;
V retracei =
V i + tmp ; end ( θ π , σ ) ← ( θ π , σ ) + α ∇ θ π ,σ 1 T (cid:80)
Ti =
A tracei π i µ i ; θ v ← θ v − α ∇ 1 T (cid:80) Ti =1 (cid:16) V retracei − ˆ V ( s i ) (cid:17) 2 ; σ ← clip ( σ, 0 . 1 , 0 . 5) end
D
= D ∪ { τ new } ;
Remove oldest N trajectory from D end
[Table]
Function
ReLU
[Table] [Table]
[Table]
64 (Newest) +
(Old)
[Table] [Figure]