R ANKING P OLICY G RADIENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Sample inefﬁciency is a long-lasting problem in reinforcement learning (RL).
The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization.
Towards the sample-efﬁcient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions.
To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles.
These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sampleefﬁciency.
We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.
I NTRODUCTION
One of the major challenges in reinforcement learning (RL) is the high sample complexity (Kakade et al. , 2003), which is the number of samples must be collected to conduct successful learning.
There are different reasons leading to poor sample efﬁciency of RL (Yu, 2018).
Because policy gradient algorithms directly optimizing return estimated from rollouts (e.g., R EINFORCE (Williams, 1992)) could suffer from high variance ( Sutton & Barto, 2018), value function baselines were introduced by actor-critic methods to reduce the variance and improve the sample-efﬁciency.
However, since a value function is associated with a certain policy, the samples collected by former policies cannot be readily used without complicated manipulations ( Degris et al., 2012) and extensive parameter tuning (Nachum et al., 2017).
Such an on-policy requirement increases the difﬁculty of sampleefﬁcient learning.
On the other hand, off-policy methods, such as one-step Q -learning (Watkins & Dayan, 1992) and variants of deep Q networks (DQN) ( Mnih et al., 2015; Hessel et al., 2017; Dabney et al., 2018;
Van Hasselt et al., 2016; Schaul et al., 2015), enjoys the advantage of learning from any trajectory sampled from the same environment (i.e., off-policy learning), are currently among the most sampleefﬁcient algorithms.
These algorithms, however, often require extensive searching ( Bertsekas &
Tsitsiklis, 1996, Chap. 5) over the large state-action space to estimate the optimal action value function.
Another deﬁciency is that, the combination of off-policy learning, bootstrapping, and function approximation, making up what Sutton & Barto (2018) called the "deadly triad", can easily lead to unstable or even divergent learning (Sutton & Barto, 2018, Chap. 11).
These inherent issues limit their sample-efﬁciency.
Towards addressing the aforementioned challenge, we approach the sample-efﬁcient reinforcement learning from a ranking perspective.
Instead of estimating optimal action value function, we concentrate on learning optimal rank of actions.
The rank of actions depends on the relative action values .
As long as the relative action values preserve the same rank of actions as the optimal action values ( Q -values), we choose the same optimal action.
To learn optimal relative action values, we propose the ranking policy gradient (RPG) that optimizes the actions’ rank with respect to the long-term reward by learning the pairwise relationship among actions.
Ranking Policy Gradient (RPG) that directly optimizes relative action values to maximize the return is a policy gradient method.
The track of off-policy actor-critic methods (Degris et al., 2012; Gu et al., 2016; Wang et al., 2016) have made substantial progress on improving the sample-efﬁciency of policy gradient.
However, the fundamental difﬁculty of learning stability associated with the bias-variance trade-off remains (Nachum et al., 2017).
In this work, we ﬁrst exploit the equivalence between RL optimizing the lower bound of return and supervised learning that imitates a speciﬁc optimal policy.
Build upon this theoretical foundation, we propose a general off-policy learning framework that equips the generalized policy iteration (Sutton & Barto, 2018, Chap. 4) with an external step of supervised learning.
The proposed off-policy learning not only enjoys the property of optimality preserving (unbiasedness), but also largely reduces the variance of policy gradient because of its independence of the horizon and reward scale.
Besides, we empirically show that there is a trade-off between optimality and sample-efﬁciency.
Last but not least, we demonstrate that the proposed approach, consolidating the RPG with off-policy learning, signiﬁcantly outperforms the state-of-the-art (Hessel et al., 2017; Bellemare et al., 2017; Dabney et al., 2018; Mnih et al., 2015).
R ELATED WORKS
Sample Efﬁciency.
The sample efﬁcient reinforcement learning can be roughly divided into two categories.
The ﬁrst category includes variants of Q -learning (Mnih et al., 2015; Schaul et al., 2015;
Van Hasselt et al., 2016; Hessel et al., 2017).
The main advantage of Q -learning methods is the use of off-policy learning, which is essential towards sample efﬁciency.
The representative DQN (Mnih et al., 2015) introduced deep neural network in Q -learning, which further inspried a track of successful DQN variants such as Double DQN ( Van Hasselt et al., 2016), Dueling networks (Wang et al., 2015), prioritized experience replay (Schaul et al., 2015), and R AINBOW (Hessel et al., 2017).
The second category is the actor-critic approaches.
Most of recent works (Degris et al., 2012; Wang et al., 2016; Gruslys et al., 2018) in this category leverage importance sampling by re-weighting the samples to correct the estimation bias and reduce variance.
Its main advantage is in the wall-clock times due to the distributed framework, ﬁrstly presented in (Mnih et al., 2016), instead of the sampleefﬁciency.
As of the time of writing, the variants of DQN ( Hessel et al., 2017; Dabney et al., 2018;
Bellemare et al., 2017; Schaul et al., 2015; Van Hasselt et al., 2016) are among the algorithms of most sample efﬁciency, which are adopted as our baselines for comparison.
RL as supervised learning.
Numerous amount of works have developed the connections between RL and supervised learning such as Expectation-Maximization algorithms ( Dayan & Hinton, 1997; Peters & Schaal, 2007; Kober & Peters, 2009; Abdolmaleki et al., 2018), Entropy-Regularized RL (Oh et al., 2018; Haarnoja et al., 2018), and Interactive Imitation Learning (IIL) (Daumé et al., 2009;
Syed & Schapire, 2010; Ross & Bagnell, 2010; Ross et al., 2011; Sun et al., 2017; Hester et al., 2018; Osa et al., 2018).
EM-based approaches utilize the probabilistic framework to transfer RL maximizing lower bound of return as a re-weighted regression problem while it requires on-policy estimation on the expectation step.
Entropy-Regularized RL optimizing entropy augmented objectives can lead to off-policy learning without the usage of importance sampling while it converges to soft optimality ( Haarnoja et al., 2018).
Of the three tracks in prior works, the IIL is most closely related to our work.
The IIL works ﬁrstly pointed out the connection between imitation learning and reinforcement learning ( Ross & Bagnell, 2010; Syed & Schapire, 2010; Ross et al., 2011) and explore the idea of facilitating reinforcement learning by imitating experts.
However, most of imitation learning algorithms assume the access to the expert policy or demonstrations.
Our off-policy learning framework can be interpreted as an online imitation learning approach that constructs expert demonstrations during the exploration without soliciting experts, and conducts supervised learning to maximize return at the same time.
In conclusion, our approach is different from the prior work in terms of at least one of the following aspects: objectives, oracle assumptions, the optimality of learned policy, and on-policy requirement.
More concretely, the proposed method is able to learn both deterministic and stochastic optimal policy in terms of long-term reward, without access to the oracle (such as expert policy or expert demonstration) and it can be trained both empirically and theoretically in an off-policy fashion.
Due to the space limits, we defer the detailed discussion of the related work in the Appendix Section 9.1.
N OTATIONS AND P ROBLEM S ETTING
In this paper, we consider a ﬁnite horizon T , discrete time Markov Decision Process (MDP) with a ﬁnite discrete state space S and for each state s 2 S ,
the action space
A s is ﬁnite.
The environment dynamics is denoted as P = f p ( s ′ j s; a ) ; 8 s; s ′ 2 S ; a 2 A s g .
We note that the dimension of action space can vary given different states.
We use m = max s ∥A s ∥ to denote the maximal action dimension among all possible states.
Our goal is to maximize the expected sum of positive rewards, or return J ( (cid:18) )
=
E (cid:28);(cid:25) (cid:18)
[ ∑ T t =1 r ( s t ; a t )] , where 0 < r ( s; a ) < 1 ; 8 s; a .
In this case, the optimal deterministic Markovian policy always exists (Puterman, 2014)[Proposition 4.4.3].
The upper bound of trajectory reward ( r ( (cid:28) ) ) is denoted as R max = max (cid:28) r
( (cid:28) ) .
A comprehensive list of notations are elaborated in the Appendix Table 1.
R ANKING P OLICY G RADIENT
Value function estimation is widely used in advanced RL algorithms (Mnih et al., 2015; 2016; Schulman et al., 2017;
Gruslys et al., 2018; Hessel et al., 2017; Dabney et al., 2018) to facilitate the learning process.
In practice, the on-policy requirement of value function estimations in actor-critic methods has largely increased the difﬁculty of sample-efﬁcient learning (Degris et al., 2012; Gruslys et al., 2018).
With the advantage of off-policy learning, the DQN (Mnih et al., 2015) variants are currently among the most sample-efﬁcient algorithms (Hessel et al., 2017; Dabney et al., 2018; Bellemare et al., 2017).
For complicated tasks, the value function can align with the relative relationship of action’s return, but the absolute values are hardly accurate (Mnih et al., 2015; Ilyas et al., 2018).
The above observations motivate us to look at the decision phase of RL from a different prospect:
Given a state, the decision making is to perform a relative comparison over available actions and then choose the best action, which can lead to relatively higher return than others.
Therefore, an alternative solution is to learn the optimal rank of the actions.
In this section, we show how to optimize the rank of actions to maximize the return, and thus avoid the necessity of accurate estimation for optimal action value function.
To learn the rank of actions, we focus on learning relative action value ( (cid:21) -values), deﬁned as follows: Deﬁnition 1 (Relative action value ( (cid:21) -values)) . For a state s , the relative action values of m actions ( (cid:21) ( s; a k ) ; k = 1 ; :::; m ) is a list of scores that denotes the rank of actions.
If (cid:21) ( s; a i ) > (cid:21) ( s; a j ) , then action a
i is ranked higher than action a j .
The optimal relative action values should preserve the same optimal action as the optimal action values:
[Equation]
where Q (cid:25) [Equation] ( s; a i ) and (cid:21) ( s; a i ) represent the optimal action value and the relative action value of action a i , respectively.
We omit the model parameter (cid:18) in (cid:21) (cid:18) ( s; a i ) for concise presentation. Remark 1.
The (cid:21) -values are different from the advantage function A (cid:25) ( s; a )
= Q (cid:25) ( s; a ) (cid:0) V (cid:25) ( s ) .
The advantage functions quantitatively show the difference of return taking different actions following the current policy (cid:25) .
The (cid:21) -values only determine the relative order of actions and its magnitudes are not the estimations of returns.
To learn the (cid:21) -values, we can construct a probabilistic model of (cid:21) -values such that the best action has the highest probability to be selected than others.
Inspired by learning to rank ( Burges et al., 2005), we consider the pairwise relationship among all actions, by modeling the probability (denoted as p ij ) of an action a i to be ranked higher than any action a j as follows:
[Equation]
(1) where p ij = 0 : 5 means the relative action value of a i is same as that of the action a j , p ij > 0 :indicates that the action a i is ranked higher than a j .
Given the independent Assumption 1, we can represent the probability of selecting one action as the multiplication of a set of pairwise probabilities in Eq (1).
Formally, we deﬁne the pairwise ranking policy in Eq (2).
Please refer to Section 9.10 in the Appendix for the discussions on feasibility of Assumption 1.
Deﬁnition 2.
The pairwise ranking policy is deﬁned as:
[Equation]
where the p ij is deﬁned in Eq (1).
The probability depends on the relative action values q =
[ (cid:21) 1 ; :::; (cid:21) m ] .
The highest relative action value leads to the highest probability to be selected.
Assumption 1.
For a state s , the set of events E = f e ij j8 i ̸ =
j g are conditionally independent, where e ij denotes the event that action
a
i is ranked higher than action a j . The independence of the events is conditioned on a MDP and a stationary policy.
Our ultimate goal is to maximize the long-term reward through optimizing the pairwise ranking policy or equivalently optimizing pairwise relationship among the action pairs.
Ideally, we would like the pairwise ranking policy selects the best action with the highest probability and the highest (cid:21) -value.
To achieve this goal, we resort to the policy gradient method.
Formally, we propose the ranking policy gradient method (RPG), as shown in Theorem 1.
Theorem 1 (Ranking Policy Gradient Theorem) .
For any MDP, the gradient of the expected longterm reward J ( (cid:18) )
=
∑ (cid:28) p (cid:18) ( (cid:28) )
r ( (cid:28) ) w.r.t.
the parameter (cid:18) of a pairwise ranking policy (Def 2) can be approximated by:
[Equation]
and the deterministic pairwise ranking policy (cid:25) (cid:18) is: a = arg max i (cid:21) i ; i = 1 ; : : : ; m , where (cid:21) i denotes the relative action value of action a i ( (cid:21) (cid:18) ( s t ; a t ) , a i = a t ), s t and a t denotes the t -th stateaction pair in trajectory (cid:28) , (cid:21) j ;
8 j ̸ = i denote the relative action values of all other actions that were not taken given state s t in trajectory (cid:28) , i.e., (cid:21) (cid:18) ( s t ; a j ) , 8 a j ̸ = a t .
The proof of Theorem 1 is available in Appendix Section 9.2.
Theorem 1 states that optimizing the discrepancy between the relative action values of the best action and all other actions, is optimizing the pairwise relationships that maximize the return.
One limitation of RPG is that it is not convenient for the tasks where only optimal stochastic policies exist since the pairwise ranking policy takes extra efforts to construct a probability distribution [see Section 9.3 in Appendix].
In order to learn the stochastic policy, we introduce Listwise Policy Gradient (LPG) that optimizes the probability of ranking a speciﬁc action on the top of a set of actions, with respect to the return.
In the context of
RL, this top one probability is the probability of action a i to be chosen, which is equal to the sum of probability all possible permutations that map action a i in the top.
Inspired by listwise learning to rank approach ( Cao et al., 2007), the top one probability can be modeled by the softmax function.
Therefore, LPG is equivalent to the R EINFORCE ( Williams, 1992) algorithm with a softmax layer.
LPG provides another interpretation of R EINFORCE algorithm from the perspective of learning the optimal ranking and enables the learning of both deterministic policy and stochastic policy.
Due to the space limit, we defer the detailed description of LPG in Appendix Section 9.4.
To this end, seeking sample-efﬁciency motivates us to learn the relative relationship (RPG (Theorem 1) and LPG (Theorem 4)) of actions, instead of seeking accurate estimation of optimal action values and then choosing action greedily.
However, both of the RPG and LPG belong to policy gradient methods, which suffers from large variance and the on-policy learning requirement ( Sutton & Barto, 2018).
Therefore, the direct implementation of RPG or LPG is still far from sample-efﬁcient.
In the next section, we will describe a general off-policy learning framework empowered by supervised learning, which provides an alternative way to accelerate learning, preserve optimality, and reduce variance.
O FF - POLICY LEARNING AS SUPERVISED LEARNING
In this section, we discuss the connections and discrepancies between RL and supervised learning, and our results lead to a sample-efﬁcient off-policy learning paradigm for RL.
The main result in this section is Theorem 2, which casts the problem of maximizing the lower bound of return into a supervised learning problem, given one relatively mild Assumption 2 and practical Assumptions 1,3.
As we show by Lemma 4 in the Appendix that assumptions are valid in a range of RL tasks.
The central idea is to collect only the near-optimal trajectories when the learning agent interacts with the environment, and imitate the near-optimal policy by maximizing the log likelihood of the stateaction pairs from near-optimal trajectories.
With the road map in mind, we then begin to introduce our approach as follows.
In a discrete action MDP with ﬁnite states and horizon, given the near-optimal policy (cid:25) (cid:3) , the stationary state distribution is given by: p (cid:25) (cid:3) ( s ) =
∑ (cid:28) p ( s j (cid:28) ) p (cid:25) (cid:3) ( (cid:28) ) , where p ( s j (cid:28) ) is the probability of a certain state given a speciﬁc trajectory (cid:28) and is not associated with any policies, and only p (cid:25) (cid:3) ( (cid:28) ) is related to the policy parameters.
The stationary distribution of state-action pairs is thus: p (cid:25) (cid:3) ( s; a ) =
p (cid:25) (cid:3) ( s ) (cid:25) (cid:3) ( a j s ) .
In this section, we consider the MDP that each initial state will lead to at least one (near)-optimal trajectory.
For a more general case, please refer to the discussion in
Appendix 9.5.
In order to connect supervised learning (i.e., imitating a near-optimal policy) with
RL and enable sample-efﬁcient off-policy learning, we ﬁrst introduce the trajectory reward shaping (TRS), deﬁned as follows:
Deﬁnition 3 (Trajectory Reward Shaping, TRS) .
Given a ﬁxed trajectory (cid:28) , its trajectory reward is shaped as follows:
[Equation]
where c = R max (cid:0) ϵ is a problem-dependent near-optimal trajectory reward threshold that indicates the least reward of near-optimal trajectory, ϵ (cid:21) 0 and ϵ ≪ R max .
We denote the set of all possible near-optimal trajectories as T = f (cid:28) j w ( (cid:28) [Equation]
= 1 g , i.e., w ( (cid:28) )
= 1 ; 8 (cid:28) 2 T .
Remark 2.
The threshold c indicates a trade-off between the sample-efﬁciency and the optimality.
The higher the threshold, the less frequently it will hit the near-optimal trajectories during exploration, which means it has higher sample complexity, while the ﬁnal performance is better (see
Figure 3).
Remark 3.
The trajectory reward can be reshaped to any positive functions that are not related to policy parameter (cid:18) .
For example, if we set w ( (cid:28) )
=
r ( (cid:28) ) , the conclusions in this section still hold (see Eq (38) in Appendix, Section 9.6).
For the sake of simplicity, we set w ( (cid:28) )
= 1 .
Different from the reward shaping works ( Ng et al., 1999), we directly shape the trajectory reward, which will enable the smooth transform from RL to SL.
After shaping the trajectory reward, we can transfer the goal of RL from maximizing the return to maximize the long-term performance (Def 4).
Deﬁnition 4 (Long-term Performance) .
[Equation]
The long-term performance is the expected shaped trajectory reward, as shown in Eq (4).
By Def 3, the expectation over all trajectories is the equal to that over the near-optimal trajectories in T , i.e., ∑
[Equation]
The optimality is preserved after trajectory reward shaping ( ϵ = 0 ; c = R max ) since the optimal policy (cid:25) (cid:3) maximizing long-term performance is also an optimal policy for original MDP, i.e., ∑ (cid:28) p (cid:25) (cid:3) ( (cid:28) )
r ( (cid:28) ) =
∑ (cid:28) 2T p (cid:25) (cid:3) ( (cid:28) ) r ( (cid:28) )
=
R max , where (cid:25) (cid:3) = arg max (cid:25) (cid:18)
∑ (cid:28) p (cid:25) (cid:18) ( (cid:28) ) w ( (cid:28) ) and p (cid:25) (cid:3) ( (cid:28) )
= 0 ; 8 (cid:28)
= 2 T (see Lemma 2 in Appendix 9.6).
Similarly, when ϵ > 0 , the optimal policy after trajectory reward shaping is a near-optimal policy for original MDP.
Note that most policy gradient methods use softmax function, in which we have 9 (cid:28) = 2 T ; p (cid:25) (cid:18) ( (cid:28) ) >
(see Lemma 3 in
Appendix 9.6).
Therefore when softmax is used to model a policy, it will not converge to an exact optimal policy.
On the other hand, ideally, the discrepancy of the performance between them can be arbitrarily small based on the universal approximation ( Hornik et al., 1989) with general conditions on the activation function and Theorem 1.
in (Syed & Schapire, 2010).
Essentially, we use TRS to ﬁlter out near-optimal trajectories and then we maximize the probabilities of near-optimal trajectories to maximize the long-term performance.
This procedure can be approximated by maximizing the log-likelihood of near-optimal state-action pairs, which is a supervised learning problem.
Before we state our main results, we ﬁrst introduce the deﬁnition of uniformly near-optimal policy (Def 5) and a prerequisite (Asm. 2) specifying the applicability of the results.
Deﬁnition 5 (Uniformly Near-Optimal Policy, UNOP) .
The Uniformly Near-Optimal Policy (cid:25) (cid:3) is the policy whose probability distribution over near-optimal trajectories ( T ) is a uniform distribution. i.e. p (cid:25) (cid:3) ( (cid:28) )
=
1 jT j ; 8 (cid:28) 2 T , where jT j is the number of near-optimal trajectories.
When we set c = R max , it is an optimal policy in terms of both maximizing return and long-term performance.
In the case of c = R max , the corresponding uniform policy is an optimal policy, we denote this type of optimal policy as uniformly optimal policy (UOP).
Assumption 2 (Existence of Uniformly Near-Optimal Policy) .
We assume the existence of Uniformly Near-Optimal Policy (Def 5).
Based on Lemma 4 in Appendix Section 9.9, Assumption 2 is satisﬁed for certain MDPs that have deterministic dynamics.
Other than Assumption 2, all other assumptions in this work (Assumptions 1,3) can almost always be satisﬁed in practice, based on empirical observation.
With these relatively mild assumptions, we present the following long-term performance theorem, which shows the close connection between supervised learning and RL.
Theorem 2 (Long-term Performance Theorem) .
Maximizing the lower bound of expected longterm performance (Eq (4)) is maximizing the log-likelihood of state-action pairs sampled from an uniformly (near)-optimal policy (cid:25) (cid:3) , which is a supervised learning problem:
[Equation]
The optimal policy of maximizing the lower bound is also the optimal policy of maximizing the long-term performance and the return.
Remark 4.
It is worth noting that Theorem 2 does not require a uniformly near-optimal policy (cid:25) (cid:3) to be deterministic.
The only requirement is the existence of a uniformly near-optimal policy.
Remark 5.
Maximizing the lower bound of long-term performance is to maximize the lower bound of long-term reward since we can set w ( (cid:28) )
=
r ( (cid:28) ) and ∑ (cid:28) p (cid:18) ( (cid:28) ) r ( (cid:28) ) (cid:21)
∑ T p (cid:18) ( (cid:28) ) w ( (cid:28) ) .
An optimal policy of maximizing this lower bound is also an optimal policy of maximizing the longterm performance when c = R max , thus maximizing the return.
The proof of Theorem 2 can be found in Appendix, Section 9.6.
Theorem 2 indicates that we break the dependency between current policy (cid:25) (cid:18) and the environment dynamics, which means off-policy learning is able to be conducted by the above supervised learning approach.
Furthermore, we point out that there is a potential discrepancy between imitating UNOP by maximizing log likelihood (even when the optimal policy’s samples are given) and the reinforcement learning since we are maximizing a lower bound of expected long-term performance (or equivalently the return over the near-optimal trajectories only) instead of return over all trajectories.
In practice, the state-action pairs from an optimal policy is hard to construct while the uniform characteristic of UNOP can alleviate this issue (see Sec 6).
Towards sample-efﬁcient RL, we apply Theorem 2 to RPG, which reduces the ranking policy gradient to a classiﬁcation problem by Corollary 1.
Corollary 1 (Ranking performance policy gradient) .
Optimizing the lower bound of expected longterm performance (deﬁned in Eq (4)) using pairwise ranking policy (Eq (2)) can be approximately optimized by the following loss:
[Equation]
where margin is a small positive value.
We set margin equal to one in our experiments.
The proof of Corollary 1 can be found in Appendix, section 9.7.
Similarly, we can reduce LPG to a classiﬁcation problem (see Appendix 9.7.1).
One advantage of casting RL to SL is variance reduction.
With the proposed off-policy supervised learning, we can reduce the upper bound of the policy gradient variance, as shown in the Corollary 2.
Before introducing the variance reduction results, we ﬁrst make the following standard assumption similar to ( Degris et al., 2012, A1).
Furthermore, the assumption is guaranteed for bounded continuously differentiable policy such as softmax function.
Assumption 3.
We assume the maximum norm of policy gradient is ﬁnite, i.e.
[Equation]
Corollary 2 (Policy gradient variance reduction) .
The upper bound of the variance of each dimension of policy gradient is O ( T 2 C 2 R 2max ) .
The upper bound of gradient variance of maximizing the lower bound of long-term performance Eq (5) is O
( C 2 ) , where C is the maximum norm of log gradient based on Assumption 3.
The upper bound of gradient variance by supervised learning compared to that of the regular policy gradient is reduced by an order of O ( T 2 R 2max ) , given R max (cid:21) 1 ; T (cid:21) 1 , which is a very common situation in practice, and a stationary policy.
The proof of Corollary 2 can be found in Appendix 9.8.
This corollary shows that the variance of regular policy gradient is upper-bounded by the square of time horizon and the maximum trajectory reward.
It is aligned with our intuition and empirical observation: the longer the horizon the harder the learning.
Also, the common reward shaping tricks such as truncating the reward to [ (cid:0) 1 ; 1] (Castro et al., 2018) can help the learning since it reduces variance by decreasing R max .
With supervised [Figure] learning, we concentrate the difﬁculty of long-time horizon into the exploration phase, which is an inevitable issue for all RL algorithms, and we drop the dependence on T and R max for policy variance.
Thus, it is more stable and efﬁcient to train the policy using supervised learning.
One potential limitation of this method is that the trajectory reward threshold c is task-speciﬁc, which is crucial to the ﬁnal performance and sample-efﬁciency.
In many applications such as Dialogue system ( Li et al., 2017), recommender system (Melville & Sindhwani, 2011), etc., we design the reward function to guide the learning process, in which c is naturally known.
For the cases that we have no prior knowledge on the reward function of MDP, we treat c as a tuning parameter to balance the optimality and efﬁciency, as we empirically veriﬁed in Figure 3.
The major theoretical uncertainty on general tasks is the existence of a uniformly near-optimal policy, which is negligible to the empirical performance.
The rigorous theoretical analysis of this problem is beyond the scope of this work.
A N ALGORITHMIC FRAMEWORK FOR OFF - POLICY LEARNING
Based on the discussions in Section 5, we exploit the advantage of reducing RL into supervised learning via a proposed two-stages off-policy learning framework.
As we illustrated in Figure 1, the proposed framework contains the following two stages:
Generalized Policy Iteration for Exploration.
The goal of the exploration stage is to collect different near-optimal trajectories as frequently as possible.
Under the off-policy framework, the exploration agent and the learning agent can be separated.
Therefore, any existing RL algorithm can be used during the exploration.
The principle of this framework is using the most advanced
RL agents as an exploration strategy in order to collect more near-optimal trajectories and leave the policy learning to the supervision stage.
Supervision.
In this stage, we imitate the uniformly near-optimal policy, UNOP (Def 5).
Although we have no access to the UNOP, we can approximate the state-action distribution from UNOP by collecting the near-optimal trajectories only.
The near-optimal samples are constructed online and we are not given any expert demonstration or expert policy beforehand.
This step provides a sampleefﬁcient approach to conduct exploitation, which enjoys the superiority of stability (Figure 2), variance reduction (Corollary 2), and optimality preserving (Theorem 2).
The two-stage algorithmic framework can be directly incorporated in RPG and LPG to improve sample efﬁciency.
The implementation of RPG is given in Algorithm 1, and LPG follows the same procedure except for the difference in the loss function.
The main requirement of Alg. 1 is on the exploration efﬁciency and the MDP structure.
During the exploration stage, a sufﬁcient amount of the different near-optimal trajectories need to be collected for constructing a representative supervised learning training dataset.
Theoretically, this requirement always holds [see Appendix Section 9.9,
Lemma 5], while the number of episodes explored could be prohibitively large, which makes this algorithm sample-inefﬁcient.
This could be a practical concern of the proposed algorithm.
However, according to our extensive empirical observations, we notice that long before the value function based state-of-the-art converges to near-optimal performance, enough amount of near-optimal trajectories are already explored.
Therefore, we point out that instead of estimating optimal action value functions and then choosing action greedily, using value function to facilitate the exploration and imitating UNOP is a more sample-efﬁcient approach.
As illustrated in Figure 1, value based methods with off-policy learning, bootstrapping, and function approximation could lead to a divergent optimization (Sutton & Barto, 2018, Chap. 11).
In contrast to resolving the instability, we circumvent this issue via constructing a stationary target using the samples from (near)-optimal trajectories, and perform imitation learning.
This two-stage approach can avoid the extensive exploration of the suboptimal state-action space and reduce the substantial number of samples needed for estimating optimal action values.
In the MDP where we have a high probability of hitting the near-optimal trajectories (such as P ONG ), the supervision stage can further facilitate the exploration.
It should be emphasized that our work focuses on improving the sample-efﬁciency through more effective exploitation, rather than developing novel exploration method.
Please refer to the Appendix Section 9.11 for more discussion on exploration efﬁciency.
E XPERIMENTAL R ESULTS
To evaluate the sample-efﬁciency of Ranking Policy Gradient (RPG), we focus on Atari 2600 games in OpenAI gym Bellemare et al.
; Brockman et al.
(2016), without randomly repeating the previous action.
We compare our method with the state-of-the-art baselines including DQN Mnih et al.
(2015), C51 Bellemare et al.
(2017), IQN Dabney et al.
(2018), and R AINBOW Hessel et al.
For reproducibility, we use the implementation provided in Dopamine framework Castro et al.
(2018) for all baselines and proposed methods.
Follow the standard practice Oh et al.
Hessel et al.
; Dabney et al.
; Bellemare et al.
(2017), we report the training performance of all baselines as the increase of interactions with the environment, or proportionally the number of training iterations.
We run the algorithms with ﬁve random seeds and report the average rewards with 95 % conﬁdence intervals.
The implementation details of the proposed RPG and its variants are given as follows:
EPG : EPG is the stochastic listwise policy gradient (see Appendix Eq (18)) incorporated with the proposed off-policy learning.
More concretely, we apply trajectory reward shaping (TRS, Def 3) to all trajectories encountered during exploration and train vanilla policy gradient using the off-policy samples.
This is equivalent to minimizing the cross-entropy loss (see Appendix Eq (69)) over the near-optimal trajectories.
LPG : LPG is the deterministic listwise policy gradient with the proposed off-policy learning.
The only difference between EPG and LPG is that LPG chooses action deterministically (see Appendix Eq (17)) during evaluation.
RPG : RPG explores the environment using a separate EPG agent in P ONG and IQN in other games.
Then RPG conducts supervised learning by minimizing the hinge loss Eq (6).
It is worth noting that the exploration agent (EPG or IQN) can be replaced by any existing exploration method.
In our
RPG implementation, we collect all trajectories with the trajectory reward no less than the threshold c without eliminating the duplicated trajectories and we empirically found it is a reasonable simpliﬁcation.
More details of hyperparameters are provided in the Appendix Section 9.12.
Sample-efﬁciency: As the results shown in Figure 2, our approach, RPG, signiﬁcantly outperform the state-of-the-art baselines in terms of sample-efﬁciency at all tasks.
Furthermore, RPG not only achieved the most sample-efﬁcient results, but also reached the highest ﬁnal performance at R OBOTANK , D OUBLE D UNK , P ITFALL , and P ONG , comparing to any model-free state-of-the-art.
In reinforcement learning, the stability of algorithm should be emphasized as an important issue.
As we can see from the results, the performance of baselines varies from task to task.
There is no single baseline consistently outperforms others.
In contrast, due to the reduction from RL to supervised learning, RPG is consistently stable and effective across different environments.
In addition to the stability and efﬁciency, RPG enjoys simplicity at the same time.
In the environment P ONG , it is surprising that RPG without any complicated exploration method largely surpassed the sophisticated value-function based approaches.
7. A BLATION STUDY
The effectiveness of pairwise ranking policy and off-policy learning as supervised learning.
To get a better understanding of the underlying reasons that RPG is more sample-efﬁcient than DQN variants, we performed ablation studies in the P ONG environment by varying the combination of policy functions with the proposed off-policy learning.
The results of EPG, LPG, and RPG are shown in the bottom right, Figure 2.
Recall that EPG and LPG use listwise policy gradient (vanilla policy gradient using softmax as policy function) to conduct exploration, the off-policy learning minimizes the cross-entropy loss Eq ( 69).
In contrast, RPG shares the same exploration method as
EPG and LPG while uses pairwise ranking policy Eq (2) in off-policy learning that minimizes hinge loss Eq (6).
We can see that RPG is more sample-efﬁcient than EPG/LPG.
We also compared the most advanced on-policy method Proximal Policy Optimization (PPO) Schulman et al.
(2017) with
EPG, LPG, and RPG.
The proposed off-policy learning largely surpassed the best on-policy method.
Therefore, we conclude that off-policy as supervised learning contributes to the sample-efﬁciency substantially, while pairwise ranking policy can further accelerate the learning. In addition, we compare RPG to off-policy policy gradient approaches: ACER Wang et al. (2016) and self-imitation learning
Oh et al.
As the results shown, the proposed off-policy learning framework is more sample-efﬁcient than the state-of-the-art off-policy policy gradient approaches.
The optimality-efﬁciency trade-off.
As reported in Figure 3, we empirically demonstrated the trade-off between the sample-efﬁciency and optimality, which is controlled by the trajectory reward threshold (as deﬁned in Def 3).
The higher value of trajectory reward threshold suggests we have higher requirement on deﬁning near-optimal trajectory.
This will increase the difﬁculty of collecting near-optimal samples during exploration, while it ensures a better ﬁnal performance.
These experimental results also justiﬁed that RPG is also effective in the absence of prior knowledge on trajectory reward threshold, with a mild cost on introducing an additional tuning parameter.
C ONCLUSIONS
In this work, we introduced ranking policy gradient (RPG) methods that, for the ﬁrst time, resolve RL problem from a ranking perspective.
Furthermore, towards the sample-efﬁcient RL, we propose an off-policy learning framework that allows RL agents to be trained in a supervised learning paradigm.
The off-policy learning framework uses generalized policy iteration for exploration and exploit the stableness of supervised learning for policy learning, which accomplishes the unbiasedness, variance reduction, off-policy learning, and sample efﬁciency at the same time.
Last but not least, empirical results show that RPG achieves superior performance as compared to the state-of-the-art.
A PPENDIX
In Table 1 we provide a brief summary of important notations used in the paper:
Notations
[Table]
(cid:21) ij
The discrepancy of the relative action value of action i and action j . (cid:21) ij = (cid:21) i (cid:0) (cid:21) j , where (cid:21) i = (cid:21) ( s; a i ) : Notice that the value here is not the estimation of return, it represents which action will have relatively higher return if followed.
Q (cid:25) ( s; a )
The action value function or equivalently the estimation of return taking action a at state s , following policy (cid:25) .
p ij p [Equation]
= P ( (cid:21) [Equation] > (cid:21) j ) denotes the probability that i -th action is to be ranked higher than j -th action.
Notice that p ij is controlled by (cid:18) through (cid:21) i ; (cid:21) j (cid:28)
A trajectory (cid:28) =
f s ( (cid:28); t ) ; a ( (cid:28); t ) g
Tt =1 collected from the environment.
It is worth noting that this trajectory is not associated with any policy.
It only represents a series of state-action pairs.
We also use the abbreviation s t = s ( (cid:28); t ) , a t = a ( (cid:28); t ) .
r ( (cid:28) )
The trajectory reward r ( (cid:28) )
=
∑ Tt =1 r ( s t ; a t ) is the sum of reward along one trajectory.
R max
R max is the maximal possible trajectory reward, i.e., R max = max (cid:28) r
( (cid:28) ) .
Since we focus on MDPs with ﬁnite horizon and immediate reward, therefore the trajectory reward is bounded.
∑ (cid:28)
The summation over all possible trajectories (cid:28) .
p (cid:18) ( (cid:28) )
The probability of a speciﬁc trajectory is collected from the environment given policy
[Table] [Equation] [Table] [Equation]
[Table]
The set of all possible near-optimal trajectories.
jT j denotes the number of near-optimal trajectories in T .
[Table]
The number of training samples or equivalently state action pairs sampled from uniformly optimal policy.
[Table]
9. A DISCUSSION OF PRIOR WORKS ON REDUCING RL TO SL.
There are two main distinctions between supervised learning and reinforcement learning.
In supervised learn- ing, the data distribution D is static and training samples are assumed to be sampled i.i.d.
from D .
On the contrary, the data distribution is dynamic in RL.
It is determined by both environment dynamics and the learn- ing policy.
The policy keeps evolving during the learning process, which results in the dynamic data distribution in RL.
Secondly, the training samples we collected are not independently distributed due to the change of learning policy.
These intrinsic difﬁculties of RL make the learning algorithm unstable and sample-inefﬁcient.
However, if we review the state-of-the-art in RL community, every algorithm eventually acquires the policy, either explicitly or implicitly, which is a mapping from the state to an action or a probability distribution over the action space.
Ultimately, there exists a supervised learning equivalent to the RL problem, if the optimal policies exist.
The paradox is that it is almost impossible to construct this supervised learning equivalent on the ﬂy, without knowing any optimal policy.
Although what is the proper supervision still lingered in the RL community, pioneers have developed a set of insightful approaches to reduce RL into its SL counterpart over the past several decades.
Roughly, we can classify the prior work into the following categories: (cid:15) Expectation-Maximization (EM) Dayan & Hinton Peters & Schaal (
Kober & Peters Abdolmaleki et al.
, etc.
(cid:15)
Entropy-Regularized RL (ERL) O’Donoghue
et
[Table]
[Table]
[Table]
Oh et al.
[Table]
[Table]
Haarnoja
[Table]
[Table]
etc.
(cid:15)
Interactive Imitation Learning (IIL) Daumé
[Table]
al.
[Table]
; Syed & Schapire
[Table]
; Ross & Bagnell
[Table]
; Ross
[Table]
al.
[Table]
; Sun
[Table]
al. [Table] , etc. The early work in the EM track transfers objective by Jensen’s inequality and the maximizing the lower bound of the original objective, which resembles Expectation-Maximization procedure and provides policy improvement guarantee.
While pioneering at the time, these works typically focus on the simpliﬁed RL setting, such as in Dayan & Hinton (1997)
the reward function is not associated with the state or in Peters & Schaal (2008) the goal is to maximize the expected immediate reward and the state distribution is assumed to be ﬁxed.
Later on in Kober & Peters (2009), the authors extended the EM framework from immediate reward into episodic return.
Recent advance Abdolmaleki et al.
(2018) utilizes the EM-framework on a relative entropy objective, which adds a parameter prior as regularization.
As mentioned in the paper, the evaluation step using Retrace Munos et al.
(2016) can be unstable even with linear function approximation Touati et al.
In general, the estimation step in EM-based algorithms involves on-policy evaluation, which is one difﬁculty shared for any policy gradient methods.
One of the main motivation that we want to transfer the RL into a supervised learning task is the off-policy learning enable sample efﬁciency.
To achieve off-policy learning, PGQ O’Donoghue et al. (2016) connected the entropy-regularized policy gra- dient with Q-learning under the constraint of small regularization.
In the similar framework, Soft Actor-Critic Haarnoja et al.
(2018) was proposed to enable sample-efﬁcient and faster convergence under the framework of entropy-regularized RL.
It is able to converge to the optimal policy that optimizes the long-term reward along with policy entropy.
It is an efﬁcient way to model the suboptimal behavior and empirically it is able to learn a reasonable policy.
Although recently the discrepancy between the entropy-regularized objective and original long-term reward has been discussed in O’Donoghue (2018); Eysenbach & Levine (2019), they focus on learning stochastic policy while the proposed framework is feasible for both learning deterministic optimal policy (Corollary 1) and stochastic optimal policy (Corollary 6).
In Oh et al.
(2018), this work shares similarity to our work in terms of the method we collecting the samples.
They collect good samples based on the past experience and then conduct the imitation learning w.r.t those good samples.
However, we differentiate at how do we look at the problem theoretically.
This self-imitation learning procedure was eventually connected to lower-bound-soft-Q-learning, which belongs to entropy-regularized reinforcement learning.
We comment that there is a trade-off between sample-efﬁciency and modeling suboptimal behaviors.
The more strict requirement we have on the samples collected we have less chance to hit the samples while we are more close to imitating the optimal behavior.
From the track of interactive imitation learning, initial representative works such as Ross & Bagnell (2010); Ross et al.
(2011) ﬁrstly pointed out the main discrepancy between imitation learning and reinforcement learning is the i.i.d.
assumption does not hold and provides SMIL E Ross & Bagnell (2010) and
DA GGER Ross et al.
(2011) to overcome the distribution mismatch.
The theorem 2.1 in Ross & Bagnell (2010) ﬁrstly analyzed if the learned policy fails to imitate the expert with a certain probability, what is the performance degradation comparing to the expert.
While the theorem seems to resemble the long-term performance theorem 2, it considers the learning policy is trained through the state distribution induced by the expert, instead of state-action distribution as we did in Theorem 2.
Their theorem thus may be more applicable to the situation where an interactive procedure is needed, such as querying the expert during the training process.
On the contrary, we [Table] focus on directly applying supervised learning approach without having access to the expert to label the data.
The optimal state-action pairs are collected during exploration and conducting supervised learning on the replay buffer will provide a performance guarantee in terms of long-term expected reward.
Concurrently, a resemble of theorem 2.1 in Ross & Bagnell (2010) is Theorem 1 in Syed & Schapire (2010), the authors reduce the apprenticeship learning to classiﬁcation, under the assumption that the apprentice policy is deterministic and the misclassiﬁcation rate at all time steps is bounded, which we do not make.
Within the IIL track, later on the
A GGRE V A T E Ross & Bagnell (2014) was proposed to incorporate the information of action costs to facilitate imitation learning, and a differentiable version called A GGRE V A T E D Sun et al.
(2017) was recently developed and achieved impressive empirical results.
Recently, hinge loss was combined with regular Q -learning loss as a pre-training step for learning from demonstration Hester et al.
(2018) or as a surrogate loss for imitating optimal trajectories Osa et al.
In this work, we show that hinge loss constructs a new type of policy gradient method and can learn optimal policy directly.
In conclusion, our method approaches the problem of reducing RL to SL from a different perspective than all prior works mentioned.
With our construction from RL to SL, the samples collected in replay buffer can satisfy the i.i.d.
assumption exactly since the state action pairs are sampled from the data distribution of UNOP.
A summary of similarity and discrepancy between our method the current best method in each category is shown in the Table 2.
9. R ANKING P OLICY G RADIENT T HEOREM
The proof of Theorem 1 can be found as follows:
Proof.
The following proof is based on direct policy differentiation Peters & Schaal (2008); Williams (1992).
For concise presentation, the subscript t for action value (cid:21) i ; (cid:21) j , and p ij is omitted.
[Equation]
T ∑ t = ∇ (cid:18) log((cid:5) mj =1 ;j ̸ = i p ij ) r ( (cid:28) )] =
E (cid:28) (cid:24) (cid:25) (cid:18)
T ∑ t = ∇ (cid:18) m ∑ j =1 ;j ̸ = i log( e (cid:21) ij 1 + e (cid:21) ij ) r ( (cid:28) )]
= E (cid:28) (cid:24) (cid:25) (cid:18) [
T ∑ t = ∇ (cid:18) m
∑ j =1 ;j ̸ = i log( 1 + e (cid:21) ji ) r ( (cid:28) )] (7) (cid:25) E (cid:28) (cid:24) (cid:25) (cid:18) [
T ∑ t = ∇ (cid:18) @ m
∑ j =1 ;j ̸ = i ( (cid:21) i (cid:0) (cid:21) j ) =
A r ( (cid:28) )] (8) where the trajectory is a series of state-action pairs from t = 1 ; :::; T , i:e:(cid:28) = s 1 ; a 1 ; s 2 ; a 2 ; :::; s T .
From Eq (7) to Eq (8), we use the ﬁrst-order Taylor expansion of log(1 + e x ) j x =
= log 2 + 12 x + O ( x 2 ) to further simplify the ranking policy gradient.
9. D ISCUSSION ON THE PROBABILITY DISTRIBUTION OF RPG
Corollary 3.
The pairwise ranking policy as shown in Eq (2) constructs a probability distribution over the set of actions when the action space m is equal to 2 , given any relative action values (cid:21) i ; i = 1 ; 2 .
For the cases with m > 2 , this conclusion does not hold in general.
It is easy to verify that (cid:25) ( a i j s ) > 0 , ∑ 2 i =1 (cid:25) ( a i j s ) = 1 holds and the same conclusion cannot be applied to m > 2 by constructing counterexamples.
However, we can introduce a dummy action a ′ to form a probability distribution for RPG.
During policy learning, the algorithm will increase the probability of best actions and the probability of dummy action will decrease.
Ideally, if RPG converges to an optimal deterministic policy, the probability of taking best action is equal to one and (cid:25) ( a ′ j s ) = 0 .
Similarly, we can introduce a dummy trajectory (cid:28) ′ with trajectory reward r ( (cid:28) ′ )
= 0 and p (cid:18) ( (cid:28) ′ )
= 1 (cid:0)
∑ (cid:28) p (cid:18) ( (cid:28) ) .
The trajectory probability forms a probability distribution since ∑ (cid:28) p (cid:18) ( (cid:28) )
+ p (cid:18) ( (cid:28) ′ )
= 1 and p (cid:18) ( (cid:28) )
(cid:21) 0 8 (cid:28) and p (cid:18) ( (cid:28) ′ )
(cid:21) 0 .
The proof of a valid trajectory probability is similar to the following proof on (cid:25) ( a j s ) is a valid probability distribution with a dummy action.
The practical inﬂuence of this is negligible since our goal is to increase the probability of (near)-optimal trajectories.
To present in a clear way, we avoid mentioning dummy trajectory (cid:28)
′ in Proof 9.
while it can be seamlessly included.
Condition 1 (The range of (cid:21) -value) .
We restrict the range of (cid:21) -values in RPG so that it satisﬁes (cid:21) m (cid:21) ln
m 1 m (cid:0) 1 (cid:0) 1) , where (cid:21) m
= min i;j (cid:21) ji , m is the action dimension.
This condition can be easily satisﬁed since in RPG we only focus on the relative relationship of (cid:21) -values and we can constrain its range so that (cid:21) m satisﬁes the condition 1.
Furthermore, since we can see that m 1 m (cid:0) 1 >is decreasing w.r.t to action dimension m .
The larger the action dimension, the less constraint we have on the (cid:21) -values.
Corollary 4.
Given Condition 1, we introduce a dummy action a ′ and set (cid:25) ( a = a ′ j s ) = 1 (cid:0)
∑ i (cid:25) ( a = a i j s ) , which will construct a valid probability distribution ( (cid:25) ( a j s )) over the action space
A [ a ′ .
Proof.
Since we have (cid:25) ( a = a i j s ) > 0 8 i = 1 ; :::; m and ∑ i (cid:25) ( a = a i j s ) + (cid:25) ( a = a ′ j s )
= 1 .
To prove this is a valid probability distribution, we only need to show that (cid:25) ( a = a ′ j s ) (cid:21) 0 ; 8 m (cid:21) 2 , i.e. ∑ i (cid:25) ( a = a i j s ) (cid:20) 1 ; 8 m (cid:21) 2 .
Let (cid:21) m
= min i;j (cid:21) ji , ∑
[Equation] [Equation]
[Equation] [Equation] [Equation]
Condition
[Equation] [Equation]
9. L ISTWISE P OLICY G RADIENT
In order to learn the stochastic policy that optimizes the ranking of actions with respect to the return, we now introduce the Listwise Policy Gradient (LPG) method.
In RL, we want to optimize the probability of each action ( a i ) to be ranked higher among all actions, which is the sum of the probabilities of all permutations such that the action a i in the top position of the list.
This probability is computationally prohibitive since we need to consider the probability of m !
permutations.
Luckily, based on Cao et al.
[Theorem 6], we can model the such probability of action a i to be ranked highest given a set of relative action values by a simple softmax formulation, as described in Theorem 3.
Theorem 3 (Theorem 6 Cao et al.
Given the relative action values q =
[ (cid:21) 1 ; :::; (cid:21) m ] , the probability of action i to be taken (i.e. to be ranked on the top of the list) is:
[Equation]
where ϕ ( (cid:3) ) is any increasing, strictly positive function.
A common choice of ϕ is the exponential function.
Closely built upon the foundations from learning to rank Cao et al.
(2007), we present the listwise policy gradient method, as introduced in Theorem 4.
Theorem 4 (Listwise Policy Gradient Theorem) .
For any MDP, the gradient of the long-term reward J ( (cid:18) ) =
∑ (cid:28) p (cid:18) ( (cid:28) )
r ( (cid:28) ) w.r.t.
the parameter (cid:18) of listwise ranking policy takes the following form:
[Equation]
[Equation] where the listwise ranking policy (cid:25) (cid:18) parameterized by (cid:18) is given by Eq (17) for tasks with deterministic optimal policies:
[Equation]
or Eq (18) for stochastic optimal policies:
[Equation]
where the policy takes the form as in Eq (19)
[Equation]
is the probability that action i being ranked highest, given the current state and all the relative action values (cid:21) 1 : : : (cid:21) m .
The proof of Theorem 4 exactly follows the direct policy differentiation Peters & Schaal (2008); Williams (1992) by replacing the policy to the form of the softmax function.
The action probability (cid:25) ( a i j s ) ; 8 i = 1 ; :::; m forms a probability distribution over the set of discrete actions [Cao et al.
Lemma 7].
Theoremstates that the vanilla policy gradient Williams (1992) parameterized by a softmax layer is optimizing the probability of each action to be ranked highest, with respect to the long-term reward.
9. D ISCUSSIONS ON THE OPTIMALITY PRESERVING
Condition 2 (Initial States) .
The (near)-optimal trajectories will cover all initial states of MDP.
i.e. f s ( (cid:28); 1) j 8 (cid:28) 2 T g = f
s ( (cid:28); 1) j 8 (cid:28) g , where T = f (cid:28) j w ( (cid:28) )
= 1 g = f
(cid:28) j r ( (cid:28) ) (cid:21) c
g .
The Condition 2 describes what type of MDPs is directly applicable to the trajectory reward shaping (TRS, Def 3).
The MDPs satisfying this condition cover a wide range of tasks such as Dialogue System Li et al.
(2017), Go Silver et al.
video games Bellemare et al.
(2013) and all MDPs
with only one initial state.
If we want to preserve the optimality by TRS, the optimal trajectories of MDP needs to cover all initial states or equivalently, all initial states will lead to at least one optimal trajectory.
Similarly, the near-optimality is preserved for all MDPs that its near-optimal trajectories cover all initial states.
Theoretically, it is possible to transfer more general MDPs to satisfy Condition 2 and preserve the optimality with potential-based reward shaping Ng et al.
More concretely, consider the deterministic binary tree
MDP ( M 1 ) with the set of initial states S 1 = f s 1 ; s ′ 1 g as deﬁned in Figure 4.
There are eight possible trajectories in M 1 .
Let r ( (cid:28) 1 ) =
10 =
R max ; r ( (cid:28) 8 ) = 3 ; r ( (cid:28) i ) = 2 ; 8 i = 2 ; ::: 7 .
Therefore, this MDP does not satisfy Condition 2.
We can compensate the trajectory reward of best trajectory starting from s ′ 0 to the
R max by shaping the reward with the potential-based function ϕ ( s ′ 7 )
= 7 and ϕ ( s )
= 0 ; 8 s ̸
= s ′ 7 .
This reward shaping requires more prior knowledge, which may not be feasible in practice.
A more realistic method is to design a dynamic trajectory reward shaping approach.
In the beginning, we set c ( s ) =
min s 2S 1 r ( (cid:28) j s ( (cid:28); 1) = s ) ; 8 s 2 S 1 .
Take M 1 as an example, c ( s ) = 3 ; 8 s 2 S 1 .
During the exploration stage, we track the current best trajectory of each initial state and update c ( s ) with its trajectory reward.
Nevertheless, if the Condition 2 is not satisﬁed, we need more sophisticated prior knowledge other than a predeﬁned trajectory reward threshold c to construct the replay buffer (training dataset of UNOP).
The practical implementation of trajectory reward shaping and rigorously theoretical study for general MDPs are beyond the scope of this work.
9. P ROOF OF L ONG - TERM P ERFORMANCE T HEOREM
In this subsection, we reduce maximizing RL objective into a supervised learning problem with Theorem 2.
Before that, we ﬁrst prove Lemma 1 to link the log probability of a trajectory (cid:28) to its state action distribution.
Then using this lemma, we can connect the trajectory probability of UNOP with its state-action distribution, from which we prove the Theorem 2.
Lemma 1.
Given a speciﬁc trajectory (cid:28) , the averaged state-action pair log-likelihood over horizon T is equal to the weighted sum over the entire state-action space, i.e.:
[Equation]
where the sum in the right hand side is the summation over all possible state-action pairs.
It is worth noting that p ( s; a j (cid:28) ) is not related to any policy parameters.
It is the probability of a speciﬁc state-action pair ( s; a ) in a speciﬁc trajectory (cid:28) .
Proof.
Given a trajectory (cid:28) =
f ( s ( (cid:28); 1) ; a ( (cid:28); 1)) ; :::; ( s ( (cid:28); T ) ; a ( (cid:28); T ))
g
= f ( s 1 ; a 1 ) ; :::; ( s T ; a T ) g , denote the unique state action pairs in this trajectory as U ( (cid:28) )
=
f ( s i ; a i ) g ni
=1 , where n is the number of unique state- action pairs in (cid:28) and n (cid:20) T .
The number of occurrences of a state-action pair ( s i ; a i ) in the trajectory (cid:28) is denoted as j ( s i ; a i ) j . [Equation] [Equation] [Equation]
[Equation]
∑
[Equation]
[Equation] where from Eq (24) to Eq (25) we use ∑ ( s;a ) 2 U ( (cid:28) ) p ( s; a j (cid:28) )
=
∑ ni =1 p ( s i ; a i j (cid:28) )
=
∑ ni =1 j ( s i ;a i )
j T = 1 , ) p ( s; a j (cid:28) )
= 0 ; 8 ( s; a )
= 2 U ( (cid:28) ) .
This thus completes the proof.
Now we are ready to prove the Theorem 2:
Proof.
The following proof holds for arbitrary subset of trajectories T which is determined by the threshold c in Def 5.
The (cid:25) (cid:3) is associated with c and this subset of trajectories.
[Equation]
The lower bound holds when p (cid:18) ( (cid:28) ) w ( (cid:28) )
=
1 jT j ; 8 (cid:28) 2 T .
To this end, we maximize the lower bound of the expected long-term performance. arg
[Equation]
This is the reason that w ( (cid:28) ) can be set as arbitrary positive constant [Equation] [Equation] = arg
[Equation]
= arg max
[Equation]
t ) Use Assumption 2 the existence of UNOP.
[Equation]
= arg
[Equation]
∑
[Equation]
42) where (cid:25) (cid:3) is a UNOP (Def 5) : (43) ) p (cid:25) (cid:3) [Equation] (cid:28) [Equation]
{+= 
+}
[Equation]
Eq (44) can be established based on ∑
[Equation]
45) = arg
[Equation]
∑
[Equation]
Use lemma= arg max [Equation] ∑ [Equation]
∑
[Equation]
The 2nd sum is over all possible state-action pairs.
( s; a ) represents a speciﬁc state-action pair. = arg max [Equation] ∑
[Equation]
∑
[Equation]
= arg
[Equation]
∑
[Equation]
[Equation] In this proof we use s t = s ( (cid:28); t ) and a t = a ( (cid:28); t ) as abbreviations, which denote the t -th state and action in the trajectory (cid:28) , respectively.
jT j denotes the number of trajectories in T .
We also use the deﬁnition of w ( (cid:28) ) to only focus on near-optimal trajectories.
We set w ( (cid:28) )
= 1 for simplicity
but it will not affect the conclusion if set to other constants.
Optimality: Furthermore, the optimal solution for the objective function Eq (49) is a uniformly (near)-optimal policy (cid:25) (cid:3) .
[Equation]
Therefore, the optimal solution of Eq (49) is also the (near)-optimal solution for the original RL problem since ∑
[Equation]
ϵ .
The optimal solution is obtained when we set c = R max .
Lemma 2.
Given any optimal policy (cid:25) of MDP satisfying Condition 2, 8 (cid:28) = 2 T , we have p (cid:25) ( (cid:28) )
= 0 , where T denotes the set of all possible optimal trajectories in this lemma.
If 9 (cid:28) = 2 T , such that p (cid:25) ( (cid:28) ) > 0 , then (cid:25) is not optimal policy.
Proof.
We prove this by contradiction.
We assume (cid:25) is an optimal policy.
If 9 (cid:28) ′
= 2 T , such that 1) p (cid:25) ( (cid:28) ′ )
̸ = 0 , or equivalently: p (cid:25) ( (cid:28) ′ ) > 0 since p (cid:25) ( (cid:28) ′ ) 2 [1 ; 0] .
and 2) (cid:28) ′
= 2 T .
We can ﬁnd a better policy (cid:25) ′ by satisfying the following three conditions:
[Equation]
[-( (cid:28) ′ ) = 
 and p (cid:25) ′ ( (cid:28) 1 )-] = p (cid:25) ( (cid:28) 1 ) + p (cid:25) ( (cid:28) ′ ) ; (cid:28) 1 2 T and p (cid:25) ′ ( (cid:28) )
=
p (cid:25) ( (cid:28) ) ; 8 (cid:28) =
2 f (cid:28) ′ ; (cid:28) 1 g
Since p (cid:25) ′
( (cid:28) )
(cid:21) 0 ; 8 (cid:28) and ∑ (cid:28) p (cid:25) ′ ( (cid:28) )
= 1 , therefore p (cid:25) ′ constructs a valid probability distribution.
Then the expected long-term performance of (cid:25) ′ is greater than that of (cid:25) :
∑ (cid:28) p (cid:25) ′
( (cid:28) )
w ( (cid:28) ) (cid:0) ∑ (cid:28) p (cid:25) ( (cid:28) ) w ( (cid:28) )
=
∑ (cid:28)= 2f (cid:28) ′ ;(cid:28) 1 g p (cid:25) ′ ( (cid:28) ) w ( (cid:28) )
+ p (cid:25) ′ ( (cid:28) 1 ) (cid:3) w ( (cid:28) 1 ) + p (cid:25) ′
( (cid:28) ′ )
(cid:3) w ( (cid:28) ′ ) (cid:0) ( ∑ (cid:28)= 2f (cid:28) ′ ;(cid:28) 1 g p (cid:25) ( (cid:28) ) w ( (cid:28) ) + p (cid:25) ( (cid:28) 1 ) (cid:3) w ( (cid:28) 1 ) + p (cid:25) ( (cid:28) ′ ) (cid:3) w ( (cid:28) ′ ))
= p (cid:25) ′ ( (cid:28) 1 ) (cid:3) w ( (cid:28) 1 ) [Table]
Essentially, we can ﬁnd a policy (cid:25) ′
that has higher probability on the optimal trajectory (cid:28) 1 and zero probability on (cid:28) ′ .
This indicates that it is a better policy than (cid:25) .
Therefore, (cid:25) is not an optimal policy and it contradicts our assumption, which proves that such (cid:28) ′ does not exist.
Therefore, 8 (cid:28) = 2 T , we have p (cid:25) ( (cid:28) )
= 0 .
Lemma 3 (Policy Performance) .
8 (cid:28) , p (cid:18) ( (cid:28) ) > 0 , if the policy takes the form as in Eq (15) or Eq (2).
This means for all possible trajectories allowed by the environment, the policy takes the form of either ranking policy or softmax will generate this trajectory with probability p (cid:18) ( (cid:28) ) > 0 .
It is worth noting that because of this property, (cid:25) (cid:18) is not an optimal policy according to Lemma 2, though it can be arbitrarily close to the optimal policy.
Proof.
[Equation]
55) and (cid:25) (cid:18) ( a t j s t ) > 0 ; p ( s 1 ) > 0 ; p ( s t +1 j s t ; a t ) > 0 :
[Equation]
56) if p ( s t +1 j s t ; a t ) = 0 or p ( s 1 ) =
; then the probability of sampling (cid:28) from any policy is zero.
This trajectory does not exist.
[Equation]
This thus completes the proof.
9. P ROOF OF P ERFORMANCE P OLICY G RADIENT C OROLLARIES
Corollary 5 (Ranking performance policy gradient) .
Optimizing the lower bound of expected long-term per- formance (deﬁned in Eq (4)) using pairwise ranking policy (Eq (2)) can be approximately optimized by the following loss:
[Equation]
where margin is a small positive value.
We set margin equal to one in our experiments.
Proof.
In RPG, the policy (cid:25) (cid:18) ( a j s ) is deﬁned as in Eq (2).
We then replace the action probability distribution in Eq (5) with the RPG policy.
[Equation]
Because RPG is ﬁtting a deterministic optimal policy, we denote the optimal action given sate s as a i ; then we have [Equation] ∑ [Equation]
∑
[Equation]
ﬁrst order Taylor expansion
[Equation]
∑
[Equation]
[Equation]
where the pairwise loss L ( s; a i ) is deﬁned as:
[Equation]
max
[Equation]
(67) where the margin in Eq (66) is a small positive constant. From Eq (65) to Eq (66), we consider learning a deterministic optimal policy a i = (cid:25) (cid:3) ( s ) , where we use index i to denote the optimal action at each state.
{+The optimal (cid:21) -values minimizing Eq (65) (denoted by (cid:21) 1 ) need to satisfy (cid:21) 
 i = (cid:21) 1 j + c; 8 j ̸ = i; s . The optiaml (cid:21) -values minimizing Eq (66) (denoted by (cid:21) 2 ) need to satisfy (cid:21) 2 i = max j ̸ = i (cid:21) 2 j + margin ;+}
8 j ̸ = i; s .
In both cases, the optimal policies from solving Eq (65) and Eq (65) are the same: (cid:25) ( s ) = arg max k (cid:21) 1 k = arg max k (cid:21) 2 k = a i . Therefore, we use Eq (66) as a surrogate optimization problem of Eq (65).
9.7. L ISTWISE PERFORMANCE POLICY GRADIENT
Corollary 6 (Listwise performance policy gradient) .
Optimizing the lower bound of expected long-term per- formance by the listwise ranking policy (Eq (19)) is equivalent to:
[Equation]
The proof of Corollary 6 is a direct application of theorem 2 by replacing policy with the softmax function.
9. P OLICY GRADIENT VARIANCE REDUCTION
Corollary 7 (Variance reduction) .
The upper bound of the variance of each dimension of policy gradient is O ( T 2 C 2 R 2max ) .
The upper bound of gradient variance of maximizing the lower bound of long-term performance Eq (5) is O
( C 2 ) , where C is the maximum norm of log gradient based on Assumption 3.
The upper bound of gradient variance by supervised learning compared to that of the regular policy gradient is reduced by an order of O ( T 2 R 2max ) , given R max (cid:21) 1 ; T (cid:21) 1 , which is a very common situation in practice, and a stationary policy.
Proof.
The regular policy gradient of policy (cid:25) (cid:18) is given as Williams (1992):
[Equation]
The regular policy gradient variance of the i -th dimension is denoted as follows:
[Equation]
We denote x i ( (cid:28) )
=
∑ Tt =1 ∇ (cid:18) log( (cid:25) (cid:18) ( a ( (cid:28); t )
j s ( (cid:28); t )) i ) r ( (cid:28) ) for convenience.
Therefore, x i is a random variable.
Then apply var ( x ) =
E p (cid:18) ( (cid:28) )
[ x 2 ] (cid:0) E p (cid:18) ( (cid:28) )
[ x ] 2 , we have:
[Equation]
The policy gradient of long-term performance (Def 4) ∑
[Equation]
The policy gradient variance of the i -th dimension is denoted as
[Equation]
Then the upper bound is given by
[Equation]
This thus completes the proof.
9. D ISCUSSIONS OF A SSUMPTION
In this section, we show that UNOP exists in a range of MDPs.
Notice that the lemma 4 shows the sufﬁcient conditions of satisfying Asumption 2 rather than necessary conditions.
Lemma 4.
For MDPs deﬁned in Section 3 satisfying the following conditions:
(cid:15)
Each initial state leads to one optimal trajectory.
This also indicates jS 1 j = jT j , where
T denotes the set of optimal trajectories in this lemma, S 1 denotes the set of initial states.
(cid:15)
Deterministic transitions, i.e., p ( s ′ j s; a ) 2 f 0 ; 1 g .
(cid:15) Uniform initial state distribution, i.e., p ( s 1 ) = 1 jT j ; 8 s 1 2 S 1 .
Then we have: 9 (cid:25) (cid:3) , where s.t. p (cid:25) (cid:3) ( (cid:28) )
=
1 jT j ; 8 (cid:28) 2 T .
It means that a deterministic uniformly optimal policy always exists for this MDP.
Proof.
We can prove this by construction.
The following analysis applies for any (cid:28) 2 T .
[Equation]
where we use a t ; s t as abbreviations of a ( (cid:28); t ) ; s ( (cid:28); t ) :
We denote D ( (cid:28) ) =
[Equation]
p
[Equation]
∑
[Equation]
) we can obtain a uniformly optimal policy by solving the nonlinear programming: T ∑
[Equation]
[Equation]
[Figure]
Use the condition p ( s 1 )
= 1 jT j , then we have:
[Equation]
Also the deterministic optimal policy satisﬁes the conditions in Eq (95 96).
Therefore, the deterministic optimal policy is a uniformly optimal policy.
This lemma describes one type of MDP in which UOP exists.
From the above reasoning, we can see that as long as the system of non-linear equations Eq (94 95 96) has a solution, the uniformly (near)-optimal policy exists.
Lemma 5 (Hit optimal trajectory) .
The probability that a speciﬁc optimal trajectory was not encountered given an arbitrary softmax policy (cid:25) (cid:18) is exponentially decreasing with respect to the number of training episodes. No matter a MDP has deterministic or probabilistic dynamics.
Proof.
Given a speciﬁc optimal trajectory (cid:28) =
f s ( (cid:28); t ) ; a ( (cid:28); t ) g
Tt =1 , and an arbitrary stationary policy (cid:25) (cid:18) , the probability that has never encountered at the n -th episode is [1 (cid:0) p (cid:18) ( (cid:28) )] n = (cid:24) n , based on lemma 3, we have p (cid:18) ( (cid:28) ) > 0 , therefore we have (cid:24) 2 [0 ; 1) .
9. D ISCUSSIONS OF A SSUMPTION
Intuitively, given a state and a stationary policy (cid:25) , the relative relationships among actions can be independent, considering a ﬁxed MDP M .
The relative relationship among actions is the relative relationship of actions’ return.
Starting from the same state, following a stationary policy, the actions’ return is determined by MDP properties such as environment dynamics, reward function, etc.
More concretely, we consider a MDP with three actions ( a 1 ; a 2 ; a 3 ) for each state.
The action value Q (cid:25) M satisﬁes the Bellman equation in Eq (100).
Notice that in this subsection, we use Q (cid:25) M to denote the action value that estimates the absolute value of return in M .
[Equation]
As we can see from Eq (100), Q (cid:25) M ( s; a i ) ; i = 1 ; 2 ; 3 is only related to s; (cid:25) , and environment dynamics P .
It means if (cid:25) , M and s are given, the action values of three actions are determined.
Therefore, we can use a directed graph Bishop (2006) to model the relationship of action values, as shown in Figure 5 (a).
Similarly, if we only consider the ranking of actions, this ranking is consistent with the relationship of actions’ return, which is also determined by s; (cid:25) , and P .
Therefore, the pairwise relationship among actions can be described as the directed graph in Figure 5 (b), which establishes the conditional independence of actions’ pairwise relationship.
Based on the above reasoning, we conclude that Assumption 1 is realistic.
9. E XPLORATION E FFICIENCY
The proposed off-policy learning framework indicates the sample complexity is related to exploration efﬁciency and supervised learning efﬁciency.
Given a speciﬁc MDP, the exploration efﬁciency of an exploration strategy can be quantiﬁed by how frequently we can encounter different (near)-optimal trajectories in the ﬁrst k episodes.
The supervised learning efﬁciency under the probably approximately correct framework Valiant (1984) is how many samples we need to collect so that we can achieve good generalization performance with high probability.
Jointly consider the efﬁciency in two stages, we can theoretically analyze the sample complexity of the proposed off-policy learning framework, which will be provided in the long version of this work.
Improving exploration efﬁciency is not the focus of this work.
In general, exploration efﬁciency is highly related to the properties of MDP, such as transition probabilities, horizon, action dimension, etc.
The exploration strategy should be designed according to certain domain knowledge of the MDP to improve the efﬁciency. Therefore, we did not specify our exploration strategy but adopt the state-of-the-art to conduct exploration. Based on the above discussion, we can see that how frequently we can encounter different (near)-optimal trajectories is a bottleneck of sample efﬁciency for RPG. In the MDPs with small the transition probabilities of reaching the near-optimal trajectories, we rarely collect any near-optimal trajectories during the early stage of exploration.
The beneﬁt of applying the proposed off-policy framework would be limited.
9. H YPERPARAMETERS
We present the training details of ranking policy gradient in Table 3.
The network architecture is the same as the convolution neural network used in DQN Mnih et al.
We update the RPG network every four timesteps with a minibatch of size 32.
The replay ratio is equal to eight for all baselines and RPG (except for ACER we use the default setting in openai baselines Dhariwal et al.
(2017) for better performance).
[Table]