S UPERVISED Q-L EARNING CAN BE A S TRONG B ASELINE FOR C ONTINUOUS C ONTROL
A BSTRACT
Policy gradient (PG) algorithms have been widely used in reinforcement learning ( RL
However, PG algorithms rely on exploiting the value function being learned with the first- order update locally , which results in limited sample efficiency.
In this work we propose an alternative method called Zeroth-Order Supervised Policy
Improvement (ZOSPI).
ZOSPI exploits the estimated value function Q globally while preserving the local exploitation of the PG methods based on zeroth-order policy optimization.
This learning paradigm follows Q-learning but overcomes the difficulty of efficiently operating argmax in continuous action space.
It finds maxvalued action within a small number of samples.
The policy learning of ZOSPI has two steps: First, it samples actions and evaluates those actions with a learned value estimator and then it learns to perform the action with the highest value through supervised learning.
We further demonstrate such a supervised learning framework can learn multi-modal policies.
Experiments show that ZOSPI achieves competitive results on the continuous control benchmarks with a remarkable sample efficiency
I NTRODUCTION
Model-free Reinforcement Learning achieves great successes in many challenging tasks Mnih et al. ; Vinyals et al. Pachocki et al. , however one hurdle for its applicability to realworld control problems is the low sample efficiency.
To improve the sample efficiency, off-policy methods Degris et al.
; Gu et al.
; Wang et al. ; Lillicrap et al. ; Fujimoto et al. (2018a) reuse the experiences generated by previous policies to optimize the current policy, therefore can obtain a much higher sample efficiency than the on-policy methods Schulman et al.
Alternatively, SAC Haarnoja et al. (2018) improves sample efficiency by conducting more active exploration with maximum entropy regularizer Haarnoja et al. (2017) to the off-policy actor critic Degris et al. ; Zhang et al.
(2019a).OAC Ciosek et al. (2019) further improves SAC by combining it with the Upper Confidence Bound heuristics Brafman and Tennenholtz (2002) to incentivize more informed exploration.
Nonetheless, all of those previous methods rely on Gaussianparameterized policies and conducts local exploration strategies that simply add noises on the action space.
Consequentially, those methods can converge to sub-optimal solutions, as has been shown in (Tessler et al., 2019).
In this work, we explore an alternative approach to the conventional policy gradient paradigm for continuous control.
We introduce a zeroth-order method that can be optimized through supervised learning.
The proposed method carries out non-local exploration through global value-function exploitation to achieve higher sample efficiency for continuous control tasks.
To better exploit the learned value function Q , we propose to search the action space globally for a better target action, which is in contrast to previous policy gradient methods that only utilize the local information of the learned value functions (e.g. the Jacobian matrix Silver et al.
The key insight is to improve sample efficiency at expense of additional computation, which is acceptable as samples are much more expensive than computations in real-world applications such as Healthcare and Robotics.
[Figure]
The idea behind our work is related to the value-based policy gradient methods Lillicrap et al. Fujimoto et al. (2018a), where the policy gradient optimization step takes the role of finding a well-performing action given a learned state-action value function.
In previous work, the policy gradient step tackles the curse of dimensionality for deep Q-learning since it is intractable to directly search for the maximal value in the continuous action space Mnih et al. (2015).
Differently, we circumvent searching for the action with the maximal value in the continuous action space by finding the max-valued action within a small set of sampled actions.
A Motivating Example
To perform a global exploitation of the learned value function Q , we propose to apply a zeroth-order optimization scheme and update the target policy through supervised learning, which is inspired by works of evolution strategies Salimans et al. ; Conti et al. ; Mania et al. (2018) that adopt zeroth-order optimizations in the parameter space.
Figureshows a motivating example that demonstrates the difference between the zeroth-order and first-order optimizations.
Different from the standard policy gradient, combining the zeroth-order optimization with supervised learning forms a new way of policy update.
Such an update avoids the local improvement of policy gradient: when policy gradient is applied, the target policy uses policy gradient to adjust its predictions according to the deterministic policy gradient theorem Silver et al. (2014), but such adjustments can only lead to local improvements and may induce sub-optimal policies due to the non-convexity of the policy function Tessler et al. (2019); on the contrary, our integrated approach of zeroth-order optimization and supervised learning can greatly improve the non-convex policy optimization and more likely escape the potential local minimum.
Contributions of this paper can be summarized as follows:
We introduce a simple yet effective policy optimization method called Zeroth-Order Supervised Policy Improvement (ZOSPI), as an alternative to the policy gradient approaches for continuous control.
The policy from ZOSPI exploits global information of the learned value function Q and updates itself through sample-based supervised learning.
Practically, we point out the flexibility of the proposed supervised learning paradigm and show ZOSPI is capable of working with multi-modal policy class.
And introduce a hybrid method that leverages both zeroth-order and first-order gradients for policy learning.
Empirically, we demonstrate the improved performance of ZOSPI over policy gradient methods on the continuous control benchmarks, in terms of both higher sample efficiency and asymptotic performance.  R ELATED W ORK
Policy Gradient Methods
The policy gradient methods solve an MDP by directly optimizing the policy to maximize the cumulative reward Williams (1992); Sutton and Barto (1998).
While the prominent on-policy policy gradient methods like TRPO Schulman et al. and PPO Schulman et al. (2017) improve the learning stability of vanilla policy gradient Williams (1992) via trust region updates, the off-policy methods such as Degris et al. ; Lillicrap et al.
; Wang et al. (2016) employ an experience replay mechanism to achieve higher sample efficiency. The work of TD3 Fujimoto et al. (2018a) further addresses the function approximation error and boosts the stability of DDPG with several improvements.
Another line of works is the combination of policy gradient methods and the max-entropy regularizer, which leads to better exploration and stable asymptotic performances Haarnoja et al.
All of these approaches adopt function approximators Sutton et al. (2000) for state or state-action value estimation and take directionallyuninformed Gaussian as the policy parameterization, which lead to a local exploration behavior Ciosek et al.
; Tessler et al. RL by Supervised Learning Iterative supervised learning and self-imitate learning are becoming an alternative approach for model-free RL.
Instead of applying policy gradient for policy improvement, methods based on supervised methods update policies by minimizing the mean square error between target actions and current actions predicted by a policy network Sun et al. (2019a;b); Liu et al. (2021), or alternatively by maximizing the likelihood for a stochastic policy class Ghosh et al.
While those previous works focus on the Goal-Conditioned tasks in RL, in this work we aim to tackle more general RL tasks.
Some other works use supervised learning to optimize the policy towards manually selected policies to achieve better training stability under both offline Wang et al. Yang et al. and online settings Zhang et al. (2019b)
; Abdolmaleki et al. ; Song et al. Differently, in our work the policy learning is based on a much simpler formulation than the previous attempts Abdolmaleki et al. ; Lim et al.
Simmons-Edler et al.
, and it does not rely on any expert data to achieve competitive performance. Zeroth-Order Methods Zeroth-order optimization methods, also called gradient-free methods, are widely used when it is difficult to compute the gradients.
They approximate the local gradient with random samples around the current estimate. The works in Wang et al.
Golovin et al. (2019) show that a local zeroth-order optimization method has a convergence rate that depends logarithmically on the ambient dimension of the problem under some sparsity assumptions.
It can also efficiently escape saddle points in non-convex optimizations Vlatakis-Gkaragkounis et al.
Bai et al. In RL, many studies have verified an improved sample efficiency of zeroth-order optimization Usunier et al. ; Mania et al. ; Salimans et al.
In this work we provide a novel way of combining the local sampling and the global sampling to ensure that our algorithm can approximate the gradient descent locally and also find a better global region.
P RELIMINARIES 3. M ARKOV D ECISION P ROCESSES
We consider the deterministic Markov Decision Process (MDP) with continuous state and action spaces in the discounted infinite-horizon setting.
Such MDPs can be denoted as M = ( S , A , T , r, γ ) , where the state space S and the action space A are continuous, and the unknown state transition probability representing the transition dynamics is denoted by T :
S × A (cid:55)→ S .[-r : S ⇥ A 7! 
-]
r : S × A (cid:55)→ [0 , 1] is the reward function and γ ∈ [0 , 1] is the discount factor.
An MDP M and a learning algorithm operating on M with an arbitrary initial state s 0 ∈ S constitute a stochastic process described sequentially by the state s t visited at time step t , the action a t chosen by the algorithm at step t , the reward r t = r
( s t , a t ) and the next state s t
+1 =
T ( s t , a t ) for any t = 0 , . .
, T .
Let
H t = { s 0 , a 0 , r 0 , . . .
, s t , a t , r t } be the trajectory up to time t .
Our algorithm finds the policy that maximizes the discounted cumulative rewards (cid:80) Tt
=
γ t r t .
Our work follows the general Actor-Critic framework Konda and Tsitsiklis (2000); Peters and Schaal (2008); Degris et al.
; Wang et al.
(2016), which learns in an unknown environment using a value network denoted by Q w t : S ×A (cid:55)→ R for estimating Q values and a policy network for learning the behavior policy π θ t : S (cid:55)→ A .
Here w t and θ t are the parameters of these two networks at step t , respectively.
3. M IXTURE D ENSITY N ETWORKS
The optimal policies in many environments tend to have multi-modal property.
Mixture Density
Networks (MDN), previously proposed in Bishop (1994) to solve the stochastic prediction problem, is a potential multi-modal policy function to be introduced in this work.
One challenge here is that previous policy gradient methods are not compatible with multi-modal policies based on MDN.
While in normal regression tasks, the learning objective is to maximize the log-likelihood of observed data given the current parameters of a predictive model, MDN uses the mixture of Gaussian with multiple parameters as D =
{ X i , Y i } N .
Then the likelihood is given by
[Equation]
where θ denotes the parameters of neural networks with three branches of outputs { w i , µ i , σ i } K , ϕ is the probability density function of normal distribution and (cid:80)
Ki
{+=
+}
w i = 1 .
Z EROTH -O RDER S UPERVISED C ONTINUOUS C ONTROL
Q-learning in the tabular setting finds the best discrete action given the current state, which can be difficult in the continuous action space due to the non-convexity of Q .
A policy network is thus trained to approximate the optimal action.
However, gradient-based algorithm may be trapped by local minima or saddle point depending on the initialization.
In this section, we introduce a hybrid method with a global policy and a perturbation policy.
The global policy predicts a coarse action by supervised learning on uniformly sampled actions and the perturbation network iterates based on the coarse prediction.
In the following section, we give a motivating example to demonstrate the benefits of the hybrid framework.
Figure 1 shows a motivating example to demonstrate the benefits of applying zeroth-order optimization to policy updates.
Consider we have a learned Q function with multiple local optima.
Here we assume the conventional estimation of Q function is sufficient for a global exploitation Fujimoto et al. (2018a); Haarnoja et al. and we will discuss an improved estimation method in the Appendix.
Our deterministic policy selects a certain action at this state, denoted as the red dot in Figure 1(a). In deterministic policy gradient methods Silver et al.
; Lillicrap et al. , the policy gradient is conducted according to the chain rule to update the policy parameter θ with regard to Q -value by timing up the Jacobian matrix ∇ θ π θ ( s ) and the derivative of Q , i.e. , ∇ a Q ( s, a ) .
Consequently, the policy gradient can only guarantee to find a local minima, and similar local improvement behaviors are also observed in stochastic policy gradient methods like PPO and SAC Schulman et al.
Haarnoja et al. ; Tessler et al. ; Ciosek et al.
Instead, if we can sample sufficient random actions in a broader range of the action space, denoted as blue dots in Figure 1(b), and then evaluate their values respectively through the learned Q estimator, it is possible to find a better initialization, from which the policy gradient can more likely find the global minima.
Figure 1(c) shows the simulation result using different sample ranges for the sample-based optimization starting from the red point.
It is clear that a larger sample range improves the chance of finding the global optima.
Utilizing such a global exploitation on the learned value function is the key insight of this work.
4. Z EROTH -O RDER S UPERVISED P OLICY I MPROVEMENT
Based on the above motivation, we propose a framework called ZOSPI (Zeroth-Order Supervised
Policy Improvement).
ZOSPI consists of two policy networks: the global policy network that gives us a coarse estimate on the optimal action and a perturbation network that iterates based on the output of the global policy network using policy gradient.
Supervised learning for global policy network.
We denote the global policy network by π θ .
At any step t , we sample a set of actions uniformly over the entire action space as well as a set of actions sampled from Gaussian distribution centered at current prediction π θ t .
We denote by a + t , the action that gives the highest Q value with respect to current state s t .
Then we apply the supervised policy improvement that minimizes the L 2 distance between a + t and π θ t ( s t ) , which gives the descent direction:
[Equation]
The global samples can help finding the regions that are better in the whole space.
The local samples from Gaussian distribution accelerate later-stage training when the prediction is accurate enough and most global samples are not as good as the current prediction.
The implementation detail is shown in
Algorithm 1.
Policy gradient for perturbation network.
We introduce another perturbation network, denoted by π ϕ t , as Fujimoto et al. (2018b).
Such a perturbation network is parameterized by ϕ t that performs fine-grained control on top of the global policy network π θ t .
Different from the π θ t , the perturbation network π ϕ t takes both the current state s t and the predicted action π θ t ( s t ) as inputs, thus the final executed action is as follows:
[Equation]
The range of the outputs for the perturbation network is limited to 0 .
05 times the value of maximal action.
Therefore, it is only able to perturb the action provided by the policy network π θ t .
The perturbation network π ϕ t is trained with the policy gradient:
[Equation]
The intuition behind such an empirical design can be drawn from the example of Figure 1: although the global sampling step as well as the zeroth-order method helps policy optimization escape suboptimal regions of the non-convex value function, the first order method can help to optimize the decision afterwards inside the locally convex region more efficiently.
Eq.
(4) shows that the updates are accessed only through the gradient, ∇ a Q w .
4. A NALYSES ON THE B ENEFITS OF G LOBAL S AMPLING
In this section, we give some analyses on the benefits of global sampling in terms of the sampling efficiency.
Our analyses does not consider the improvement from the local sampling set, with which the quality of supervised learning can only be even better.
Error rate of the a + t .
The performance of the supervised policy improvement heavily depends on the goodness of a + t , the best action in the sampling set.
By assuming the continuity of the estimated
Q function, we give an upper bound on the error rate ∥ a + t − a ∗ ( s ) ∥ , where a ∗ ( s ) is the true optimal action given the current state s .
Lemma 1.
Assume the estimated Q function Q w ( s t , · ) given any s t is L -Lipschitz with respect to action inputs.
Let the action space
A ⊂ R d for some positive integer d and assume that any action a ∈ A , ∥ a ∥ 2 ≤ k for k > 0 .
Then with a probability at least 1 − δ , we have
[Equation]
Though the dependence of n is 1 /n 1 /d , the estimation of a + t is sufficient for a coarse prediction, because our prediction prediction π θ aggregates all the historical information, which further reduces the error and we only need π θ in a local convex region of the global optima.
This rate can be much lower when we have a fairly better prediction and the best actions are given by the local sampling set instead.
[-w 
-]
[-Sample a mini-batch of transition tuples D 
-]
[-=1 , 2 Q w 
-]
Quality of π θ .
It is hard to directly evaluate the error rate of π θ ( s t ) because of the online stochastic gradient descent applied on a non-stationary distribution of states and a Q-function that is continuously being updated.
Thus we take a step back and consider a simpler scenario.
We assume a fixed Q function and a fixed state distribution.
We assume that the policy network is a linear model with θ ∈ R p .
We consider a global empirical risk minimizer, so called ERM, since it is easy to achieve for linear models.
Let our dataset with T samples be { ( s t , a + t ) } Tt =1 .
Our ERM estimate is defined by
[Equation]
Theorem 1.
If assumptions in Lemma 1 hold, with a high probability we have
[Equation]
where ˜ O hides all the constant and logarithmic terms.
Theorem 1 implies that though the error of a ∗ t at one step can be high, the error of our policy network can be further reduced by aggregating information from multiple steps.
However, when π θ is nonlinear functions, the quality may be worse than the case in our analyses.
That is why we need some local perturbation through policy gradient to further improve our policy.
Missing proofs in this section are given by Appendix B. Pseudo code of ZOSPI is provided in Appendix D.
[Equation] C ONTROL WITH ZOSPI
Different from standard policy gradient methods, the policy optimization step in ZOSPI can be considered as sampling-based supervised learning.
Such a design enables many extensions of the proposed learning paradigm.
In this section, we introduce the combination of ZOSPI and first-order method for stabilized training and the combination of ZOSPI and MDN for multi-modal policy learning.
𝒂𝒂𝒂𝓠 𝝅 𝑷𝑮 𝒑𝒑𝒑𝝅 𝑫𝑷𝑮 𝓐 𝝅 𝒁𝑶𝑺𝑷𝑰 W/ MDN 𝒂𝒂𝒂𝝅 𝑫𝑷𝑮 𝝅 𝑷𝑮 𝝅 𝒁𝑶𝑺𝑷𝑰 when multiple optimal actions exist for a certain state: (left) Supermario is going to collect the mushroom, both a 2 and a 3 are the optimal actions for the current location.
While deterministic policy gradient methods (green color) are only able to learn one of those optimal actions, ZOSPI with MDN is able to learn both. Policy gradient methods normally learn a Gaussian policy class. Learning Multi-Modal Policies with
Mixture Density Networks
In the context of RL, there might be multiple optimal actions for some certain states, e.g., stepping up-ward and then right-ward may lead to the same state as stepping right
and then upward, therefore both choices result in identical return in Figure 2.
However, normal deterministic policy gradient methods can not capture such multi-modality due to the limitation that the DPG theorem is not applicable to a stochastic policy class.
In this section, we further integrate
ZOSPI with the Mixture Density Networks (MDNs) Bishop (1994), which was introduced for multi-modal regression.
Applying MDNs to ZOSPI leads to a more flexible and interpretable multi-modal policy class Tessler et al. (2019): different from normal Dirac policy parameterization used in TD3, ZOSPI with MDN predicts a mixture of Dirac policies, i.e. , the policy π MDN ( s ) predicts K choices for action ⃗a
= { a 1 , ..., a K } ∈ A K , with their corresponding probability ⃗p = { p 1 , ..., p K } ∈ R K , and (cid:80)
Ki p
i
= 1 .
Then the action is sampled by
[Equation]
[Figure]
is able to reach
Thereby, instead of learning the mean value of multiple optimal actions, ZOSPI with MDN is able to learn multiple optimal actions.
Different from previous stochastic multi-modal policies discussed in Haarnoja et al. (2018), the learning of multi-modal policy does not aim to fit the entire Q function distribution.
Rather, ZOSPI with MDN focuses on learning the multi-modality in the best choices of actions, thus addresses the difficulty in generating multi-modal policies Haarnoja et al. Tessler et al.
Figure 2 illustrates the difference between ZOSPI with MDN and the previous policy gradient methods.
E XPERIMENTS
In this section, we conduct experiments on five MuJoCo locomotion benchmarks to demonstrate the effectiveness of the proposed method.
Specifically, we validate the following statements:
If we use ZOSPI with locally sampled actions, the performance of ZOSPI should be the same as its policy gradient counterpart like TD3; if we increase the sampling range, ZOSPI can better exploit the Q function thus find better solution than the policy gradient methods. 2.
If we continuously increase the sampling range, it will result in an uniform sampling, and the Q function can be maximally exploited.
The perturbation network can help to improve the sample efficiency of the primal ZOSPI frame work purely based on zeroth-order optimization.
The supervised learning policy update paradigm of ZOSPI permits it to be flexibly work with multi-modal policy class.
ZOSPI ON THE M U J O C O L OCOMOTION T ASKS .
We evaluate ZOSPI on the Gym locomotion tasks based on the MuJoCo engine Brockman et al. ; Todorov et al.
The five locomotion tasks are Hopper-v2, Walker2d-v2, HalfCheetahv2, Ant-v2, and Humanoid-v2.
We compare our method with TD3 and SAC, the deterministic and the stochastic SOTA policy gradient methods.
We also include PPO and OAC Ciosek et al. (2019) to better show the learning efficiency of ZOSPI.
We compare different methods within 0 .
5 M environment interactions in the two easy tasks to demonstrate the high learning efficiency of ZOSPI, and present results during 1 M interactions for the other tasks.
The results of TD3 and OAC are obtained by running the code released by the authors, the results of PPO are obtained through the
[Table] al results SAC are from the training logs Haarnoja et
The learning curves of those tasks are shown in Figure 3.
It is worth noting that in all of the results we reported, only 50 actions are sampled and it is sufficient to learn well-performing policies.
With a high sampling efficiency, ZOSPI works well in the challenging environments that have high-dimensional action spaces such as Ant-v2 and Humanoid-v2.
In all the tasks the sample efficiency is consistently improved over TD3, which is the DPG counterpart of ZOSPI.
While a total of 50 sampled actions should be very sparse in the high dimensional space, we attribute the success of ZOSPI to the generality of the policy network as well as the sparsity of meaningful actions.
For example, even in the tasks that have high dimensional action spaces, only limited dimensions of the action are crucial.
We provide quantitative comparison in Table 1.
We report both the averaged score and the standard deviation of different methods after 1 M step of interactions with the environment.
Moreover, we provide the quantitative results of ZOSPI with 0 .
5 M interactions during training to emphasize on its high sample efficiency.
In 4 out of the 5 environments (except Humanoid), ZOSPI is able to achieve on-par performance compared to TD3 and SAC, with only half number of interactions.
The last line of Table 1 reports the improvement of ZOSPI over TD3 and SAC respectively.
All results are collected by experimenting with 10 random seeds.
The last plot in Figure 3 shows the ablation study on the sampling range in ZOSPI, where a sampling method based on a zero-mean Gaussian is applied and we gradually increase its variance from 0 .to 0 .
We also evaluate the uniform sampling method with radius of 0 . 5 , which is denoted as U 0 .
5 in the Figure.
The results suggest that zeroth-order optimization with local sampling performs similarly to the policy gradient method, and increasing the sampling range can effectively improve the performance.
5. A BLATION S TUDY
In this section, we conduct a series of experiments as the ablation study to evaluate different components of ZOSPI.
Specifically, we investigate the performance of ZOSPI under different number of samples, with or without perturbation networks, and Number of Diracs (NoD) used in the policy network.
Number of Samples Used in ZOSPI
The first column of Figure 4 shows the performance of
ZOSPI when using different number of samples.
In both environments, more samples lead to better performance, but in the easier task of Hopper, there are only tiny differences, while in the task of Walker2d, using more than 50 samples noticeably improves the performance.
Trading off the improvements and the computational costs when using more samples, we choose to use 50 samples for the experiments we reported in the previous section. [Figure]
The second column of Figure 4 shows the ablation studies on the perturbation network.
While in the simpler control task of Hopper, the perturbation network helps improve the learning efficiency only a little bit, in more complex task the perturbation network is crucial for efficient learning.
This result demonstrate the effectiveness of the combination of global and local information in exploiting of the learned value function.
In all of our main experiments, we use a perturbation network with a perturbation range of 0 . 05 .
The Application of MDNs Our ablation studies on the usage of MDNs are shown in the last column of Figure 4.
In the easier task of Hopper, using a larger number of NoD in ZOSPI with
MDN only improves a little but introduce several times of extra computation expense.
However, such multi-modal policies clearly improves the performance in the complex tasks like Humanoid.
In our experiments, we find larger NoD benefits the learning for the Walker2d and Humanoid tasks, and do not improve performance in the other three environments.
More implementation details of ZOSPI with MDN can be found in Appendix E.
C ONCLUSION
In this work, we propose the method of Zeroth-Order Supervised Policy Improvement (ZOSPI) as an alternative approach of policy gradient algorithms for continuous control.
ZOSPI improves the learning efficiency of previous policy gradient learning by exploiting the learned Q functions globally and locally through sampling the action space.
Different from previous policy gradient methods, the policy optimization of ZOSPI is based on supervised learning so that the learning of actor can be implemented with regression.
Such a property enables potential extensions such as multi-modal policies, which can be seamlessly cooperated with ZOSPI.
We evaluate ZOSPI on five locomotion tasks, where it remarkably improves the performance in terms of both sample efficiency and asymptotic performance compared to previous policy gradient methods. A V ISUALIZATION OF Q-L ANDSCAPE Figure 5 shows the visualization of learned policies (actions given different states) and Q values in TD3 during training in the Pendulum-v0 environment, where the state space is 3-dim and action space is 1-dim.
The red lines indicates the selected action by the current policy.
The learned Q function are always non-convex and locally convex .
As a consequence, in many states the TD3 is not able to find globally optimal solution and local gradient information may be misleading in finding actions with the highest Q values.
[Figure]
B M ISSING P ROOFS IN S ECTION 4.
Proof of Lemma 1.
By the properties of Euclidean metric
[Figure]
we have the following lemma.
Lemma 2.
Let A ⊂ R d with Euclidean metric.
Any a ∈ A
[Figure]
∥ x ∥ 2 ≤ k .
Then there exists a set { a 1 , . .
. , a N } that is a 2 k √ d N 1 /d -covering of N .
In other words, for all a ∈ A , ∃ i ∈
[ N ] ,
[Equation]
Define A
i as the set of all the actions that is closest to a i :
[Equation]
Let P ( · ) be the probability measure of uniform distribution over A We have P ( A i ) ≥ 1 / (2 N ) .
Now since we have n uniform samples from set A .
Let N = n/ log 2 ( n/δ ) .
Lemma 3 (Coupon Collector’s problem) .
It takes O ( N log 2 ( N/δ )) rounds of random sampling to see all N distinct options with a probability at least 1 − δ .
Proof.
Consider a general sampling problem: for any finite set N with |N| = N .
For any n , whose sampling probability is p ( c ) , with a probability at least 1 − δ , it requires at most
[Equation]
Since log(1 + x ) ≥ x − 12 x 2 for all x > 0 , we have
[Equation]
Searching the whole space N with each new element being found with probability N − i N at round i , it requires at most
[Equation]
with a probability at most 1 − δ .
By Lemma 3 We have with a probability at least 1 − δ , there exists a sample in each A i described above.
To proceed, we apply the Lipschitz of the Q function, Lemma 1 follows.
Proof of Theorem 1.
Now we proceed to show Theorem 1.
We denote 2 k √ dL log 1 /d
[Equation]
n
[Equation]
n
[Equation]
d by σ 2 .
Our global policy network gives a prediction from a linear model.
Let our dataset be { s t , a + t } Tt =1 .
Let ϵ t
=
a + t − a ∗ t .
Let S =
( s 1 , . . .
, s T ) T and a + , a ∗ , ϵ be the corresponding vector for ( a + t )
Tt =1 , ( a ∗ t )
Tt =1 , ( ϵ t )
Tt =1 .
We have any ϵ i , ϵ j are independent for i ̸
=
j .
We further make an assumption that E [ ϵ t ] = 0 .
Then we immediately have E [ ϵ i
ϵ
j ] = 0 as well.
This assumption is just for simplifying the proof, we can show similar results without assume the unbiasedness as the bias can be bounded.
To proceed, let θ ∗ = arg min θ
E s ∥ s T θ − a ∗ ( s ) ∥ 22 .
Since the ERM solution is simply OLS (ordinary least square).
We have the estimate
[Equation]
S T S ) − 1 S T a + .
We have
[Equation]
Since ∥ ϵ t ∥ 2 ≤ σ 2 , we have Var( ϵ t ) ≤ σ 2 .
We observe that
Var(( S T S ) −
[Equation]
S T ϵ
[Equation]
σ 2 (
[Equation]
S T S ) −
[Equation]
= O ( σ 2 p/T ) .
The generalization error is given by
[Equation]
C O NE - STEP Z EROTH -O RDER O PTIMIZATION WITH C ONSISTENT I TERATION
Algorithm
Require
Objective function Q , domain A , current point a 0 , number of local samples n 1 , number of global samples n 2 , local scale η > 0 and step size h , number of steps m .
for t = 1 , . . .
n 2 do
Globally sampling
Sample a point uniformly in the entire space by a t 0 ∼ U A where U A is the uniform distribution over A .
for i = 1 , . . .
, m do
Locally sampling
Sample n 1 points around a t,i − 1 by ˜ a j = a t,i − 1 + µe j for e j ∼ N (0 , I d ) , j = 1 , . . .
n 1 , where N (0 , I d ) is the standard normal distribution centered at 0.
Update
{+Set a t,i = a t,i − 
+}
+ h (arg max a ∈{ ˜ a j } Q ( a ) − a t,i − 1 ) end for end for return max a ∈{ a t m } n 2 t =1 Q ( a ) .
D P SEUDO C ODE OF ZOSPI
Algorithm 3 Zeroth-Order Supervised Policy
1: Require 2:
Number of epochs M , size of mini-batch N , momentum τ > 0 .
Random initialized policy network π θ , target policy network π θ ′ , perturbation network π ϕ , θ ′ ← θ .
Two random initialized Q networks, and corresponding target networks, parameterized by w 1 , w 2 , w ′ 1 , w ′ 2 .
w ′
i ← w i .
Empty experience replay buffer D = {} .
6: for iteration = 1 , 2 , ... do 7: for t = 1 , 2 , ..., T do 8: # Interaction 9:
Rollout with a t =
π θ t ( s t ) + π ϕ t
s t , π θ t ( s t )) , store transition ( s t , a t , s t +1 , r t ) in D .
10: for epoch
= 1 , 2 , ..., M do 11:
Sample a mini-batch of transition tuples D ′
= { ( s t j , a t j , s t j +1 , r t j ) } Nj =1 .
12: # Update Q 13:
Calculate target Q value y j
= r t j + min i
=1 , 2 Q w ′
i ( s t j +1 , π θ ′ ( s t j )) .
Update w i with one step gradient descent on the loss (cid:80) j ( y j − Q w i ( s t j , a t j )) 2 , i = 1 , 2 .
15: # Update π 16:
Call Algorithm 1 for policy optimization to update θ and ϕ .
17: end for 18: θ ′ ←
[Equation]
; w ′
[Equation]
←
[Equation]
) w ′ i 19: end for 20: end for
E I MPLEMENTATION D ETAILS E. N ETWORK S TRUCTURE A ND H YPER -P ARAMS
In our experiments, we follow Fujimoto et al.
(2018a) to use a 3 -layer MLP with 256 hidden units for both critic and actor networks.
We also follow Fujimoto et al.
(2018a) to use 25000 timesteps for worm-up and use a batch-size of 256 : 1 training-interaction proportion during training.
E. M IXTURE D ENSITY N ETWORKS
Our implementation of ZOSPI with MDN is based on neural network with multiple outputs.
For
MDN with K Gaussian mixture outputs, the neural network has 3 × K -dim output.
The first Kdim units are normalized with softmax activation as the probability of selecting the Gaussians, the following K -dim units are corresponding K mean values of the Gaussians, and the last K -dim units are standard deviation of the K Gaussians.
In our experiments, we use Diracs instead of Gaussians for parameterization.
Therefore, our networks output 2 × K -dim units for each action dimension, where the first K -dimensions denote the probabilities and the latter K -dimensions denote the mean values.
In our experiments, we use K = 1 for Hopper, HalfCheetah and Ant, K = 5 for Walker2d, and K = 5 for Humanoid.
( K = 5 and K = 10 achieve on-par performance for the Humanoid environment, though the K = 10 setting spend roughly one more time computational expense).
More implementation details are provided with the code in the supplementary material.
E. R UNNING T IME OF ZOSPI
We conduct our experiments with 8 GTX TITAN X GPUs and 32 Intel(R)
Xeon(R) E5-2640 v3 @ 2.60GHz CPUs.
The wall clock time of our proposed method is roughly 3-times slower than running
TD3, without application of MDNs (i.e., NoD = 1 ).
It takes roughly 20 hours to train Hopper with 10 seeds, and takes about 120 hours to train Humanoid when NoD = 10 with 10 seeds.
[Equation]
ETTER E XPLORATION WITH B OOTSTRAPPED N ETWORKS
Sample efficient RL requires algorithms to balance exploration and exploitation.
One of the most popular way to achieve this is called optimism in face of uncertainty (OFU) Brafman and Tennenholtz (2002); Jaksch et al.
; Azar et al.
; Jin et al.
(2018), which gives an upper bound on Q estimates and applies the optimal action corresponding to the upper bound.
The optimal action a t is given by the following optimization problem:
[Equation]
where Q + is the upper confidence bound on the optimal Q function.
A guaranteed exploration performance requires both a good solution for (6) and a valid upper confidence bound.
While it is trivial to solve (6) in the tabular setting, the problem can be intractable in a continuous action space.
Therefore, as shown in the previous section, ZOSPI adopts a local set to approximate policy gradient descent methods in the local region and further applies a global sampling scheme to increase the potential chance of finding a better maxima.
As for the requirement of a valid upper confidence bound, we use bootstrapped Q networks to address the uncertainty of Q estimates as in Osband et al.
(2016; 2018); Agarwal et al.
; Kumar et al.
; Ciosek et al.
Specifically, we keep K estimates of Q , namely Q 1 , . . .
Q K with bootstrapped samples from the replay buffer.
Let Q = 1 K (cid:80)
k Q k ( s, a ) .
An upper bound Q + is
[Equation]
(7) where ϕ is the hyper-parameter controlling the failure rate of the upper bound.
Another issue is on the update of bootstrapped Q networks.
Previous methods Agarwal et al.
(2019) usually update each Q network with the following target r t + γQ k
( s t +1 , π θ t ( s t +1 )) , which violates the Bellman equation as π θ t is designed to be the optimal policy for Q + rather than Q k .
Using π θ t also introduces extra dependencies among the K estimates.
We instead employ a global random sampling method to correct the violation as
[Equation]
The correction also reinforces the argument that a global random sampling method yields a good approximation to the solution of the optimization problem (6).
The detailed algorithm is provided in
Algorithm 4 in Appendix F.1.
F. A LGORITHM 4: ZOSPI WITH B OOTSTRAPPED Q NETWORKS
Require
• The number of epochs M , the size of mini-batch N , momentum τ > 0 and the number of Bootstrapped Q-networks K .
• Random initialized policy network π θ 1 , target policy network π θ ′ 1 , θ ′ 1 ← θ 1 .
• K random initialized Q networks, and corresponding target networks, parameterized by w k, 1 , w ′ k, 1 , w ′ k, 1 ← w k, 1 for k = 1 , . . .
, K .
[Table]
[Table]
[Table]
[Table]
[Table]
G G AUSSIAN P ROCESSES FOR C ONTINUOUS C ONTROL
Different from previous policy gradient methods, the self-supervised learning paradigm of ZOSPI permits it to learn both its actor and critic with a regression formulation.
Such a property enables the learning of actor in ZOSPI to be implemented with either parametric models like neural networks or non-parametric models like Gaussian Processes (GP).
Although plenty of previous works have discussed the application of GP in RL by virtue of its natural uncertainty capture ability, most of these works are limited to model-based methods or discrete action spaces for value estimation Kuss and Rasmussen (2004); Engel et al.
; Kuss (2006); Levine et al.
; Grande et al.
Fan et al.
On the other hand, ZOSPI formulates the policy optimization in continuous control tasks as a regression objective, therefore empowers the usage of GP policy in continuous control tasks.
As a first attempt of applying GP policies in continuous control tasks, we simply alter the actor network with a GP to interact with the environment and collect data, while the value approximator is still parameterized by a neural network.
We leave the investigation of better consolidation design in future work.
[Figure]
H E XPERIMENTS ON THE F OUR -S OLUTION -M AZE .
The Four-Solution-Maze (FSM) environment is a diagnostic environment where four positive reward regions with a unit side length are placed in the middle points of 4 edges of a N × N map.
An agent starts from a uniformly initialized position in the map and can then move in the map by taking actions according to the location observations (current coordinates x and y ).
Valid actions are limited to [ − 1 , 1] for both x and y axes.
Each game consists of 2 N timesteps for the agent to navigate in the map and collect rewards.
In each timestep, the agent will receive a +10 reward if it is inside one of the 4 reward regions or a tiny penalty otherwise.
For simplicity, there are no obstacles in the map, the optimal policy thus will find the nearest reward region, directly move towards it, and stay in the region till the end.
Figure 6(a) visualizes the environment and the ground-truth optimal solution.
Although the environment is simple, we found it extremely challenging due to existence of multiple sub-optimal policies that only find some but not all four reward regions.
We do not conduct grid search on hyper-parameters of the algorithms compared in our experiments but set them to default setting across all experiments.
Though elaborated hyper-parameter tuning may benefit for certain environment.
On this environment we compare ZOSPI to on-policy and off-policy SOTA policy gradient methods in terms of the learning curves, each of which is averaged by 5 runs.
The results are presented in
Figure 6(b).
And learned policies from different methods are visualized in Figure 6(c)-6(i).
For each method we plot the predicted behaviors of its learned policy at grid points using arrows (although the environment is continuous in the state space), and show the corresponding value function of its learned policy with a colored map.
All policies and value functions are learned with 0 .
3M interactions except for SAC whose figures are learned with 1 . 2M interactions as it can find 3 out of 4 target regions when more interactions are provided.
We use 4 bootstrapped Q networks for the upper bound estimation in consideration of both better value estimation and computational cost for ZOSPI with UCB.
And in ZOSPI with GP, a GP model is used to replace the actor network in data-collection, i.e. ,exploration.
The sample efficiency of ZOSPI is much higher than that of other methods.
Noticeably ZOSPI with UCB exploration is the only method that can find the optimal solution i.e. a policy directs to the nearest region with a positive reward.
All other methods get trapped in sub-optimal solutions by moving to only part of reward regions they find instead of moving toward the nearest one.