Truly Deterministic Policy Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we present a policy gradient method that avoids exploratory noiseinjection and performs policy search over the deterministic landscape, with thegoal of improving learning with long horizons and non-local rewards.
By avoidingnoise injection all sources of estimation variance can be eliminated in systems withdeterministic dynamics (up to the initial state distribution).
Since deterministic policy regularization is impossible using traditional non-metric measures such as the
KL divergence, we derive a Wasserstein-based quadratic model for our purposes.
We state conditions on the system model under which it is possible to establish amonotonic policy improvement guarantee, propose a surrogate function for policygradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic.
Finally,we describe two novel robotic control environments—one with non-local rewardsin the frequency domain and the other with a long horizon (8000 time-steps)—forwhich our policy gradient method (TDPO) signiﬁcantly outperforms existing methods (PPO, TRPO, DDPG, and TD3).
Our implementation with all the experimentalsettings is available at https://anonymous.4open.science/r/code_tdpo-D23A.
Policy Gradient (PG) methods can be broadly characterized by three deﬁning elements: the policygradient estimator, the regularization measures, and the exploration proﬁle.
For gradient estimation,episodic [56], importance-sampling-based [48], and deterministic [52] gradients are some of the mostcommon estimation oracles.
As for regularization measures, either an Euclidean distance within theparameter space [56, 52, 33], or dimensionally consistent non-metric measures [48, 24, 50, 25, 57]have been frequently adapted.
Common exploration proﬁles include Gaussian [48] and stochasticprocesses [33].
These elements form the basis of many model-free and stochastic policy optimizationmethods successfully capable of learning high-dimensional policy parameters.
Both stochastic and deterministic policy search can be useful in applications.
A stochastic policyhas the effect of smoothing or ﬁltering the policy landscape, which is desirable for optimization.
Searching through stochastic policies has enabled the effective control of challenging environmentsunder a general framework [48, 50].
The same method could either learn robotic movements or playbasic games (1) with minimal domain-speciﬁc knowledge, (2) regardless of function approximationclasses, and (3) with less human intervention (ignoring reward engineering and hyper-parametertuning) [12].
Using stochasticity for exploration, although it imposes approximations and variance,has provided a robust way to actively search for higher rewards.
Despite many successes, there arepractical environments which remain challenging for current policy gradient methods.
For example,non-local rewards (e.g., those deﬁned in the frequency domain), long time horizons, and naturallyresonant environments all occur in realistic robotic systems [32, 36, 45] but can present issues forpolicy gradient search.
To tackle challenging environments such as these, this paper considers policy gradient methodsbased on deterministic policies and deterministic gradient estimates, which could offer advantages byallowing the estimation of global reward gradients on long horizons without the need to inject noiseinto the system for exploration.
To facilitate a dimensionally consistent and low-variance deterministic policy search, a compatible policy gradient estimator and a metric measure for regularization shouldbe employed.
For gradient estimation we focus on Vine estimators [48], which can be easily appliedto deterministic policies.
As a metric measure, we use the Wasserstein distance, which can measuremeaningful distances between deterministic policy functions that have non-overlapping supports (incontrast to the Kullback-Liebler (KL) divergence and the Total Variation (TV) distance).
The Wasserstein metric has seen substantial recent application in a variety of machine-learningdomains, such as the successful stable learning of generative adversarial models [4].
Theoretically,this metric has been studied in the context of Lipschitz-continuous Markov decision processesin reinforcement learning [20, 13].
Pirotta et al.
[43] deﬁned a policy gradient method using the
Wasserestein distance by relying on Lipschitz continuity assumptions with respect to the policygradient itself.
Asadi et al.
[5] and Rachelson and Lagoudakis [46] used the Wasserstein distance toderive model-based value-iteration and policy-iteration methods, respectively.
On a more practicalnote, Ciosek et al.
[8] introduced a method which in the deterministic mode optimized policies usingthe Wasserstein distance.
Pacchiano et al.
[41] utilized Wasserstein regularization for behavior-guidedstochastic policy optimization.
Moreover, Abdullah et al.
[1] has proposed another robust stochasticpolicy gradient formulation.
Estimating the Wasserstein distance for general distributions is morecomplicated than typical KL-divergences [54].
This fact constitutes and emphasizes the contributionsof Abdullah et al.
[1] and Pacchiano et al.
However, for our deterministic observation-conditionalpolicies, closed-form computation of Wasserstein distances is possible without any approximation.
Existing deterministic policy gradient methods (e.g., DDPG and TD3) use deterministic policies[52, 33, 14], meaning that they learn a deterministic policy function from states to actions.
However,such methods still use stochastic search (i.e., they add stochastic noise to their deterministic actions toforce exploration during policy search).
In contrast, we will be interested in a method which not onlyuses deterministic policies , but also uses deterministic search (i.e., without constant stochastic noiseinjection).
We call this method truly deterministic policy optimization (TDPO) and it may have lowerestimation variances and better scalability to long horizons, as we will show in numerical examples.
Scalability to long horizons is one of the most challenging aspects of policy gradient methods that usestochastic search.
This issue is sometimes referred to as the curse of horizon in reinforcement learning[34].
General worst-case analyses suggest that the sample complexity of reinforcement learning isexponential with respect to the horizon length [26, 30, 29].
Deriving polynomial lower-bounds for thesample complexity of reinforcement learning methods is still an open problem [23].
Lower-boundingthe sample complexity of reinforcement learning for long horizons under different settings andsimplifying assumptions has been a topic of theoretical research [9, 55].
Some recent work hasexamined the scalability of importance sampling gradient estimators to long horizons in terms of boththeoretical and practical estimator variances [34, 27, 28].
All in all, long horizons are challenging forall reinforcement learning methods, especially the ones suffering from excessive estimation variancedue to the use of stochastic policies for exploration, and our truly deterministic method may haveadvantages in this respect.
In this paper, we focus on continuous-domain robotic environments with reset capability to previouslyvisited states.
The main contributions of this work are: (1) we introduce a Deterministic Vine(DeVine) policy gradient estimator which avoids constant exploratory noise injection; (2) we derive anovel deterministically-compatible surrogate function and provide monotonic payoff improvementguarantees; (3) we show how to use the DeVine policy gradient estimator with the Wasserstein-basedsurrogate in a practical algorithm (TDPO: Truly Deterministic Policy Optimization); (4) we illustratethe robustness of the TDPO policy search process in robotic control environments with non-localrewards, long horizons, and/or resonant frequencies.
Background
MDP preliminaries.
An inﬁnite-horizon discounted Markov decision process (MDP) is speciﬁedby ( S , A , P, R, µ,  ) , where S is the state space, A is the action space, P : S ⇥ A !
( S ) is the 89 transition dynamics, R :
S ⇥ A !
[0 , R max ] is the reward function,  2 [0 , 1) is the discount factor,and µ ( s ) is the initial state distribution of interest (where  ( F ) denotes the set of all probabilitydistributions over F , otherwise known as the Credal set of F ).
The transition dynamics P is deﬁnedas an operator which produces a distribution over the state space for the next state s 0 ⇠ P ( s, a ) .
The transition dynamics can be easily generalized to take distributions of states or actions as input
(i.e., by having P deﬁned as P :  ( S ) ⇥
A !
( S ) or P : S ⇥  ( A ) !
( S ) ).
We may abuse 95 the notation and replace  s and  a by s and a , where  s and  a are the deterministic distributionsconcentrated at the state s and action a , respectively.
A policy ⇡ : S !
( A ) speciﬁes a distributionover actions for each state, and induces trajectories from a given starting state s as follows:
s 1 = s ,a 1 ⇠ ⇡ ( s 1 ) , r 1 =
R ( s 1 , a 1 ) , s 2 ⇠ P ( s 2 , a 2 ) , a 2 ⇠ ⇡ ( s 2 ) , etc.
We will denote trajectories as stateaction tuples ⌧
=
( s 1 , a 1 , s 2 , a 2 , . . . ) .
One can generalize the dynamics (1) by using a policy insteadof an action distribution P ( µ s , ⇡ ) :
=
E s ⇠ µ s [ E a ⇠ ⇡ ( s ) [ P ( s, a )]] , and (2) by introducing the t -steptransition dynamics recursively as P t ( µ s , ⇡ ) :
=
P ( P t  1 ( µ s , ⇡ ) , ⇡ ) with P 0 ( µ s , ⇡ ) :
= µ s , where µ s isa distribution over S .
The visitation frequency can be deﬁned as ⇢ ⇡ µ :=
(1   )
[Equation]
Table 2 of the Supplementary Material summarizes all MDP notation.
The value function of ⇡ is deﬁned as V ⇡ ( s ) :
= E [
P 1 t =1  t  1 r t | s 1 = s ; ⇡ ] .
Similarly, one candeﬁne Q ⇡ ( s, a ) by conditioning on the ﬁrst action.
The advantage function can then be deﬁned astheir difference (i.e. A ⇡ ( s, a ) :
= Q ⇡ ( s, a )  V ⇡ ( s ) ).
Generally, one can deﬁne the advantage/valueof one policy with respect to another using A ⇡ ( s, ⇡ 0 ) :
= E [ Q ⇡ ( s, a )  V ⇡ ( s ) | a ⇠ ⇡ 0 ( ·| s )] and
Q ⇡ ( s, ⇡ 0 ) :
= E [ Q ⇡ ( s, a ) | a ⇠ ⇡ 0 ( ·| s )] .
Finally, the payoff of a policy ⌘ ⇡ :=
E [ V ⇡ ( s ); s ⇠ µ ] is theaverage value over the initial states distribution of the MDP.
Probabilistic and mathematical notations.
Sometimes we refer to
R f ( x ) g ( x ) dx integrals ash f, g i x Hilbert space inner products.
Assuming that ⇣ and ⌫ are two probabilistic densities,the Kulback-Liebler (KL) divergence is D KL ( ⇣
| ⌫ ) :
= h ⇣ ( x ) , log( ⇣ ( x ) ⌫ ( x ) )
i x , the Total-Variation(TV) distance is TV( ⇣ , ⌫ )
=:
12 h | ⇣ ( x )  ⌫ ( x ) | , 1 i x , and the Wasserstein distance is W ( ⇣ , ⌫ ) = 114 inf  2  ( ⇣ , ⌫ ) hk x  y k ,  ( x, y ) i x,y where  ( ⇣ , ⌫ ) is the set of couplings for ⇣ and ⌫ .
We deﬁne Lip( f ( x, y ); x ) :
= sup x kr x f ( x, y ) k 2 and assume the existence of Lip( Q ⇡ ( s, a ); a ) and Lip( r s Q ⇡ ( s, a ); a ) constants.
Under this notation, the Rubinstein-Kantrovich (RK) duality statesthat the | h ⇣ ( x )  ⌫ ( x ) , f ( x ) i x |  W ( ⇣ , ⌫ ) · Lip( f ; x ) bound is tight for all f .
For brevity, we mayabuse the notation and denote sup s W ( ⇡ 1 ( ·| s ) , ⇡ 2 ( ·| s )) with W ( ⇡ 1 , ⇡ 2 ) (and similarly for othermeasures).
For parameterized policies, we deﬁne r ⇡ f ( ⇡ ) :
= r ✓ f ( ⇡ ) where ⇡ is parameterized bythe vector ✓ .
Table 1 of the Supplementary Material summarizes all these mathematical deﬁnitions.
Policy gradient preliminaries.
The advantage decomposition lemma provides insight into therelationship between payoff improvements and advantages [24].
That is,
[Equation]
We will denote the current and the candidate next policy as ⇡ 1 and ⇡ 2 , respectively.
Taking derivativesof both sides with respect to ⇡ 2 at ⇡ 1 yields
[Equation]
Since ⇡ 1 does not have any advantage over itself (i.e., A ⇡ 1 ( · , ⇡ 1 ) = 0 ), the ﬁrst term is zero.
Thus,the Policy Gradient (PG) theorem is derived as
[Equation]
For policy iteration with function approximation, we assume ⇡ 2 and ⇡ 1 to be parameterized by ✓ 2and ✓ 1 , respectively.
One can view the PG theorem as a Taylor expansion of the payoff at ✓ 1 .
A brief introduction of the Conservative Policy Iteration (CPI) [24], the Trust Region Policy Optimization (TRPO) [48], the Proximal Policy Optimization (PPO) [49], the Deep Deterministic Policy
Gradient (DDPG) [33], and the Twin-Delayed Deterministic Policy Gradient (TD3) [14] policygradient methods is left to the Supplementary Material.
Whether using deterministic policy gradients(e.g.
, DDPG and TD3) or stochastic policy gradients (e.g., TRPO and PPO), all these methods stillperform stochastic search by adding stochastic noise to the deterministic policies to force exploration.
Monotonic Policy Improvement Guarantee
We use the Wasserstein metric because it allows the effective measurement of distances betweenprobability distributions or functions with non-overlapping support, such as deterministic policies, unlike the KL divergence or TV distance which are either unbounded or maximal in this case.
Thephysical transition model’s smoothness enables the use of the Wasserstein distance to regularizedeterministic policies.
Therefore, we make the following two assumptions about the transition model:
[Equation]
Also, we make the dynamics stability assumption sup
[Equation]
1 , withthe deﬁnitions of the new constants and further discussion of the implications deferred to the
Supplementary Material where we also discuss Assumptions 4 and 5 and the existence of other
Lipschitz constants which appear as coefﬁcients in the ﬁnal lower bound.
The advantage decomposition lemma can be rewritten as
[Equation]
The h ⇢ ⇡ 2 µ  ⇢ ⇡ 1 µ , A ⇡ [Equation] · , ⇡ 2 ) i term has zero gradient at ⇡ 2 = ⇡ 1 , which qualiﬁes it to be crudely called“the second-order term”.
We dedicate a full section of our Supplementary Material to the theoreticalderivations and proofs necessary to lower-bound this second-order term into an objective in a formwell-suited for practical optimization.
Next, we present the ﬁnal bound: [Equation] ◆ [Equation]
[Equation]
For brevity, we denote the  r
[Equation]
W
[Equation]
[Equation]
s [Equation] s  2 2 expression as L G 2 ( ⇡ 1 , ⇡ 2 ; s ) in the rest of the paper.
We have ⌘ ⇡ 2  L sup ⇡ 1 ( ⇡ 2 ) and L sup ⇡ 1 ( ⇡ 1 ) = ⌘ ⇡ 1 .
This facilitates the application of Theorem 2.1 as an 152 instance of Minorization-Maximization algorithms [21].
Theorem 2.1.
Successive maximization of L sup yields non-decreasing policy payoffs.
Proof.
With ⇡ 2 = arg max ⇡ L sup ⇡ 1 ( ⇡ ) , we have L sup ⇡ 1 ( ⇡ 2 )  L sup ⇡ 1 ( ⇡ 1 ) .
Thus,⌘ ⇡ 2  L sup ⇡ 1 ( ⇡ 2 ) and ⌘ ⇡ 1 = L sup ⇡ 1 ( ⇡ 1 ) = ) ⌘ ⇡
[Equation]
[Equation]
Successive optimization of L sup ⇡ 1 ( ⇡ 2 ) generates non-decreasing payoffs.
However, it is impractical dueto the large number of “sup” terms that need to be expensively statistically estimated and implementedas constraints [48].
To mitigate this, we take a similar approach to TRPO and replace the maximumswith expectations over the observations.
The Truly Deterministic Policy Optimization (TDPO) method is given in Algorithm 1. In the basicvariant , the coefﬁcients C 1 and C 2 are constant and a trust region is used.
The coefﬁcients C 1 and C 2are dynamics-dependent and the Supplementary Material provides practical notes on their choice.
Inthe advanced variant , we use a line search similar to Schulman et al.
[48] and an adaptive tuning ofthe exploration scale using an importance sampling derivative estimate, as described in Algorithm 1. We note that, for deterministic policies, the squared Wasserstein distance W ( ⇡ 2 ( a | s ) , ⇡ 1 ( a | s ))
2degenerates to the Euclidean distance over the action space. Any policy deﬁnes a sensitivity matrixat a given state s , which is the Jacobian matrix of the policy output with respect to s .
The policysensitivity term L G 2 ( ⇡ 1 , ⇡ 2 ; s ) is essentially the squared Euclidean distance over the action-toobservation Jacobian matrix elements.
In other words, our surrogate prefers to step in directionswhere the action-to-observation sensitivity is preserved within updates.  Model-Free Estimation of Policy Gradient The DeVine advantage estimator is formally deﬁned in Algorithm 2.
Unlike DDPG and TD3, the DeVine estimator allows our method to perform deterministic search by not consistently injectingnoise in actions for exploration. Algorithm 2 uses an exploration index sampler ⌫ , which samples a set Require:
Initial policy ⇡ 0 and exploration scale  .
Require:
Advantage estimator and sample collector oracle A ⇡ of Algorithm 2.
1: for i = 0 , 1 , 2 , . . .
do 2:
Collect trajectories and construct A ⇡ i using Algorithm 2.
Compute the policy gradient g at ✓ i : g
[Equation]
Construct a surrogate Hessian vector product oracle v !
H · v such that for ✓ 0 = ✓ i + ✓
[Equation]
[Equation]
where h.o.t.
denotes higher order terms in ✓ .
Find the optimal update direction ✓ ⇤
=
H  1 g using the Conjugate Gradient algorithm.
6: (Basic Variant) Determine the best step size ↵ ⇤ within the trust region:
[Equation]
[Equation] 7: (Advanced Variant) Determine the best step size ↵ ⇤ using a line-search procedure and pick the best one; each coefﬁcient’s performance can be evaluated by sampling from the environment.
8: (Advanced Variant) Update the exploration scale  in A ⇡ using the collected samples.
Update the policy parameters: ✓ i + ✓ [Equation] + ↵ ⇤ ✓ ⇤ .
10: end for
Require:
The number of workers K , policy ⇡ , initial state distribution µ ( s ) , and discount factor  . Require: An exploration index set distribution ⌫ , exploration scale  , and maximal horizon H .
1: Sample an initial state s 0 from µ , and then roll out a trajectory ⌧ = ( s 0 , a 0 , s 1 , a 1 , · · · ) using ⇡ .
2: Sample the exploration indices set X K : = { ( t 1 , j 1 ) , ( t 2 , j 2 ) , · · · , ( t K , j K ) } from ⌫ . 3: for k = 1 , 2 , · · · , K do 4:
Compute the value V ⇡ 1 ( s t k ) =
[Equation]
[Equation] , a [Equation]
Reset the initial state to s t k , set a 0 t k : = ⇡ ( s t k ) +  · e j k , and use ⇡ for the rest of the trajectory, with e j being the j th basis element for A .
This will create ⌧ 0 =
( s t k , a 0 t k , s 0 t k +1 , a 0 t k +1 , · · · ) .
Compute the value Q ⇡ 1 ( s t k , a 0 t k ) =
[-P 1 i = t  t  i R ( s 
 i , a 0 i ) .-][Equation]
Compute the advantage A ⇡ 1 ( s t k , [Equation] t [Equation]
[Equation] Q [Equation] ( s
[Equation]
s t k
[Equation]
8: end for 9: Deﬁne A ⇡ 1 ( ⇡
[Equation]
) :=
[Equation]
K
[Equation]
10: Return A ⇡ 1 ( ⇡ 2 ) and r ⇡ 2 A ⇡ 1 ( ⇡ 2 ) as unbiased estimators for E s ⇠ ⇢ ⇡ 1 µ [ A ⇡ 1 ( s, ⇡ 2 )] and the PG.
of time-steps and action dimensions for exploration perturbation.
The truly deterministic version of
TDPO uses the deterministic ⌫ det which always returns the complete covering of { 1 , · · · , dim( A ) } ⇥{ 1 , · · · , H } .
Using ⌫ det , DeVine produces exact policy gradients in the limit of small  as stated in 177 Theorem 3.1, whose proof is deferred to the Supplementary Material.
Theorem 3.1.
Assume a ﬁnite horizon MDP with both deterministic transition dynamics P andinitial distribution µ , with maximal horizon length of H .
Deﬁne K = H · dim( A ) and ⌫ := ⌫ det ,where ⌫ det always returns the complete covering of { 1 , · · · , dim( A ) } ⇥ { 1 , · · · , H } . Then we have
[Equation]
[Equation]
Although this theorem sets the stage for computing a fully deterministic gradient, stochastic approximation can be used in Algorithm 2 by randomly sampling a small set of states from for advantageestimation.
In other words, Theorem 3.1 would use ⌫ to deterministically sample all trajectory states, whereas this is not a practical requirement for Algorithm 2 and the gradients are still unbiased if arandom set of vine branches is used.
The DeVine estimator can be advantageous in at least two scenarios.
First, in the case of rewards thatcannot be decomposed into summations of immediate rewards.
For example, overshoot penalizationsor frequency-based rewards as used in robotic systems are non-local.
DeVine can be robust tonon-local rewards as it is insensitive to whether the rewards were applied immediately or after a longperiod.
Second, DeVine can be an appropriate choice for systems that are sensitive to the injection ofnoise, such as high-bandwidth robots with natural resonant frequencies.
In such cases, using white(or colored) noise for exploration can excite these resonant frequencies and cause instability, makinglearning difﬁcult.
DeVine avoids the need for constant noise injection.
Experiments
The next three subsections show challenging robotic control tasks including frequency-based nonlocal rewards, long horizons, and sensitivity to resonant frequencies.
In Sections 4.1 and 4.2, we usethe basic variant of our method (i.e., ﬁxed exploration scale and update coefﬁcient hyper-parametersthroughout the training).
This will facilitate a better understanding of our core method’s capabilitieswithout any additional tweaks.
See the Supplementary Material for a comparison on traditional gymenvironments, where the basic variant of TDPO works similarly to existing methods.
Section 4.3includes the most difﬁcult setting in our paper, where we use the advanced variant of our method (i.e.,with line-search the update coefﬁcient and adaptive exploration scales).
4.1 An Environment with Non-Local Rewards
The ﬁrst environment that we consider is a simple pendulum.
The transition function is standard—thestates are joint angle and joint velocity, and the action is joint torque.
The reward function is non- trajectory payoffs standard—rather than deﬁne a local reward in the time domain with the goal of making the pendulumstand upright (for example), we deﬁne a non-local reward in the frequency domain with the goal ofmaking the pendulum oscillate with a desired frequency and amplitude about a desired offset.
Inparticular, we compute this non-local reward by taking the Fourier transform of the joint angle signalover the entire trajectory and by penalizing differences between the resulting power spectrum and adesired power spectrum.
We apply this non-local reward at the last time-step of the trajectory.
Allmethods here only used the current state of the systems, despite the fact that these environments arein all cases partially observable or history-dependent. Nevertheless, the agents are able to achievehigh-reward behaviors, since these environments have weak history dependence. Note that thisenvironment is a toy-problem to illustrate frequency dependence, and may as such be solved using Wavelet or short-term Fourier transformations in conjunction with the existing PG methods.
However,our focus is on the representative features of this example, rather than this particular problem itself. Implementation details and similar results for more variants are left to the Supplementary Material.
Figure 1 shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG, and
TD3.
These results were averaged over 25 experiments in which the desired oscillation frequency was1 .
7 Hz (different from the pendulum’s natural frequency of 0 .
5 Hz), the desired oscillation amplitudewas 0 .
28 rad, and the desired offset was 0 .
52 rad.
Figure 1 also shows trajectories obtained by thebest agents from each method.
TDPO (our method) was able to learn high-reward behavior and toachieve the desired frequency, amplitude, and offset.
TRPO was able to learn the correct offset butdid not produce any oscillatory behavior.
TD3 also learned the correct offset, but not the desiredoscillation.
PPO and DDPG failed to learn any desired behavior.
We hypothesize that our methodwas able to learn good behaviors here because using a deterministic policy for exploration avoidsexcitation of spurious frequencies and the DeVine estimator is accurate even for the non-local reward.
resource limitations and initial performance
4.2 An Environment with Long Horizon and Resonant Frequencies
The second environment that we consider is a single leg from a quadruped robot [42].
This leg hastwo joints, a “hip” and a “knee,” about which it is possible to exert torques.
The hip is attached to aslider that conﬁnes motion to a vertical line above ﬂat ground.
We assume the leg is dropped fromsome height above the ground and the task is to recover from this drop and to stand upright at restafter impact.
States given to the agent are the angle and velocity of each joint (slider position andvelocity are hidden), and actions are the joint torques.
The reward function penalizes difference froman upright posture, slipping or chattering at the contact between the foot and the ground, non-zerojoint velocities, and steady-state joint torque deviations.
We use the open-source MuJoCo softwarefor simulation [53], with high-ﬁdelity models of ground compliance, motor nonlinearity, and jointfriction.
The control loop rate is 4000 Hz and the rollout length is 2 s, resulting in a horizon of 8000steps.
Implementation details are left to the Supplementary Material.
Figure 2 shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG and TD3.
These results were averaged over 75 experiments.
A discount factor of  = 0 .
99975 was chosenfor all methods, where (1   )  1 is half the trajectory length.
Similarly, the GAE factors for PPOand TRPO were scaled up to 0.99875 and 0.9995, respectively, in proportion to the trajectory length.
Figure 2 also shows trajectories obtained by the best agents from each method.
TDPO (our method)was able to learn high-reward behavior.
TRPO, PPO, DDPG, and TD3 were not.
We hypothesize that the reason for this difference in performance is that TDPO better handles thecombination of two challenges presented by the leg environment—an unusually long time horizon (8000 steps) and the existence of a resonant frequency that results from compliance between the footand the ground (note the oscillations at a frequency of about 100 Hz that appear in the trajectoriesafter impact).
Both high-speed control loops and resonance due to ground compliance are commonfeatures of real-world legged robots to which TDPO seems to be more resilient.
4.3 Practical Training and Hardware Implementation
For the most realistic setting, we take the environment from the previous section and make it highlystochastic by (a) injecting physical modeling noise into the transition dynamics P , and (b) makingthe initial state distribution µ as random as physically possible.
We also systematically perform
Hyper-Parameter Optimization (HPO) on all methods to allow the most fair comparison.
The choice of the HPO method can have a signiﬁcant impact on the
RL agent’s performance.
We consider a list of ﬁve off-the-shelf HPOimplementations and run them in their default settings: Optuna [2],
BayesianOptimization [40], Scikit-Optimize [18], GPyOpt [16], and
ProSRS [51].
These implementations include a range of HPO methods, including Gaussian processes and tree Parzen estimators.
Forbetter performance, HPO methods need a reasonable set of initialhyper-parameter guesses.
For this, we perform a one-variable-ata-time parameter sweep along every hyper-parameter near the RLmethod’s default hyper-parameters.
These parameter sweep resultsare then input to each HPO method for full optimization.
Using all
HPO algorithms for all RL methods in the long-horizon environment(where each full training run takes 5 billion samples) is computationally infeasible.
To pick the best HPO method, we benchmark ashort-horizon environment with only 200 time-steps in a trajectory.
The result is shown in Figure 3 (see the Supplementary Material forfull details on the HPO methods).
Overall, we found that Optuna and ProSRS are the best HPO methods on the test problem.
Since Optuna is widely-tested and arguablythe most popular HPO library, we pick it as the main HPO method for our long-horizon environment.
We repeat the same HPO procedure on the long-horizon environment using Optuna, and pick thebest hyper-parameters found in the course of HPO for a ﬁnal training.
Figure 4 shows this ﬁnaltraining.
TDPO shows superior performance in this highly stochastic environment, and such beneﬁtscannot be obtained by merely performing HPO on other methods.
To showcase the practicality of ourmethod, we picked the best TDPO trained agent, and implemented it on the physical hardware.
Thetransferred agent was able to successfully perform drop-and-catch tests on the robot system at 4 kHz,with both global control and suppression of high-frequency transients.
Figure 5 shows a glimpse ofthis test, and a short video is also included in the code repository.
Discussion
We proposed a deterministic policy gradient method (TDPO: Truly Deterministic Policy Optimization)based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric.
We proved monotonic payoff guarantees for our method, and deﬁned a novel surrogate for policyoptimization.
We believe that using deterministic policies for exploration and avoiding the need forconsistent noise injection results in lower gradient estimation variances, enabling our method to solvetasks with longer horizons and non-local rewards.
We introduced several realistic robotic controltasks that have such features and we showed that TDPO performs well on them, in contrast to existingpolicy gradient methods. There are a number of limitations of this paper. First, we assumed continuous environments andrequired a state reset capability of the environment, which is commonly available in simulators butwould prevent learning on hardware. Second, TDPO relies on local gradients and thus may not beable to learn effectively in environments with payoffs that are sparse in either action or state.
Suchenvironments typically need substantial exploration which TDPO may not be able to achieve. Third,the analysis in this paper does not speciﬁcally pinpoint the reasons why existing methods fail on thelong-horizon non-local-reward robotic control environments. For instance, such failures might bemore speciﬁcally attributed to either (1) the frequency-based nature of the reward, (2) the non-localityof the reward signal, or (3) the long horizon. Understanding this is an important prerequisite forfuture work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] The claims are well-supported with both theoretical derivations
and numerical examples.
(b) Did you describe the limitations of your work?
[Yes] The last sentence of the discussion
section talks about the current limitation of our paper to non-discrete environments.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We dedicated a
section of the appendix to discussing potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All theoretical results
and mathematical notations are clearly stated in the form of lemmas and theorems, both in themain paper and the appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] All the theoretical theorems
and lemmas are proven either in the main paper or in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] We have included detailed
instructions to reproduce all experiments in the appendix, and our anonymized code repository
includes all the code we used to produce the paper and the appendix results.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] Yes, training details and experiment speciﬁcations are discussed at length in the appendix.
For best reproducibility, we also released the code along with the speciﬁc
hyper-parameters (in JSON format) in our anonymized code repository.
Our code repository includes all software/library version details that we used to produce the results.
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] All our experiments include 95% conﬁdence intervals, and are ran for
100 independent random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[Yes] This was left to the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the scholar articlefor the Mujoco physical simulator used in our paper. [Table]
(b) Did you mention the license of the assets?
[Yes] After de-anonymization, we will open-sourceour code with an MIT license.
We used the open-source Mujoco simulator which is availableunder the Apache 2.0 lisence. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Table]
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
[Table] (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]
[Table]