Truly Deterministic Policy Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we present a policy gradient method that avoids exploratory noiseinjection and performs policy search over the deterministic landscape.
By avoidingnoise injection all sources of estimation variance can be eliminated in systems withdeterministic dynamics (up to the initial state distribution).
Since deterministic policy regularization is impossible using traditional non-metric measures such as the
KL divergence, we derive a Wasserstein-based quadratic model for our purposes.
We state conditions on the system model under which it is possible to establish amonotonic policy improvement guarantee, propose a surrogate function for policygradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic.
Finally,we describe two novel robotic control environments—one with non-local rewardsin the frequency domain and the other with a long horizon (8000 time-steps)—forwhich our policy gradient method (TDPO) signiﬁcantly outperforms existing methods (PPO, TRPO, DDPG, and TD3).
Our implementation with all the experimentalsettings is available at https://anonymous.4open.science/r/code_tdpo-D23A.
Policy Gradient (PG) methods can be broadly characterized by three deﬁning elements: the policygradient estimator, the regularization measures, and the exploration proﬁle.
For gradient estimation,episodic [46], importance-sampling-based [38], and deterministic [42] gradients are some of the mostcommon estimation oracles.
As for regularization measures, either an Euclidean distance within theparameter space [46, 42, 28], or dimensionally consistent non-metric measures [38, 19, 40, 20, 47]have been frequently adapted.
Common exploration proﬁles include Gaussian [38] and stochasticprocesses [28].
These elements form the basis of many model-free and stochastic policy optimizationmethods successfully capable of learning high-dimensional policy parameters.
Both stochastic and deterministic policy search can be useful in applications.
A stochastic policyhas the effect of smoothing or ﬁltering the policy landscape, which is desirable for optimization.
Searching through stochastic policies has enabled the effective control of challenging environmentsunder a general framework [38, 40].
The same method could either learn robotic movements or playbasic games (1) with minimal domain-speciﬁc knowledge, (2) regardless of function approximationclasses, and (3) with less human intervention (ignoring reward engineering and hyper-parametertuning) [8].
Using stochasticity for exploration, although it imposes approximations and variance,has provided a robust way to actively search for higher rewards.
Despite many successes, there arepractical environments which remain challenging for current policy gradient methods.
For example,non-local rewards (e.g., those deﬁned in the frequency domain), long time horizons, and naturallyresonant environments all occur in realistic robotic systems [27, 30, 36] but can present issues forpolicy gradient search.
To tackle challenging environments such as these, this paper considers policy gradient methodsbased on deterministic policies and deterministic gradient estimates, which could offer advantages byallowing the estimation of global reward gradients on long horizons without the need to inject noiseinto the system for exploration.
To facilitate a dimensionally consistent and low-variance deterministic policy search, a compatible policy gradient estimator and a metric measure for regularization shouldbe employed.
For gradient estimation we focus on Vine estimators [38], which can be easily appliedto deterministic policies.
As a metric measure we use the Wasserstein distance, which can measuremeaningful distances between deterministic policy functions that have non-overlapping supports (incontrast to the Kullback-Liebler (KL) divergence and the Total Variation (TV) distance).
The Wasserstein metric has seen substantial recent application in a variety of machine-learningdomains, such as the successful stable learning of generative adversarial models [3].
Theoretically,this metric has been studied in the context of Lipschitz-continuous Markov decision processesin reinforcement learning [16, 9].
Pirotta et al.
[35] deﬁned a policy gradient method using the
Wasserestein distance by relying on Lipschitz continuity assumptions with respect to the policygradient itself.
Furthermore, for Lipschitz-continuous Markov decision processes, Asadi et al.
[4]and Rachelson and Lagoudakis [37] used the Wasserstein distance to derive model-based valueiteration and policy-iteration methods, respectively.
On a more practical note, Pacchiano et al.
[33]utilized Wasserstein regularization for behavior-guided stochastic policy optimization.
Moreover, Abdullah et al.
[1] has proposed another robust stochastic policy gradient formulation.
Estimatingthe Wasserstein distance for general distributions is more complicated than typical KL-divergences[44].
This fact constitutes and emphasizes the contributions of Abdullah et al.
[1] and Pacchiano et al.[33].
However, for our deterministic observation-conditional policies, closed-form computation of Wasserstein distances is possible without any approximation.
Existing deterministic policy gradient methods (e.g., DDPG and TD3) use deterministic policies[42, 28, 10], meaning that they learn a deterministic policy function from states to actions.
However,such methods still use stochastic search (i.e., they add stochastic noise to their deterministic actions toforce exploration during policy search).
In contrast, we will be interested in a method which not onlyuses deterministic policies , but also uses deterministic search (i.e., without constant stochastic noiseinjection).
We call this method truly deterministic policy optimization (TDPO) and it may have lowerestimation variances and better scalability to long horizons, as we will show in numerical examples.
Scalability to long horizons is one of the most challenging aspects for policy gradient methods thatuse stochastic search.
This issue is sometimes referred to as the curse of horizon in reinforcementlearning [29].
General worst-case analyses suggests that the sample complexity of reinforcementlearning is exponential with respect to the horizon length [21, 25, 24].
Deriving polynomial lowerbounds for the sample complexity of reinforcement learning methods is still an open problem [18].
Lower-bounding the sample complexity of reinforcement learning for long horizons under differentsettings and simplifying assumptions has been a topic of theoretical research [6, 45].
Some recentwork has examined the scalability of importance sampling gradient estimators to long horizons interms of both theoretical and practical estimator variances [29, 22, 23].
All in all, long horizons arechallenging for all reinforcement learning methods, especially the ones suffering from excessiveestimation variance due to the use of stochastic policies for exploration, and our truly deterministicmethod may have advantages in this respect.
In this paper we focus on continuous-domain robotic environments with reset capability to previouslyvisited states.
The main contributions of this work are: (1) we introduce a Deterministic Vine(DeVine) policy gradient estimator which avoids constant exploratory noise injection; (2) we derive anovel deterministically-compatible surrogate function and provide monotonic payoff improvementguarantees; (3) we show how to use the DeVine policy gradient estimator with the Wasserstein-basedsurrogate in a practical algorithm (TDPO: Truly Deterministic Policy Optimization); (4) we illustratethe robustness of the TDPO policy search process in robotic control environments with non-localrewards, long horizons, and/or resonant frequencies.
Background
MDP preliminaries.
An inﬁnite-horizon discounted Markov decision process (MDP) is speciﬁedby ( S , A , P, R, µ,  ) , where S is the state space, A is the action space, P : S ⇥ A !
( S ) is the 88 transition dynamics, R :
S ⇥ A !
[0 , R max ] is the reward function,  2 [0 , 1) is the discount factor,and µ ( s ) is the initial state distribution of interest (where  ( F ) denotes the set of all probabilitydistributions over F , otherwise known as the Credal set of F ).
The transition dynamics P is deﬁnedas an operator which produces a distribution over the state space for the next state s 0 ⇠ P ( s, a ) .
The transition dynamics can be easily generalized to take distributions of states or actions as input
(i.e., by having P deﬁned as P :  ( S ) ⇥
A !
( S ) or P : S ⇥  ( A ) !
( S ) ).
We may abuse 94 the notation and replace  s and  a by s and a , where  s and  a are the deterministic distributionsconcentrated at the state s and action a , respectively.
A policy ⇡ : S !
( A ) speciﬁes a distributionover actions for each state, and induces trajectories from a given starting state s as follows:
s 1 = s ,a 1 ⇠ ⇡ ( s 1 ) , r 1 =
R ( s 1 , a 1 ) , s 2 ⇠ P ( s 2 , a 2 ) , a 2 ⇠ ⇡ ( s 2 ) , etc.
We will denote trajectories as stateaction tuples ⌧
=
( s 1 , a 1 , s 2 , a 2 , . . . ) .
One can generalize the dynamics (1) by using a policy insteadof an action distribution P ( µ s , ⇡ ) :
=
E s ⇠ µ s [ E a ⇠ ⇡ ( s ) [ P ( s, a )]] , and (2) by introducing the t -steptransition dynamics recursively as P t ( µ s , ⇡ ) :
=
P ( P t  1 ( µ s , ⇡ ) , ⇡ ) with P 0 ( µ s , ⇡ ) :
= µ s , where µ s isa distribution over S .
The visitation frequency can be deﬁned as ⇢ ⇡ µ :=
(1   )
[Equation]
Table 2 of the Supplementary Material summarizes all MDP notation.
The value function of ⇡ is deﬁned as V ⇡ ( s ) :
= E [
P 1 t =1  t  1 r t | s 1 = s ; ⇡ ] .
Similarly, one candeﬁne Q ⇡ ( s, a ) by conditioning on the ﬁrst action.
The advantage function can then be deﬁned astheir difference (i.e. A ⇡ ( s, a ) :
= Q ⇡ ( s, a )  V ⇡ ( s ) ).
Generally, one can deﬁne the advantage/value [Figure] of one policy with respect to another using A ⇡ ( s, ⇡ 0 )
= E [ Q ⇡ ( s, a )  V ⇡ ( s ) | a ⇠ ⇡ 0 ( ·| s )] and
Q ⇡ ( s, ⇡ 0 ) :
= E [ Q ⇡ ( s, a ) | a ⇠ ⇡ 0 ( ·| s )] .
Finally, the payoff of a policy ⌘ ⇡ :=
E [ V ⇡ ( s ); s ⇠ µ ] is theaverage value over the initial states distribution of the MDP.
Probabilistic and mathematical notations.
Sometimes we refer to
R f ( x ) g ( x ) dx integrals ash f, g i x Hilbert space inner products.
Assuming that ⇣ and ⌫ are two probabilistic densities,the Kulback-Liebler (KL) divergence is D KL ( ⇣
| ⌫ ) :
= h ⇣ ( x ) , log( ⇣ ( x ) ⌫ ( x ) )
i x , the Total-Variation(TV) distance is TV( ⇣ , ⌫ )
=:
12 h | ⇣ ( x )  ⌫ ( x ) | , 1 i x , and the Wasserstein distance is W ( ⇣ , ⌫ ) = 113 inf  2  ( ⇣ , ⌫ ) hk x  y k ,  ( x, y ) i x,y where  ( ⇣ , ⌫ ) is the set of couplings for ⇣ and ⌫ .
We deﬁne Lip( f ( x, y ); x ) :
= sup x kr x f ( x, y ) k 2 and assume the existence of Lip( Q ⇡ ( s, a ); a ) andk Lip( r s Q ⇡ ( s, a ); a ) k 2 constants.
Under this notation, the Rubinstein-Kantrovich (RK) dualitystates that the | h ⇣ ( x )  ⌫ ( x ) , f ( x ) i x |  W ( ⇣ , ⌫ ) · Lip( f ; x ) bound is tight for all f .
For brevity, wemay abuse the notation and denote sup s W ( ⇡ 1 ( ·| s ) , ⇡ 2 ( ·| s )) with W ( ⇡ 1 , ⇡ 2 ) (and similarly for othermeasures).
For parameterized policies, we deﬁne r ⇡ f ( ⇡ ) :
= r ✓ f ( ⇡ ) where ⇡ is parameterized bythe vector ✓ .
Table 1 of the Supplementary Material summarizes all these mathematical deﬁnitions.
Policy gradient preliminaries.
The advantage decomposition lemma provides insight into therelationship between payoff improvements and advantages [19].
That is,
[Equation]
We will denote the current and the candidate next policy as ⇡ 1 and ⇡ 2 , respectively.
Taking derivativesof both sides with respect to ⇡ 2 at ⇡ 1 yields
[Equation]
Since ⇡ 1 does not have any advantage over itself (i.e., A ⇡ 1 ( · , ⇡ 1 ) = 0 ), the ﬁrst term is zero.
Thus,the Policy Gradient (PG) theorem is derived as
[Equation]
For policy iteration with function approximation, we assume ⇡ 2 and ⇡ 1 to be parameterized by ✓ 2and ✓ 1 , respectively.
One can view the PG theorem as a Taylor expansion of the payoff at ✓ 1 .
A brief introduction of the Conservative Policy Iteration (CPI) [19], the Trust Region Policy Optimization (TRPO) [38], the Proximal Policy Optimization (PPO) [39], the Deep Deterministic Policy
Gradient (DDPG) [28], and the Twin-Delayed Deterministic Policy Gradient (TD3) [10] policygradient methods is left to the Supplementary Material.
Whether using deterministic policy gradients(e.g.
, DDPG and TD3) or stochastic policy gradients (e.g., TRPO and PPO), all these methods stillperform stochastic search by adding stochastic noise to the deterministic policies to force exploration. Reinforcement learning challenges.
Non-local rewards and soft horizon scalability are two majorchallenges in reinforcement learning.
Due to space constraints, we leave the discussion of thesechallenges with simple and intuitive numerical examples to the Supplementary Material.
The rest ofthe paper assumes the reader’s familiarity with these concepts.
Monotonic Policy Improvement Guarantee
We use the Wasserstein metric because it allows the effective measurement of distances betweenprobability distributions or functions with non-overlapping support, such as deterministic policies,unlike the KL divergence or TV distance which are either unbounded or maximal in this case.
Thephysical transition model’s smoothness enables the use of the Wasserstein distance to regularizedeterministic policies.
Therefore, we make the following two assumptions about the transition model:
[Equation]
Also, we make the dynamics stability assumption sup
[Equation]
, withthe deﬁnitions of the new constants and further discussion of the implications deferred to the
Supplementary Material where we also discuss Assumptions 4 and 5 and the existence of other
Lipschitz constants which appeare as coefﬁcients in the ﬁnal lower bound.
The advantage decomposition lemma can be rewritten as
[Equation]
The h ⇢ ⇡ 2 µ  ⇢ ⇡ 1 µ , A ⇡ 1 ( · , ⇡ 2 ) i term has zero gradient at ⇡ 2 = ⇡ 1 , which qualiﬁes it to be crudely called“the second-order term”.
We dedicate a full section of our Supplementary Material to the theoreticalderivations and proofs necessary to lower-bound this second-order term into an objective well-suitedform for practical optimization.
Next, we present the theoretical bottom line and the ﬁnal bound:
[Equation]
For brevity, we denote the  r s 0 W
[Equation]
[Equation]
[Equation]
2 2 expression as L G 2 ( ⇡ 1 , ⇡ 2 ; s ) in the rest of the paper.
We have ⌘ ⇡ 2  L sup ⇡ 1 ( ⇡ 2 ) and L sup ⇡ 1 ( ⇡ 1 ) = ⌘ ⇡ 1 .
This facilitates the application of Theorem 2.1 as an 155 instance of Minorization-Maximization algorithms [17].
Theorem 2.1.
Successive maximization of L sup yields non-decreasing policy payoffs.
Proof.
With ⇡ 2 = arg max ⇡ L sup ⇡ 1 ( ⇡ ) , we have L sup ⇡ 1 ( ⇡ 2 )  L sup ⇡ 1 ( ⇡ 1 ) .
Thus,⌘
[Equation]
and
[Equation]
Successive optimization of L sup ⇡ 1 ( ⇡ 2 ) generates non-decreasing payoffs.
However, it is impracticaldue to the large number of constraints and statistical estimation of maximums.
To mitigate this, wetake a similar approach to TRPO and replace the maximums with expectations over the observations.
The coefﬁcients C 1 and C 2 are dynamics-dependent. In the basic variant of our method, we usedconstant coefﬁcients and a trust region. This yields the Truly Deterministic Policy Optimization(TDPO) as given in Algorithm 1.
See the Supplementary Material for practical notes on the manualchoice of C 1 and C 2 .
Alternatively, one could adopt processes similar to Schulman et al.
[38] wherethe IS-based advantage estimator used a line search for proper step size selection, or the adaptivepenalty coefﬁcient setting in Schulman et al. In the advanced variant of our method, weimplement such a line search procedure by collecting samples from the environment and picking thecoefﬁcient yielding the best improvement.
This comes at an increase in the sample complexity, butthe beneﬁts can outweigh the added costs. Furthermore, the exploration scale in the sampling oracleis adaptively tuned using the collected payoffs in the advanced variant; by constructing an importancesampling derivative estimate for the exploration scale parameter, stochastic gradient descent can beused to tune the exploration scale individually.
2.1 On the interpretation of the surrogate function
For deterministic policies, the squared Wasserstein distance W ( ⇡ 2 ( a | s ) , ⇡ 1 ( a | s )) 2 degenerates tothe Euclidean distance over the action space. Any policy deﬁnes a sensitivity matrix at a given
Require:
Initial policy ⇡ 0 .
Require:
Advantage estimator and sample collector oracle A ⇡ .
1: for k = 1 , 2 , . . .
do 2:
Collect trajectories and construct the advantage estimator oracle A ⇡ k .
Compute the policy gradient g at ✓ k : g r
[Equation]
Construct a surrogate Hessian vector product oracle v !
H · v such that for ✓ 0 = ✓ k + ✓ ,
[Equation]
s
[Equation]
[Equation]
where h.o.t.
denotes higher order terms in ✓ .
Find the optimal update direction ✓ ⇤
=
H  1 g using the Conjugate Gradient algorithm.
6: (Basic Variant) Determine the best step size ↵ ⇤ within the trust region:
[Equation]
7: (Advanced Variant) Determine the best step size ↵ ⇤ using a line-search procedure and pick the best one; each coefﬁcient’s performance can be evaluated by sampling from the environment.
8: (Advanced Variant) Update the exploration scale in A ⇡ using the collected samples.
Update the policy parameters: ✓ k + ✓ k + ↵ ⇤ ✓ ⇤ .
10: end for
Algorithm
Require:
The number of parallel workers K Require: A policy ⇡ , an exploration policy q , discrete time-step distribution ⌫ ( t ) , initial state distribution µ ( s ) , and the discount factor  .
1: Sample an initial state s 0 from µ , and then roll out a trajectory ⌧ = ( s 0 , a 0 , s 1 , a 1 , · · · ) using ⇡ .
2: for k = 1 , 2 , · · · , K do 3: Sample the integer number t = t k from ⌫ .
Compute the value V ⇡ 1 ( s t ) =
P 1 i =
t  t  i R ( s i , a i ) .
Reset the initial state to s t , sample the ﬁrst action a 0 t according to q ( ·| s t ) , and use ⇡ for the rest of the trajectory.
This will create ⌧ 0 =
( s t , a 0 t , s 0 t +1 , a 0 t +1 , · · · ) .
Compute the value Q ⇡ 1 ( s t , a 0 t ) =
P 1 i =
t {+ t  i R ( s 
+}
i , a 0 i ) .
Compute the advantage A ⇡ 1 ( s t , a 0 t ) =
Q ⇡ ( s t , a 0 t )  V ⇡ ( s t ) .
8: end for 9: Deﬁne A ⇡ 1 ( ⇡
[Equation]
) :=
[Equation]
K
[Equation]
10: Return A ⇡ 1 ( ⇡ 2 ) and r ⇡ 2 A ⇡ 1 ( ⇡ 2 ) as unbiased estimators for E s ⇠ ⇢ ⇡ 1 µ [ A ⇡ 1 ( s, ⇡ 2 )] and PG, respectively.
state s , which is the Jacobian matrix of the policy output with respect to s . The policy sensitivityterm L G 2 ( ⇡ 1 , ⇡ 2 ; s ) is essentially the squared Euclidean distance over the action-to-observation Jacobian matrix elements. In other words, our surrogate prefers to step in directions where theaction-to-observation sensitivity is preserved within updates.
Although our surrogate uses a metric distance instead of the traditional non-metric measures for regularization, we do not consider this sole replacement a major contribution. The squared Wasseresteindistance and the KL divergence of two identically-scaled Gaussian distributions are the same upto a constant (i.e., D KL ( N ( m 1 ,  ) k N ( m 2 ,  )) = W ( N ( m 1 ,  ) , N ( m 2 ,  )) 2 / 2  2 ).
On the otherhand, our surrogate’s compatibility with deterministic policies makes it a valuable asset for ourpolicy gradient algorithm; both W ( ⇡ 2 ( a | s ) , ⇡ 1 ( a | s )) 2 and L G 2 ( ⇡ 1 , ⇡ 2 ; s ) can be evaluated for twodeterministic policies ⇡ 1 and ⇡ 2 numerically without any approximations to overcome singularities.  Model-Free Estimation of Policy Gradient The DeVine advantage estimator is formally deﬁned in Algorithm 2. Unlike DDPG and TD3, the
DeVine estimator allows our method to perform deterministic search by not consistently injectingnoise in actions for exploration. Under deterministic dynamics and policies, if DeVine samples eachdimension at each time-step exactly once then in the limit of small exploration scale  it can produceexact advantages, as stated in Theorem 3.1, whose proof is deferred to the Supplementary Material.
Theorem 3.1.
Assume a ﬁnite horizon MDP with both deterministic transition dynamics P andinitial distribution µ , with maximal horizon length of H .
Deﬁne K = H · dim( A ) , a uniform ⌫ , andq ( s ;  ) = ⇡ 1 ( s ) +  e j in Algorithm 2 with e j being the j th basis element for A . If the ( j, t k ) pairsare sampled to exactly cover { 1 , · · · , dim( A ) } ⇥ { 1 , · · · , H } , then we have
[Equation]
Theorem 3.1 provides a guarantee for recovering the exact policy gradient if the initial state distribution was deterministic and all time-steps of the trajectory were used to branch vine trajectories.
Although this theorem sets the stage for computing a fully deterministic gradient, stochastic approximation can be used in Algorithm 2 by randomly sampling a small set of states for advantageestimation.
In other words, Theorem 3.1 would use ⌫ to deterministically sample all trajectory states,whereas this is not a practical requirement for Algorithm 2 and the gradients are still unbiased if arandom set of vine branches is used.
The DeVine estimator can be advantageous in at least two scenarios.
First, in the case of rewards thatcannot be decomposed into summations of immediate rewards.
For example, overshoot penalizationsor frequency-based rewards as used in robotic systems are non-local.
DeVine can be robust tonon-local rewards as it is insensitive to whether the rewards were applied immediately or after a longperiod.
Second, DeVine can be an appropriate choice for systems that are sensitive to the injection ofnoise, such as high-bandwidth robots with natural resonant frequencies.
In such cases, using white(or colored) noise for exploration can excite these resonant frequencies and cause instability, makinglearning difﬁcult.
DeVine avoids the need for constant noise injection.
Experiments
The next three subsections show challenging robotic control tasks including frequency-based nonlocal rewards, long horizons, and sensitivity to resonant frequencies.
In Sections 4.1 and 4.2, we usethe basic variant of our method (i.e., ﬁxed exploration scale and update coefﬁcient hyper-parametersthroughout the training).
This will facilitate a better understanding of our core method’s capabilitieswithout any additional tweaks.
See the Supplementary Material for a comparison on traditional gymenvironments, where the basic variant of TDPO works similarly to existing methods.
Section 4.3includes the most difﬁcult setting in our paper, where we use the advanced variant of our method (i.e.,with line-search for the update coefﬁcient and adaptive exploration scales).
4.1 An Environment with Non-Local Rewards
The ﬁrst environment that we consider is a simple pendulum.
The transition function is standard—thestates are joint angle and joint velocity, and the action is joint torque.
The reward function is nonstandard—rather than deﬁne a local reward in the time domain with the goal of making the pendulumstand upright (for example), we deﬁne a non-local reward in the frequency domain with the goalof making the pendulum oscillate with a desired frequency and amplitude about a desired offset.
In particular, we compute this non-local reward by taking the Fourier transform of the joint anglesignal over the entire trajectory and by penalizing differences between the resulting power spectrumand a desired power spectrum.
We apply this non-local reward at the last time step of the trajectory.
Implementation details and similar results for more pendulum variants
Due to resource limitations and poor initial performance, we excluded TD
Figure 1 shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG, and
TD3.
These results were averaged over 25 experiments in which the desired oscillation frequency was1 .
7 Hz (different from the pendulum’s natural frequency of 0 .
5 Hz), the desired oscillation amplitudewas 0 .
28 rad, and the desired offset was 0 .
52 rad.
Figure 1 also shows trajectories obtained by thebest agents from each method.
TDPO (our method) was able to learn high-reward behavior and toachieve the desired frequency, amplitude, and offset.
TRPO was able to learn the correct offset butdid not produce any oscillatory behavior.
TD3 also learned the correct offset, but could not producedesirable oscillations.
PPO and DDPG failed to learn any desired behavior.
4.2 An Environment with Long Horizon and Resonant Frequencies 2
The second environment that we consider is a single leg from a quadruped robot [34].
This leg hastwo joints, a “hip” and a “knee,” about which it is possible to exert torques.
The hip is attached to aslider that conﬁnes motion to a vertical line above ﬂat ground.
We assume the leg is dropped fromsome height above the ground and the task is to recover from this drop and to stand upright at restafter impact.
States given to the agent are the angle and velocity of each joint (slider position andvelocity are hidden), and actions are the joint torques.
The reward function penalizes difference froman upright posture, slipping or chattering at the contact between the foot and the ground, non-zerojoint velocities, and steady-state joint torque deviations.
We use the open-source MuJoCo softwarefor simulation [43], with high-ﬁdelity models of ground compliance, motor nonlinearity, and jointfriction.
The control loop rate is 4000 Hz and the rollout length is 2 s, resulting in a horizon of 8000steps.
Implementation details are left to the Supplementary Material.
Figure 2 shows training curves for TDPO (our method) as compared to TRPO, PPO, DDPG and TD3.
These results were averaged over 75 experiments.
A discount factor of  = 0 .
99975 was chosenfor all methods, where (1   )  1 is half the trajectory length.
Similarly, the GAE factors for PPOand TRPO were scaled up to 0.99875 and 0.9995, respectively, in proportion to the trajectory length.
Figure 2 also shows trajectories obtained by the best agents from each method.
TDPO (our method)was able to learn high-reward behavior.
TRPO, PPO, DDPG, and TD3 were not.
We hypothesize that the reason for this difference in performance is that TDPO better handles thecombination of two challenges presented by the leg environment—an unusually long time horizon(8000 steps) and the existence of a resonant frequency that results from compliance between the footand the ground (note the oscillations at a frequency of about 100 Hz that appear in the trajectoriesafter impact).
Both high-speed control loops and resonance due to ground compliance are commonfeatures of real-world legged robots to which TDPO seems to be more resilient.
4.3 Practical Training and Hardware Implementation
For the most realistic setting, we take the environment from the previous section and make it highlystochastic by (a) injecting noise into the transition dynamics P , and (b) making the initial statedistribution µ as random as physically possible.
We also systematically perform Hyper-Parameter
Optimization (HPO) on all methods to allow the most fair comparison.
The choice of the HPO method can have a signiﬁcant impact on the
RL agent’s performance.
We consider a list of ﬁve off-the-shelf HPOimplementations and run them in their default settings: Optuna [2],
BayesianOptimization [32], Scikit-Optimize [14], GPyOpt [12], and
ProSRS [41].
These implementations include a range of HPO methods, including Gaussian processes and tree Parzen estimators.
Forbetter performance, HPO methods need a reasonable set of initialhyper-parameter guesses.
For this, we perform a one-variable-ata-time parameter sweep along every hyper-parameter near the RLmethod’s default hyper-parameters.
These parameter sweep resultsare then input to each HPO method for full optimization.
Using all
HPO algorithms for all RL methods in the long-horizon environment(where each full training run takes 5 billion samples) is computationally infeasible.
To pick the best HPO method, we benchmark ashort-horizon environment with only 200 time-steps in a trajectory.
The result is shown in Figure 3 (see the Supplementary Material forfull details on the HPO methods).
Overall, we found that Optunaand ProSRS are the best HPO methods on the test problem.
Since Optuna is widely-test and arguablythe most popular HPO library, we pick it as the main HPO method for our long-horizon environment.
We repeat the same HPO procedure on the long-horizon environment using Optuna, and pick thebest hyper-parameters found in the course of HPO for a ﬁnal training.
Figure 4 shows this ﬁnaltraining.
TDPO shows superior performance in this highly stochastic environment, and such beneﬁtscannot be obtained by merely performing HPO on other methods.
To showcase the practicality of ourmethod, we picked the best TDPO trained agent, and implemented it on the physical hardware.
Thetransferred agent was able to successfully perform drop-and-catch tests on the robot system at 4 kHz,with both global control and suppression of high-frequency transients.
Figure 5 shows a glimpse ofthis test, and a short video is also included in the code repository.
Discussion
We proposed a deterministic policy gradient method (TDPO: Truly Deterministic Policy Optimization)based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric.
We proved monotonic payoff guarantees for our method, and deﬁned a novel surrogate for policyoptimization.
We showed numerical evidence for superior performance with non-local rewardsdeﬁned in the frequency domain and a realistic long-horizon resonant environment.
This methodenables applications of policy gradient to customize frequency response characteristics of agents.
Our work assumed continuous environments, and future work should include the adaptation of ourmethod to environments with discrete state and action spaces.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] The claims are well-supported with both theoretical
derivations and numerical examples.
(b) Did you describe the limitations of your work?
[Yes] The last sentence of the discussion
section talks about the current limitation of our paper to non-discrete environments.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We
dedicated a section of the appendix to discussing potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All theoret-
ical results and mathematical notations are clearly stated in the form of lemmas andtheorems, both in the main paper and the appendix.(b)
Did you include complete proofs of all theoretical results?
[Yes] All the theoretical
theorems and lemmas are proven either in the main paper or in the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We have
included detailed instructions to reproduce all experiments in the appendix, and our
anonymized code repository includes all the code we used to produce the paper and the appendix results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Yes, training details and experiment speciﬁcations are discussed atlength in the appendix.
For best reproducibility, we also released the code along with
the speciﬁc hyper-parameters (in JSON format) in our anonymized code repository.
Our code repository includes all software/library version details that we used to produce the results.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] All our experiments include 95% conﬁdence intervals,
and are ran for 100 independent random seeds.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] This was left to the appendix.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the scholar article for the Mujoco physical simulator used in our paper.
(b) Did you mention the license of the assets?
[Yes] After de-anonymization, we will open-source our code with an MIT license.
We used the open-source Mujoco simulator which is available under the Apache 2.0 lisence.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]