V ARIANCE R EDUCED D OMAIN R ANDOMIZATION FOR P OLICY G RADIENT
Anonymous authors
Paper under double-blind review
A BSTRACT
By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capability of generalization.
The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance due to trajectory sampling.
Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing low sample efﬁciency during the training of DR.
In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR.
We further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity in practice.
By dividing the entire space of environments into some subspaces and estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of faster convergence than the state-dependent baseline.
We conduct empirical evaluations on six robot control tasks with randomized dynamics.
The results demonstrate that VRDR can consistently accelerate the convergence of policy training in all tasks, and achieve even higher rewards in some speciﬁc tasks.
I NTRODUCTION
Deep reinforcement learning (DRL) has achieved an impressive success on complex sequential decision-making tasks, such as playing video games at top human-level (Ye et al., 2020), continuous robot manipulation (Agarwal et al., 2021) and trafﬁc signal control (Egeaa et al., 2021).
For an extensive exploration, DRL requires a massive amount of samples to train the policy, which is usually done in a simulator constructed for the task.
Due to the lack of diversity, however, policy trained by DRL tends to overﬁt to a speciﬁc training environment (Cobbe et al., 2019), causing possibly a severe performance degradation (i.e., reality gap) when the policy learned in simulator is transferred to the real deployment (Peng et al., 2018; Kang et al., 2019).
To close this gap, domain randomization (DR) has been proposed to randomize the environment parameters that may vary the dynamics and observation in simulator, imposing diversity on the trained policy (Tobin et al., 2017;
Jiang et al., 2021).
Recent success in robot control has shown that DR enables the trained policy to generalize in real deployment, even with presence of fundamental modeling errors in the simulator (Muratore et al., 2021; Xie et al., 2020).
DR has also demonstrated its capability of robustness to contend with extremely rare environments (Muratore et al., 2021).
Direct applications of DR on policy gradient methods can be thought as policy optimization on multiple randomly generated environments, which incurs additional variability of the observed data.
Therefore, a critical challenge is the low sample efﬁciency due to the extremely high variance of the gradient estimator, where the variability not only stems from the gradient approximation of expected return from a batch of sampled trajectories (Greensmith et al., 2004), but is also imposed by the randomization of environment parameters.
In standard DRL, a commonly used method to reduce variance is to construct a bias-free and action-independent baseline that is subtracted from the expected return (Sutton & Barto, 2018).
For example, a typical choice of baseline is the state value function of a policy.
When directly applied to DR, as proposed by Andrychowicz et al.
(2020), it is equivalent to learning a state value function that predicts the expected return over all possible environments.
Though remaining unbiased, such a state-dependent baseline may be a poor choice in DR, since the additional variability of randomized environments is not taken into account.
In this paper, we aim to address the high variance issue of policy gradient methods for domain randomization, with a particular focus on reducing the additional variance imposed by randomization of environments.
Our key insight is that the additional information on varying environments can be incorporated into the baseline to further reduce this variance.
Theoretically, we derive the optimal state/environment-dependent baseline, and demonstrate that it improves consistently the variance reduction over baselines that is constant or uses state information only.
In order for the practical implementation to strike the tradeoff between the variance reduction performance and computational complexity for maintaining the state/environment-dependent baselines, we propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, which can improve the sample efﬁciency while maintaining a reasonable number of baselines associated with a set of speciﬁcally designed environment subspaces.
Our main contributions can be summarized as follows
• Theoretically optimal state/environment-dependent baseline for DR.
For the policy gradient in a variety of environments that differ in dynamics, we derive theoretically an optimal baseline that depends on both the states and environments.
We further quantify the variance reduction improvement achieved by the proposed state/environment-dependent baseline over two common choices of baselines for DR, i
e., the constant and state-dependent baselines.
• Criterion for constructing practical state/subspace-dependent baseline.
Since the accurate estimation of state/environment-dependent baseline for each possible environment is infeasible during practical implementation of RL, we propose to alternatively divide the entire space of environment parameters into a limited number of subspaces, and estimate instead the optimal baseline for every pair of state and environment subspace.
We further show that the clustering of environments into subspaces should follow the policy’s expected returns on these environments, which can guarantee an improvement of variance reduction over the state-dependent baseline.
• Variance reduced domain randomization (VRDR) with empirical evaluation.
To strike the tradeoff between the variance reduction performance and computational complexity for maintaining optimal baselines, we develop a variance reduced domain randomization
(VRDR) approach for policy gradient methods.
Speciﬁcally, VRDR learns an acceptable number of baselines for each pair of state and environment subspace, where the environment subspaces are determined based on the above criterion.
We then conduct experiments on six robot control tasks with their fundamental physical parameters randomized, demonstrating that VRDR can accelerate the convergence of policy training in DR settings, as compared to the standard state-dependent baselines.
In some speciﬁc tasks, VRDR can even achieve a higher reward.
B ACKGROUND
Notation.
Under the standard reinforcement learning (RL) setting, the environment is modeled as a Markov decision process (MDP) deﬁned by a tuple < S , A , T p , Φ > , where S and A denote the state and action spaces, respectively.
For the convenience of derivation, we assume that they are ﬁnite.
T p : S ×
A × S → [0 , 1] is the environment transition model that is essentially determined by environment parameter p ∈ P , with P denoting the space of environment parameters.
In robot control, for example, environment parameter p can be a vector containing the rolling friction of each joint and the mass of torso and arms.
Throughout the rest of this paper, by environment p we mean that the environment has dynamics determined by parameter p .
Φ : S ×
A → R is the reward function.
At each time step t , the agent observes its state s t ∈ S and takes an action a t ∈
A under the guidance of policy π θ ( a t | s t ) parameterized by θ .
It will then receive a reward r t =
Φ( s t , a t ) , while the environment shifts to the next state s t +1 with probability T ( s t +1 | s t , a t , p ) .
The goal of standard RL is to search for a policy π that maximizes the expected discounted return η ( π, p ) =
E τ [ R ( τ )] over all possible trajectories τ = { s t , a t , r t , s t +1 } ∞ t =0 of states and actions, where R ( τ ) = (cid:80) ∞ t
=
γ t r t and γ ∈ [0 , 1] is the discount factor.
We can then deﬁne the state value function as V π ( s, p ) =
E (cid:2)(cid:80) ∞ k =0 γ k r t + k | s t =
s (cid:3) , the action value function as Q π ( s, a, p ) =
E (cid:2)(cid:80) ∞ k =0 γ k r t + k | s t = s, a t = a (cid:3) , and the advantage function as A π ( s, a, p ) =
Q π ( s, a, p ) − V π
( s, p ) .
Policy gradient methods for DR.
In DR, the environment parameter p is a random variable that follows the probability distribution P over P .
For the convenience of derivation, we assume a ﬁnite en- vironment parameter space P with cardinality |P| .
By introducing DR, the goal of policy optimization is to maximize the expected return over all possible environment parameters: E p ∼ P [ η ( π, p )] .
The policy gradient with action-independent baseline (Sutton & Barto, 2018) can be formulated as:
[Equation]
where we deﬁne µ pπ ( s ) = (cid:80) ∞ t =0 γ t P π ( s t = s | p ) as the discounted state visitation frequency, with P π ( s t = s | p ) denoting the probability of shifting from the initial state s 0 to state s after t steps under policy π in environment p .
For convenience, we further denote g ( θ, s, a, p ) (cid:44) ∇ θ log π θ ( a | s )
[ Q π ( s, a, p ) − b ] , which is the gradient estimator for the state-action pair under environment p .
As long as the baseline b is action-independent, we have E a [ ∇ θ log π θ ( a | s ) b ] = ∇ θ E a [ b ] = 0 and thus E P,µ pπ ,π [ g ]
=
E P,µ pπ ,π [ ∇ θ log π θ ( a | s ) Q π ( s, a, p )] (cid:44) E [ g ] , where the subscript is dropped for convenience.
Therefore, by subtracting this action-independent baseline b , the variance of gradient estimator V ar ( g ) =
E [ g T g ] − E
[ g ] T E [ g ] can be reduced without introducing any bias (Greensmith et al., 2004).
O PTIMAL B ASELINES FOR D OMAIN R ANDOMIZATION
To derive the optimal baselines for DR, we formulate the following optimization problem that aims to minimize the variance of gradient estimate w.r.t.
the baseline b (Greensmith et al., 2004): min b E
[Equation]
where we denote G ( a, s ) (cid:44) ∇ θ log π θ ( a | s ) T ∇ θ log π θ ( a | s ) .
Due to its independence of b , the second term E [ g ] T E [ g ] does not affect the minimizer, and is thus omitted on RHS of Eq.
In the following, we ﬁrst derive for DR two common choices of baselines that are constant or depend on the state only.
We then propose the optimal state/environment-dependent baseline, and show its ability to further reduce the variance incurred by the randominzation of environment parameters.
3. T WO C OMMON C HOICES OF A CTION -I NDEPENDENT B ASELINES
Optimal constant baseline.
We ﬁrst consider a constant baseline b c that depends neither on the action nor on the state.
The optimization problem in Eq.
(2) then becomes minimizing the expectation of a quadratic function, which can be proved to be convex.
Referring to the detailed derivation in
Appendix A.1, the optimal constant baseline b ∗ c for DR-based policy gradient methods is:
[Equation]
This optimal baseline b ∗ c can be understood as the expected state value function V (cid:48)
( s, p ) over all states and possible environments, where the state value function V (cid:48) ( s, p ) is computed as the weighted average of the action value function: V (cid:48) ( s [Equation]
p [Equation]
E π
[Equation]
Optimal state-dependent baseline.
As in the standard RL, the utilization of state-dependent baseline in the DR setting can be considered as an expected value function b ( s ) =
E P (cid:2)
E π [ Q π ( s, a, p )] (cid:3) for each state over all possible environments, which predicts the expected return over the distribution of dynamics caused by the variation of environment parameters.
Still referring to Appendix A.1, the optimal state-dependent baseline for DR is given as follow:
[Equation]
3. O PTIMAL S TATE /E NVIRONMENT -D EPENDENT B ASELINE
In DR, the randomization of environment parameters imposes additional variability for the estimate of gradients, which aggravates the instability of RL training process.
In the following subsections, we show that this variance can be effectively reduced by further incorporating into the baseline the information about varying environments.
To this end, we propose a state/environment-dependent baseline b ( s, P ) =
{ b ( s, p i ) } |P| i =1 for DR, and rewrite the variance minimization in Eq.
(2) as:
[Equation]
In Appendix A.1, following the similar deduction as in (Greensmith et al., 2004), the optimal state/environment baseline can be derived as:
[Equation]
which indicates requirements of maintaining a speciﬁc baseline for each environment to incorporate the environment-speciﬁc information.
Note that if environment parameter p i is time-varying along episodes and considered as stochastic input process that partially affects the dynamics, this optimal baseline b ∗ ( s, P ) is equivalent to the optimal input-driven baseline as derived by Mao et al.
V ARIANCE R EDUCTION I MPROVEMENT OF b ∗ ( s, P )
Theorem 1.
Let V ar b ( s ) ( g ) and V ar b ∗ ( s, P ) ( g ) denote the variance of gradient estimator by incorporating an arbitrary state-dependent baseline b ( s ) and the optimal state/environment-dependent baseline b ∗ ( s, P ) , respectively.
Compared to the state-dependent baseline b ( s ) , the variance can be further reduced by b ∗ ( s, P ) , with the following improvement:
V
[Equation]
Proof.
See Appendix A.2.
Guided by Theorem 1, we can then quantify the variance improvement achieved by the optimal state/environment-dependent baseline b ∗ ( s, P ) over the optimal state-dependent baseline b ∗ ( s ) by letting b ( s )
=
b ∗ ( s ) , as shown in the following corollary.
Corollary 1.
Let V ar b ∗ ( s ) ( g ) denote the variance of gradient estimator with the optimal statedependent baseline b ∗ ( s ) .
The variance reduction improvement of b ∗ ( s, P ) over b ∗ ( s ) is:
[Equation]
Since the optimal constant baseline b ∗ c can be thought as letting b ( s ) =
b ∗ c for each state s ∈ S , we show in Appendix A.3 that further variance reduction is achieved by b ∗ ( s, P ) over b ∗ c .
Therefore, we analytically justify that by incorporating the additional information about environments, the optimal state/environment-dependent baseline b ∗ ( s, P ) can obtain the minimum variance for DR within the baselines that consider both state and environment parameters.
V ARIANCE R EDUCED D OMAIN R ANDOMIZATION
For the single environment case in standard RL, the theoretically optimal state-dependent baseline is rarely used in practice due to the computational concern (Wu et al., 2018).
Rather, for both implementational and conceptual beneﬁt, a common alternative choice of b ( s ) is the state value function V ( s ) .
Similarly in DR, even though the optimal state/environment-dependent baseline is known and given in Eq.
(6), the computation of |P| baselines for all the possible environments requires a very high computational and sampling complexity, which is therefore infeasible during the practical training process.
Fortunately, similar system dynamics often emerge on environments having the similar parameters, on which the same policy will also obtain the similar expected returns.
In the following, we aim to strike a tradeoff between the variance reduction performance and the computational complexity required for maintaining multiple state/environment-dependent baselines.
Speciﬁcally, we propose to maintain an acceptable number M of state/environment-dependent baselines in practice, where M < |P| .
The entire environment parameter space P is then divided as M subspaces {P j } Mj =1 , each of which has a size |P
j | .
For each subspace P j , we can compute the optimal state-only baseline as b ∗ ( s, P j ) = E P j ( p | s )
E π
[ G ( a | s ) Q π ( s, a, p )] / E π
[ G ( a, s )] .
We then denote the state/subspace-dependent baseline as b M ( s, P ) =
{ b ∗ ( s, P j ) } Mj =1 .
Assuming a uniform sampling on P , and referring to Appendix A.4, the total variance of the gradient estimator by incorporating the state/subspace-dependent baseline b M ( s, P ) can be written as:
[Equation]
where p ji is the i -th environment parameter in subspace j with sampling distribution
|P j | , and
V ar ( g jM ) is computed over p ji ∈ P j , following s ∼ µ π ( ·| p ji ) and a ∼ π .
Theorem 2.
The variance difference of gradient estimate between the optimal state-dependent baseline and the state/subspace-dependent baseline is:
[Equation]
where Y p ( s ) (cid:44) E π [ G ( a, s ) Q π ( s, a, p )] .
For a policy π to be updated at a certain step, and assuming the uniform distribution for sampling environment parameters, we have V ar b ∗ ( s ) ( g ) ≥ V ar M ( g ) if for each subspace P j :
[Equation]
where the variance V ar P ( Y p ( s )) and V ar P j
( Y p ( s )) is computed over P and P j , respectively.
Proof.
See Appendix A.5.
Theorem 2 provides a practical guideline for constructing subspaces to reduce the variance of gradient estimate, by holding the variance of Y p ( s ) in each subspace not exceeding that in the entire environment space.
However, it still introduces extra computation to estimating G ( a, s ) in Y p ( s ) .
When the inner product of log policy G ( s, a ) is loosely correlated to the Q -value function 1 , we have E a [ G ( a, s ) Q π ( s, a, p )]
=
E a [ G ( a, s )]
E a [ Q π ( s, a, p )] , and Eq.
(11) is thus equivalent to:
[Equation]
It indicates that by clustering the entire environment space into some subspaces and holding the variance of expected return in each subspace not greater than that in the entire environment space, the state/subspace-dependent baseline can further reduce the variance of gradient estimate compared to the optimal state-only baseline.
Note that by increasing the number of subspaces, the state/subspace-dependent baseline b M ( s, P ) will approach the optimal sate/environment-dependent baseline b ∗ ( s, P ) .
However, we argue that the more baselines we employ, the more samples are required for estimating these baselines.
As will be shortly shown by the experiment, increasing M may not monotonically improve the policy’s performance during the training process.
Based on our theory, we propose a practical variance reduced domain randomization approach (VRDR) for policy gradient and summarize it in Algorithm 1, where the state/subspace-dependent baseline b ( s, P j ) =
E p ∼ P j V ( s, p ) (cid:44) V ( s, P i ) is employed for each state and subspace pair.
At each clustering Determine corresponding
Determine
Sample a set of environment parameters { p
H and determine their cluster label by min i | p
e − u i | to obtain the cluster set { Sample L trajectories Use PPO for policy optimization to get the updated policy π k +1 on trajectories, with the baseline V ( s |P i ) of the cluster that has been grouped to.
Update |P i ) on the trajectories in cluster C i .
15: end for iteration k , we sample M environments and group them into M clusters by ﬁnding the nearest subspace prototype.
Then, we train the policy on the trajectories sampled on each environment with the corresponding baseline b ( s, P j ) and update the baseline b ( s, P j ) on the corresponding trajectories.
During the clustering phase, as guided by Eq.
(12), we perform the clustering method on { p c } H − 1 c = w.r.t.
the average return ˆ V π ( p c ) to get the M clusters with corresponding subspace prototypes u (cid:48) i .
The clustering methods are usually applied to group a ﬁxed dataset, with the labeling done once and for all.
In VRDR, however, we perform clustering multiple times during training, hence the prototypes between two clustering phases would be similar but have different labels.
To avoid the case where the new prototype u (cid:48) i close to a certain current prototype is assigned with a different label, we relabel these new prototypes u (cid:48) , and update the current ones u according to u (cid:48) as described in Line 8 of Algorithm 1, where we aim to ﬁnd a set of labels l = { l i } that minimizes the distance | u (cid:48) l i − u i | between new prototype u (cid:48) and current prototype u .
Then, after the clustering phase, the subspace P j can be reconstructed according to the sampled environments in cluster C j . We utilize loss function L V = (cid:80) s t ∈ τ i | V ( s t , P i ) − (cid:80) ∞
t (cid:48) = t γ t (cid:48) − t r t | 2 to update the baseline V ( s, P i ) for each cluster C i . It has been proved by Ghadimi & Lan (2013) that the optimization of gradient variance can accelerate the convergence of gradient descent methods.
In Appendix A.7, we show that our proposed
VRDR method gains a κ = ( κ > 1) acceleration on the convergence of policy gradient methods, as compared to the direct application of optimal state-dependent baseline for DR.
E XPERIMENTS
In this section, we evaluate our proposed VRDR method on six simulated robot control tasks, where the fundamental environment parameters that affect dynamics are randomized for the generalization of trained policy.
The experimental environments are implemented based on the robot control simulator as described in (Brockman et al., 2016).
We compare VRDR with two baseline algorithms: the uniform domain randomization (DR) (Mehta et al., 2020) and the multiple value network (MVN) (Mao et al., 2018).
We utilize the proximal policy optimization (PPO) (Schulman et al., 2017) to train a generalized policy for all the comparison algorithms.
In DR, we uniformly sample the environment parameter p and collect trajectories by using the current policy on the sampled environments for policy update, where a value function that is learned from trajectories from all sampled environments is employed as the state-dependent baseline.
In MVN, multiple value functions are learned for the pre-sampled environments at the beginning of training, while each value function V i [Table] is utilized as a baseline for the gradient estimator of trajectories collected on environment parameters that are closest to the corresponding pre-sampled p i .
For VRDR, we apply hierarchical cluster method (Jain, 2010) to divide the entire environment space during training, and group the trajectories collected at each interaction phase as described in Algorithm 1. See Appendix A.14 for detail of hierarchical clustering, and Appendix A.10 for comparison of clustering methods.
We keep all the hyper-parameters about policy optimization for PPO the same as the implementation in Brockman et al.
(2016), and purely discuss the gain brought by variance reduction for gradient estimate in policy gradient methods.
Our experiments are designed to answer the following questions:
• Can VRDR effectively improve the convergence rate of the policy training.
Will the ﬁnal reward achieved by the converged policy trained by VRDR outperform the other two baseline algorithms?
• How would the policy performance of all the three algorithms degrade on the out-of-distribution unseen environments/dynamics?
• How would the speciﬁc hyper-parameters in VRDR affect the performance of training process.
5. T RAINING C URVES WITH R ANDOMIZED D YNAMICS
The six robot control tasks are as follows.
1) Walker2D : control a planar biped robot to run as fast as possible; 2) Hopper : control a planar monopod robot to hop as fast as possible; 3) Halfcheetah : control a planar cheetah robot to run fast; 4 & 5)
Pendulum & Pendulum2D : apply the force to swing a pendulum and keep it upright; 6) InvertedDoublePendulum : control a cart (attached to a two-link pendulum) to balance the whole system and keep the pendulum upright.
We modify the environment parameters in the conﬁgure ﬁle to generate six environments with the same control task and different dynamics.
The possible sampling range of the environment parameters is shown in Table 1.
Speciﬁcally, we randomize the material density that determines the mass and inertia, and the sliding friction acting along the tangent plane in Walker2D, Hopper, and Halfcheetah.
We randomize the pole length, respectively, in Pendulum, Pendulum2D and InvertedDoublePendulum, and the pole mass additionally in Pendulum2D.
We run the training process on environments sampled on the preset parameter range for the same number of iterations N until all the algorithms are converged.
At each iteration k , we generate
H = 100 environments by uniformly sampling the parameter range in all algorithms.
We run at least 15 seeds for each algorithm on all the environments to obtain the training results.
The policy and value networks in each algorithm are trained for the same number of epochs and sampled trajectories at each iteration.
We show the training curves of the six tasks, respectively, in Figs. 1(a)-1(f).
The solid curve is the average returns on all the seeds at a certain iteration, while the shaded area denotes the standard deviations.
It can be seen that by applying DR on PPO, we can improve the average return on the whole training range, while VRDR can accelerate the training process on the six tasks.
In some speciﬁc tasks, such as Walker2D, Hopper and Halfcheetah, VRDR even achieves a higher average return than DR and MVN.
In balancing tasks like Pendulum and Pendulum2D, VRDR has shown a signiﬁcant acceleration on the convergence of policy training.
5. G ENERALIZATION TO U NSEEN E NVIRONMENTS
In this subsection, we evaluate the generalization capability of trained policies on the unseen range during training.
We learn 15 policies for DR, MVN and VRDR, respectively, on training environments with different random seeds and apply the trained policies on corresponding testing unseen environments. The generalization performance for each algorithm is measured by the average score over 15 policies.
The generalization evaluation of DR, MVN and VRDR are shown in Figs. 2(a) and 2(b).
VRDR shows the best generalization performance on Walker2D. DR has a broader range of generalization than MVN, although MVN achieves higher score on the training range than DR.
[Figure] [Figure] For Pendulum2D, however, all the algorithms present a close performance.
This result indicates that the generalization performance is consistent with the average return achieved after convergence in training, and would not be affected by the convergence rate.
5. A BLATION S TUDY OF VRDR
Number of baselines M :
In Theorem 1, we show that by applying b ∗ ( s, P ) , we maintain a state/environment baseline b ∗ ( s, p i ) for each environment p i and can obtain the minimum variance of gradient estimate.
Theoretically, increasing M to |P| will achieve a better variance reduction improvement.
Figs. 3(a)-3(c) show the training curves of VRDR on Walker2D, Pendulum, and
Pendulum2D with different numbers of baselines.
Note that VRDR is reduced to DR when setting
M = 1 .
We ﬁnd that the increase of M will decrease the number of training samples for approximation of each baseline if we sample the same number of environment at each iteration, which hence retards the convergence rate of VRDR.
This suggests that we can set M to a small number in practice to strike the tradeoff between the variance reduction and computational complexity.
Clustering interval N c : Each new clustering result may change the group that each environment belong to, hence altering the value function and affecting the policy training in the next clustering interval.
The underlying dynamic mechanisms vary in different environments, and thus the shifting of value function may affect the training with varying degrees.
We show the training curves of
VRDR with different clustering interval N c on Walker2D, Hopper, and Halfcheetah in Figs.
4(a)- 4(c), and the clustering results in Appendix A.9.
The results suggest the need of a speciﬁc tuning of
N c for each task.
R ELATED W ORK
State-of-the-art methods that have endeavored to tackle the generalization and robustness of DRL, including DR, meta RL and robust RL, all implicitly or explicitly impose diversity on the policy training.
DR and meta RL both tend to learn a policy that can generalize to a variety of environments (Jiang et al., 2021; Lin et al., 2020), which may differ in dynamics and reward function during training.
By introducing an adversary that acts to disturb the dynamics or the observation of the environment during training (Zhang et al., 2020), the learned policy can gain robustness on deployment.
[Figure]
[Figure]
The introduced diversity for training environments enables the generalization of DRL policy, while the consequent high variance aggravates the sample efﬁciency of DRL training
[Figure]
For policy optimization under DR, Mehta et al.
(2020) declare that the uniform sampling of DR may lead to suboptimal and high-variance policy
[Figure]
and propose active DR to encourage the sampling distribution to select informative environments.
Several works have been proposed to optimize the sampling distribution for DR (Andrychowicz et al., 2020; Paul et al., 2019), while the baseline optimization is seldom considered in the policy search using policy gradient and the direct application of state-dependent baseline is commonly used as formulated in Eq.
Policy gradient methods usually suffer high variance due to the estimation of gradient from sampled trajectories.
By learning a Q -function (critic) instead of using its sampling estimation, actor-critic methods reduce the variance with the expense of introducing a bias (Sutton & Barto, 2018).
Another line of research follows the idea of control variate (Cheng et al., 2020), by subtracting an commonly used state-dependent baseline from the gradient estimate to reduce the variance without introducing bias.
By assuming the independence of each dimension in the action vector, Wu et al.
(2018) propose a action/state-dependent baseline, which reduces more variance than the state-only baseline.
These methods are designed for standard RL and require further discussion under the context of DR.
Variance reduced gradient algorithms like stochastic variance-reduced gradient (SVRG) (Johnson & Zhang, 2013) and momentum-based variance reduction (Cutkosky & Orabona, 2019) methods have been studied in the policy gradient for RL (Huang et al., 2020; Xu et al., 2020).
These methods, however, do not consider the variance reduction for the DR setting.
Mao et al.
(2018) propose an input-dependent baseline to reduce the variance of stochastic input process, which is similar to our state/environment-dependent baseline.
Then, the multi-value-network (MVN) approach and metalearning approach are proposed to learn the baseline.
Meanwhile, Liu et al.
(2019) propose to learn the per-task control variate and the meta control variate for each meta-RL task, where the per-task variate is constructed by the action-dependent baseline.
In essence, the MVN and per-task control variates approaches are the same, since they both aim to learn a baseline for each input or task.
And both of these meta-learning approaches are built upon model-agnostic meta learning (MAML)
(Finn et al., 2017) to scale when the number of input instantiations or tasks are large.
C ONCLUSION
In this paper, we have focused on reducing the high variance of gradient estimator in policy gradient methods for DR with additional randomness on environment parameters.
Speciﬁcally, we derived the optimal state/environment-dependent baseline, and veriﬁed that by incorporating the environment information further variance reduction could be achieved over the optimal constant or state-only baselines.
We then proposed a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity in practical implementation.
We have validated VRDR on several robot control tasks, demonstrating an overall faster convergence speed of policy training, and even better policy performance in some speciﬁc tasks.
A A PPENDIX A. D ERIVATION OF O PTIMAL B ASELINES
Following the derivation in (Greensmith et al., 2004), we now give the derivation of three kinds of optimal baselines by solving the optimization problem as proposed in Eq.
Derivation of optimal constant baseline.
It can be veriﬁed that the optimization problem in Eq.
(2) is convex w.r.t.
the constant baseline b c , and we denote:
[Equation]
By letting the ﬁrst-order derivative equal to zero:
[Equation]
14) we have the optimal constant baseline for DR as:
[Equation]
Derivation of optimal state-dependent baseline.
The variance minimization problem w.r.t.
b ( s ) is denoted as:
[Equation]
Following the generating process of joint distribution of p , s and a , we have P ( p, s, a ) =
P ( p ) P ( s | p ) P ( a | s, p )
=
P ( p ) µ π ( s | p ) π ( a | s ) , where the last equality is because that policy π is constructed without knowing p .
The RHS of Eq.
(16) can be expanded as:
[Equation] By letting the ﬁrst-order partial derivative equal to zero: [Equation]
where b ( s ) = { b ( s j ) }
[Equation]
we have:
[Equation]
b
[Equation]
It can be further written as
[Equation] = [Equation] 20) and |P| (cid:88) i =
P
[Equation]
[Equation]
where P ( s j ) can be further canceled on both sides.
We can then get the optimal state-dependent baseline for s j as
[Equation]
For a continuous state space
[Equation]
we have
[Equation]
Derivation of optimal state/environment-dependent baseline.
The variance minimization problem w.r.t.
b ( s, P ) in Eq.
(5) is denoted as:
[Equation]
By letting the ﬁrst-order partial dirivative equal to zero:
[Equation]
25) we obtain the optimal state/environment-dependent baseline
[Equation] ( s, P ) = { b ∗ ( s, p i ) } |P| i =1 , where b ∗ [Equation]
s
[Equation]
p i
[Equation]
= E π [ G
[Equation]
a
[Equation]
s
[Equation]
s
[Equation]
a, p i )] / E π
[ G ( a, s )] . A. P ROOF OF T HEOREM The variance of policy gradient using the optimal state/environment-dependent baseline b ∗ ( s, P ) is:
[Equation]
) where we denote X p ( s ) (cid:44) E π (cid:2) G ( a, s ) Q 2 π ( s, a, p ) (cid:3) , Y p ( s ) (cid:44) E π [ G ( a, s ) Q π ( s, a, p )] and Z (cid:44)
E π
[ G ( a, s )] .
For an arbitrary state-dependent baseline b ( s ) , the variance difference between b ( s ) and b ∗ ( s, P ) is:
[Equation]
A. V ARIANCE R EDUCTION I MPROVEMENT OF b ∗ ( s, P ) OVER b ∗ c
The application of optimal constant baseline b ∗ c can be considered as letting b ( s ) =
b ∗ c for each state s ∈ S .
Hence, by direct applying Theorem 1, we can obtain the variance reduction improvement of b ∗ ( s, P ) over b ∗ c as:
[Equation]
A. V ARIANCE OF P OLICY G RADIENT E STIMATOR BY A PPLYING b M ( s, P )
Under uniform domain randomization, we demonstrate that the variance of policy gradient in the whole environment space can be formulated as weighted sum of the variance in subspaces:
V ar M ( g ) =
|
[Equation]
| (cid:88) k
=
[Equation]
| (cid:88)
[Equation]
where g j is the gradient estimator of environment in subspace j and p ji is the i − th environment in subspace j .
A. P ROOF OF T HEOREM
In this subsection, we study how to divide subspace to reduce variance of gradient estimator.
First, we derive the variance difference in Eq.
For a certain state-dependent baseline b ( s ) and an arbitrary sampling distribution P of environment parameters, the variance of gradient estimate is:
[Equation]
The variance of gradient estimate by incorporating the state/subspace-dependent baseline is
[Equation] and variance of gradient estimate by incorporating the optimal state-dependent baseline is [Equation] The difference between them is
[Equation] Thus, Eq. (10) can be obtained by letting P ( p i ) = 1 |P| .
Then
[Equation]
we discuss the dividing process of subspaces to make the variance reduction improvement V ar b ∗ ( s ) ( g ) ≥ V ar M ( g ) hold.
Under the uniform sampling distribution P ( p ) = 1 |P| , we have
[Equation]
i =
[Equation]
=
[Equation]
{+The ﬁrst equality holds since (cid:80) Mj =
+}
|P j | |P| = 1 .
And
M (cid:88) j
= |P
j | |P
[Equation]
= |
[Equation]
=
[Equation]
s
[Equation]
( s )
[Equation]
=
M (cid:88) j
= |P
j | |P| |P
j | (cid:88) i =  |
[Equation]
|
[Equation]
p
[Equation]
[Equation]
=
[Equation]
j = | [Equation] | (cid:88) i =  | [Equation]
|
[Equation]
Note that P ( s | p ji ) = µ π ( s | p ji ) . Thus, as long as V ar P ( Y p ( s )) ≥ V ar P j ( Y p ( s )) , we have
V ar P
( Y p ( s ))
= (cid:80) Mj
=
|P
j | |P| V ar P
( Y p ( s )) ≥ (cid:80) Mj
=
|P j | |P| V ar P j ( Y p ( s )) . Then, combing Eq. (10), we have V ar b ∗ ( s ) ( g ) ≥ V ar M ( g ) . A. D OMAIN R ANDOMIZATION WITH A N A RBITRARY S AMPLING D ISTRIBUTION
In this subsection, we study how to divide subspace to reduce variance of gradient estimator in domain randomization with an arbitrary sampling distribution.
Speciﬁcally, we prove that for an arbitrary sampling distribution of environment parameters, as long as Eq.
(11) holds, we still have V ar b ∗ ( s ) ( g ) ≥ V ar M ( g ) . Under arbitrary sampling distribution P , the variance difference of gradient estimate between the optimal state-dependent baseline and the state/subspace-dependent baseline is: [Equation]
36) where P ( p i ) is the sampling probability of environment p i from the entire environment space P .
Hence,
P
[Equation]
p i
[Equation]
pi
[Equation]
j P ( p i ) can be used to denote the sampling probability of environment p i within the subspace P j .
For V ar P j ( Y p ( s )) , we have
[Equation]
37) and thus
M
[Equation]
[Equation]
For V ar P
( Y p ( s )) , we have
[Equation]
where the ﬁrst equality follows that (cid:80) Mj
=1 (cid:104)(cid:80) p i ∈P j P ( p i ) (cid:105) =
1 and the second quality is from the deﬁnition of V ar P
( Y p ( s )) .
Note that P ( s | p ji ) = µ π ( s | p ji ) .
Thus, as long as V ar P
( Y p ( s )) ≥ V
ar P j ( Y p ( s )) , we have (cid:80) Mj =1 (cid:104)(cid:80) p i ∈P j P ( p i ) (cid:105) V ar P ( Y p ( s )) ≥ (cid:80) Mj =1 (cid:104)(cid:80) p i ∈P j P ( p i ) (cid:105) V ar P j
( Y p ( s )) .
Then, combing
Eq.
(36), we have V ar b ∗ ( s ) ( g ) ≥ V ar M ( g ) .
A. C ONVERGENCE A NALYSIS OF VRDR
Let ∇ J ( θ ) =
∇ θ E p ∼ P [ η ( π, p )] be the policy gradient.
In this section, we present the theorem proposed by Ghadimi & Lan (2013) under the DR setting.
It shows that under Assumptionthat ∇ J ( θ ) is Lipschitz continuous w.r.t.
policy parameter θ , the variance of gradient estimator dominates the convergence rate.
Assumption 1.
[Equation]
Applying an arbitrary gradient estimator, the convergence of policy gradient is upper bounded as follows.
Theorem 3.
(Ghadimi & Lan, 2013)
Under Assumption 1, and assuming that the step size α k = min { 1 L , 1 √ V ar ( g k ) K } , we have
[Equation]
where J ( θ ∗ ) is the maximum expected cumulative discounted reward that can be achieved by policy π θ ∗ , K is the maximum number of iterations for policy gradient method, g k
= g ( θ k , s, a ) is the gradient estimator at the k -th iteration for ( s, a ) , and
[Equation]
41) is the expectation w.r.t.
distribution P R ( θ k ) =
[Equation]
and all the possible randomness.
By holding Eq.
(12), we have V ar b ∗ ( s ) ( g ) V ar M ( g ) > 1 and gains a κ = (cid:113) [Equation] ( κ > 1) acceleration on the convergence of policy gradient methods. A. A NALYSIS OF A DDITIONAL C OMPUTATIONAL C OST OF VRDR
In DR, the computational cost contains mainly the feed-forward computation and back-propagation computation of the policy network and the value network.
For N iterations, H trajectories with length of L h , size of state space |S| and constant computational cost of feed-forward and backpropagation computation d 1 , the computational cost of DR is O ( N · H · L h · |S| · d 1 ) . The additional computational cost introduced by VRDR is mainly from the clustering phase. Considering the hierachical clustering method in Algorithm 2. The maximum number of outer loop is H − M < H .
At each outer loop, the computational cost of distance matrix is m 2 ≤ H 2 .
Then, let the cost of distance computation in Line 6 of Algorithm 2 be d 2 , the computational cost is
O ( N · H 3 · d 2 ) .
Note that compared with the simple computation of distance between two scalars u i and u j , the computational cost of feed-forward and back-propagation computation are implemented for the entire policy and value networks, both of which contain multiple layers and neurons.
Therefore, in practice, d 1 is usually multiple order-of-magnitude larger than d 2 .
Thus, under the condition that
L h and |S| are comparable with H , term O ( N · H · L h · |S| · d 1 ) will become the dominate term in the overall computational complexity of VRDR, while the additional computational complexity introduced by the clustering phase of VRDR can be neglected.
A. C LUSTERING R ESULTS D URING T RAINING
In this subsection, we show the clustering results of three successive clustering updates (iteration k = nN c , ( n + 1) N c , ( n + 2) N c ) on Walker2d, Hopper, and Halfcheetah in Figs. 5(a)-5(c). A. A BLATION S TUDY OF VRDR ON C LUSTERING METHODS
In this subsection, we study the inﬂuence of different clustering methods on VRDR.
Considering that we require to specify the number of clusters in VRDR and the clusters should be generated with similar state values to satisfy Eq. (12), we additionally chose k-means for the comparison to the originally adopted hierachical clustering. The comparison results of different clustering methods are shown in Fig. 6(a) and Fig. 6(b). It can be seen that applying k-means and hierachical clustering methods present similar average scores on Hopper and Halfcheetah.
A. C OMPARISON TO C OMBINED S TATE - DEPENDENT B ASELINE
The optimal state/environment baseline can be considered as combined state-dependent baseline, where combined state is composed of original state s and environment parameter p , if we treat p as part of state of the MDP.
In this subsection, we conduct experiment to evaluate DR with combined state-dependent baseline (DRCS) and the training curves are shown in Fig.
7(a) and Fig. 7(b).
The curves of DRCS show that applying regular state-dependent baseline on this combined state degrades the average score during training as compared to DR and VRDR.
[Table] [Equation] [Table] [Figure]
n
[Figure]
) N C
[Figure]
friction
[Figure]
e n s i t
y k = nN C
[Figure]
n
[Figure]
A.
C OMPARISON TO O THER B ASELINE ALGORITHMS
In this subsection, we compare VRDR with other baseline algorithms including DR with meta baseline (DRMB) (Mao et al., 2018; Liu et al., 2019) and Epopt (Rajeswaran et al., 2016).
We use PPO with the same setting as in Section 5.
In DRMB, a meta baseline is applied to learn the simpler
[Figure]
[Figure]
[Figure]
alternative E π
[ Q π ( s, a, p )] instead of state/environment-dependent baseline b ∗ ( s, p ) .
The training curves are shown in Fig. 8(a) and Fig. 8(b).
VRDR shows better average scores than DRMB on
Walker2d and a similar average score on Hopper.
In Epopt, we use purely trajectories from the α -percentile worst-case environments for training, where we set α = 5 .
The training curves are shown in Fig.
9(a) and Fig.9(b).
And VRDR outperform than Epopt on both environments.
[Figure]
[Figure]
A. V ARIANCE OF G RADIENT E STIMATES D URING T RAINING
In this subsection
[Figure]
we empirically evaluate the variance of the gradient estimates that are practically achieved by DR, MVN and VRDR on Pendulum2D. It can be seen that at the beginning of training
[Figure]
VRDR can achieve a signiﬁcantly lower variance of gradient estimates than the other two baselines, which coincides with the faster convergence behavior of VRDR as shown in Fig. 1(e).
A. H IERACHICAL C LUSTERING FOR VRDR
In this subsection
[Figure]
we detail the hierarchical clustering method used in the implementation of VRDR, and summarize it in Algorithm
[Figure]