Neural Posterior Domain Randomization
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Combining domain randomization and reinforcement learning is awidely used approach to obtain control policies that can bridge the gap betweensimulation and reality.
However, existing methods make limiting assumptions onthe form of the domain parameter distribution which prevents them from utilizingthe full power of domain randomization.
Typically, a restricted family of probability distributions (e.g., normal or uniform) is chosen a priori for every parameter.
Furthermore, straightforward approaches based on deep learning require differentiable simulators, which are either not available or can only simulate a limitedclass of systems.
Such rigid assumptions diminish the applicability of domain randomization in robotics.
Building upon recently proposed neural likelihood-free inference methods, we introduce Neural Posterior Domain Randomization (NPDR),an algorithm that alternates between learning a policy from a randomized simulator and adapting the posterior distribution over the simulator’s parameters in a
Bayesian fashion.
Our approach only requires a parameterized simulator, coarseprior ranges, a policy (optionally with optimization routine), and a small set ofreal-world observations.
Most importantly, the domain parameter distribution isnot restricted to a speciﬁc family, parameters can be correlated, and the simulatordoes not have to be differentiable.
We show that the presented method is ableto efﬁciently adapt the posterior over the domain parameters to closer match theobserved dynamics.
Moreover, we demonstrate that NPDR can learn transferablepolicies using fewer real-world rollouts than comparable algorithms.
Introduction
Learning control policies on a physical robot is time- and resource-intensive.
Crucially, Reinforcement Learning (RL) relies on random exploration, which in most cases can not be executed directly on the device.
Training in simulation promises to alleviate these problems by generating vastamounts of diverse data faster and cheaper.
However, all simulators are only models of reality andtherefore guaranteed to be ﬂawed.
Thus, using data from a single simulation instance is often notsufﬁcient to learn a control policy which transfers to the real-world counterpart, and might even leadto dangerous overﬁtting since the optimization process is optimistically biased [1].
The sim-to-real robot learning community suggested numerous approaches to bridge the ‘realitygap’ within the last few years.
Motivated by superior results and ﬂexibility, there is a clear trendtowards methods that automatically tune a randomized physics simulator’s parameter distribution tocloser match the reality [2, 3, 4, 5, 6, 7], also called guided domain randomization.
Such approaches involve a metric to quantify how well the simulated data is matching the observed data, combined with a mechanism to update the domain parameters or their distribution. Likelihood-
Free (LFI) methods measure the closeness between two observations by comparing their (logarithmic) probabilities, and provide various ways to update the density estimator which is either connected to the posterior [8, 9, 10] or the likelihood [11, 12, 13]
We contribute to the state-of-the-artin robot learning by proposing Neural Posterior Domain Randomization (NPDR), an algorithm which intertwines LFI, domain randomization, and RL to infer a distribution over simulators which is subsequentlyused to train transferable policies.
Most notably, NPDRdoes not make any assumptions on the simulator apartfrom requiring the ability to sample from it, works withcoarse priors, and can be conﬁgured to strictly yieldphysically plausible domain parameters.
Our approachallows to seamlessly integrate any (parameterizable)physics engine, hence beneﬁts from the rapid progressin the simulation community.
Moreover, we synchronize the simulator segmentwise with the trajectories recorded in the target domain.
This technique facilitates domain parameter inference forhighly dynamical systems.
We evaluate NPDR in one sim-to-sim and two sim-to-real experiments,and compare it with (online) BayesSim [14, 15] as well as Bayesian linear regression [16].
Our experiments demonstrate the generality of the proposed approach and show the advantages of ﬂexibleposterior inference in challenging continuous control problems.
The source code of all experimentswill be made available in a uniﬁed open-source library.
Background and Notation
Inferring the dynamics of an environment while solving a task is at the core of model-based RL [17].
We describe this problem using the framework of Markov Decision Processes (MDPs) in which thedomain parameters are modeled as random variables with an unknown distribution.
This frameworkis described in Section 2.1.
To infer the distribution over the domain parameters without makingany additional assumptions on the structure of the model of the MDP, we rely on the LFI methodsdescribed in Section 2.2, which only require the ability to sample trajectories from the model.
2.1 Markov Decision Processes with Randomized Dynamics
Consider a discrete-time dynamical system
[Equation]
t , a t ) , s 0 ∼ µ ξ ( s 0 ) , a t ∼ π θ ( a t | s t ) , ξ ∼ p ( ξ ) , with the continuous state s t ∈ S ξ ⊆ R n s and continuous action a t ∈ A ξ ⊆ R n a at time step t .
Theenvironment, also called domain, is characterized by its parameters ξ ∈ R n ξ (e.g., masses, frictioncoefﬁcients, or time delays) which are assumed to be random variables distributed according toan unknown probability distribution p : R n ξ → R + .
The domain parameters determine the transition probability density function P ξ : S ξ ×
A ξ × S ξ → R + that describes the system’s stochasticdynamics.
The initial state s 0 is drawn from the start state distribution µ ξ : S ξ → R + .
Togetherwith the reward function r ξ : S ξ × A ξ → R , and the temporal discount factor γ ∈ [0 , 1] , the systemforms an MDP described by the tuple M ξ = {S ξ , A ξ , P ξ , µ ξ , r ξ , γ } .
The goal of an RL agent is to maximize the expected (discounted) return, a numeric scoring functionwhich measures the policy’s performance.
The expected discounted return of a policy π θ ( a t | s t )with the parameters θ ∈ Θ ⊆ R n θ is deﬁned as J ( θ , ξ )
=
E τ ∼ p ( τ ) (cid:104)(cid:80) T − 1 t =
γ t r ξ ( s t , a t ) | θ , ξ (cid:105) .
While learning from experience, the agent adapts its policy parameters.
The resulting state-actionreward tuples are collected in trajectories, a.k.a.
rollouts, τ = { s t , a t , r t } T − 1 t =0 ∈ T with r t =r ξ ( s t , a t ) .
When augmenting the RL setting with domain randomization, the goal becomes tomaximize the expected (discounted) return for a distribution of domain parameters
[Equation]
The outer expectation with respect to the domain parameter distribution p ( ξ ) is the key differencecompared to the standard MDP formulation.
It enables the learning of robust policies that work fora whole set of environments instead of overﬁtting to a particular problem instance.
2.2 Sequential Neural Posterior Estimation (SNPE)
A simulator is a generative model g : Ξ → X mapping a set of domain parameters ξ ∈ Ξ toobservations x ∈ X .
Generating data from the simulator can be interpreted as sampling from anintractable likelihood x ∼ p ( x | ξ ) .
Given a prior belief p ( ξ ) over the domain parameters and anobservation x obs , LFI provides a way to learn an approximation ˆ p φ ( ξ | x ) of the underlying truedomain parameter distribution p (cid:0) ξ | x = x obs (cid:1) , where φ ∈ Φ are the parameters of the density estimator, e.g., a neural network or a mixture model.
When operating on time series data, LFI requiresan embedding to construct features of constant size.
This function f : T → X computes features,also called observations, from the rollouts τ ∈ T .
It may depend on parameters ψ ∈ Ψ whichstate-of-the-art implementations optimize jointly with the parameters φ of the density estimator.
Neural Posterior Domain Randomization (NPDR), as introduced in Section 3, is agnostic to theinference subroutine.
In this paper, a neural LFI algorithm SNPE-C [10] is used for its superior performance compared to the alternative approaches described in Section 5.3.
SNPE-C approximatesthe true posterior using normalizing ﬂows, a generative model which produces tractable distributions where both sampling and density evaluation can be efﬁcient and exact [18].
To overcome theintractability of the likelihood, the posterior is trained to minimize the following objective function
[Equation]
where R is the number of inference rounds N is the number of samples per round, and Z is thenormalization constant.
Note that the domain parameters are sampled from a proposal prior distribution ˜ p ( ξ ) , which is initialized with the prior p ( ξ ) , and requires importance reweighting of theposterior except for the ﬁrst round.
Sampling from the proposal prior has the advantage that the domain parameters could be sampled in more narrow regions where the actual support of the domainparameter distribution lies.
In a multi-round setup, the proposal prior ˜ p ( ξ ) is set to be the posteriorof the previous round conditioned on the previously observed data ˆ p φ (cid:0) ξ | x = x obs (cid:1) .
Neural Posterior Domain Randomization (NPDR)
The goal of NPDR is to learn a control policy in simulation such that it transfers to the real device,making as few assumptions as possible about the properties of the simulator, e.g., not assuming differentiability or use of rigid-body dynamics.
To achieve this, NPDR augments the nominal RL taskwith domain randomization and leverages LFI methods powered by normalizing ﬂows to approximate the posterior over the domain parameters.
The complete procedure is described in Algorithm 1.
NPDR does not impose any restrictions on the inference subroutine (Line 11) or on the policyoptimization subroutine (Lines 2 and 13).
A key feature of NPDR is the ability to use the currentbelief over the simulation parameters at every iteration.
This belief is inferred such that it explainsthe data observed from the real system best.
Furthermore, the algorithm was implemented in a waythat it is possible to jointly condition on multiple real-world rollouts (Lines 6 and 10).
To execute theinference (Line 11), we integrated the sbi toolbox [19], which enables us to easily switch between
LFI methods.
The sbi package expects all observations used to condition the posterior to have thesame size.
Satisfying this constrain for arbitrary time series embeddings, requires the rollouts to beof equal length.
Thus, if a policy causes an emergency stop, e.g., by exceeding the state boundaries,we pad the remainder of the rollout with zeros.
During our experiments, we did not observe anynegative effect of this measure on the inference routine.
Complementarily, the rollouts are simulated segmentwise, i.e., the states are synchronized with thereal trajectory every K time steps.
This repeated synchronization is especially beneﬁcial for systemswith a fast dynamics like the Furuta pendulum (Figure 1), since the trajectories diverge quickly evenfor well-ﬁt parameters.
Finally, if the policy has an internal model, e.g., true for MPC or the energybased control law used in Section 4.3, NPDR allows to set the parameters of the internal model tothe most likely sample from the current posterior (Line 13).
Updating the controller’s model wasalso done in [15], however not in combination with updating the policy parameters.
input : initial policy π θ ( a | s ) , generative model g sim ( ξ ) , physical device g real , prior p ( ξ ) , initial density estimator q φ ( ξ | x ) , summary statistics f ψ ( τ ) , LFI subroutine I NFER , policy optimization subroutine P OL O PT , output: approximate (conditional) posterior over the domain parameters
ˆ p φ (cid:0) ξ | x = x real (cid:1) , policy π θ (cid:63) ( a | s ) trained using the approximate (conditional) posterior 1 Initialize the parameters θ , φ , ψ randomly 2 Train an initial policy θ (cid:63) ← P OL O PT [ θ , p ( ξ )] with samples from the prior ξ ∼ p ( ξ ) 3 Initialize the approximate posterior with the prior ˆ p φ ( ξ | x ) ← p ( ξ ) 4 for each iteration i = 1: I do (cid:46) Sim-to-real loop
Execute H target domain rollouts τ real ← g real ( θ (cid:63) )
Compute the observations from the rollouts using learned summ. stats. x real ← f ψ (cid:0) τ real (cid:1) for each round r = 1: R do (cid:46) multi-round LFI, ﬁrst round is amortized
Sample N domain parameters ξ ∼ ˆ p φ (cid:0) ξ | x = x real (cid:1)
Sample N associated rollouts τ sim ← g sim ( θ (cid:63) , ξ )
Compute the observations from the rollouts using learned summ. stats. x sim ← f ψ (cid:0) τ sim (cid:1)
Optimize the density estimator and embedding φ (cid:63) , ψ (cid:63) ← I NFER (cid:2) x real , x sim , φ , ψ (cid:3)
Obtain the cond.
posterior from the density estimator
ˆ p φ (cid:0) ξ | x = x real (cid:1) ← q φ (cid:0) ξ | x = x real (cid:1)
Train the policy using the latest conditional posterior [Table]
Experiments and Evaluations
We conduct three experiments to analyze the properties of the proposed NPDR algorithm.
First,the correctness of the algorithm is evaluated on a simulated environment for which a ground truthsolution is available.
Namely, a sim-to-sim experiment on the pendulum is conducted to allow forvisual inspection of the complete posterior distribution over the domain parameters (Section 4.1).
Next, we conduct an experiment in a contact-rich mini golf environment, inferring a 10-dimensionalposterior using a state-of-the-art physics engine (Section 4.2).
Finally, we integrate policy optimization and solve an underactuated swing-up and balancing task on the Furuta pendulum (Section 4.3).
The conﬁgurations of all experiments are provided in the appendix.
Baselines Depending on the experiment, we chose BayesSim [14] or its online variant [15] asbaseline, because they are the only other approaches that work on the same assumptions and yield aﬂexible estimator of the domain parameter distribution as well as a trained policy (see Section 5.1 fora detailed description of BayesSim).
To obtain comparable results, it was necessary to adapt online
BayesSim such that the policy optimization is done until convergence (lines 13–14 of Algorithm 1 in[15]), i.e., we made this part identical to NPDR.
For the Furuta pendulum experiment (Section 4.3),we additionally validated the system identiﬁcation module against Bayesian linear regression.
Metrics In order to assess the quality of the domain parameter posteriors found by the LFI subroutines in NPDR and BayesSim, we compare the state trajectories generated by domain instancessampled from the approximate posterior against the ground truth data that was recorded on the physical system.
Before computing the performance metrics, all trajectories are normalized by the statebounds to equalize the importance of the different state dimensions.
We decided to use two metrics:the Root-Mean-Square Error (RMSE) and the Dynamic Time Warping (DTW) similarity computedby the python-dtw package [20].
Both metrics are desired to be small.
DTW is a dynamic programming algorithm that calculates an optimal match between two temporal sequences [21].
Different
DTW methods vary in the cost function and in the assignment of points to each other.
We conﬁgured
DTW to be symmetric, with the beginning and end of the assignment being chosen by the optimizer.
Code The implementations of NPDR as well as the baselines are building on the SimuRLacraframework [22], the sbi toolbox [19], and on the scikit-learn package [23], all open source. [Figure]
4.1 Sim-to-Sim Validation with a Fully Observable Posterior
The ﬁrst experiment is set up such that there is a manifold of domain parameters Ξ id ⊂ Ξ producingalmost identical trajectories τ id which are highly similar to the nominal ones.
Thus, the observationsx id = f ψ (cid:0) τ id (cid:1) are close to indistinguishable, independent of the summary statistic f .
The rationaleis that the mass m p and the length l p always appear coupled in the pendulum’s Equations of Motion (EoMs), and are dominated by the term m p l 2 p .
Thus, if we only randomize these two domainparameters, then conﬁgurations where the product m p l 2 p has a value similar to the one of the groundtruth environment should be inferred as more likely than others.
Figure 2 conﬁrms this hypothesisempirically, by clearly capturing the manifold Ξ id .
To generate the rollouts, the pendulum has beenexcited by applying a feed-forward sinusoidal action signal.
Both algorithms used the summarystatistic function proposed in [14] and the same prior.
4.2 Sim-to-Real Bayesian System Identiﬁcation in a Contact-Rich Environment
One of the main beneﬁts that NPDR and BayesSim inherit from their
LFI subroutines is the ability to incorporate state-of-the-art physicsengine which expose their parameters.
To showcase this advantage,we chose mini golf (Figure 1 and 3) since it requires correct contactmodeling to align simulation and reality.
We employed the Bulletphysics engine to randomize the ball’s radius r b , mass m b , restitutioncoefﬁcient e b , rolling friction coefﬁcient µ b , the two rails’ Cartesianoffsets ∆ x 1 , ∆ x 2 , ∆ y 1 , ∆ y 2 , as well as their angular offsets aroundthe vertical axis ∆ γ 1 , ∆ γ 2 .
The goal location was a circular pieceof double-sided tape, modeled as a material with very high rollingfriction, that stops the ball quickly.
To hit the ball reliably, the robot was driven by a hand-tuned,time-dependent policy.
For the inference, we used the ball positions, recorded by a Vicon system,and the robot’s joint angles.
Figure 4 displays a slice through the 10-dimensional domain parameter posteriors learned with
NPDR and BayesSim on the same data.
As expected, the mass of an ideally rolling object hasno inﬂuence on the dynamics, thus the marginal of m b should be uniform.
The prior for the angular offsets were chosen informatively, i.e., the object can be rotated by ± π at the beginning ofthe simulation.
Therefore, the posterior should have two modes for γ i .
Both algorithms were ableto recognize this multi-modality as well as the coupling between e b and r b .
However, the correlation between ∆ x i and ∆ y i was only inferred by NPDR.
A quantitative assessment of the modelingaccuracy is given in Table 1, which reports the best experiment for every method.
4.3 Sim-to-Real Transfer with Policy Optimization in the Loop
The ﬁnal experiment is aimed at evaluating NPDR on the challenging continuous control task ofswinging up and stabilizing a rotary inverted pendulum (Figure 1).
This system, known as the Furutapendulum [24], is particularly hard to control due to its underactuated nature and fast oscillatory r b m b e b µ b ∆ x∆ yr
b ∆ γm b e b µ b ∆ x∆ y∆ γ
The nominal values (green), were either determined by prior measurements or by coarse estimates.
Every domain off-diagonal plot shows the same 200 samples for 2 dimensions, whereas the diagonal plots show the marginal distributions.
The omitted domain parameters of the ﬁrst obstacle ∆ x 1 , ∆ y 1 , γ 1 are similar to ∆ x 2 , ∆ y 2 , γ 2 .
As described in Section 4.2, the mini golf experiment is set up such that multiple parameter conﬁgurations can result in the same trajectory.
This arises naturally from broad or nonminimal choices.
Details on the domain parameters ranges can be found in the appendix.
movements.
To capture a wide range of dynamics, we chose to randomize the link masses m r ,m p , the link lengths, l r , l p , the motor constant and resistance k m , R m , the gravity constant g , aswell as the viscous friction parameters d p , d r of both joints.
As derived in the appendix, it is notpossible to linearly separate all of these domain parameters and obtain a unique solution.
Therefore,we excluded g , k m , and R m for the Bayesian linear regression, which makes the results baselinecomparable to the others.
The policy is given as a parameterized hybrid controller consisting of twocomponents: an energy-shaping swing-up controller that brings the pendulum close to the uprightposition and a stabilizing PD controller that switches on in the vicinity of the upright position [25].
We used Policy learning by Weighting Exploration with the Returns (PoWER) [26] to optimize thepolicy parameters for all experiments reported in Table 2.
During policy execution on the physicaldevice, the parameters of the energy-shaping controller’s internal model were set to the most likelydomain parameter set, sampled from the posterior during training.
The results show that our LFI-based domain randomization approach outperforms Bayesian linearregression and the nominal baseline, both in terms of the real-world return and the RMSE error.
Webelieve that the advantage of NPDR over BayesSim is due to its superior neural LFI subroutine aswell as the fact that our method, in contrast to online BayesSim, trains the policy until convergencebefore executing the next target domain rollout.
This hypothesis is supported by the better metrics forthe trajectory differences (Table 2).
See Section 5.1 for a more in-depth discussion of the differencesbetween the two algorithms.
Comparing the trajectories resulting from the nominal simulation withthe ones resulting from the learned distribution (Figure 5), we observe that all approaches improvedover the nominal simulator.
Moreover, NPDR and BayesSim learned a predictive distribution thatincludes the ground truth trajectory, except for the segment between 3 s and 4 s . Since all simulatorconﬁgurations fail to reproduce the recorded trajectories, we conclude that the modeled EoMs aremissing effects which are decisive of the last part of the swing-up.
[Table]
α [ r a d ]        θ [ r a d ]
α [ r a d ]        θ [ r a d ] (b
time [s]   α
[ r a d ]     time [s]   θ [ r a d ]
[Figure]
Related Work
The topic of sim-to-real transfer in robot learning has grown in prominence in the last few years.
For a broad overview of the existing approaches, we refer the reader to the survey paper [27].
In thefollowing, we focus on the methods directly related to the proposed NPDR algorithm.
5.1 Adapting the Source Domain Distribution
The domain parameter distribution can be updated in multiple ways [28, 3, 5, 6, 29].
Most approaches assume independent domain parameters, i.e., they cannot represent the correlations between them.
In contrast, NPDR, BayesSim [14], and DISCO [30] model the whole posterior, andthus capture the correlations between the parameters.
As observed in the experiments in Section 4,these kinds of correlations are abundant in models of robotic systems.
BayesSim [14], especially itsonline variant [15], share the concepts such as the combination of LFI and domain randomizationwith NPDR.
However, the approaches differ in three important ways: (i) NPDR approximates theposterior using a normalizing ﬂow instead of a mixture model, which is coupled with using SNPE-Cinstead of SNPE-A as inference algorithm.
The results in Section 4 highlight the importance of the
[Table]
LFI subroutine and clearly favour the neural density estimator.
This ﬁnding is consistent with [31].(ii)
The policy optimization and domain parameter inference in NPDR are not alternated at every(time) step, but trained until convergence, which stabilizes the training.
(iii)
We demonstrated that
NPDR is able to not only update the policy’s internal model, as done with MPC in [15], but also totrain the policy parameters directly based on the latest posterior.
5.2 Bayesian System Identiﬁcation for Robotics
A crucial step in NPDR is the approximate inference of the domain parameter distribution.
While
NPDR makes no assumptions on the simulator properties, other methods provide more domainspeciﬁc solutions by assuming more structure in the model.
For example, identiﬁcation of rigid-bodydynamics can be framed as a (Bayesian) linear regression problem [32, 33].
However, the identiﬁedparameters commonly turn out to be physically implausible [16].
Nonlinear system identiﬁcationof rigid-body dynamics with neural networks [34, 35] allows for greater ﬂexibility and physicalplausibility, but it is not straightforward to obtain conﬁdence estimates based on these methods.
Formore general dynamical systems, approaches utilizing the classiﬁcation loss between simulated andreal samples have been considered [36, 29].
If the simulator model is non-differentiable, episodic
RL has been proposed to optimize the posterior using trajectory similarity as the reward signal [3].
Our NPDR method is the closest to the latter family of algorithms, but instead of episodic RL, itutilizes neural LFI methods, which enable the use of more expressive posteriors, such as normalizingﬂows, and are more sample efﬁcient, requiring only a few rollouts to be trained.
5.3 Likelihood-Free Inference
LFI is a collective term for Bayesian inference methods which are able to learn a posterior distribution in scenarios where the likelihood is either not available or too expensive to evaluate.
Approximate Bayesian Computation (ABC) applies Monte-Carlo sampling to infer the parameters bycomparing summary statistics of synthetically generated and observed data [37].
Although MCMCABC [38] and SMC-ABC [39] enhance the sample efﬁciency of the classical rejection-ABC byemploying more sophisticated sampling schemes, the ABC approach does not scale well becauseof the inherent problems of sampling in high-dimensional spaces.
Amortized methods have beenmore successful in scaling LFI to more challenging problems by utilizing ﬂexible neural densitymodels [40].
Such models can be used both to generate samples from the posterior and to evaluatethe likelihood of parameter conﬁgurations.
SNPE approaches [8, 9, 10] approximate the conditionalposterior, allowing for direct sampling from the posterior, foregoing MCMC.
Learning the likelihood [11] can be useful in the context for hypothesis testing.
Alternatively, posterior samples can begenerated from likelihood-ratios [12] which can be trained using contrastive learning [13].
A tabular summary of state-of-the-art LFI approaches can be found in the appendix.
For a comprehensivesurvey on LFI from simulations, see [41].
Conclusion
In this paper, we introduced and empirically studied a new adaptive domain randomization algorithm
Neural Posterior Domain Randomization (NPDR) that employs normalizing ﬂows for representingthe posterior distribution over the domain parameters.
Our approach makes no assumptions on thesimulator except the ability to sample from it, and it is guaranteed to identify physically plausibleparameters.
NPDR interleaves approximate posterior inference with policy optimization, leading toan iterative reﬁning of the posterior distribution from which more transferable policies are trained.
Our results showed that NPDR improves the ﬂexibility and precision of the existing methods and requires only a few real-world rollouts to train robust policies in a randomized simulation environment.
We believe that the proposed method, powered by likelihood-free inference with normalizing ﬂowsas density estimators, is a valuable addition to the toolbox of domain randomization algorithms,enabling rapid domain adaptation and training of effective control policies.
Future work will investigate alternative likelihood-free inference approaches, such as sequential neural ratio estimation.
Additionally, we plan to apply NPDR to problems involving more challengingdynamical systems such as soft body simulation and ﬂuid dynamics.
It is furthermore of interest toconsider other modalities of domain parameters, such as parameters of a vision system.