Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes
Abstract: Monte Carlo methods have become increasingly relevant for control of non-differentiable systems, approximate dynamics models and learning from data.
These methods scale to high-dimensional spaces and are effective at the non-convex optimizations often seen in robot learning.
We look at sample-based methods from the perspective of inference-based control, speciﬁcally posterior policy iteration.
From this perspective, we highlight how Gaussian noise priors produce rough control actions that are unsuitable for physical robot deployment.
Considering smoother Gaussian process priors, as used in episodic reinforcement learning and motion planning, we demonstrate how smoother model predictive control can be achieved using online sequential inference.
This inference is realized through an efﬁcient factorization of the action distribution and a novel means of optimizing the likelihood temperature to improve importance sampling accuracy.
We evaluate this approach on several high-dimensional robot control tasks, matching the sample efﬁciency of prior heuristic methods while also ensuring smoothness.
Simulation results can be seen at monte-carlo-ppi.github.io .
020 r
Baseline (i cem , coloured noise)
Ta
HA c t i o n s a m p l e 020 r
Ours ( lbps , se kernel)
Ta
HA c t i o n p r i
o r s a m p l e smoothness in the downstream actuation a , resulting in aggressive control (center).
We use smooth Gaussian process priors to infer posterior actions (right), which preserves smoothness while maintaining performance and sample efﬁciency, as both are using only 32 samples.
Rewards r show quartiles over 25 seeds.
Introduction
Learning robot control requires optimization to be performed on sampled transitions of the environment [1].
Monte Carlo methods [2] provide a principled means to approach such algorithms, bridging black-box optimization and approximate inference techniques.
These methods have been adopted extensively by the community for their impressive simulated [3, 4, 5, 6] and real-world [7, 8, 9, 10, 11, 12, 13, 14, 15] robot learning results.
Their appeal includes requiring only function evaluations of the dynamics and objective, so can be applied to complex environments with minimal overhead (Figure 1).
Moreover, their stochastic nature also avoids issues with local minima that occur with gradient-based solvers [16, 17].
Finally, while Monte Carlo sampling is expensive, shooting methods can be effectively parallelized across processes and the advent of simulations on
GPU s also provides a means of acceleration [18, 13].
However, some aspects of black-box optimization are open to criticism.
Sample-based solvers such as the cross-entropy method ( CEM ) [19] appear ‘wasteful’, ignoring computation by throwing away the majority of samples, while others enforce high-entropy search distributions to avoid premature convergence [18].
Moreover, many design decisions and hyperparameters are heuristic in nature, which is undesirable from both the user- and research perspective when interpreting, tuning or advancing these methods.
In this work, we consider Monte Carlo optimal control through the broader perspective of inferencebased control [20, 21, 22, 23, 24, 25, 26, 27], where optimization is achieved through importance sampling [28].
This approach covers settings such as policy search [29], motion planning [8, 30] and model predictive control ( MPC ) [18].
From this view point, we highlight two key design decisions: the likelihood temperature and the distribution over action sequences.
An adaptive temperature scheme is crucial for controlling the optimization behavior across objectives and distributions, but in many methods this aspect is ignored or opaque.
Moreover, correlated action sequences are equally crucial for performing effective exploration and control in practical settings.
Smoothness , arising from such correlations, is an aspect of human motion [31].
Smooth priors have taken many forms across domains, such as movement primitives [32], smoothed- [11, 12] or coloured noise [4].
We use Gaussian processes [33] as action priors and show how they can be scaled to high-dimensional action spaces through factorization of the covariance.
Evaluating on simulated robotic systems, we reproduce prior results on policy search while transferring these ideas to MPC , matching prior performance with respect to sample efﬁciency while ensuring smooth actuation.
Contribution.
First, we present a perspective of episodic inference-based control based on Gibbs posteriors.
Using this view, we then present novel Monte Carlo variants that incorporate the approximate inference error due to importance sampling, simplifying the hyperparameter while providing regularization.
Thirdly, we demonstrate how richer Gaussian process priors can be combined with these regularized Gibbs posteriors for Monte Carlo MPC using online sequential inference, which achieves greater smoothness and sample efﬁciency than standard white noise priors.
We highlight connections between this approach to MPC and effective prior approaches to episodic policy search.
Monte Carlo Methods for Optimal Control
This section outlines the problem setting and introduces variational optimization and posterior policy iteration methods.
We consider the standard (stochastic) optimal control setting in discretetime, with states s ∈ R d s , actions a ∈ R d a .
Optimization is framed as maximizing a reward r : R d s × R d a → R under dynamics p
( s t +1 | s t , a t ) and initial state distribution p ( s 1 ) ,
[Equation]
This work focuses on the episodic setting, where optimization is performed after evaluating the current solution over a ﬁnite-time horizon T .
We frequently use the episodic return R , where
R ( S , A ) =
(cid:80) T t =1 r
( s t , a t ) , using upper-case to denote sequences, e.g. A := { a 1 , . .
. , a T } .
2.1 Variational Optimization with Gibbs Posteriors
The optimization outlined above is amenable to gradient-based solvers such as stochastic differential dynamic programming [34].
However, to aid optimization through exploration and regularization, we can consider optimizing a parametric belief over action sequences q ∈ Q .
The variational formulation (Equation 3) generalizes Bayes’ rule beyond optimizing likelihoods and resembles many learning algorithms [35, 36].
This work concerns optimizing an open-loop action sequence to maximize an episodic return.
Bayesian inference of an action sequence from data, known as input estimation, can be performed using message passing of the appropriate probabilistic graphical model, capturing the sequential structure of the problem and necessary priors [27].
If the measurement log-likelihood is replaced with the control objective, this inference computation can be shown to have precise dualities with dynamic programming-based optimal control [37].
While this switch in objective provides a powerful suite of inference tools for efﬁcient computation, it requires treating the control objective as a Markovian log-likelihood, which is not the case for episodic objectives.
The Gibbs likelihood is a general treatment of the objective-as-likelihood (Deﬁnition 1) [38, 39].
Deﬁnition 1.
(Gibbs likelihoods and posteriors)
For a loss f and prior p ( x ) , the Gibbs posterior q α for parameter x is derived by constructing the Gibbs likelihood exp( − α f ( x )) from the loss,
[Equation]
This posterior minimizes the following objective
[Equation]
This objective appears in PAC-Bayes methods [38], mirror descent methods [40] and Bayesian inference as the evidence lower bound objective when f ( x ) is a negative log-likelihood [39].
Augmenting the variational optimization objective with prior regularization (Equation 3), we obtain an expression of the optimal belief in the action sequence (Equation 2).
The parameter α has a range of meanings, depending on context.
In PAC-Bayes it is the dataset size, in mirror descent it is an update step size and in risk-sensitive control it is the sensitivity [41, 42].
Example 1 in Appendix A examines a tractable linear-quadratic-Gaussian example of this update, demonstrating its relation to
Newton-like optimization and highlighting the effect α has on the regularized update.
2.2 Posterior Policy Iteration
The optimal control problem (Equation 1) is ambiguous regarding whether the action sequence or state-action trajectory is the optimization variable.
Applying the Gibbs posterior to the optimal control setting recovers Rawlik et al.
’s
posterior policy iteration [41], which can be implemented using the joint distribution or policy.
We consider the following joint state-action distribution, that factorizes in the following Markovian fashion p ( S , A ) =
p ( s 1 ) (cid:81) Tt =1 p ( s t +1 | s t , a t ) p ( a t
| s t ) .
Pos- terior policy iteration updates the state-action distribution through the policy, constructing a Gibbs likelihood from the reward, as the dynamics and initial state distribution are constant.
Deﬁnition 2.
(Posterior policy iterations ( PPI )
As the initial distribution and dynamics are shared by the prior and posterior joint state-action distribution, the joint Gibbs posterior q α ( S , A ) ∝ exp( αR ( S , A )) p ( S , A ) can be alternatively expressed using the policy posterior update q α ( A | S ) ∝ exp( αR ( S , A )) p ( A | S ) .
Using this update, the key decisions are choosing p ( A | S ) , α and the inference approximation.
If p and q α are Gaussian, then PPI involves iterative reﬁnement of the distribution.
In the Monte Carlo setting, q α takes the form of an importance-weighted empirical distribution.
To apply iteratively, p is updated using the M-projection, following the objective (Equation 3), i.e. a weighted maximum likelihood ﬁt of the policy parameters [29].
This approach is a stochastic approximate expectation maximization ( SAEM ) method [44] and described fully in Algorithm 1 in the Appendix.
We argue a key aspect of PPI methods is how to specify the inverse temperature α during optimization (Section 3), as it has a strong inﬂuence on the posterior, which is important when ﬁtting rich distributions such as Gaussian processes (Section 4) from samples.
Gaussian process action priors can be applied to several control settings, such as policy search and model predictive control (Section 6).
Posterior Policy Constraints for Monte Carlo Optimization
The Gibbs posterior in Deﬁnition 2 has been adopted widely in control, albeit from a range of different perspectives, such as Bayesian smoothing [23], solutions to the Feynman-Kac equation [45], maximum entropy [26], mirror descent [46] and entropy-regularized reinforcement learning [47].
An open question is how best to set α for Monte Carlo optimization?
Relative entropy policy search (Deﬁnition 3), provides a principled and effective means of deriving α for stochastic optimization, using the constrained optimization view of entropy-regularized optimal control.
Deﬁnition 3.
(Episodic relative entropy policy search (e REPS ) [29]) Maximize the expected return, subject to a hard KL bound (cid:15) on the policy update,
[Equation]
The posterior policy takes the form q θ ( A | S )
∝ exp( αR ) p ( A | S ) , where α is derived from Lagrange multiplier calculated by minimizing the empirical dual G ( · ) using N samples,
[Equation]
While REPS is a principled approach to stochastic optimization, we posit two weaknesses: The hard
KL constraint (cid:15) is difﬁcult to specify, as it depends on the optimization problem, distribution family and dimensionality.
Secondly, the Monte Carlo approximation of the dual has no regularization and may poorly adhere to the KL constraint without sufﬁcient samples.
Therefore, we desire an alternative approach that resolves these two issues, capturing the Monte Carlo approximation error with a simpler hyperparameter.
To tackle this problem, we interpret the REPS update as a pseudo-posterior, where the temperature is calculated using the KL constraint.
We make this interpretation concrete by reversing the objective and constraint, switching to an equality constraint for the expectation,
[Equation]
This objective is a minimum relative entropy problem [48], which yields the same Gibbs posterior as e REPS (Lemma 1, Appendix A).
With exact inference, a suitable prior and oracle knowledge of the maximum return, this program computes the optimal policy in a single step by setting R ∗ to the optimal value.
However, in this work, the expectation constraint requires self-normalized importance sampling ( SNIS ) on sampled returns R ( n ) using samples from the current policy prior,
E
[Equation]
Rather than specifying R ∗ here, we identify that this estimator is fundamentally limited by inference accuracy.
We capture this error by applying an IS -derived concentration inequality to this estimate (Theorem 1) [49].
This lower bound can be used as an objective for optimizing α , balancing policy improvement with approximate inference accuracy.
Theorem 1.
(Importance sampling estimator concentration inequality (Theorem 2, [49])) Let q and p be two probability densities such that q (cid:28) p and d 2 [ q || p ] < + ∞ .
Let x 1 , x 2 , . . .
x N i.i.d.
random variables sampled from p and f :
X → R be a bounded function ( || f || ∞ < + ∞ ) .
Then, for any 0 < δ ≤ 1 and N > 0 with probability at least 1 − δ :
[Equation]
The divergence term d 2 [ q || p ] is the exponentiated R ´ enyi-2 divergence, exp D 2 [ q || p ] .
While this is tractable for the multivariate Gaussian, it is otherwise not available in closed form.
Fortunately, we can use the effective sample size ( ESS ) [50] as an approximation, as ˆ N α ≈ N / d 2 [ q α || p ] [49, 51] (Lemma 2, see Section A of the Appendix).
Combining Equation 4 with our constraint, instead of setting R ∗ , we maximize the IS lower bound R ∗ LB to form an objective for the inverse temperature α which incorporates the inference accuracy due to the sampling given inequality probability [Equation] δ ,
[Equation]
We refer to this approach as lower-bound policy search ( LBPS ).
This objective combines the expected performance of q α , based on the IS estimate E q α /p
[ · ] , with regularization E R based on the return and inference accuracy.
Treating p , N , || R || ∞ as task-speciﬁc hyperparameters, the only al- gorithm hyperparameter δ ∈ [0 , 1) deﬁnes the probability of the bound.
In practice, self-normalized importance sampling is used for PPI , as the normalizing constants of the Gibbs likelihoods are not available.
While Metelli et al.
also derive an SNIS lower bound [49], we found, as they did, that the IS lower bound with SNIS estimates work better in practice due to the conservatism of the SNIS bound.
An interpretation of this approach is that the R´enyi-2 regularization constrains the Gibbs posterior to be one that can be estimated from the ﬁnite samples, as the divergence is used in evaluating IS sample complexity [52, 53].
Moreover, the role of the ESS for regularization is similar to the ‘elite’ samples in CEM .
Connecting these two mechanisms as robust maximum estimators (Section A), we also propose effective sample size policy search ( ESSPS ), which optimizes α to achieve a desired ESS N ∗ , i.e. a R´enyi-2 divergence bound, using the objective min α | ˆ N α − N ∗ | .
More details regarding PPI (Section A) and temperature selection methods (Table 1) are in the Appendix.
This section introduces two methods, LBPS and ESSPS , for constraining the Gibbs posteriors for
Monte Carlo optimization.
These methods provide statistical regularization through soft and hard constraints involving the effective sample size, which avoids the pitfall of ﬁtting high-dimensional distributions to a few effective samples.
A popular setting for these methods is MPC , which performs episodic optimization over short planning horizons while adapting each time step to the current state.
Moreover, for optimal control, we also need to specify a suitable prior over action sequences.
To apply PPI to the MPC setting, we must implement online optimization given this prior over actions.
a t a
Coloured Noise ( β =2.5) t
SE Kernel ( l =0.2) t
Periodic Kernel ( l =0.2, T =0.5)
Online Posterior Policy Iteration & Prior Design
In this section, we derive the online realization of posterior policy iteration that uses and maintains correlated action priors, computing the ﬁnite-horizon H future actions given a likelihood on a subset of actions from the past.
R represents the return-based Gibbs likelihood term (Deﬁnition 1),
[Equation]
(6) where τ ≤ t + H .
As an analogy, this is equivalent to combining forecasting with state estimation, i.e. p ( x t : t + H | y 1: t ) for states x and measurements y .
For correlated priors on the action space, this computation is tractable if working with Gaussian processes.
In fact, a recurring aspect across several many posterior policy iteration-like approaches is the use of Gaussian process policies,
[Equation]
(Independent Gaussian noise, e.g. [18]),
[Equation]
) , φ [Equation] t [Equation]
(cid:62) [Equation] , (Bayesian linear regression, e.g. P ro MP [32]) ,
[Equation]
(time-varying linear Gaussian e.g. [41, 54, 42]),
GP ( µ ( s ) , [Equation] , (non-parametric Gaussian process [55]).
Despite the simplicity of Gaussian action noise, for robotics, more sophisticated noise is often desired for safety and effective exploration [56, 29].
Prior work has proposed ﬁrst-order smoothing [11, 12].
Using v ( n ) t ∼ N [Equation] 0 , I ) , β ∈ [0 , 1] and Σ t =
L t L (cid:62) t , actions are sampled using
[Equation]
However, in practice it is also implemented as a ( n ) t
[Equation]
While this approach directly smooths the actuation, it also introduces a lag, which may deteriorate performance.
Other approaches have used colored noise for sampling the noise n [4].
Contrast these approaches to Gibbs sampling a multivariate Gaussian joint distribution with 1-step cross-correlations [58], which is a ( n ) t | t − 1 = µ ( n ) t | t − 1 + L t | t − 1 v ( n ) t , where µ ( n ) t | t − 1 = µ t + Σ t,t − 1 Σ − 1 t − 1 ( a ( n ) t − 1 − µ t − 1 ) , and Σ t | t − 1 = Σ t − Σ t,t − 1 Σ − 1 t − 1 Σ (cid:62) t,t − 1 .
The differences are subtle, but important.
The initial proposed sampling scheme essentially adds correlated noise to the mean for exploration, but does not consider the smoothness of the mean itself.
The practical implementation incorporates the previous action, but through exponential smoothing, which introduces a ﬁxed lag that potentially degrades the quality of the mean action sequence.
Correct sampling of the joint distribution has neither of these issues and naturally extends to correlations over several time steps.
We do this in a general fashion by considering the (continuous time)
Gaussian process (see Section G, Appendix), so p ( a t ) =
N ( µ t i : t j , Σ t i : t j ) =
GP ( µ ( t ) , Σ ( t )) for a discrete-time sequence t =
[ t i , . .
t j ] .
Propo- sition 1 shows how the time shift for MPC can be implemented in a general fashion when using GP s.
Proposition 1.
Given a Gaussian process prior GP ( µ ( t ) , Σ ( t )) and multivariate normal posterior q α ( a t 1 : t 2 ) = N ( µ t 1 : t 2 | R , Σ t 1 : t 2 | R ) for t 1 to t 2 , the posterior for t 3 to t 4 is expressed as
[Equation]
(7) where ν t 1 : t 2 = Σ − 1 t 1 : t 2 ( µ t 1 : t 2 | R − µ t 1 : t 2 ) and Λ t 1 : t 2 = Σ − 1 t 1 : t 2 ( Σ t 1 : t 2 − Σ t 1 : t 2 | R ) Σ − 1 t 1 : t 2 .
This update combines the new sequence prior from t 3 to t 4 and the previous likelihood used in the update for t 1 to t 2 , obtained from the posterior and prior.
Note, the cross-covariance Σ t 3 : t 4 ,t 1 : t 2 is computed using the covariance function of the prior GP .
The proof is in Appendix A.
For a stationary kernel, ﬁxed planning horizon and ﬁxed control frequency, the term Σ − 1 t 1 : t 2 is Σ − 1 t : t + H and is constant, so can be computed at initialization to avoid repeated inversion.
Figure 2 demonstrates how this update lets us combine our prior with previous posterior in a principled fashion.
Moreover, its continuous-time construction means that the time resolution can be updated, not just the time window, for planning at different timescales [30].
Compared to the independence assumption, modeling correlations between actions introduces complexity.
The full covariance over (ﬂattened) time and action has a complexity R ( T 3 d 3 a ) , which is infeasible to work with.
Assuming independence between actions, a GP per action has a complexity of R ( T 3 d a ) , requiring d a GP s to be ﬁt, which is not desirable for online methods such as MPC .
To avoid the linear scaling w.r.t.
d a , we propose using the matrix Normal distribution (Deﬁnition 4) for scalability, as it is parameterized into single T and d a -dimensional covariances,
Deﬁnition 4.
(Matrix Normal Distribution (MaVN) [59])
For a random matrix X ∈ R n × p , it follows a matrix normal distribution X ∼ MN
( M , K , Σ ) , where M ∈ R n × p , K ∈ S n + and Σ ∈ S p + , if and only if vec ( X ) ∼ N ( vec ( M ) , Σ ⊗ K ) , where ⊗ denotes the Kronecker product.
Using the Kronecker-structured covariance provides a useful decomposition of the time-based covariance K , that deﬁnes correlations between time steps, and an action covariance Σ that captures correlations between actions.
Typically we assume actions are independent, but cross-correlations could be learned from experience for richer coordination.
While this Kronecker structure does not fully capture the correlations between time and actions, the structure is very useful for MPC on robotic systems, where the actions space could be very high but the planning horizon is sufﬁciently small for covariance estimation using a reasonable number of Monte Carlo rollouts.
Feature Approximations.
Despite the matrix Normal factorization, computing the correlations between actions still requires a dense H × H covariance matrix K for planning horizon H .
To sparsify this quantity, we consider kernel approximations, such as the canonical basis functions (cid:80) n k ( · , x n ) and spectral approximations using random features (cid:80) n φ n ( · ) [60], for a Bayesian linear model φ (cid:62) t W .
Focusing on the squared exponential ( SE ) kernel, this results in radial basis function ( RBF ) and random Fourier features ( RFF ) respectively.
Interestingly, RBF features are closely related to probabilistic movement primitives, used extensively in policy search for robotics [32].
For onedimensional inputs, RFF s are effectively approximated by applying Gauss-Hermite quadrature [61] to the random weights [62].
RBF features and RFF s approximate w.r.t.
time and frequency respectively and could be combined [63].
Using these continuous-time features, the optimization is now abstracted from planning horizon and control frequency, providing much greater ﬂexibility.
Secondly, due to the features, a factorized weight covariance approximation does not sacriﬁce smoothness.
Moreover, the moment updates described above are not needed, as only φ t is updated.
Related Work
Inference-based control.
Posterior policy iteration was proposed by Rawlik et al.
[41] and covers prior methods developed from Bayesian smoothing [23, 37], expectation maximization [22, 56], entropy regularization [47, 9] and path integral [64] perspectives.
For MPC speciﬁcally, the path integral-based MPPI was proposed [18], with alternative formulations based on mirror descent [46] and variational inference [5, 65, 66].
Mukadam et al.
[25] models the optimal state-action distribution as a sparse Gaussian process and uses linearization for approximate inference.
The same approach is used for Gaussian process motion planning [30], which are also optimized using sampling [8].
Gaussian quadrature is also used for inference-based MPC [27].
Concurrent work uses the
ESS for a temperature adjusting heuristic for MPPI [15] and also combines policy search with MPC using PPI techniques [67].
See Section B for a more in-depth discussion on these related works.
Policy design and regularization.
Smooth actuation is important in robot learning for safety and exploration, having been proposed for Monte Carlo MPC [11, 12, 4] and more broadly incorporated using augmented objectives, parameter sampling and policy design, e.g. [68, 69, 70].
Stochastic search.
Probabilistic interpretations of black-box optimization algorithms are well established [71, 72, 73], however prior work did not connect the ESS and elite samples.
CEM and extensions have also been adopted widely as a solver for MPC [3, 4, 6].
Gaussian processes for control.
This work adopts GP s for correlated action priors.
This is distinct from prior work which uses GP s to approximate dynamics or value functions, e.g. [74, 75, 76, 77].
[Figure]
R e t u r n ↑
HumanoidStandup-v  2 ,4 ,door-v     · 10 4 hammer-v
[Figure]
S m oo t hn e ss ↓
[Figure]
These priors require a large number of samples for good performance, across methods and tasks.
Experimental Results
We assess the Gibbs posterior methods and policy design empirically across various settings.
Blackbox optimization (Section 6.1) considers standard benchmarks, while policy search (Section 6.2) optimizes action sequences for a robotic task.
For MPC (Section 6.3), we evaluate online PPI approaches with white noise and smooth priors on high-dimensional, contact-rich tasks.
For the code, see github.com/JoeMWatson/monte-carlo-posterior-policy-iteration .
6.1 Black-box Optimization
To understand the behaviour of the proposed PPI variants, the performance of LBPS and ESSPS on a range of standard black-box optimization functions over a range of hyperparameters, with e REPS and
CEM as baselines, are shown in Appendix D.1.
Figures 7 – 10 show that ESS is a useful metric for these methods, as each solver exhibits consistent ESS for a given hyperparameter value.
However, the uniform weights used by CEM (Figure 7) maintain entropy longer than ESSPS (Figure 8), which can lead to better optima, so the ESS is not sufﬁcient to fully capture the behavior of these solvers.
6.2 Policy Search
As LBPS and ESSPS are closely related to e REPS , we repeat the experiment from prior work performing the ‘ball in a cup’ task using policy search using a Barret WAM [14], which has been shown to transfer to the physical system [78, 14, 79], as a benchmark task.
Moreover, we replace P ro MP s with
Matrix normal RBF and RFF policies.
From the kernel perspective, this feature approximation is motivated by the large ( T (cid:39) 1000 ) task horizon.
The results in Appendix D.2 conﬁrm that these solvers are all capable of solving the task, based on success rate, where RBF (Figure 11) and RFF features (Figure 12) perform equally well w.r.t.
the convergence of the success rate for each approach.
6.3 Model Predictive Control with Oracle Dynamics
We evaluate online PPI across a range of high-dimensional robotic control tasks in MuJoCo [80], including HumanoidStandup-v2 in Gym [81] and door-v0 , hammer-v0 from mj envs , using the
Adroit hand (Figure 1) [82].
To measure smoothness, we adopt the FFT -based score 2 Nf s (cid:80) Ni
=1 a i f i [68], with sampling frequency f s and N resolvable frequencies f with amplitudes a .
We compute the Euclidean norm of the action sequence over time and apply the smoothness measure to this signal.
For the evaluation, we focus on a low computational budget, with 1 or 2 iterations per timestep.
To assess the impact of approximate inference, we assess performance over an logarithmic range of sample rollouts, following prior work [4].
Details may be found in Appendix E.2.
White noise priors.
Figure 3 shows MPC with white noise priors using LBPS and ESSPS , with MPPI ,
CEM and PI 2 baselines (see Table 1).
While each solver performs comparably for 1024 rollouts, the low sample regime shows greater performance variance.
While MPPI seems particularly effective,
Figure 13 shows that its average ESS is particularly low, (cid:39) 1 for many cases.
Combined with the ﬁxed variances, this suggests optimization is closer to greedy random search than importance sampling.
The poor door-v0 performance of ESSPS is due to slow opening, rather than task failure.
i CEM (coloured noise)
[Figure]
R
[Figure]
t u r n ↑
HumanoidStandup-v
[Figure]
door-v
[Figure]
S m oo t hn e ss
[Figure]
Policy design for smooth control.
Figure 4 shows online PPI with action priors.
LBPS , ESSPS and
MPPI use the SE kernel, with i CEM [4] and MPPI with smooth actions and noise as a baseline.
The smooth action distributions greatly improves performance across models, tasks and sample sizes, due to effective exploration.
As desired, the richer SE kernel provides much greater smoothness, by up to a factor of 2 compared to baselines, with limited impact to performance.
It is unsurprising that the smoothness bias reduces performance if optimal behavior is non-smooth, as illustrated in Figure 2.
Appendix D.3.2 shows some of the action sequences from Figure 4, where we see GP smoothness varies with increasing rollout samples and also results in signiﬁcant actuator amplitude reduction.
Comparing kernel- and feature-based policies.
To assess feature approximations for smooth MPC , we replace the SE kernel with RBF and RFF features, while keeping the lengthscale ﬁxed.
These policies perform worse given fewer samples, but are comparable to the true kernel with sufﬁcient samples (Figure 15).
We attribute this to the compounding errors of kernel approximation and fewer effective samples.
In contrast to the policy search task, RFF s appear superior to RBF features.
Learning priors from data.
A beneﬁt of using GP priors is the ability to optimize hyperparameters from expert demonstrations through the likelihood or a divergence.
Moreover, the matrix normal distribution is useful for analyzing high-dimensional action sequences, as it decomposes temporaland action correlations into viewable covariance matrices.
Section D.3.3 shows the matrix normal distributions of expert demonstrations of the tasks, obtained through human and RL experts.
The results show that, surprisingly, the demonstrations are rougher than the smoothness achievable with
MPC .
We attribute this to control artifacts from demonstration collection and the use of Gaussian noise by RL agents.
Applying the same methodology to the demonstrations of the smooth MPC agents proposed here extracts the expected action correlations across tasks.
This analysis also raises the question of whether smoothness is an inductive bias we enforce for practicality, or a phenomena we expect to arise from optimality.
If the latter, it may be that the simulated environments or objectives considered are lacking components that encourage smoothness, such as energy efﬁciency.
Conclusion
We present a broad perspective on episodic posterior policy iteration method for robotics and new methods for the Monte Carlo setting, based on regularizing the IS approximations.
By considering vector-valued Gaussian processes for action priors, we have demonstrated how sample-efﬁcient MPC can be performed as online inference and with greater control over actuator smoothness, connecting
Monte Carlo MPC to prior work on policy search.
This approach was validated on a set of highdimensional MPC tasks closely matching baseline performance while achieving greater smoothness.
Limitations.
Much of the prior work is motivated by simplicity, minimizing hyperparameter tuning and numerical procedures such as matrix inversion [64].
In contrast, the contributions of this work introduces complexity, i.e. online temperature optimization and the use of dense covariance matrices in order to perform more sophisticated approximate inference.
While this additional complexity has an impact on execution time (Table 2, Appendix), we hope the sample-efﬁciency when combined with accelerations such as GPU -integration should produce real-time algorithms [13].
Acknowledgments
This work built on prior codebases developed by Johannes Silberbauer, Michael Lutter and Hany
Abdulsamad.
The large-scale experiments and ablations were conducted on the Lichtenberg high performance computer of the TU Darmstadt.
The authors wish to thank Hany Abdulsamad, Boris
Belousov, Michael Lutter, Fabio Muratore, Pascal Klink, Georgia Chalvatzaki, Kay Hansel, Oleg
Arenz and the anonymous conference reviewers for helpful feedback on earlier drafts.