Risk-Averse Zero-Order Trajectory Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We introduce a simple but effective method for managing risk in zeroorder trajectory optimization that involves probabilistic safety constraints andbalancing of optimism in the face of epistemic uncertainty and pessimism in theface of aleatoric uncertainty of an ensemble of stochastic neural networks.
Variousexperiments indicate that the separation of uncertainties is essential to performingwell with data-driven MPC approaches in uncertain and safety-critical controlenvironments.
Introduction
Data-driven approaches to sequential decision-making are becoming increasingly popular [1, 2, 3, 4].
They hold the promise of reducing the number of prior assumptions about the system that are imposedby traditional approaches that are based on nominal models.
Such approaches come in several different ﬂavors [5].
Model-free approaches attempt to extractclosed-loop control policies directly from data, while model-based approaches rely on a learned modelof the dynamics to either generate novel data to extract a policy or to be used in a model-predictivecontrol fashion (MPC).
This study belongs to the latter line of work.
Model-based methods have several advantages over pure model-free approaches.
Firstly, humanstend to have a better intuition on how to incorporate prior knowledge into a model rather than into apolicy or value function.
Secondly, most model-free policies are bounded to a speciﬁc task, whilemodels are task-agnostic and can be applied for optimizing arbitrary cost functions, given sufﬁcientexploration.
Nevertheless, learning models for control come with certain caveats.
Traditional MPC methodsrequire the model and cost function to permit a closed-form solution which restricts the function classprohibitively.
Alternatively, gradient-based iterative optimization can be employed, which allows fora larger class of functions but typically fails to yield satisfactory solutions for complicated functionapproximators such as deep neural network models.
In addition, calculating ﬁrst-order or evensecond-order information for trajectory optimization tends to be computationally costly, which makesit hard to meet the time constraints of real-world settings.
This motivates the usage of zero-order, i.egradient-free or sample-based methods, such as the Cross-entropy Method (CEM) that do not rely ongradient information but are efﬁciently parallelizable.
Many methods relying on a learned model and zero-order trajectory optimizers have been proposed[6, 7, 8], but all share the same problem: compounding of errors through auto-regressive modelprediction.
This naturally brings us to the question of how can we effectively manage model errorsand uncertainty to be more data-efﬁcient and safe.
Arguably, this is one of the main obstacles toapplying data-driven model-based methods to the real world, e.g. to robotics settings.
In this work, we introduce a risk-averse zero-order trajectory optimization method (RAZER) formanaging errors and uncertainty in zero-order MPC and test it on challenging scenarios (Fig. 1).
Weargue that it is essential to differentiate between the two types of uncertainty in the model-predictivesetting: the aleatoric uncertainty arising from inherent noise in the system and epistemic uncertaintyarising from the lack of knowledge [9, 10].
We measure these uncertainties by making use ofprobabilistic ensembles with trajectory sampling [6] (PETS).
Our contributions can be summarized
[Figure]
as follows: (i) method for separation of uncertainties in probabilistic ensembles (termed PETSUS);(ii) efﬁcient use of aleatoric and epistemic uncertainty in model-based zero-order trajectory optimizers;(iii) an simple but practical approach to probabilistic safety constraints in zero-order MPC.
Related Work
Uncertainty Estimation.
In the typical model-based reinforcement learning (MBRL) setting, thetrue transition dynamics function is modeled through an approximator.
Impressive results have beenachieved by both parametric models [11, 12, 13, 14], such as neural networks, and nonparametricmodels [15, 16, 17, 18], such as Gaussian Processes (GP).
The latter inspired seminal work on theincorporation of the dynamics model’s uncertainty for long-term planning [18, 19].
However, theirusability is limited to low-data, low-dimensional regimes with smooth dynamics [20, 21], which is notideal for robotics applications.
Alternative parametric approaches include ensembling of deep neuralnetworks, used both in the MBRL community [6, 22], and outside [23, 24].
In particular, ensemblesof probabilistic neural networks established state-of-the-art results in the MBRL community [6],but focus mainly on estimating the expected cost and disregard the underlying uncertainties.
Incomparison, we propose a treatment of the resulting uncertainties of the ensemble model.
Zero-order MPC.
The learned model can be used for policy search like in PILCO [25, 18, 19, 26] orfor online model-predictive control (MPC) [27, 28, 6].
In this work, we do planning in an MPC fashionand employ a zero-order method as a trajectory optimizer, since less sensitive to hyperparametertuning and less likely to get stuck in local minima of complex objective functions.
Speciﬁcally, weconsider a sample-efﬁcient implementation of the Cross-Entropy method [29, 30] introduced in [31].
Safe MPC.
Separating the sources of uncertainty is of particular importance for AI applicationsdirectly affecting humans’ safety, as self-driving cars, elderly care systems, or in general any application that involves a physical interaction between the AI system and humans.
Disentanglingepistemic from aleatoric uncertainty allows for separate optimization of the two, as they representsemantically different objectives: efﬁcient exploration and risk-awareness.
Extensive research onuncertainty decomposition has been done in the Bayesian setting and the context of safe policysearch [32, 33, 34, 35], MPC planning [36, 37, 38], and distributional RL [39, 40].
On the other side,a state-of-the-art baseline for ensemble learning like PETS [6], despite estimating both uncertainties,only optimizes for the expected cost during action evaluation.
Our work aims at ﬁlling this gap byexplicitly integrating the propagated uncertainty information in the zero-order MPC planner.
Method
Our approach concerns itself with the efﬁcient usage of uncertainties in zero-order trajectory optimization and is therefore generally applicable to such optimizers.
We are interested in modeling noisysystem dynamics x t +1 = f
( x t , u t , w ( x t , u t )) where f is a nonlinear function, x t the observationvector, u t applied control input and w ( x t , u t ) a noise term sampled from an arbitrary distribution.
Consequently, in the absence of prior knowledge about the function f , the system needs to bemodeled by a complex function approximator such as a neural network.
Furthermore, we areinterested in managing uncertainties based on our ﬁtted model, which is erroneous.
To this end, weuse stochastic ensembles of size K , where the output of each model ϑ k ( x t , u t ) are parameters of a normal distribution depending on input observation x t and control u t .
As a by-product, our autoregressive model prediction based on controls u becomes a predictive distribution over trajectoriesτ ; ψ τ ( x t , u ) :
= p ( τ | x t , u ; θ ) where θ denotes the parameters of the ensemble.
For convenience,from this point onward we will differentiate between multiple usages of ψ τ .
We denote with ψ x ∆ t thedistribution p ( x t +∆ t | x t , u t : t + h ; θ ) over states at time step t + ∆ t and ψ ϑ ∆ t the distribution over the
Gaussian parameter outputs p ( ϑ t +∆ t | x t , u t : t + h ; θ ) at time step t + ∆ t of the planner.
3.1 Planning and Control
To validate our hypothesis that accounting for uncertainty in the environment and model prediction isessential to develop risk-averse policies, we use the Cross-Entropy Method (CEM) with improvementssuggested in Pinneri et al.
Accordingly, at each time step t we sample a ﬁnite number of controlsequences u for a ﬁnite horizon H from an isotropic Gaussian prior distribution which we evaluatefrom the state x t using an auto-regressive forward-model and the cost function.
The samplingdistribution is reﬁtted in multiple rounds based on good-performing (elite) trajectories.
After thisoptimization step, the ﬁrst action of the mean of the ﬁtted Gaussian distribution is executed.
Sincethis approach utilizes a predictive model for a ﬁnite horizon at each time step, it naturally falls intothe category of Model Predictive Control (MPC) methods.
Although we use CEM, our approach of managing uncertainty can generically be applied to otherzero-order trajectory optimizers such as MPPI [28], by a modiﬁcation of the trajectory cost function.
3.2 The Problem of Uncertainty Estimation
Since we have a stochastic model of the dynamics, at the model prediction time step t we observe adistribution over potential outcomes.
Indeed, since our model outputs are parameters of a Gaussiandistribution, with auto-regressive predictions we end up with a distribution over possible Gaussiansfor a certain time step t .
Given a sampled action sequence u and the initial state x t we observe a distribution over trajectoriesψ τ .
To efﬁciently sample from the trajectory distribution ψ τ we use the technique introduced by Chuaet al.
[6] (PETS) which involves prediction particles that are sampled from the probabilistic modelsand randomly mixed between ensemble members at each prediction step.
In this way, the sampledtrajectories are used to perform a Monte Carlo estimate of the expected trajectory cost E τ ∼ ψ τ [ c ( τ )] .
However, this does not take the properties of ψ τ into account, which might be a high-entropydistribution and may lead to very risky and unsafe behavior.
In this work, we alleviate this by lookingat the properties of ψ τ , i.e. different kinds of uncertainties arising from the predictive distribution.
3.3 Learned Dynamics Model
We learn a dynamics model f θ that approximates the true system dynamics x t + =f ( x t , u t , w ( x t , u t )) .
As a model class, we use an ensemble of neural networks with stochasticoutputs as in Chua et al.
Each model k , parameterizes a multivariate Gaussian distribution withdiagonal covariance, f kθ ( x t , u t ) = N ( x t +1 ; x t + µ kθ ( x t , u t ) , Σ kθ ( x t , u t )) where µ kθ ( · , · ) and Σ kθ ( · , · )are model functions outputting the respective parameters.
Iteratively, while interacting with the environment, we collect a dataset of transitions D and traineach model k in the ensemble by the following negative log-likelihood loss on the Gaussian outputs:
[Equation]
In addition, we use several regularization terms to make the model training more stable.
We providemore details on this in Suppl.
A.
3.4 Separation of Uncertainties
In the realm of parametric estimators, two uncertainties are of particular interest.
Aleatoric uncertaintyis the kind that is irreducible and results from inherent noise of the system, e.g. sensor noises in robots.
On the other hand, we have epistemic uncertainty resulting from lack of data or knowledge which isreducible.
This begs the question, how can we separate these uncertainties given an auto-regressive dynamics model f θ ?
The way that we efﬁciently sample from ψ τ is by mixing sampled predictionparticles, similarly as in PETS[6].
This process is illustrated by the red lines in Fig.
Simple model prediction disagreement is not a good measure for aleatoric uncertainty since it can beentangled with epistemic uncertainty.
Given our assumptions about the system dynamics, we measurealeatoric uncertainty as the entropy of the predicted normal distributions of the ensemble models.
More concretely, given a sampled particle state ˜ x t , we deﬁne the estimated aleatoric uncertainty forensemble model associated to particle b at time step t as:
[Equation]
Where ψ x ∆ t,b is the output distribution of ensemble model based on inputs ˜ x t , u t .
Since in the end weare interested in the aleatoric uncertainty incurred from applying the action sequence u from initialstate x t , the quantity of interest for us is the expected aleatoric uncertainty for time slice t :
[Equation]
Intuitively, because we only have access to the ensemble for sampling, we take a timeslice in the sampled trajectories from ψ τ and compute the output entropies.
Moreover,since we assume a Gaussian 1-step predictive distribution this is an expectation over differential Gaussian entropy.
An alternative way of computation which we also explore in thiswork is calculating the expected particle variance for time slice t of the prediction horizon: [Figure] + [Figure]
+
[Figure]
NN
[Figure]
[Equation]
For estimating the epistemic uncertainty, one would be tempted to lookat the disagreement between ensemble models in parameter space Var[ θ ] ,but this is not completely satisfying,since neural networks tend to be overparametrized and variance within theensemble still may exist albeit the optimum has been reached by all ensemblemodels.
An alternative would be tocalculate the Fisher information met-
[Equation]
where L denotes the likelihood function, but this tends to be expensive to compute.
Given the assumption of local Gaussianity, the true epistemic uncertainty for this case is the predictiveentropy over the Gaussian parameters ϑ at time step t + h .
[Equation]
It is easy to verify that this quantity is 0 given perfect predictions of the model.
Note that, because ofauto-regressive predictions of a nonlinear model, this is a very difﬁcult object to handle.
Nevertheless,since our predictive distribution p ( x | x t , u t ; ϑ ) is parametrized by model outputs, we may utilizedisagreement in ϑ t to approximate E .
To get correct estimations, we need to propagate meanpredictions
¯ x in addition to the particles as illustrated as the yellow lines in Fig.
We quantifyepistemic uncertainty as ensemble disagreement at time step t :
[Equation]
where Var e is the empirical variance over the k = 1 . .
K ensembles.
3.5 Probabilistic Safety Constraints
When applying data-driven control algorithms to real systems, safety is of utmost importance.
In therealm of zero-order optimization, safety constraints can be easily introduced by putting an inﬁnitecost on constraint-violating trajectories.
Nevertheless, we are dealing with erroneous stochasticnonlinear models which lead to nontrivial predictive distributions of future states, based on the control sequence u .
For this reason, we want to control the risk of violating the safety constraints that we,as practitioners, are willing to tolerate.
If we denote the observation space as X , given a violationset C ⊂ X , we deﬁne the probability of the control sequence u to enter the violation set at timet + ∆ t as p ( x ∈ C | x t , u ) =
(cid:82) x ∈ C ψ x ∆ t ( x | x t , u ) .
In practice, it is hard to compute this integralefﬁciently, since our distribution ψ x ∆ t is nontrivial as a result of nonlinear propagation of uncertainty.
Furthermore, the violation set C might not have the structure necessary to allow an efﬁcient solutionto the integral, in which case one needs to resort to Monte Carlo estimation.
To simplify computation and gain speed, we consider box violation sets resulting in each dimensionof x being constrained to be outside of [ a, b ] ∈ { a, b | a, b ∈ R 2 , a < b } .
By performing momentmatching by a Gaussian in each time-slice ψ x ∆ t , the probability of ending up in state x at time stept + ∆ t is given by integrating N ( x ; µ t +∆ t , Σ t +∆ t ) , where µ and Σ are estimated by Monte Carlosampling.
If we further assume a diagonal covariance Σ , this integral can be deconstructed into dunivariate Gaussian integrals, which can be computed fast and in closed form (error function).
Hence,the probability of a constraint violation happening at time step t is deﬁned by:
[Equation]
3.6 Implementing Risk-Averse ZERo-Order Trajectory Optimization (RAZER)
We assume the task deﬁnition is provided by the cost c ( x t , u ) .
For trajectory optimization, we startfrom a state x t and predict with an action sequence u the future development of the trajectory τ .
Along this trajectory, we want to compute a single cost term which is conveniently deﬁned as theexpected cost of all particles ˜ x summed over the planning horizon H :
[Equation]
The optimizer, in our case CEM, will optimize the action sequence u to minimize the cost in aprobabilistic sense, i.e. p ( u | x ) ∝ exp( − β c ( x, u )) where β reﬂects the strength of the optimizer(the higher the more likely it ﬁnds the global optimum).
To make the planner uncertainty-aware, weneed to make sure it avoids unpredictable parts of the state space by making them less likely.
Usingthe aleatoric uncertainty provided by PETSUS Eq.
4, we deﬁne the aleatoric penalty as
[Equation]
where w A > 0 is a weighting constant.
The larger the aleatoric uncertainty, the higher the cost.
To guide the exploration to states where the model has epistemic uncertainty Eq.
6 (due to lack ofdata), we use an epistemic bonus:
[Equation]
where w E > 0 is a weighting constant.
To be able to operate on a real system, the most importantpart is to adhere to safety constraints.
As formulated in Eq.
7, the predicted safety violations need tobe uncertainty aware, independent of the source of uncertainty.
We integrate this into the planningmethod by adding:
[Equation]
11) where (cid:74) · (cid:75) is Iverson bracket and w S is either a large penalty c max or 0 to disable safety.
An alternativefor implementing safety constraints into CEM is by changing the ranking function [41].
The overallalgorithm used in a model-predictive control fashion is outlined in Suppl.
B.
Experiments
We study our uncertainty-aware planner in 4 continuous state-/action-space environments and compareresults with PETS [6].
PETS does not use the model’s uncertainties explicitly, instead, they enteronly by taking the mean over the particle costs.
We ﬁrst describe the environments.
[Figure] BridgeMaze This toy environment (see Fig. 1c), we speciﬁcally designed to study the different aspects of uncertainty independently.
The agent (blue cube) starts on the left platform and has toreach the goal platform on the right.
To reach the goal platform, the agent has to move over one ofthree bridges without falling into the lava.
The upper bridge is safeguarded by walls; hence, it isthe safest path to the goal but also the longest.
The lower bridge has no walls and therefore is moredangerous for an unskilled agent to cross but the path is shorter.
The middle bridge is the shortestpath to the goal.
However, randomly appearing strong winds perpendicular to the bridge might causethe agent to fall off the bridge with some probability, rendering this bridge dangerous.
Noisy-HalfCheetah
This environment is based on HalfCheetah-v3 from the OpenAI Gym toolkit.
We introduce aleatoric uncertainty to the system by adding Gaussian noise ξ ∼ N ( µ, σ 2 ) to theactions when the velocity is above 6 .
The action noise translates into a non-Gaussian and potentiallyvery complicated state-space noise distribution that makes the control problem very challenging.
Noisy-FetchPickAndPlace Based on the FetchPickAndPlace-v1 gym environment.
Additive actionnoise is applied to the gripper so that its grip on the box might become tighter or looser.
The noise isapplied for x -positions < 0 .
8 which is illustrated in Fig.
1a by a blue line causing the agent to dropthe box with high probability if it tries to lift the box too early.
Solo8-LeanOverObject
In this robotic environment, the task of a quadrupedal robot [42] is to standup and lean forward to reach a target position (purple markers need to reach green dots in Fig.
1b)without hitting an object visualized by the red cube representing the unsafe zone.
The robot starts in alaying position as shown in the inset of Fig.
1b.
As in the Noisy-HalfCheetah environment, Gaussianaction noise is applied to mimic real-world perturbances.
4.1 Algorithmic Choices and Training Details
For the control part, we use the CEM implementation from [31], both for RAZER and PETS.
See Suppl. A.2 for further details on the parameters we used in each experiment.
For planning, we usethe same architecture for the ensemble of probabilistic models, both in RAZER and in PETS.
Theonly difference is that in RAZER we also forward propagate the mean state predictions in addition tothe sampled state predictions.
Further details can be found in Suppl.
A.1.
For training the predictive model, we alternate between two phases: data collection and model ﬁtting. In the BridgeMaze environment, we collect 5 rollouts of length 80 each time and append them to theprevious rollouts and ﬁt the model for 25 epochs. For Noisy-HalfCheetah we collect 1 rollout everytime and ﬁt for 50 epochs.
Next, we will present RAZER’s exploration and safety in the BridgeMaze environment.
Afterward,we are going to discuss planning with external safety constraints in the Solo8 environment.
Wecomplete this section with results on Noisy-HalfCheetah and Noisy-FetchPickAndPlace .
4.2 Active Learning for Model Improvement
If model uncertainties are used for risk-averse planning, they are only meaningful if the model hasthe right training data. Only from good data can the parameters of the approximate noise model belearned correctly.
In case of too little data, the agent might avoid parts of the state space due to an
0.0PETS 0.
Aleatoric Penalty 0.
S u cc e ss R a t e
[Figure]
w [Figure] 50 runs.
5.0 Gripper Noise 0.
D r
o
pp i n g R a t e
RAZER PETS
[Figure]
Figure 4a shows this active data gathering process for the BridgeMaze environment.
PETS ﬁnds oneparticular solution to the problem of reaching the goal platform. It chooses the path over the safer,lower bridge over the dangerous middle path and the longer path via the upper bridge (Fig. 3b).
Once,one solution is found, the model overﬁts to it without exploring any other parts of the state space. This is also reﬂected in the plateauing of the red curve in Fig. 3a.
In comparison, RAZER actively explores larger and larger parts of the state space with an increasingweight of the epistemic bonus (Fig. 3a).
RAZER not only ﬁnds the easy solution found by PETS butalso extensively explores other parts of the state space (Fig. 3c).
To not get stuck at the middle bridgeduring exploration due to the inherent noise in the associated states, it is important to separate betweenepistemic and aleatoric uncertainties. Only the former should be used for exploration.
With enoughdata, our model can correctly capture the uncertainties of these states resulting in the epistemicuncertainty approaching zero.
4.3 Risk-Averse Planning
Once a good model is learned, the estimated noise model can be actively used for safe planning.
Whatdifferentiates RAZER from PETS is that our method makes full usage of the information from thenoise model while in PETS the uncertainties only enter planning by taking the mean over the particlecosts.
First, the PETS approach does not differentiate between aleatoric and epistemic uncertainty,and second, the sample-based approach suffers from the typically low ( ∼ 5 ) sample size.
BridgeMaze:.
Figure 4a shows the success rate of PETS and RAZER in the BridgeMaze.
Inboth cases, we use the same model that was trained from data collected during a training run withw E = 0 . 05 .
Hence, the model saw enough training data from all parts of the state space.
The noise inthe env is tuned such that there is a chance to cross the bridge without falling.
While in Fig.
3b PETSavoided this path because of an overestimation of the state due to a lack of enough training data nowit sometimes sees a chance to cross the bridge.
However, these attempts are very likely to fail becauseof stronger winds that occur randomly, resulting in a success rate of only 58% .
RAZER does not relyon sampling for the aleatoric part and can thus avoid risk.
With a higher penalty constant the successrate increase up to 96% but only as long as the agent is willing to take a risk at all.
For large values ofw A the agent becomes so afraid that it only moves slowly (decreasing reward in Fig. 4a).
Noisy-HalfCheetah.
How does RAZER perform on the Noisy-HalfCheetah environment whenlearning models from scratch?
Without aleatoric penalty, the planner is optimistic.
Risky situationsare only detected if a failing particle is sampled.
Thus, the noise is mostly neglected and the robotincreases its velocity, gets destabilized, and ends up slower than with the aleatoric penalty (Fig. 5a).
FetchPickAndPlace.
In this environment, a 7-DoF robot arm should bring the box to a targetposition – starting and target positions are at the opposite sides of the table.
The shortest path is to liftthe box and move in a straight line to the target.
However, with noise applied to the gripper action,there is a certain probability to drop the box along the way.
When penalizing aleatoric uncertainty,this is avoided and also fewer trajectory samples are “wasted” in high-entropic regions, as presented [Figure] [Figure]
in Fig.
1a.
Figure 4b shows the number of times the box is dropped on the table depending on thealeatoric penalty.
RAZER adopts a cautious behavior, preferring to slide the box on the table andlifting it only in the area without action noise, keeping below 20%, even when considerable noise isapplied.
4.4 Planning with External Safety Constraints
Noisy-HalfCheetah:.
We consider a safety constraint on the height of the body above groundsimulating a narrow passage.
Figure 5b shows the number of safety violations.
Note that PETS hasthe same penalty cost for hard violations.
Solo8-LeanOverObject:.
In this experiment, the robot has to move to two target points with itsfront and rear of the trunk while avoiding entering a speciﬁed rectangular area (fragile object).
Thefront feet are ﬁxed.
To track the points, the robot has to lean forward, such that it can lose balancedue to noisy actions.
RAZER successfully manages to satisfy the safety constraints almost always,which is in contrast to PETS, as shown in Fig.
However, satisfying the safety constraint comeswith the cost of reduced tracking accuracy.
Conclusion
In this work, we have provided a methodology to separate uncertainties in stochastic ensemblemodels (PETSUS) which can be used as a tool to build risk-averse model-based planners that arealso data-efﬁcient and enforce safety through probabilistic safety constraints (RAZER).
This type ofrisk-averseness can be achieved by a simple modiﬁcation of the cost function in form of uncertaintypenalties in zero-order trajectory optimizers.
Furthermore, the separation of uncertainties allows us to do proper exploration via epistemic bonuswhich beneﬁts generalization of the model and therefore makes it applicable to more settings.
Asfuture work, it would be of interest to see this approach applied to a proper transfer learning settingfrom simulations to real systems, where risk-averseness combined with exploratory behavior is crucialfor efﬁcient learning and safe operation.