Neural Solvers for Fast and Accurate Numerical Optimal Control
Abstract
Synthesizing optimal controllers for dynamical systems in practice involves solving real–time optimization problems with hard time constraints.
These constraints restrict the class of numerical methods that can be applied; indeed, computationally expensive but accurate numerical routines often have to be replaced with fast and inaccurate methods, trading inference time for worse theoretical guarantees on solution accuracy.
This paper proposes a novel methodology to accelerate numerical optimization of optimal control policies via hypersolvers , hybrids of a base solver and a neural network.
In particular, we apply low–order explicit numerical methods for the ordinary differential equation (ODE) associated to the numerical optimal control problem, augmented with an additional parametric approximator trained to reduce local truncation errors introduced by the base solver.
Given a target system to control, we ﬁrst pre-train hypersolvers to approximate base solver residuals by sampling plausible control inputs.
Then, we use the trained hypersolver to obtain fast and accurate solutions of the target system during optimization of the controller.
The performance of our approach is evaluated in direct and model predictive optimal control settings, where we show consistent Pareto improvements in terms of solution accuracy and control performance.
Introduction
Emerging applications of optimal control to complex, high–dimensional systems require the implementation of computationally expensive numerical methods for ordinary differential equations (ODEs) (Wanner & Hairer, 1996; Butcher, 1997).
Here, real–time and hardware constraints render certain numerical methods not applicable, forcing the application of computationally cheaper but otherwise less accurate algorithms.
While the paradigm of optimal control has successfully been applied in various domains (Vadali et al., 1999; Lewis et al., 2012; Zhang et al., 2016), these computation budget constraints are still a great challenge (Ross & Fahroo, 2006; Baoti´c et al., 2008).
Furthermore, lower–order explicit ODE solvers are known to be brittle in the presence of stiff dynamical systems (Wanner & Hairer, 1996), resulting in numerical issues and unreliable controllers.
This work alleviates these limitations by detailing a systematic procedure for the ofﬂine optimization and subsequence online application of ODE solvers equipped with a neural network approximator of higher–order local truncation residuals.
These hybrid solvers, known as hypersolvers (Poli et al., 2020a), employ a low–order, computationally cheaper solver while preserving theoretical local and global solution accuracy guarantees of traditional higher order methods.
We extend the range of applicability of hypersolvers to controlled dynamical systems, investigating and developing several strategies to enable their application during online solution of the optimal control problem without ﬁnetuning .
Namely, we consider hypersolvers whose neural network residual approximators are a priori optimized on an uncontrolled target system and then applied to signiﬁcantly speed up iterated simulations of the system later required by the control policy optimization algorithm.
The proposed two-stage scheme is tested in both direct and model predictive control settings and shows promising
Pareto optimal results in terms of solution accuracy and subsequent control performance.
[Figure]
Hypersolver Training
[Figure]
Numerical Optimal Control
Let us consider the following controlled dynamical system described:
[Equation]
) with state x ∈ X ⊂ R n x , input u θ ∈ U ⊂ R n u deﬁned on a compact time domain T :=
[ t 0 , T ] where θ is a set of free parameters of the controller.
Solutions of equation 1 are denoted with x ( t ) =
Φ( x ( s ) , s, t ) for all s, t ∈ T .
2.1 Optimal Control Problem
Given some objective function J : X × U → R ; x 0 , u θ (cid:55)→ J ( x 0 , u θ ( t )) , and a distribution ρ 0 ( x 0 ) of initial conditions with support in
X , we consider the following nonlinear program, constrained to the dynamics 1:
[Equation]
to
[Equation]
where the controller parameters θ are optimized.
We will omit the use of θ and write u ( t ) =
u θ ( t ) from now on for simplifying the notation.
Since analytic solutions of 2 exist only for limited classes of systems and objectives, numerical solvers are often applied to iteratively ﬁnd a solution instead: for this reason we refer to problem 2 as numerical optimal control .
2.2 Solver Residuals for Controlled Dynamical Systems
Given nominal solutions Φ of equation 1 we can deﬁne the residual of a numerical ODE solver as the normalized error accumulated in a single step size of the method, i.e.
[Equation]
where p is the order of the numerical solver corresponding to ψ (cid:15) .
Hypersolvers for Optimal Control 3.1 Formulation
Given a p –order base solver update map ψ (cid:15) , the corresponding hypersolver is the discrete iteration
[Equation]
(4) where g ω ( t k , x k , u k ) is some o (1) parametric function with free parameters ω .
The core idea is to select g ω as some function with universal approximation properties and ﬁt the higher-order terms of the base solver by explicitly minimizing the residuals over a set of state and input samples.
3.2 Training
We assume the considered optimal control problems to feature a control input constrained in a set
U .
Such constraints are often present either due to physical limitations of the actuators or safety restraints of the workspace; while this is not directly enforced nor necessary, the limited dataset of constrained controls ensures that hypersolver pre–training is performed successfully.
The analysis considers the case of time–invariant systems (A.3) in which dynamics of equation 1 is not directly dependent on time t , thus we can rewrite g w ( t, x ( t ) , u ( t ))
=
g w ( x, u ) .
We also check the generalization properties of hypersolvers: details are available in the Appendix B.
The training procedure can be written as the following optimization problem
[Equation]
(5) where ξ ( x, u ) is a distribution with support in X ×U of the state and controller spaces.
To guarantee sufﬁcient exploration of the state-controller space, we use Monte Carlo sampling (Robert & Casella, 2013) from the given distribution.
In particular, batches of initial conditions { x i 0 } , { u i 0 } are sampled from ξ and the optimization problem of equation 5 is solved via backpropagation for updating the parameters of the hypersolver g w using a stochastic gradient descent (SGD) algorithm, e.g., Adam (Kingma & Ba, 2017) and we repeat the procedure for every training epoch.
[Figure] [Figure]
p
P h a s e
S p a c e [Figure] q ( t ) P o s i
t i o n
[Figure]
p ( t ) M o m e n t u m [Figure] [Figure]
a
[Figure]
s i d [Figure] a l [Figure]
RK 103 ×
[Figure]
− [Figure] 6 × [Figure]
2 ×
[Figure]
F i n a l P
o s i t i o n E rr o r 3.3 Optimal Control Optimization with Hypersolvers
The pre-trained hypersolver model is employed for obtaining solutions to the trajectories of the optimal control problem of equation 2.
We then optimize the control policy for minimizing the cost function J ; we repeat the process iteratively until a solution is found.
Figure 1 shows an overview of the proposed two–stage approach consisting in hypersolver pre–training and subsequent optimal control.
Evaluation 4.1 Direct Optimal Control of a Pendulum
We consider the inverted pendulum model with a torsional spring described in equation 9.
After pre– training the hypersolver by sampling from the sets of possible states X and controllers U , we apply direct optimal control to optimize over full unrolled trajectories obtained via different ﬁxed-step solvers.
The hypersolved version of the Euler scheme, HyperEuler, improves control performance while introducing a minimal overhead as shown in Figure 2.
Further details are available in the Appendix C.1.
4.2 Model Predictive Control of a Quadcopter
We consider the three-dimensional quadcopter model of equation 10.
The controller was optimized online via the Model Predictive Control (MPC) (Garcia et al., 1989) algorithm whose receding horizon was calculated through different standard ODE solvers and our hypersolver approach.
Figure 3 shows that HyperEuler reduces mean local residuals even compared to the Midpoint solver, while control performance shows the lowest error on ﬁnal positions even compared to higher–order solvers.
Additional experimental details are available in the Appendix C.2.
Conclusion
We presented a novel method for obtaining fast and accurate control policies.
Hypersolver models were ﬁrstly pre–trained on distributions of states and controllers to approximate higher–order residuals of base ﬁxed–step ODE solvers.
The obtained models were then employed to improve the accuracy of trajectory solutions over which control policies were optimized.
We veriﬁed that our method shows consistent improvements in the accuracy of ODE solutions and thus on the quality of control policies optimized through numerical solutions of the system.
We envision the proposed approach to beneﬁt the control ﬁeld and robotics in both simulated and potentially real–world environments by efﬁciently solving high–dimensional time–continuous problems.
A Additional Hypersolver Material A.1 Bounds on Solution Errors
From the deﬁnition of residual in equation 3, we can deﬁne the local truncation error e k :=
(cid:13)(cid:13) (cid:15)
p {++
+} R
k
(cid:13)(cid:13) 2 .
Given a p -th order explicit solver, we have e k
= o ( (cid:15) p +1 )
(Butcher, 1997).
For hypersolvers, similar error bounds are deﬁned as e k ≤ o ( δ(cid:15) p +1 ) , where δ > 0 depends on the hypersolver training results (Poli et al., 2020a, Theorem 1).
This result practically guarantees that if g ω is a good approximator for R , i.e. δ (cid:28) 1 , then the overall local truncation error of the hypersolved ODE is signiﬁcantly reduced.
A.2 Explicit HyperEuler Formulation
During the experiments, we restricted our analysis to the hypersolved version of the Euler scheme, namely HyperEuler because of the highest potential speed-up it can achieve: since Euler is a ﬁrst-order method, it requires the least number of function evaluations (NFE) of the vector ﬁeld f in equation 1 and yields a second order local truncation error e k :=
(cid:13)(cid:13) (cid:15) 2 R k (cid:13)(cid:13) 2 , which is larger than other ﬁxed-step solvers.
In particular, the base solver scheme ψ of equation 4 can be written as ψ (cid:15) ( t k , x k , u k ) =
f ( t k , x k , u k ) , which is approximating the next state by adding an evaluation of the vector ﬁeld multiplied by the step size (cid:15) .
The HyperEuler scheme can be explicitly written [-as(cid:26) x k +
-]as
[Equation]
while its residual can be written as
[Equation]
A.3 Hypersolvers for Time-invariant Systems
A time-invariant system with time-invariant controller can be described as following
[Equation]
in which f and u do not explicitly depend on time.
A.4 Architecture
We designed the hypersolver network g w as a feed–forward neural network.
Table 1 summarizes the parameters used for the considered controlled systems, where activation functions SoftPlus : x (cid:55)→ log(1 + e x ) and Tanh : x (cid:55)→ e x − e − x [Equation] x +
[Equation]
− x .
[Table]
We also used the vector ﬁeld f as an input of the hypersolver, which, however, does not require a further evaluation since it is pre-evaluated at runtime by the base solver ψ .
We emphasize that the size of the network should depend on the application: a too-large neural network may require more computations than just increasing the numerical solver’s complexity: Pareto optimality of hypersolvers also depends on their complexity.
By keeping their neural network small enough, we can guarantee that evaluating the hypersolvers is cheaper than resorting to more complex numerical routines.
[-Tanh x → e x − e − x e x + e − x − 4 π − 2 π 0 
 π 4 π p ReLU x → max ( 0, x ) − 4 π − 2 π 0 2 π 4 π p-][Figure]
[Figure]
B Generalization Study
We assumed the state and controller spaces to be bounded and that training is performed sampling for their known distributions.
While this is sufﬁcient for optimal control problems given a priori known bounds, we also investigate how the system generalizes to unseen states and control input values.
[Figure]
In particular, we found that activation functions have an impact on the end result of generalization beyond known boundaries.
We took into consideration two commonly used activation functions, Tanh : x → e x − e − x e x + e − x and ReLU : x → max(0 , x ) , along with network architectures which employ activation functions containing periodic components: SIREN : x → sin ( Wx + b ) (Sitzmann et al., 2020) and
Snake : x → x + 1 a sin 2 ( ax ) (Ziyin et al., 2020).
We trained hypersolver models with the different activation functions for the inverted pendulum of equation 9 with common settings: time step (cid:15) was chosen as 0 .
1 s , networks were trained for 10 5 epochs with the Adam optimizer and learning rate of 10 − 3 , where sampling was done on from the uniform distribution ξ ( x, u ) with support in X × U with X =
[ − 2 π, 2 π ] × [ − 2 π, 2 π ] and U =
[ − 10 , 10] .
Fig- ure 4 and Figure 5 show generalization outside the states and controllers distribution respectively.
We notice that while Tanh and ReLU perform well on the training set of interest, performance degrades rapidly outside of it.
On the other one hand, SIREN and Snake manage to extrapolate the periodicity of the residual distribution even outside of the training region, thus providing further empirical evidence of the universal approximation theorem (Ziyin et al., 2020, Theorem 3).
C Experimental Details C.1 Inverted Pendulum
System Dynamics We model the inverted pendulum with elastic joint with Hamiltonian dynamics via the following:
[Equation]
(cid:21) u (9) where m =
[ Kg ] , k = 0 .
5 [ N/ rad] , r = 1 [ m ] , β = 0 .
[ Ns/ rad] , g = 9 .
[ m/s 2 ] .
{+− 10 u [ N ] 10 −10 −10 −
+}
M e a n R e s i
d u a l
{+R e = 0.
+}
{+[ s ] − 10 u [ N ] e = 0.03 [ s ] − 10 u [ N ] e = 0.
+}
[ s ] − 10 u [ N ] e = 0.3 [ s ] − 10 u [ N ] e = 1.0 [ s ] HyperEuler Euler Midpoint RK Hypersolver pre–training We select ξ ( x, u ) as a uniform distribution with support in X × U where
X =
[ − 2 π, 2 π ] × [ − 2 π, 2 π ] and U = [ − 5 , 5] to guarantee sufﬁcient exploration of the state- controller space.
Nominal solutions Φ are calculated using the Tsitouras 5 (Tsitouras, 2011) adaptive–step solver with absolute and relative tolerances set to 10 − 5 .
We train the hypersolver on local residuals using the Adam optimizer with learning rate of 3 × 10 − 4 for 3 × 10 5 epochs.
Figure 6 shows mean local residuals with hypersolver models trained on different step sizes.
Direct optimal control The goal is to stabilize the inverted pendulum in the vertical position x (cid:63) =
We choose t ∈ [0 , 3] and a step size (cid:15) = 0 .
2 s for the experiment.
The control input is assumed continuously time–varying.
The neural controller is optimized via SGD with Adam with learning rate of 3 × 10 − 3 for 1000 epochs.
Trajectories obtained with the controller optimized with HyperEuler reach ﬁnal positions q = (1 . 6 ± 17 . 6) × 10 − 2 while Midpoint and RK4 ones q = ( − 0 . 6 ± 12 . 7) × 10 − 2 and q = (1 . 1 ± 12 .
8) × 10 − 2 respectively. On the other hand, the controller optimized with the Euler solver fails to control some trajectories obtaining a ﬁnal q = (6 . 6 ± 19 . 4) × 10 − 1 . HyperEuler considerably improved on the Euler baseline while requiring only 1 . 2% more Floating Point Operations (FLOPs) and 49 . 5% less compared to Midpoint.
C.2 Quadcopter
System Dynamics
The quadcopter model is a suitably modiﬁed version of the explicit dynamics update in (Panerati et al., 2021) for batched training in PyTorch.
The dynamic model is described by the following accelerations update:
[Equation]
where x = [ x, y, z ] corresponds to the drone positions and ψ = [ φ, θ, ψ ] to its angular positions;
R and J are its rotation and inertial matrices respectively, τ ( · ) is a function calculating the torques induced by the motor speeds ω i , while arm length l , mass m , gravity acceleration constant g along with k F and k T are scalar variables describing the quadcopter’s physical properties.
Hypersolver pre–training We select ξ ( x, u ) as a uniform distribution with support in X × U where X is chosen as a distribution of possible visited states and each of the four motors i ∈ [0 , 3] has control inputs u i ∈ [0 , 2 . 17] × 10 5 rpm .
Nominal solutions Φ are calculated using the Dormand-Prince 5 (Dormand & Prince, 1980) adaptive–step solver with relative and absolute tolerances set to 10 − 7 and 10 − 9 respectively.
We train HyperEuler on local residuals using the Adam optimizer with learning rate of 10 − 3 for 10 5 epochs.
Model predictive control The control goal is to reach a ﬁnal positions ( x, y, z ) (cid:63) =
(8 , 8 , 8) m .
We choose t ∈ [0 , 3] and a step size (cid:15) = 0 .
02 s for the experiment.
The control input is assumed piece–wise constant during MPC sampling times.
The receding horizon is chosen as 0 . 5 s . The neural controller is optimized via SGD with Adam with learning rate of 10 − 2 for 20 iterations at each sampling time. 30 experiments are conducted starting at random initial conditions which are kept common for the different ODE solvers.
HyperEuler requires a single function evaluation per step as for the Euler solver compared to two function evaluations per step for Midpoint and four for RK4. Controlled trajectories optimized with Euler, Midpoint and RK4 collect an error on ﬁnal positions of (1 . 09 ± 0 . 37) m , (0 . 71 ± 0 . 17) m , (0 . 70 ± 0 . 19) m respectively while HyperEuler achieves the lowest terminal error value of (0 . 66 ± 0 .
24) m .
C.3 Hardware and Software
Experiments were carried out on a machine equipped with an AMD R YZEN T HREADRIPPER 3960X CPU and two NVIDIA RTX 3090 graphic cards.
Software–wise, we used
PyTorch (Paszke et al., 2019) for deep learning and the torchdyn (Poli et al., 2020b) and torchdiffeq (Chen et al., 2019) libraries for ODE solvers. D Broader Impact
We envision our hypersolver approach to optimal control to be applicable to real–world online control problems.
Let us present some considerations that should be taken into account in these regards.
Firstly, implementation–wise, ad–hoc hardware choices may need to be done for fast inference of the approximator depending on its nature, e.g., neural networks (Sze et al., 2017); additionally, depending on the computational requirements of the control algorithms (Torrente et al., 2021).
Secondly, unlike in simulated environments, imperfections in the modeling itself or disturbances can lead to sub–optimal controllers by incorrectly solving the control problem’s associated ODEs.
However, correction terms can be learned and combined with the dynamics model to improve the control policy (Carron et al., 2019;
Hewing et al., 2020).
Similarly, hypersolver models trained on residuals derived from nominal data–driven trajectories may be able to correct modeling errors and disturbances to a certain extent.
Nonetheless, we believe future works on hypersolvers for optimal control represent an exciting avenue for obtaining fast and accurate optimal control policies not only in simulated, but also in real–world and production environments.