N EURAL S OLVERS FOR F AST AND A CCURATE N UMERICAL O PTIMAL C ONTROL
A BSTRACT
Synthesizing optimal controllers for dynamical systems often involves solving op- timization problems with hard real time constraints.
These constraints determine the class of numerical methods that can be applied: computationally expensive but accurate numerical routines are replaced by fast and inaccurate methods, trading inference time for solution accuracy.
This paper provides techniques to improve the quality of optimized control policies given a ﬁxed computational budget.
We achieve the above via a hypersolvers (Poli et al., 2020a) approach which hybridizes a differential equation solver and a neural network.
The performance is evaluated in direct and receding–horizon optimal control tasks in both low and high dimensions, where the proposed approach shows consistent Pareto improvements in solution accuracy and control performance
I NTRODUCTION
Optimal control of complex, high–dimensional systems requires computationally expensive numerical methods for differential equations (Pytlak, 2006; Rao, 2009).
Here, real–time and hardware constraints preclude the use of accurate and expensive methods, forcing instead the application of cheaper and less accurate algorithms.
While the paradigm of optimal control has successfully been applied in various domains (Vadali et al., 1999; Lewis et al., 2012; Zhang et al., 2016), improving accuracy while satisfying computational budget constraints is still a great challenge (Ross & Fahroo, 2006; Baoti´c et al., 2008).
To alleviate computational overheads, we detail a procedure for ofﬂine optimization and subsequent online application of hypersolvers (Poli et al., 2020a) to optimal control problems.
These hybrid solvers achieve the accuracy of higher–order methods by augmenting numerical results of a base solver with a learning component trained to approximate local truncation residuals.
When the cost of a single forward–pass of the learning component is kept sufﬁciently small, hypersolvers improve the computation–accuracy Pareto front of low–order explicit solvers (Butcher, 1997).
However, direct application of hybrid solvers to controlled dynamical system involves learning truncation residuals on the higher–dimensional spaces of state and control inputs.
To extend the range of applicability of hypersolvers to controlled dynamical systems, we propose two pretraining strategies designed to improve, in the set of admissible control inputs, on the average or worst–case hypersolver solution.
With the proposed methodology, we empirically show that Pareto front improvements of hypersolvers hold even for optimal control tasks.
In particular, we then carry out performance and generalization evaluations in direct and model predictive control tasks.
Here, we conﬁrm Pareto front improvements in terms of solution accuracy and subsequent control performance, leading to higher quality control policies and lower control losses.
In high–dimensional regimes, we obtain the same control policy as the one obtained by accurate high–order solvers with more than 3 × speedup.
{𝑥𝑥
[Figure]
𝑢𝑢
[Figure]
Hypersolver Training
[Figure]
𝑅𝑅
[Figure]
𝒘𝒘
[Figure]
𝑥𝑥
[Figure]
𝑢𝑢 )
[Figure]
𝑢𝑢 [Figure] 𝑥𝑥
[Figure]
𝜉𝜉
[Figure]
𝑥𝑥
[Figure]
𝑢𝑢)
[Figure]
𝒈𝒈 ℓ 𝒙𝒙 𝒖𝒖 {𝑥𝑥
[Figure]
𝑢𝑢 1 } {𝑥𝑥 2 ,𝑢𝑢 2 } {𝑥𝑥 𝑛𝑛
[Figure]
𝑢𝑢 𝑛𝑛
[Figure]
Optimal Control
[Figure]
N UMERICAL O PTIMAL C ONTROL
We consider control of general nonlinear systems of the form
[Equation] (1) with state x ∈ X ⊂ R n x , input u θ ∈ U ⊂ R n u deﬁned on a compact time domain T :=
[ t 0 , T ] where θ is a ﬁnite set of free parameters of the controller.
Solutions of (1) are denoted with x ( t ) =
Φ( x ( s ) , s, t ) for all s, t ∈ T .
Given some objective function J : X ×U → R ; x 0 , u θ (cid:55)→ J ( x 0 , u θ ( t )) and a distribution ρ 0 ( x 0 ) of initial conditions with support in X , we consider the following nonlinear program, constrained to the system dynamics:
[Equation]
where the controller parameters θ are optimized.
We will henceforth omit the subscript θ and write u ( t ) =
u θ ( t ) .
Since analytic solutions of (2) exist only for limited classes of systems and objectives, numerical solvers are often applied to iteratively ﬁnd a solution.
For these reasons, problem 2 is often referred to as numerical optimal control .
Direct optimal control If the problem (2) is solved ofﬂine by directly optimizing over complete trajectories, we call it direct optimal control.
The inﬁnite–dimensional optimal control problem is time–discretized and solved numerically: the obtained control policy is then applied to the real target system without further optimization.
Model predictive control Also known in the literature as receding horizon control, Model Predictive Control (MPC) is a class of ﬂexible control algorithms capable of taking into consideration constraints and nonlinearities (Mayne & Michalska, 1988; Garcia et al., 1989).
MPC considers ﬁnite time windows which are then shifted forward in a receding manner.
The control problem is then solved for each window by iteratively forward–propagating trajectories with numerical solvers i.e. predicting the set of future trajectories with a candidate controller u ( t ) and then adjusting it iteratively to optimize the cost function J (further details on the MPC formulation in Appendix B.2).
The optimization is reiterated online until the end of the control time horizon.
2. S OLVER R ESIDUALS
Given nominal solutions Φ of (1) we can deﬁne the residual of a numerical ODE solver as the normalized error accumulated in a single step size of the method, i.e. [Equation]
[Equation]
where (cid:15) is the step size and p is the order of the numerical solver corresponding to ψ (cid:15) .
From the deﬁnition of residual in (3), we can deﬁne the local truncation error e k :=
(cid:13)(cid:13) (cid:15)
p
+
R
k
(cid:13)(cid:13) 2 which is the error accumulated in a single step;
while the global truncation error E k = (cid:107) x ( t k ) − x k (cid:107) 2 represents the error accumulated in the ﬁrst k steps of the numerical solution.
Given a p –th order explicit solver, we have e k
= O ( (cid:15) p +1 ) and E k
= O ( (cid:15) p ) (Butcher, 1997).
H YPERSOLVERS FOR O PTIMAL C ONTROL
We extend the range of applicability of hypersolvers (Poli et al., 2020a) to controlled dynamical systems.
In this Section we discuss the proposed hypersolver architectures and pre–training strategies of the proposed hypersolver methodology for numerical optimal control of controlled dynamical systems.
3. H YPERSOLVERS
Given a p –order base solver update map ψ (cid:15) , the corresponding hypersolver is the discrete iteration
[Equation]
where g ω ( t k , x k , u k ) is some o (1) parametric function with free parameters ω .
The core idea is to select g ω as some function with universal approximation properties and ﬁt the higher-order terms of the base solver by explicitly minimizing the residuals over a set of state and control input samples.
This procedure leads to a reduction of the overall local truncation error e k , i.e. we can improve the base solver accuracy with the only computational overhead of evaluating the function g ω .
It is also proven that, if g ω is a δ –approximator of R , i.e. ∀ k ∈ N ≤ K
[Equation]
then e k ≤ o ( δ(cid:15) p +1 ) , where δ > 0 depends on the hypersolver training results (Poli et al., 2020a,
Theorem 1).
This result practically guarantees that if g ω is a good approximator for R , i.e. δ (cid:28) 1 , then the overall local truncation error of the hypersolved ODE is signiﬁcantly reduced with guaranteed upper bounds.
3. N UMERICAL O PTIMAL C ONTROL WITH H YPERSOLVERS
Our approach relies on the pre–trained hypersolver model for obtaining solutions to the trajectories of the optimal control problem (2).
After the initial training stage, control policies are numerically optimized to minimize the cost function J (see Appendix B.3 for further details).
Figure 1 shows an overview of the proposed approach consisting in pre–training and system control.
H YPERSOLVER P RE – TRAINING AND A RCHITECTURES
We introduce in Section 4.1 loss functions which are used in the proposed pre–training methods of Section 4.2 and Section 4.3.
We also check the generalization properties of hypersolvers with different architectures in Section 4.4.
In Section 4.5 we introduce multi–stage hypersolvers in which an additional ﬁrst–order learned term is employed for correcting errors in the vector ﬁeld.
4. L OSS F UNCTIONS
Residual ﬁtting Training the hypersolver on a single nominal trajectory { x ( t k ) } k results in a supervised learning problem where we minimize point–wise the Euclidean distance between the [Figure] [Figure] e a n R e
s i d u a l
R e = 0.
[Figure] [ s ] [Figure] [Figure]
u [ N ] e = 0.
[Figure]
[ s ] −
[Figure]
20 u [ N ] e = 0.
[Figure]
[ s ] −
[Figure]
20 u [ N ] e = 0.3 [ s ] − 20 u
[ N ] e = 1.
[Figure]
[ s ] [Figure] Euler Midpoint
[Figure]
residual (3) and the output of g ω , resulting in an optimization problem minimizing a loss function (cid:96) of the form
[Equation]
which is also called residual ﬁtting since the target of g w is the residual R .
The optimization can also be carried out via trajectory ﬁtting as following
[Equation]
[Equation] where x ( t k +1 ) corresponds to the exact one–step trajectory and x k +1 is its approximation, derived via (4) for standard hypersolvers or via (11) for their multi–stage counterparts.
This method can also be used to contain the global truncation error in the T domain.
We will refer to (cid:96) as a loss function of either residual or trajectory ﬁtting types; we note that these loss functions may also be combined depending on the application.
The goal is to train the hypersolver network to explore the state– control spaces so that it can effectively minimize the truncation error.
We propose two methods with different purposes: stochastic exploration aiming at minimizing the average truncation error and active error minimization whose goal is to reduce the maximum error i.e., due to control inputs yielding high losses.
4. S TOCHASTIC E XPLORATION
Stochastic exploration aims to minimize the average error of the visited state–controller space i.e., to produce optimal hypersolver parameters ω ∗ as the solution of a nonlinear program
[Equation]
where ξ ( x, u ) is a distribution with support in X × U of the state and controller spaces and (cid:96) is the training loss function.
In order to guarantee sufﬁcient exploration of the state–controller space, we use Monte Carlo sampling (Robert & Casella, 2013) from the given distribution.
In particular, batches of initial conditions { x i 0 } , { u i 0 } are sampled from ξ and the loss function (cid:96) is calculated with the given system and step size (cid:15) .
We then perform backpropagation for updating the parameters of the hypersolver using a stochastic gradient descent (SGD)
algorithm
e.g., Adam (Kingma & Ba, 2017) and repeat the procedure for every training epoch.
Figure 2 shows pre–training results with stochastic exploration for different step sizes (see Appendix C.2).
We notice how higher residual values generally correspond to higher absolute values of control inputs.
Many systems in practice are subject to controls that are constrained in magnitude either due to physical limitations of the actuators or safety restraints of the workspace.
This property allows us to design an exploration strategy that focuses on worst-case scenarios i.e. largest control inputs. [Figure]
4. A CTIVE E RROR M INIMIZATION
[Figure] The goal of active error minimization is to actively reduce the highest losses in terms of the control inputs,
i
[Figure]
e., to obtain w ∗ as the solution to a minmax problem:
[Equation]
Similarly to stochastic exploration, we create distribution ξ ( x, u ) with support in X × U and perform Monte Carlo sampling of n batches { ( x i , u i ) } , from ξ .
Then, losses are computed pair–wisely for each state x j , j = 0 , . . .
, n −with each control input u k , k = 0 , . . .
, n − 1 .
We then take the ﬁrst n controllers { u i (cid:48) 0 } yielding the maximum loss for each state.
The loss is recalculated using these controller values with their respective states and SGD updates to hypersolver parameters are performed.
Figure 3 shows a comparison of the pre–training techniques (further experimental details in Appendix C.2).
The propagated error on trajectories for the hypersolver pre–trained via stochastic exploration is lower on average with random control inputs compared with the one pre– trained with active error minimization.
The latter accumulates lower error for controllers yielding high residuals.
Different exploration strategies may be used depending on the down–stream control task.
4. G ENERALIZATION P ROPERTIES OF D IFFERENT A RCHITECTURES
We have assumed the state and controller spaces to be bounded and that training be performed by sampling for their known distributions.
While this is sufﬁcient for optimal control problems given a priori known bounds, we also investigate how the system generalizes to unseen states and control input values.
In particular, we found that activation functions have an impact on the end result of generalization beyond training boundaries.
We take into consideration two commonly used activation functions, Tanh : x → e x − e − x e x + e − x and ReLU : x → max(0 , x ) , along with network architectures which employ activation functions containing periodic components: SIREN : x → sin ( Wx + b ) (Sitzmann et al., 2020) and Snake : x → x + 1 a sin 2 ( ax ) (Ziyin et al., 2020).
We train hypersolver models with the different activation functions for the inverted pendulum model of (18) with common experimental settings (see Appendix C.3).
Figure 4 shows generalization outside the training states (see Figure 9 in Appendix C.3 for generalization of controllers and step sizes).
We notice that while Tanh and ReLU perform well on the training set of interest, performance degrades rapidly outside of it.
On the other one hand, SIREN and Snake manage to extrapolate the periodicity of the residual distribution even outside of the training region, thus providing further empirical evidence of the universal extrapolation theorem (Ziyin et al., 2020, Theorem 3).
Activation function choice plays an important role in Hypersolver performance and generalization.
4. M ULTI – STAGE H YPERSOLVERS
We have so far considered the case in which the vector ﬁeld (1) fully characterizes the system dynamics.
However, if the model does not completely describe the actual system dynamics, ﬁrst–order errors are introduced.
We propose Multi-Stage Hypersolvers to correct these errors: an additional term is introduced in order to correct the inaccurate dynamics f .
The resulting procedure is a modiﬁed version of (4) in which the base solver ψ (cid:15) ( t k , x k , u k ) does not iterate over the modeled vector ﬁeld f
but over its corrected version f (cid:63) :
[Equation]
where h w is a function with universal approximation properties.
While the inner stage h w is a ﬁrst– order error approximator, the outer stage g ω further reduces errors approximating the p –th order residuals:
[Equation]
We note that f (cid:63) is continuously adjusted due to the optimization of h w .
For this reason, it is not possible to derive the analytical expression of the residuals to train the stages with the residual ﬁtting loss function (6).
Instead, both stages can be optimized at the same time via backpropagation calculated on one–step trajectory ﬁtting loss (7) which does not require explicit residuals calculation.
E XPERIMENTS
We introduce the experimental results divided for each system into hypersolver pre–training and subsequent optimal control.
We use as accurate adaptive step–size solvers the Dormand/Prince method dopri5 (Dormand & Prince, 1980) and an improved version of it by Tsitouras tsit5 (Tsitouras, 2011) for training the hypersolvers and to test the control performance at runtime.
5. D IRECT OPTIMAL CONTROL OF A P ENDULUM
Hypersolver pre–training We consider the inverted pendulum model with a torsional spring described in (18).
We select ξ ( x, u ) as a uniform distribution with support in X × U where
X =
[ − 2 π, 2 π ] × [ − 2 π, 2 π ] and U =
[ − 5 , 5] to guarantee sufﬁcient exploration of the state- controller space.
Nominal solutions are calculated using tsit5 with absolute and relative tolerances set to 10 − 5 .
We train the hypersolver on local residuals via stochastic exploration using the Adam optimizer with learning rate of 3 × 10 − 4 for 3 × 10 5 epochs.
Direct optimal control The goal is to stabilize the inverted pendulum in the vertical position x (cid:63) =
We choose t ∈ [0 , 3] and a step size (cid:15) = 0 .
2 s for the experiment.
The control input is assumed continuously time–varying.
The neural controller is optimized via SGD with Adam with learning rate of 3 × 10 − 3 for 1000 epochs.
Figure 5 shows nominal controlled trajectories of HyperEuler and other baseline ﬁxed–step size solvers.
Trajectories obtained with the controller optimized with HyperEuler reach ﬁnal positions q
= (1 . 6 ± 17 . 6) × 10 − 2 while Midpoint and RK ones q
= ( − 0 .
6 ± 12 . 7) × 10 − 2 and q = (1 . 1 ± 12 .
8) × 10 − 2 respectively.
On the other hand, the controller optimized with the Euler solver fails to control some trajectories obtaining a ﬁnal q = (6 . 6 ± 19 . 4) × 10 − 1 .
HyperEuler considerably improved on the Euler baseline while requiring only 1 . 2% more Floating Point Operations (FLOPs) and 49 .
5% less compared to Midpoint.
Further details are available in Appendix C.3.
[Figure]
5. M ODEL P REDICTIVE C ONTROL OF A C ART -P OLE S YSTEM
Hypersolver pre–training We consider the partial dynamics of the cart–pole system of (19) with wrong parameters for the frictions between cart and track as well as the one between cart and pole. We employ the multi–stage hypersolver approach to correct the ﬁrst–order error in the vector ﬁeld as well as base solver residual.
We select ξ ( x, u ) as a uniform distribution with support in X × U where X
= [ − 2 π, 2 π ] × [ − 2 π, 2 π ] × [ − 2 π, 2 π ] × [ − 2 π, 2 π ] and U =
[ − 10 , 10] .
Nominal solutions are calculated on the accurate system using RungeKutta 4 instead of adaptive–step solvers due faster training times.
We train our multi–stage Hypersolver (i.e. a multi–stage hypersolver with the second–order Midpoint as base solver with the partial dynamics) on nominal trajectories of the accurate system via stochastic exploration using the Adam optimizer for 5 × 10 4 epochs, where we set the learning rate to 10 − 2 for the ﬁrst 3 × 10 4 epochs, then decrease it to 10 − 3 for 10 4 epochs and to 10 − 4 for the last 10 4 .
Model predictive control The goal is to stabilize the cart–pole system in the vertical position around the origin, e.g. x (cid:63) =
We choose t ∈ [0 , 3] and a step size (cid:15) = 0 .
05 s for the experiment.
The control input is assumed piece-wise constant during MPC sampling times.
The receding horizon is chosen as 1 s .
The neural controller is optimized via SGD with Adam with learning rate of 3 × 10 − 3 for a maximum of 200 iterations at each sampling time.
Figureshows nominal controlled trajectories of multi–stage Hypersolver and other baseline solvers.
The Midpoint solver on the inaccurate model fails to stabilize the system at the origin position x = (39 .
7 ± 97 . 7) cm , while multi–stage Hypersolver manages to stabilize the cart–pole system and improve on ﬁnal positions x
= (7 . 8 ± 3 . 0) cm .
Further details are available in Appendix C.4. [Figure]
Multi–stage Hypersolvers can correct ﬁrst–order errors on dynamic models
5. M ODEL P REDICTIVE C ONTROL OF A Q UADCOPTER
Hypersolver pre–training We consider the quadcopter model of (20).
We select ξ ( x, u ) as a uniform distribution with support in X × U where X is chosen as a distribution of possible visited states and each of the four motors i ∈ [0 , 3] has control inputs u i ∈ [0 , 2 . 17] × 10 5 rpm .
Nominal solutions are calculated on the accurate system using dopri5 with relative and absolute tolerances set to 10 − 7 and 10 − 9 respectively.
We train HyperEuler on local residuals via stochastic exploration using the Adam optimizer with learning rate of 10 − 3 for 10 5 epochs.
Model predictive control The control goal is to reach a ﬁnal positions [ x, y, z ] (cid:63) =
[8 , 8 , 8] m .
We choose t ∈ [0 , 3] and a step size (cid:15) = 0 .
02 s for the experiment.
The control input is assumed piece–wise constant during MPC sampling times.
The receding horizon is chosen as 0 .
5 s .
The neural controller is optimized via SGD with Adam with learning rate of 10 − 2 for 20 iterations at each sampling time.
Figure 7 shows local residual distribution and control performance on the quadcopter over 30 experiments starting at random initial conditions which are kept common for the different
ODE solvers.
HyperEuler requires a single function evaluation per step as for the Euler solver compared to two function evaluations per step for Midpoint and four for RK4.
Controlled trajectories optimized with Euler, Midpoint and RK4 collect an error on ﬁnal positions of (1 . 09 ± 0 . 37) m , (0 .
71 ± 0 . 17) m , (0 .
70 ± 0 . 19) m respectively while HyperEuler achieves the lowest terminal error value of (0 . 66 ± 0 .
24) m .
Additional experimental details are available in Appendix C.5.
[Figure]
M e a n R e s i d u a l R
[Figure]
z [ m
[Figure]
Midpoint
[Figure]
InitialPosition Target
HyperEuler Euler Midpoint
{+RK 103 × 10 −4 × 10 −6 × 10 −2 × 
+}
F i n a l P
o s i
t i o n E rr
o r
5. B OUNDARY C ONTROL OF A T IMOSHENKO B EAM
Hypersolver pre–training We consider the ﬁnite element discretization of the Timoshenko beam of (22).
We create ξ ( x, u ) as a distribution with support in X × U which is generated at training time via random walks from known boundary conditions in order to guarantee both physical feasibility and sufﬁcient exploration of the state-controller space (see Appendix C.6 for further details).
Nominal solutions are calculated using tsit5 with absolute and relative tolerances set to 10 − 5 .
We train the hypersolver on local residuals via stochastic exploration using the Adam optimizer for 10epochs, where we set the learning rate to 10 − 3 for the ﬁrst 8 × 10 4 epochs, then decrease it to 10 −for 10 4 epochs and to 10 − 5 for the last 10 4 .
Boundary direct optimal control The task is to stabilize the beam in the straight position, i.e. each of its elements i have velocities v it , v ir and displacements σ it , σ ir equal to 0 .
We choose t ∈ [0 , 3] and step size (cid:15) = 5 ms for the experiment.
The control input is assumed continuously time– varying.
The neural controller is optimized via SGD with Adam with learning rate of 10 − 3 forepochs.
Figure 8 shows nominal controlled trajectories for HyperEuler and other baseline ﬁxed– step size solvers.
Control policies trained with Euler and Midpoint obtain averaged ﬁnal states of ( − 2 .
8 ± 4 . 2) × 10 − 1 and ( − 0 .
04 ± 4 . 6) × 10 − 1 thus failing to stabilize the beam, while HyperEuler and RK4 obtain ( − 0 .
6 ± 4 . 9) × 10 − 3 and ( − 0 .
5 ± 3 . 3) × 10 − 3 respectively.
HyperEuler considerably improves on both the Euler and Midpoint baselines obtaining a very similar performance to RK4, while requiring 72 .
9% less FLOPs;
the mean runtime per training iteration was cut from 8 .
24 s for
RK4 to just 2 . 53 s for HyperEuler. Further details on this experiment are available in Appendix C.6. t 0  x 0.0 0.5 1.
σ t ( x , t ) −
HyperEuler t 0  x 0.0 0.5 1.
σ t ( x , t ) −
Euler t 0  x 0.0 0.5 1.
σ t ( x , t ) −
Midpoint t 0  x 0.0 0.5 1.
σ t ( x , t ) −
RK t 0  x 0.0 0.5 1.
σ r ( x , t ) −  t 0  x 0.0 0.5 1.
σ r ( x , t ) −0t 0  x 0.0 0.5 1.
σ r ( x , t ) − t 0  x 0.0 0.5 1.
σ r ( x , t ) −
R ELATED W ORK
This work is rooted in the broader literature on surrogate methods for speeding up simulations and solutions of dynamical systems (Grzeszczuk et al., 1998; James & Fatahalian, 2003; Gorissen et al., 2010).
Differently from these approaches, we investigate a methodology to enable faster solution during a downstream, online optimization problem involving a potential mismatch compared to data seen during pre–training.
We achieve this through the application of the hypersolver (Poli et al., 2020a) paradigm.
Modeling mismatches between approximate and nominal models is explored in (Saveriano et al., 2017) where residual dynamics are learned efﬁciently along with the control policy while (Fisac et al., 2018;
Taylor et al., 2019) model systems uncertainties in the context of safety– critical control.
In contrast to previous work, we model uncertainties with the proposed multi–stage hypersolver approach by closely interacting with the underlying ODE base solvers and their residuals to improve solution accuracy.
The synergy between machine learning and optimal control continues a long line of research on introducing neural networks in optimal control (Hunt et al., 1992), applied to modeling (Lin & Cunningham, 1995), identiﬁcation (Chu et al., 1990) or parametrization of the controller itself (Lin et al., 1991).
Existing surrogate methods for systems (Grzeszczuk et al., 1998; James & Fatahalian, 2003) pay a computational cost upfront to accelerate downstream simulation.
However, ensuring transfer from ofﬂine optimization to the online setting is still an open problem.
In our approach, we investigate several strategies for an accurate ofﬂine–online transfer of a given hypersolver, depending on desiderata on its performance in terms of average residuals and error propagation on the online application.
Beyond hypersolvers, our approach further leverages the latest advances in hardware and machine learning software (Paszke et al., 2019) by solving thousands of ODEs in parallel on graphics processing units (GPUs) .
C ONCLUSION
We presented a novel method for obtaining fast and accurate control policies.
Hypersolver models were ﬁrstly pre–trained on distributions of states and controllers to approximate higher–order residuals of base ﬁxed–step ODE solvers.
The obtained models were then employed to improve the accuracy of trajectory solutions over which control policies were optimized.
We veriﬁed that our method shows consistent improvements in the accuracy of ODE solutions and thus on the quality of control policies optimized through numerical solutions of the system.
We envision the proposed approach to beneﬁt the control ﬁeld and robotics in both simulated and potentially real–world environments by efﬁciently solving high–dimensional space–continuous problems.
C ODE OF E THICS
We acknowledge that all the authors of this work have read and commit to adhering to the ICLR
Code of Ethics.
R EPRODUCIBILITY S TATEMENT
We share the code used in this paper and make it publicly available on Github 1 .
The following appendix also supplements the main text by providing additional clariﬁcations.
In particular, Appendix A provides further details on the considered hypersolver models.
We provide additional information on optimal control policy in Appendix B while in Appendix C we provide details on on the system dynamics, architectures and other experimental details.
Additional explanations are also provided as comments in the shared code implementation.
A A DDITIONAL H YPERSOLVER M ATERIAL A. E XPLICIT H YPER E ULER F ORMULATION
Our analysis in the experiments takes into consideration the hypersolved version of the Euler scheme, namely HyperEuler.
Since Euler is a ﬁrst–order method, it requires the least number of function evaluations (NFE) of the vector ﬁeld f in (1) and yields a second order local truncation error e k :=
(cid:13)(cid:13) (cid:15) 2 R k (cid:13)(cid:13) 2 .
This error is larger than other ﬁxed–step solvers and thus has the most room for potential improvements.
The base solver scheme ψ (cid:15) of (4) can be written as ψ (cid:15) ( t k , x k , u k ) =
f ( t k , x k , u k ) , which is approximating the next state by adding an evaluation of the vector ﬁeld multiplied by the step size (cid:15) .
We can write the HyperEuler update explicitly as
[Equation]
12) while we write its residual as
[Equation]
A. H YPERSOLVERS FOR T IME – INVARIANT S YSTEMS
A time–invariant system with time–invariant controller can be described as following
[Equation]
14) in which f and u do not explicitly depend on time.
The models considered in the experiments satisfy this property.
B C ONTROL P OLICY D ETAILS B. O
[Equation] UNCTION
The general form of the integral cost functional can be written as follows
[Equation]
where matrix P is a penalty on deviations from the target x (cid:63) of the last states, Q penalizes all deviations from the target of intermediate states while R is a regulator for the control inputs.
Evaluation of (15) usually requires numerical solvers such as the proposed hypersolvers of this work.
Discretizations of the cost functional are also called cost function in the literature.
B. M ODEL P REDICTIVE C ONTROL F ORMULATION
The following problem is solved online and iteratively until the end of the time span
[Equation]
to
[Equation] where J is a cost function and T ∈ T is the receding horizon over which predicted future trajectories are optimized.
B. N EURAL O PTIMAL C ONTROL
We parametrize the control policy of problem (2) as u θ : t, x (cid:55)→ u θ ( t, x ) where θ is a ﬁnite set of free parameters.
Speciﬁcally, we consider the case of neural optimal control in which controller u θ is a multi–layer perceptron.
The optimal control task is to minimize the cost function J described in (15) and we do so by optimizing the parameters θ via SGD; in particular, we use the Adam (Kingma & Ba, 2017) optimizer for all the experiments.
C E XPERIMENTAL D ETAILS
In this section we include additional modeling and experimental details divided into the different dynamical systems.
C. H YPERSOLVER N ETWORK A RCHITECTURE
We design the hypersolver networks g w as feed–forward neural networks.
Table 1 summarizes the parameters used for the considered controlled systems, where Activation denotes the activation functions, i.e. SoftPlus : x (cid:55)→ log(1 + e x ) , Tanh : x (cid:55)→ e x − e − x e x + e − x and Snake : x → x + 1 a sin 2 ( ax ) (Ziyin et al., 2020).
We also use the vector ﬁeld f as an input of the hypersolver, which does not
[Table]
require a further evaluation since it is pre–evaluated at runtime by the base solver ψ .
We emphasize that the size of the network should depend on the application: a too–large neural network may require more computations than just increasing the numerical solver’s order: Pareto optimality of hypersolvers also depends on their complexity.
Keeping their neural network small enough guarantees that evaluating the hypersolvers is cheaper than resorting to more complex numerical routines.
C. S PRING - MASS S YSTEM
System Dynamics
The spring-mass system considered is described in the Hamiltonian formulation by
[Equation]
(17) where m =
[ Kg ] and k = 0 .
5 [ N/m ] .
Pre-training methods comparison We select ξ ( x, u ) as a uniform distribution with support in
X × U where X
= [ − 20 , 20] × [ − 20 , 20] while U =
[ − 100 , 100] .
Nominal solutions are calculated on the accurate system using dopri5 with relative and absolute tolerances set to 10 − 7 and 10 −respectively.
We train two separate HyperEuler models with different training methods on local residual for step size (cid:15) = 0 .
03 s : stochastic exploration and active error minimization.
The optimizer used is Adam with learning rate of 10 − 3 for 10 4 epochs.
Hypersolvers with different step sizes We select ξ ( x, u ) as a uniform distribution with support in X × U where X
= [ − 5 , 5] × [ − 5 , 5] while U =
[ − 20 , 20] .
Nominal solutions are calculated on the accurate system using dopri5 with relative and absolute tolerances set to 10 − 7 and 10 −respectively.
We train separate HyperEuler models with stochastic exploration with different step sizes (cid:15) .
The optimizer used is Adam with learning rate of 10 − 3 for 10 4 epochs. [Figure]
C. I NVERTED P ENDULUM
System Dynamics We model the inverted pendulum with elastic joint with Hamiltonian dynamics via the following:
[Equation]
[Equation] u (18) where m =
[ Kg ] , k = 0 .
5 [ N/ rad] , r = 1 [ m ] , β = 0 .
[ Ns/ rad] , g = 9 .
[ m/s 2 ] .
Pre–training for the generalization study We perform sampling via stochastic exploration from the uniform distribution ξ ( x, u ) with support in X × U with X =
[ − 2 π, 2 π ] × [ − 2 π, 2 π ] and
U =
[ − 10 , 10] for the different architectures.
We choose as a common time step (cid:15) = 0 .
1 s ; the networks are trained for 100000 epochs with the Adam optimizer and learning rate of 10 − 3 .
The network architectures share the same parameters as the inverted pendulum ones in 1, while the activation functions are substituted by the ones in Figure 4.
The SIREN architecture is chosen with 2 hidden layers of size 64.
Figure 9 provides an additional empirical results on generalization properties across controller values and step sizes: we notice how Snake can generalize to unseen control values better compared to other hypersolvers.
Additional visualization Figure 10 provides an additional visualization of the inverted pendulum controlled trajectories from Figure 5 with positions q and momenta p over time.
[Figure]
C. C ART -P OLE
System Dynamics We consider a continuous version of a cart–pole system additionally taking into account the full dynamic model in Florian (2005).
This formulation considers the friction coefﬁcient between the track and the cart µ c inducing a force opposing the linear motion as well as the friction generated between the cart and the pole µ p , whose generated torque opposes the angular motion.
The full cart–pole model is described by the four variables x, ˙ x, θ, ˙ θ and the accelerations update is as following
[Equation]
where m c =
[ Kg ] , m p = 0 .
1 [ Kg ] , l = 0 .
5 [ m ] and g = 9 .
[ m/s 2 ] .
N c represents the normal force acting on the cart.
For simulation purposes, we consider its sign to [-
 [ Kg ] for-]be always positive when evaluating the sign (sgn) function as the cart should normally not jump off the track.
Setting µ c , µ p to 0 results in the same dynamic model used in the OpenAI Gym (Brockman et al., 2016) implementation.
[Figure]
training and Midpoint base solver iterating on an inaccurate vector ﬁeld agnostic of friction forces outperforms the Midpoint solver with full knowledge of the vector ﬁeld. Multistage training strategies We study two different training strategies for the inner and outer networks h w and g ω in (11).
We ﬁrst consider a joint training strategy in which both stages are trained at the same time via stochastic exploration. Secondly, we do a separated training in which only the inner stage network h w is trained ﬁrst and then the outer stage network g ω is added and only its parameters are trained in a ﬁnetuning process. We ﬁnd that, as shown in Figure 11, joint training yields slightly better results.
A further advantage of jointly training both stages is that only a single training procedure is required. Ablation study We consider the same model as our Multi–stage Hypersolver with base Midpoint solver but no ﬁrst–stage h w , which corresponds to learning the residual dynamics only, and we train this model with stochastic exploration. We show in Figure 11 that while the residual dynamics model can improve the one–step error compared to the base solver on the inaccurate dynamics, it performs worse the Multi–stage Hypersolver scheme.
We additionally study the contribution of each stage in the prediction error improvements by separately zeroing out the contributions of the inner and outer stage.
While iterating over the inner stage only improves on the base–solver error, including the outer stage further contributes in improving the error.
We notice how the excluding the inner–stage yields higher errors: this may be due to the fact that the inner–stage specializes in correcting the ﬁrst–order vector ﬁeld inaccuracies while the outer–stage corrects the one step base solver residual. Additional experimental details For the Multi–stage Hypersolver control experiment, we pre– train both inner and outer stage networks h w and g ω in (11) at the same time using stochastic exploration. The base solver is chosen as the second–order Midpoint iterating on the partial dynamics (19) with µ c , µ p set to 0 .
The nominal dynamics considers non–null friction forces: we set the cart friction coefﬁcient to µ c = 0 . 1 and the one of the pole to µ p = 0 . We note how the friction coefﬁcients make the vector ﬁeld (19) non–smooth : simulation through adaptive–step size solvers as tsit5 results experimentally time–consuming, hence we resort to RK4 for training the hypersolver networks. Nonetheless, as shown in the error propagation of Figure 12, this does not degrade the performance of the trained multi–stage hypersolver scheme.
All neural networks in the experiements, including the ablation study, are trained with the Adam optimizer for 5 × 10 4 epochs, where we set the learning rate to 10 − 2 for the ﬁrst 3 × 10 4 epochs, then decrease it to 10 − 3 for 10epochs and to 10 − 4 for the last 10 4 .
C. Q UADCOPTER
System Dynamics
The quadcopter model is a suitably modiﬁed version of the explicit dynamics update in (Panerati et al., 2021) for batched training in PyTorch.
The following accelerations update describes the dynamic model
[Equation]
(20) mwhere x = [ x, y, z ] corresponds to the drone positions and ψ
= [ φ, θ, ψ ] to its angular positions;
R and J are its rotation and inertial matrices respectively, τ ( · ) is a function calculating the torques induced by the motor speeds ω i , while arm length l , mass m , gravity acceleration constant g along with k F and k T are scalar variables describing the quadcopter’s physical properties. [Figure]
C. T IMOSHENKO B EAM
System Dynamics We consider as a system from the theory of continuum dynamics the Timoshenko beam with no dissipation described in (Macchelli & Melchiorri, 2004; Massaroli et al., 2021).
The system can be described in the coenergy formulation by the following partial differential equation (PDE)
[Equation]
where ρ is the mass density, A is the cross section area, I ρ is the rotational inertia, C s and C b are the shear and bending compliance; the discretized state variables v t , v r represent the translational and rotational velocities respectively while σ t , σ r denote the translational and rotational displacements. cantilever beam.
In order to discretize the problem, we implement a software routine based on the fenics (Alnæs et al., 2015) open–source software suite to obtain the ﬁnite–elements discretization of the Timoshenko PDE of (21) given the number of elements, physical parameters of the model and initial conditions of the beam.
We choose a 40 elements discretization of the PDE for a total ofdimensions of the discretized state z
= [ v t , v r , σ t , σ r ] (cid:62) and we initialize the beam at time t =as z ( x, 0) =
[sin( πx ) , sin(3 πx ) , 0 , 0] .
The system can thus be reduced to the following controlled [Table]
[Equation]
where the mass matrices M ρA , M I ρ , M C b , M C s , matrices D 0 , D 1 , D 2 , vectors B F , B T are computed through the fenics routine and boundary controllers u 1 ∂ and u 2 ∂ are the control torque and the control force applied at the free end of the beam.
Stochastic exploration strategy via random walks We pre–train the hypersolver model via stochastic exploration of the state-controller space
X × U .
We restrict the boundary control input values u
= [ u∂ , u 2 ∂ ] (cid:62) in [ − 1 , 1] × [ − 1 , 1] .
As for the state space, naively generating a probability distribution with box boundaries on each of the 160 dimensions of X would require an inefﬁcient search over this high-dimensional space: in fact, not every combination is physically feasible due to the Timoshenko beam’s structure.
We solve this problem by propagating batched trajectories with
RK4 from the initial boundary condition z
( x, 0) =
[sin( πx ) , sin(3 πx ) , 0 , 0] with random control actions sampled from a uniform distribution with support in [ − 1 , 1] × [ − 1 , 1] applied for a time t 1 ∼ U [0 .
002 , 1] s .
We save the states { z ( x, t 1 ) i } and forward propagate from these states again by sampling from the controller and time distributions.
We repeat the process K times and obtain a sequence [ { z ( x, t 1 ) i } , . . .
, { z ( x, t K ) i } ] of batched initial conditions characterized by physical feasibility.
Finally, we train the hypersolver with stochastic exploration by sampling from the generated distribution ξ ( x, u ) on local one–step residuals as described in Section 5.4.
This initial state generation strategy is repeated every 100 epochs for guaranteeing an extensive exploration of all possible boundary conditions.
Figure 13 shows the error propagation over controlled trajectories: the trained
HyperEuler achieves the lowest error among baseline ﬁxed–step solvers.
[Figure]
Additional details on the results We report additional details regarding the runtime of the experiments on the Timoshenko Beam.
As for the training time, training the hypersolver for 10 5 epochs takes around 80 minutes, where the time for each training epoch slightly varies depending on the length of the sequence of initial condition batches obtained via random walks.
As for the averaged runtime per training epoch during the control policy optimization, HyperEuler takes (2 . 53 ± 0 . 09)
s per training iteration, Euler (2 .[-01 ± 0 . 
-]
[Table]
s .
Experiments were run on the CPU of the machine described in Section C.7.
C. H ARDWARE AND S OFTWARE
Experiments were carried out on a machine equipped with an AMD R YZEN T HREADRIPPER 3960X CPU with 48 threads and two NVIDIA RTX 3090 graphic cards.
Software–wise, we used PyTorch (Paszke et al., 2019) for deep learning and the torchdyn (Poli et al., 2020b) and