Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition
Abstract: We present a data-efficient algorithm for learning models for modelpredictive control (MPC).
Our approach, Jacobian-Regularized Dynamic-Mode
Decomposition (JDMD), offers improved sample efficiency over traditional Koopman approaches based on Dynamic-Mode Decomposition (DMD) by leveraging Jacobian information from an approximate prior model of the system, and improved tracking performance over traditional model-based MPC.
We demonstrate JDMD’s ability to quickly learn bilinear Koopman dynamics representations across several realistic examples in simulation, including a perching maneuver for a fixed-wing aircraft with an empirically derived high-fidelity physics model.
In all cases, we show that the models learned by JDMD provide superior tracking and generalization performance within a model-predictive control framework, even in the presence of significant model mismatch, when compared to approximate prior models and models learned by standard Extended DMD (EDMD).
Introduction
In recent years, both model-based optimal-control [1, 2, 3, 4] and data-driven reinforcement-learning methods [5, 6, 7] have demonstrated impressive success on complex, nonlinear robotic systems.
However, both approaches suffer from inherent drawbacks: Data-driven methods often require extremely large amounts of data and fail to generalize outside of the domain or task on which they were trained.
On the other hand, model-based methods require an accurate model of the system to achieve good performance.
In many cases, high-fidelity models can be too difficult to construct from first principles or too computationally expensive to be of practical use.
However, low-order approximate models that can be evaluated cheaply at the expense of controller performance are often available.
With this in mind we seek a middle ground between model-based and data-driven approaches in this work.
We propose a method for learning bilinear Koopman models of nonlinear dynamical systems for use in model-predictive control that leverages derivative information from an approximate prior dynamics model of the system in the training process.
Given the increased availability of differentiable simulators [8, 9], this approximate derivative information is readily available for many systems of interest.
Our new algorithm builds on Extended Dynamic Mode Decomposition (EDMD), which learns Koopman models from trajectory data [10, 11, 12, 13, 14], by adding a derivative regularization term based on derivatives computed from a prior model.
We show that this new algorithm,
Jacobian-regularized Dynamic Mode Decomposition (JDMD), can learn models with dramatically fewer samples than EDMD, even when the prior model differs significantly from the true dynamics of the system.
We also demonstrate the effectiveness of these learned models in a model-predictive control (MPC) framework.
The result is a fast, robust, and sample-efficient pipeline for quickly training a model that can outperform MPC controllers using both approximate analytical models as well as models learned using traditional Koopman approaches.
Our work is most closely related to the recent works of Folkestad et.
al.
[13, 15, 16], which learn bilinear models and apply nonlinear model-predictive control directly on the learned bilinear dynamics.
Other recent works have combined linear Koopman models with model-predictive control [12] and Lyapunov control techniques with bilinear Koopman models [17].
Our contributions are:
• A novel extension to Dynamic Mode Decomposition, called JDMD, that incorporates gradient information from an approximate analytic model • A recursive, batch QR algorithm for solving the least-squares problems that arise when learning bilinear dynamical systems using DMD-based algorithms, including JDMD and EDMD
The remainder of the paper is organized as follows:
In Section 2 we provide some background on the application of Koopman operator theory to controlled dynamical systems and review some related works.
Section 3 then describes the proposed JDMD algorithm.
In Section 4 we outline a memory-efficient technique for solving the large, sparse linear least-squares problems that arise when applying JDMD and other DMD-based algorithms.
Section 5 then provides simulation results and analysis of the proposed algorithm applied to control tasks on a cartpole, a quadrotor, and a small foam airplane with an experimentally determined aerodynamics model, all subject to significant model mismatch.
We also discuss the suitability of the conventional open-loop prediction error as a metric for evaluating dynamics model used in closed-loop control frameworks.
In Section 6 we discuss the limitations of our approach, followed by some concluding remarks in Section 7.
Background and Related Work 2.1 Koopman Operator Theory
The theoretical underpinnings of the Koopman operator and its application to dynamical systems has been extensively studied [11, 18, 19, 20, 21, 22].
Rather than describe the theory in detail, we highlight the key concepts employed by the current work and refer the reader to the existing literature on Koopman theory for further details.
We start by assuming a controlled, nonlinear, discrete-time dynamical system,
[Equation]
where x ∈ X ⊆ R N x is the state vector, u ∈ R N u is the control vector, and x + is the state at the next time step.
Assuming the dynamics are control affine, the nonlinear finite-dimensional system (1) can be represented exactly by an infinite-dimensional bilinear system through the Koopman canonical transform [22].
This bilinear Koopman model takes the form,
[Equation]
where y
= ϕ ( x ) is a nonlinear mapping from the finite-dimensional state space X to the infinitedimensional Hilbert space of observables Y .
In practice, we approximate (2) by restricting Y to be a finite-dimensional vector space, in which case ϕ becomes a finite-dimensional nonlinear function of the state variables that can be either chosen heuristically based on domain expertise or by learning [23, 24, 25].
Intuitively, ϕ “lifts” our state x into a higher dimensional space Y where the dynamics are approximately bilinear, effectively trading dimensionality for bilinearity.
Similarly, we can perform an “unlifting” operation by projecting a lifted state y back into the original state space X .
In this work, since we embed the original state within the nonlinear mapping [11, 15, 26, 27, 28], ϕ is constructed in such a way that this unlifting is linear:
[Equation]
We note that our proposed method does not require this assumption: any mapping could be used.
The problem of finding an optimal mapping is itself a major area of research, and many recent studies have focused on jointly learning both the model and the mapping [23, 24, 25, 29, 30].
While clearly advantageous, learning an optimal mapping is not the focus of this paper.
Instead, we focus on incorporating prior information from an approximate model in a way that is applicable to any lifting function, and we rely on simple mappings that are chosen heuristically in all of our examples.
2.2 Extended Dynamic Mode Decomposition
A lifted bilinear system of the form (2) can be learned from P samples of the system dynamics ( x + j , x j , u j ) using Extended Dynamic Mode Decomposition (EDMD) [15, 21].
We first define the following data matrices:
[Equation]
where u ik is the i -th element of the control vector at time k .
We then concatenate all of the model coefficient matrices from (2) as follows:
[Equation]
The model learning problem can then be written as the following linear least-squares problem:
[Equation]
EDMD is closely related to classical feature-based machine learning approaches like the “kernel trick” used in support vector machines [31], but extends these ideas to bilinear models of controlled dynamical systems.
Jacobian-Regularized Dynamic Mode Decomposition
We now present JDMD as a straightforward adaptation of the original EDMD algorithm described in Section 2.2.
Given P samples of the dynamics ( x + j , x j , u j ) , and an approximate discrete-time dynamics model,
[Equation]
we can evaluate the Jacobians of our approximate model ˜ f
at each of the sample points: ˜ A j = ∂ ˜ f∂x , ˜ B j = ∂ ˜ f∂u .
After choosing a nonlinear mapping ϕ : R N x (cid:55)→ R N y our goal is to find a bilinear dynamics model (2) that matches the Jacobians of our approximate model, while also matching our dynamics samples.
We accomplish this by penalizing differences between the Jacobians of our learned bilinear model with respect to the original states x and controls u , and the Jacobians we expect from our analytical model.
These projected Jacobians are calculated by differentiating through the projected dynamics :
[Equation]
Differentiating (8) with respect to x and u gives us
[Equation]
9b
[Equation]
where Φ( x ) =
∂ϕ / ∂x is the Jacobian of the nonlinear map ϕ , and
[Equation]
We then solve the following linear least-squares problem: minimize E
[Equation]
[Equation]
Problem (11) has ( N y + N 2 x + N x · N u ) · P rows and N y · N z columns.
Given that the number of rows in this problem grows quadratically with the state dimension, solving it can be computationally challenging.
The next section proposes an algorithm to address this challenge without resorting to a distributed-memory system.
This solution method also has the benefit of allowing incremental updates to the bilinear system, enabling online model learning.
Efficient Recursive Least Squares
In its canonical formulation, a linear least squares problem can be represented as the following unconstrained optimization problem:
[Equation]
We assume F is a large, sparse matrix and that solving it directly using a QR or Cholesky decomposition requires too much memory for a single computer.
While solving (12) using an iterative method such as LSMR [32] or LSQR [33] is possible, we find that these methods do not work well in practice for solving (11) due to ill-conditioning.
Standard recursive methods for solving these problems are able to process the rows of the matrices sequentially to build a QR decomposition of the full matrix, but also tend to suffer from ill-conditioning [34, 35, 36].
To overcome these issues, we propose an alternative recursive method based.
We solve (12) by dividing rows of F into batches:
[Equation]
The main idea is to maintain and update an upper-triangular Cholesky factor U i of the first i terms of the sum (13).
Given U i , we can calculate U
i +1 using the QR decomposition, as shown in [37]:
[Equation]
where QR R returns the upper triangular matrix R from the QR decomposition.
For an efficient implementation, this function should be an “economy” or “Q-less” QR decomposition since the Q matrix is never needed.
We also handle regularization of the normal equations, equivalent to adding quadratic or Tikhonov regularization to the original least squares problem, during the base case of our recursion, [Equation]
[Equation]
where λ is a scalar regularization weight.
To ensure fair comparisons, the results presented in the next section for both EDMD and JDMD correspond to the best-performing λ values found by sweeping over a wide parameter range.
[Figure]
Experimental Results
This section presents the results of several simulation experiments to evaluate the performance of
JDMD.
We specify two models for each simulated system: a nominal model, which is simplified and contains both parametric and non-parametric model error, and a true model, which is used exclusively for simulating the system and evaluating algorithm performance.
All models were trained by simulating the “true” system with a nominal controller to collect data in the region of the state space relevant to the task.
A set of fixed-length trajectories were collected, each at a sample rate of 20-25 Hz.
The bilinear EDMD model was trained using the same approach introduced by Folkestad and Burdick [15].
When applying MPC to the learned Koopman models, the projected Jacobians (9) were used, since this projected system is much more likely to be controllable than the lifted one and reduces the computational complexity of the MPC controller.
This results in a nonlinear model in the original state space, which is linearized about the reference trajectory to create a linear MPC controller.
All continuous-time dynamics were discretized with an explicit fourth-order Runge-Kutta method.
Code for all experiments is available at https://github. com/bjack205/BilinearControl.jl .
5.1 Systems and Tasks
Cartpole: We perform a swing-up task on a cartpole system.
The true model includes Coulomb friction between the cart and the floor, viscous damping at both joints, and a deadband in the control input that were not included in the nominal model.
Additionally, the mass of the cart and pole model were altered by 20% and 25% with respect to the nominal model, respectively.
The following nonlinear mapping was used when learning the bilinear models: ϕ ( x ) =
[ 1 , x, sin( x ) , cos( x ) , sin(2 x ) , sin(4 x ) , T 2 ( x ) , T 3 ( x ) , T 4 ( x ) ] ∈ R 33 , where T i ( x ) is a Chebyshev polynomial of the first kind of order i .
All reference trajectories for the swing up task were generated using ALTRO [37, 39].
Quadrotor:
We track point-to-point linear reference trajectories from various initial conditions on a full 3D quadrotor model.
The true model includes aerodynamic drag terms not included in the nominal model, as well as parametric error of roughly 5% on the system parameters (e.g. mass, rotor arm length, etc.).
The model was trained using a nonlinear mapping of ϕ ( x ) =
[ 1 , x, T 2 ( x ) , sin( p ) , cos( p ) , R T v, v T RR T v, p × v, p × ω, ω × ω ] ∈ R 44 , where p is the quadrotor’s position, v and ω are the translational and angular velocities respectively, and R is the rotation matrix representation of the quadrotor’s attitude.
Airplane: We perform a post-stall perching maneuver on a high-fidelity model of a fixed-wing airplane.
The perching trajectory is produced using trajectory optimization (see Figure 1a).
Perching involves flight at high angles of attack, where the aerodynamic lift and drag forces are extremely complex and difficult to model from first principles.
We leverage previous works that fit post-stall aerodynamics models using empirical data from wind-tunnel experiments [38, 40].
The true model includes these empirical nonlinear flight dynamics [38], while the nominal model uses a simple flatplate wing model with linear lift and quadratic drag coefficient approximations.
The bilinear models use a nonlinear mapping ϕ ∈ R 68 , which includes the aircraft attitude (expressed as a vector of
Modified Rodriguez Parameters [41]), powers of the angle of attack and side slip angle, the body frame velocity, various cross products with the angular velocity, and 3rd and 4th order Chebyshev polynomials of the states.
5.2 Sensitivity to Model Mismatch
µ
N o m i n a l
E D M
D
J D M
D 0.
✓  0. ✓  0.
✗  0. ✗  0.
✗ ✗ 0. ✗ ✗ 0.
✗ ✗
While a significant amount of model mismatch is introduced in all examples, a natural argument against model-based methods is that they are only as good as the model’s ability to capture the salient dynamics of the system.
Therefore, we investigated the effect of increasing model mismatch by incrementally increasing the Coulomb friction coefficient µ between the cart and the floor for the cartpole stabilization task (recall the nominal model assumed zero friction).
The results are shown in Table 1.
As expected, the number of training trajectories required to find a good stabilizing controller increases with µ .
We achieved the results above by setting α = 0 . 01 , corresponding to a decreased confidence in our model, thereby placing greater weight on the experimental data.
The standard EDMD approach always required more samples, and was unable to find a good enough model above friction values of 0.4.
While this could likely be remedied by adjusting the nonlinear mapping ϕ , the proposed approach works well with the given basis.
Note that the nominal MPC controller failed to stabilize the system above friction values of 0.1, so again, we demonstrate that we can improve MPC performance substantially with just a few training samples by combining analytical derivative information and data sampled from the true dynamics.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
5.3 Sample Efficiency
We compare the sample efficiency of several algorithms on the cartpole swing-up task in Figure 2.
As shown, JDMD achieves the best performance overall, and does so with only two training trajectories.
In comparison, traditional EDMD requires about 10 iterations to achieve consistent performance.
Similar results were obtained for the airplane perching example (see Figure 2b), where
EDMD requires over three times the number of samples (35 vs 10) compared to JDMD and never achieves the same closed-loop performance
[Table]
5.4 Generalization
[Table]
We demonstrate the generalizability of JDMD on a quadrotor.
The task is to return to the origin, given an initial condition sampled from a uniform distribution centered at the origin.
Test initial conditions are sampled from a distribution larger than that of the training data.
Given the goal of tracking a straight line back to the origin, we test 50 initial conditions, many of which are far from the goal, have large velocities, or are nearly inverted (see Figure 3a).
The results using an MPC controller are shown in
Table 2, demonstrating the generalizability of JDMD, given that the algorithm was only trained on 30 initial conditions sampled relatively sparsely given the size of the sampling window.
EDMD only successfully brings about 18% of the samples to the origin, while the majority of the time resulting in trajectories like those in Figure 3b.
JDMD improves the tracking performance of nominal MPC, which is subject to a constant-bias error due to model mismatch, as shown in Figure 3b.
5.5 Model Prediction Error vs. Controller Performance
Much of the previous literature focuses on open-loop prediction error for evaluating learneddynamics models [11, 12, 15, 19, 22, 42].
While intuitive, we argue that this is a poor metric when the end goal is closed-loop control performance.
As shown in the histogram of open-loop prediction error in Figure 4a, the open-loop prediction error of JDMD (trained with 8 trajectories) is significantly higher over 100 test trajectories, with 74% of tests resulting in a prediction error of approximately 1 .
compared to 25% for EDMD (trained with 24 trajectories).
Despite worse open-loop prediction performance, the JDMD model outperforms the EDMD model in closed-loop
Prediction Error
N u m b e r o f
T r a j ec t o r i e s (a) 0 .
Tracking Error (b) 0 .
Jacobian Error
Nominal
EDMD
JDMD (c) tracking (see Figure 4b).
Given that MPC, like most closed-loop controllers, relies on the behavior of the model under small perturbations (i.e. derivative information), the difference in tracking performance may be explained by JDMD achieving a distribution with much lower Jacobian error than EDMD (see Figure 4c).
This suggests that open-loop prediction error is not necessarily a good metric for evaluating models that will be used in control applications, and that models sufficient for closed-loop control may be learned with far less data.
Limitations
Many of the limitations of the proposed approach derive from the limitations of Koopman approaches more broadly, such as the sensitivity to the nonlinear mapping selected and the limitation to control-affine continuous dynamics.
While the presented single rigid-body systems such as the quadrotor or airplane have similar dimensionality to many autonomous systems of interest, extensions to systems with many degrees of freedom may be difficult computationally, given that Jacobian matrices grow in size with the square of the state dimension.
As with most data-driven techniques, it is difficult to claim that our method will increase performance in all cases.
It is possible that having an extremely poor prior model may hurt rather than help the training process, especially if the derivative information from the approximate model has the incorrect sign.
Conclusions and Future Work
We have presented JDMD, a simple but powerful extension to EDMD that incorporates derivative information from an approximate prior model.
We have tested JDMD in combination with a simple linear MPC control policy across a range of systems and tasks, and have found that the resulting combination can dramatically increase sample efficiency over EDMD, often improving over a nominal MPC policy with just a few sample trajectories.
We also argued that the conventional open-loop dynamics prediction error is a poor metric for evaluating models used in closed-loop control frameworks.
Substantial areas for future work remain: most notably, demonstrating the proposed pipeline on hardware.
Additional directions include applications on sytems with many degrees of freedom such as those whose dynamics are governed by discretized PDEs, online learning or adaptive control applications, combining simulated and real data through the use of modern differentiable physics engines [9, 8], residual dynamics learning, as well as the development of specialized numerical methods for solving nonlinear optimal control problems using the learned bilinear dynamics.