Connections Between Optimization in Machine Learning and Adaptive Control
Abstract
This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.
Adaptive control as a ﬁeld has focused on mathematical rigor and guaranteed convergence.
The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.
This paper elucidates many of the numerous common connections between both ﬁelds such that results from both may be leveraged together to solve new problems.
In particular, a speciﬁc problem related to higher order learning is solved through insights obtained from these intersections.
This version is an extended abstract; refer to (Gaudio et al., 2019b) for the full paper.
1. Introduction
The ﬁelds of adaptive control and machine learning have evolved in parallel over the past few decades, with a signiﬁcant overlap in goals, problem statements, and tools.
Machine learning as a ﬁeld has focused on computer based systems that improve through experience (Duda et al., 2001;
Bishop, 2006; Hastie et al., 2009; Efron & Hastie, 2016;
Goodfellow et al., 2016; Jordan & Mitchell, 2015).
Often times the process of learning is encapsulated in the form of a parameterized model, whose parameters are learned in order to approximate a function.
Optimization methods are commonly employed to reduce the function approximation error using any and all available data.
The ﬁeld of adaptive control, on the other hand, has focused on the process of controlling engineering systems in order to accomplish regulation and tracking of critical variables of interest (e.g. position and force in robotics, Mach number and altitude in aerospace systems, frequency and voltage in power systems) in the presence of uncertainties in the underlying system models, changes in the environment, and unforeseen variations in the overall infrastructure (Narendra & Annaswamy, 1989; Sastry & Bodson, 1989; ˚Astr¨om & Wittenmark, 1995;
Ioannou & Sun, 1996; Narendra & Annaswamy, 2005).
The approach used for accomplishing such regulation and tracking in adaptive control is the learning of underlying parameters through an online estimation algorithm.
Stability theory is employed for enabling guarantees for the safe evolution of the critical variables, and convergence of the regulation and tracking errors to zero.
Learning parameters of a model in both machine learning and adaptive control occurs through the use of input-output data.
In both cases, the main algorithm used for updating the parameters is often based on a gradient descent-like algorithm.
Related tools of analysis, convergence, and robustness in both ﬁelds have a tremendous amount of similarity.
As the scope of problems in both ﬁelds increases, the associated complexity and challenges increase as well.
Therefore it is highly attractive to understand these similarities and connections so that the two communities can develop new methods for addressing new challenges.
2. Connections: Update Law
Two types of error models are common in machine learning and adaptive control, where output errors e y may be related to regressors (features) φ and parameter errors ˜ θ as
[Equation]
where W ( s ) denotes a dynamic operator and ˜ θ
= θ − θ ∗ ( θ ∗ unknown).
Our goal with both perspectives will be to adjust a parameter θ with knowledge of the regressor φ and output error e y , such that a loss function L ( θ ; e y ) is minimized.
For the adaptive control perspective we present solutions in terms of gradient ﬂow in continuous time t while the machine learning updates are presented as gradient descent in discrete time steps indexed by k , i.e.,
[Equation]
where γ > 0 is the learning rate in gradient ﬂow and γ k is the step size in gradient descent.
For a more detailed discussion of the problem statements see Appendix A.
In this section we consider the question: What common modiﬁcations to the update laws in (3) and (4) have been developed?
[Equation]
While the update laws in (3) and (4) are designed primarily to reduce the output error e y , there are several secondary reasons to modify these update laws from robustness considerations due to perturbations stemming from disturbances, noise, and other unmodeled causes.
Historically the adaptive update law in (3) has been modiﬁed to ensure robustness to bounded disturbances as
[Equation]
where σ > 0 is a tuneable parameter that scales the extra term G .
Common choices for G include the σ -modiﬁcation
G = θ (Ioannou & Kokotovic, 1984), and the e -modiﬁcation
G = (cid:107) e y (cid:107) θ
(Narendra & Annaswamy, 1987b).
Regularization is often included in a machine learning optimization problem in order to help cope with overﬁtting by including constraints on the parameter, thus resulting in an augmented loss function (Hastie et al., 2009; Bubeck, 2015): ¯
L ( θ ) =
L ( θ ) + σ R ( θ ) where σ > 0 is a tuneable parameter, often referred to as a Lagrange multiplier.
The gradient descent update (4) for this augmented loss function is often referred to as the “regularized follow the leader” algorithm in online learning (Hazan, 2016) and may be expressed as
[Equation]
The common choice of (cid:96) 2 regularization in machine learning of R =
(1 / 2) (cid:107) θ (cid:107) 22 , can be seen to coincide with the σmodiﬁcation (Ioannou & Kokotovic, 1984), as ∇ θ R = G .
2.2. Dead-Zone Modiﬁcation and Early Stopping
This subsection details common modiﬁcations of the update laws in both ﬁelds adopted to cease updating the parameter estimate after sufﬁcient tuning.
Another method in adaptive control employed to increase robustness in the presence of bounded disturbances is to employ a “dead-zone” (Peterson & Narendra, 1982), for the update in (3):
[Equation]
where d 0 > 0 is the dead-zone width that may correspond to an upper bound on the disturbance, and (cid:15) > 0 is a small constant.
The function D is a non-negative metric on the output error to stop adaptation in desired regions of the output space.
A common choice is D = (cid:107) e y (cid:107) such that adaptation stops after a small output error is achieved above a noise level with upper bound d 0 .
The training processes is often stopped in machine learning applications as a method to deal with overﬁtting (Hastie et al., 2009).
This may be done by using multiple data sets and stopping the parameter update process (4) when the loss computed for a validation data set starts to increase (Prechelt, 1998).
Early stopping is often seen to be needed for training neural networks due to their large number of parameters (Goodfellow et al., 2016) and can act as regularization (Sj¨oberg & Ljung, 1995).
2.3. Projection
It is often desirable to deﬁne a compact region a priori for the parameters θ , such that during the learning process the parameters are not allowed to leave that region.
In physical systems there are natural constraints which may aid in the design of that region, and for non-physical systems, the constraints are often engineered by the algorithm designer.
The continuous projection algorithm, commonly employed in adaptive control for increased robustness to unmodeled dynamics (Kreisselmeier & Narendra, 1982; Lavretsky et al., 2012; Hussain, 2017), is deﬁned as
[Equation]
otherwise
[Equation]
where Ω , θ i, max , θ (cid:48)
i, max deﬁne a user-speciﬁed boundary layer region inside of a compact convex set Θ .
The update law in (3) may then be modiﬁed as
[Equation]
The following projection operation commonly used in online learning (Zinkevich, 2003; Hazan et al., 2007; 2008; Hazan, 2016) ﬁnds the closest point in a convex set
[Equation]
which may be employed in the update law (4) modiﬁed as
[Equation]
2.4. Adaptive Gains and Stepsizes
The following parameter update law is one example which alters the gain of the standard update law (3) as a function of the time varying regressors φ (Narendra & Annaswamy, 1989; Ioannou & Sun, 1996):
[Equation]
where Υ ≥ 0 is a forgetting factor and N ( t ) is a normalizing signal , with common choice
[Equation]
It can be seen that the update for Γ may be used in the update for θ to result in a gain adaptive to the regressor φ .
Adaptive step size methods (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2017; Reddi et al., 2018) have seen widespread use in machine learning problems due to their ability to handle sparse and small gradients by adjusting the step size as a function of features as they are processed online.
A common update law for adaptive step size methods (Reddi et al., 2018) can then be seen to be similar to (11) as ¯
[Equation]
where m k and V k are functions of previous gradients, which can be compared to normalization by the regressor in (12).
3. Connections: Tools and Concepts
In this section we consider concepts and tools common to machine learning and adaptive control.
3.1. Lyapunov Functions and Regret
Stability and convergence tools in adaptive control and online machine learning are analyzed in this section.
Suppose we consider the error model in (2) where W ( s ) =
c ( sI − A ) − 1 b , and a corresponding state space representation of the form (Narendra & Annaswamy, 1989)
[Equation]
The term ˜ φ is due to exponentially decaying terms in the regressor φ .
That is, ˜ φ
= ˆ φ − φ and ˙˜ φ = Λ˜ φ for a Hurwitz matrix Λ ∈ R N × N . 1 Stability is often proven in adaptive control by the use of a Lyapunov function V , such as
[Equation]
Note that the last two terms in V are not needed for the algebraic error model in (1).
The time derivative of the Lyapunov function may then be stated using the update law in (3) and the KYP lemma as ˙ V =
− e T
Qe − α ˜ φ T ¯ Q ˜ φ + 2 e T Pbθ ∗ T ˜ φ , where ˙ V ≤ 0 for α > (4 (cid:107) Pb (cid:107) 2 (cid:107) θ ∗ (cid:107) 2 / (min eig ( Q ) · min eig ( ¯ Q )) .
It can be shown that δ ( t ) (cid:44) 2 e T Pbθ ∗ T ˜ φ is exponentially decaying with ˜ φ, e ∈ L 2 ∩ L ∞ .
By integrating ˙
V from t 0 to T : (cid:90) T te
[Equation]
T
[Equation] [Equation]
Given that ˙ V ≤ 0 , V ( t 0 ) − V ( T ) ≤ V ( t 0 ) < ∞ .
In online learning, efﬁciency of an algorithm is often analyzed using the notion of “regret” as
[Equation]
where regret can be seen to correspond to the sum of the time varying convex costs C k associated with the choice of the time varying parameter estimate θ k , minus the cost associated with the best static parameter estimate choice, over a time horizon of T steps (Zinkevich, 2003; Hazan et al., 2007; 2008; Hazan, 2016).
Suppose we consider a quadratic cost C k = e
Tk Qe k , Q = Q T > 0 .
A continuous time limit of (17) leads to an integral of the form continuous regret T = (cid:90) T te T Qedt − (cid:90) T t¯ δ ( t ) dt (18) where ¯ δ ( t ) is an exponentially decaying signal which is due to nonzero initial conditions in (2) or similarly in (14).
A strong similarity can thus be seen between (16) and (18).
It is desired to have regret grow sub-linearly with time, such that average regret, (1 /T ) regret T , goes to zero in the limit
T → ∞ , to provide for an efﬁcient algorithm (Hazan, 2016).
For adaptive control, convergence of state/output errors is shown from a similar integral which is akin to constant regret upper bounded by V ( t 0 ) in (16).
3.2. Unmodeled
Dynamics
and Generalization
Models used to design adaptive controllers, including the examples of (1) and (2), are approximations with a certain amount of modeling errors.
As such, they may only hold about an operating point and need to contend with unmodeled dynamics.
This implies that any stabilizing controller must be designed to not only adapt to parametric uncertainties, but also be robust to unmodeled dynamics.
In addition, constraints on the state and input may also be present in adaptive control problems (Karason & Annaswamy, 1994;
Annaswamy & K´arason, 1995).
Analysis becomes more complicated when considering unmodeled dynamics and constraints, resulting in non-global guarantees.
Many of the update law modiﬁcations in adaptive control from Sectionwere initially derived to ensure robustness in such cases.
This same notion of robustness to modeling errors exists in machine learning in which an estimator is constructed from a ﬁnite training data set.
It is then desired that this estimator produces a low prediction error based on a test data set consisting of unseen data.
Generalization thus refers to the concept of a designed estimator having low loss when applied to new problems.
In particular it can be seen that in speciﬁc cases, generalization pertains to stability, where algorithms that are stable and train in a small amount of time result in a small generalization error (Bousquet & Elisseeff, 2002; Hardt et al., 2016).
3.3. Persistent Excitation and Stochastic Perturbations
Persistent excitation (PE) of the system regressor in adaptive control is a condition that has been shown to be necessary and sufﬁcient for parameter convergence (Jenkins et al., 2018).
It can be shown that if the regressor φ is persistently exciting, then the parameter estimation error ˜ θ ( t ) converges to zero uniformly in time (Narendra & Annaswamy, 1989).
The PE condition essentially corresponds to certain spectral conditions being satisﬁed by the regressor (Boyd & Sastry, 1983; 1986).
A detailed exposition of system identiﬁcation and parameter convergence in both deterministic and stochastic cases can be found in (Goodwin & Sin, 1984;
Anderson & Johnson, 1982; Narendra & Annaswamy, 1986; 1987a; Ljung, 1987).
Another way to think of the PE condition is that it leads to a perfect test error, since it provides for convergence of the parameter error to zero, and therefore zero output error once transients decay to zero.
Many machine learning problems consider the case when stochastic perturbations are present.
In this context, signiﬁcant improvements may be possible by leveraging well known concepts in system identiﬁcation (Ljung, 1987).
For example (Dean et al., 2018) purposely includes a Gaussian random input into a dynamical system in order to provide for PE by construction.
Such stochastic perturbations can guarantee a PE condition only in the limit, when inﬁnite samples can be obtained.
In order to address the realistic case of ﬁnite samples, approaches in machine learning algorithms for system identiﬁcation and control have attempted to obtain performance bounds with probability 1 − p f for p f ∈ (0 , 1) , where the bound usually scales inversely with p f .
The probability of failure given by the choice of p f allows for error due to the presence of ﬁnite samples.
3.4. Neural Networks
Gradient based methods to solve for estimates of unknown parameters via back propagation, in what would develop into the foundations of neural networks have been used for decades in control, with early examples consisting of ﬁnding optimal trajectories (Pontryagin, 1961) in ﬂight control (Kelley, 1960), and resource allocation problems (Bryson, 1961) (see (Dreyfus, 1990) for a brief history).
Since then, the use of neural networks in control systems has expanded to include stabilizing nonlinear dynamical systems (Miller et al., 1995).
Design and analysis of stable controllers based on neural networks was taken up by the adaptive control community due to the the similarities of gradient-like update laws used in neural networks and adaptive control.
The adaptive control community developed a well established literature for the use of neural networks in nonlinear dynamical systems in the 1990s (Miller et al., 1995; Narendra &
Parthasarathy, 1990; 1991; Yu & Annaswamy, 1996; 1998).
The use of neural networks in the machine learning community greatly expanded as of recent due to the increase in computing power available and an increase in applications (Krizhevsky et al., 2012; Sutskever et al., 2013; Goodfellow et al., 2016).
Recurrent neural networks (Hopﬁeld, 1982;
Hinton & Sejnowski, 1983; Hochreiter & Schmidhuber, 1997), while often similar in structure to nonlinear dynamical systems, have historically been trained in a manner similar to feed-forward neural networks (Rumelhart et al., 1986) using back propagation through time (Werbos, 1990).
While a theoretical understanding of why deep neural networks work as well as they do for given problems has been lacking, the machine learning community has worked to rigorously analyze sub-classes of deep neural network architectures such as deep linear networks (Arora et al., 2018; 2019).
The update laws employed in training deep neural networks often include selections of modiﬁcations of the update laws as discussed in Section 2 (Schmidhuber, 2015).
4. Advantageous Combinations of Tools: Higher Order Learning
Given the many similarities in problem statements, tools, concepts, and algorithms, we now demonstrate how methods from the ﬁeld of adaptive control can be used to solve a new problem related to higher-order learning.
Many of the update laws addressed thus far were ﬁrst-order in nature, and made use of gradient-like quantities for learning.
A question of increasing interest is when accelerated learning can occur for higher-order learning methods.
In particular,
Nesterov’s accelerated method (Nesterov, 1983) was able to certify a convergence rate of O (1 /k 2 ) as compared to the standard gradient descent (4) rate of O (1 /k ) for a class of convex functions.
A parameterization of Nesterov’s higher order method may be stated as
[Equation]
where β > 0 is a design parameter that weighs the effect of past parameters.
Continuous time problem formulations have been explored in (Su et al., 2016; Wibisono et al., 2016), with rate-matching discretizations established in (Wilson et al., 2016; Wilson, 2018; Betancourt et al., 2018).
Many of these methods however become inadequate for time varying inputs and features.
Adaptive update laws which include additional levels of integration appeared in the “higher order tuners” in (Morse, 1992; Evesque et al., 2003), and take the form
[Equation]
where N t (cid:44) (1 + µφ T ( t ) φ ( t )) for a µ > 0 .
In contrast to (19), the update law in (20) can be shown to be stable in the presence of time varying regressors as in (1) and as well as in adaptive control applications with error model as in (2) (Gaudio et al., 2019a).
This solution was only possible by leveraging techniques from the ﬁeld of adaptive control.
Acknowledgments
The authors acknowledge Dr. Michael I. Jordan for useful discussions.
This work was supported by the Air Force
Research Laboratory, Collaborative Research and Development for Innovative Aerospace Leadership (CRDInAL),
Thrust 3 - Control Automation and Mechanization grant
FA 8650-16-C-2642 and the Boeing Strategic University Initiative. Supplementary Material: Appendix A. Problem Statements
In this section, we state typical problems that are addressed in the areas of adaptive control and machine learning.
In both cases, we illustrate the role of learning, the input-output data used, and the overall problem that is desired to be solved.
A.1. Adaptive Control
The main goal in adaptive control is to carry out problems such as estimation or tracking in the presence of parametric uncertainties.
The underlying model that relates inputs, outputs, and the unknown parameters is assumed to stem from either the underlying physics or from data-driven approaches.
Often these models take the form
[Equation]
or
[Equation]
where u ∈ R m is an exogenous input, x ∈ R n denotes the state, y ∈ R p corresponds to output measurements, φ ∈
R N corresponds to measured and computed variables, and θ ∗ ∈ R N denotes the uncertain parameter.
In an estimation problem, the goal is to estimate the state x in (22) and output y in both (21), (22), alongside the unknown parameter θ ∗ simultaneously, using all available variables.
In a control problem, the goal is to determine a control input u so that the output y in (22) follows a desired output ˆ y .
A typical approach taken in order to solve the estimation problem in (21) is to choose an estimator structure of the form ˆ y ( t ) =
[Equation]
[Equation] [Equation]
(23) where θ ∈ R N denotes the estimate of θ ∗ and adjust θ so that the estimation error e y
= ˆ y − y is minimized, i.e., choose a function g 1 ( e y , φ ) with ˙
[Equation]
e
[Equation] [Equation]
[Equation] [Equation]
)) (24) so that the estimator has bounded signals, e y ( t ) converges to zero and θ ( t ) converges to θ ∗ .
Similarly, the control problem consists of constructing an output tracking error e y
= ˆ
y − y , where ˆ y denotes the desired output that y is required to track.
The goal is to then choose functions g 2 ( e y , φ, θ ) and g 3 ( e y , φ, θ ) so that the control input u and parameter estimate θ can be chosen as
[Equation]
[Figure]
[Equation]
[Figure]
leading to closed-loop signals remaining bounded, e y ( t ) converging to zero and θ ( t ) converging to its true value θ ∗ .
Denote the corresponding parameter errors as ˜ θ
= θ − θ ∗ .
In order to derive the function g 1 for the estimation problem in (21) and the functions g 2 and g 3 for the control problem in (22) so as to realize the underlying goals, a stability framework together with an error model approach is often employed in adaptive control.
The error model approach consists of identifying the basic relationship between the two errors that are commonly present in these adaptive systems, which are the estimation (or tracking) error e y and the parameter error ˜
θ .
While the estimation error is measurable and correlated with the parameter error, the parameter error is unknown but adjustable through the parameter estimate.
In order to determine the update laws g i , the relationship (error model) that relates these two errors is used as a cue.
Two types of error models frequently occur in adaptive systems, and are presented below (see Figure 1).
The ﬁrst corresponds to the case when the relation in (21) is linear, and the underlying error model is simply of the form (cf.
(Narendra & Annaswamy, 2005))
[Equation] [Equation]
and as a result, the function g 1 in (24) can be determined simply using the gradient rule that minimizes (cid:107) e y
(cid:107) 2 .
The second is of the form (cf.
(Narendra & Annaswamy, 2005))
[Equation] [Equation]
where W ( s )[ ζ ] denotes a dynamic operator operating on ζ ( t ) .
It has been shown in the adaptive control literature (Narendra & Annaswamy, 1989; Sastry & Bodson, 1989; ˚Astr¨om & Wittenmark, 1995; Ioannou & Sun, 1996; Narendra & Annaswamy, 2005)
that for speciﬁc classes of dynamic operators W ( s ) , a stable, gradient-like rule can be determined for adjusting ˜ θ .
Most of these results apply uniformly to the case when u and y are scalars or vectors, with the latter introducing additional technicalities.
In this paper we consider the case where inputs and outputs are scalars for notational simplicity, and to focus on the core of the learning problem with multi-dimensional regressors φ and parameter estimates θ .
Often the unknown parameter θ ∗ is assumed to reside in a compact convex set, which we will denote as Θ .
A.2. Machine Learning
Machine learning is a broad ﬁeld encompassing a wide variety of learning techniques and problems such as classiﬁcation and regression.
A large portion of machine learning considers supervised learning problems, where regressors φ and outputs y are related to one another in an unknown algebraic manner (Duda et al., 2001; Bishop, 2006;
Hastie et al., 2009; Efron & Hastie, 2016; Goodfellow et al., 2016;
Jordan & Mitchell, 2015).
A typical approach taken in order to perform classiﬁcation or regression is to choose an output estimator
ˆ y k parameterized with adjustable weights
θ k as
[Equation]
A common form of the estimator as in (28) is that of neural networks, where the parameters θ k represent the adjustable weights in the network (Duda et al., 2001; Bishop, 2006;
Hastie et al., 2009; Efron & Hastie, 2016; Goodfellow et al., 2016).
Similar to adaptive control, θ k is often adjusted using the output error e y,k
= ˆ y k − y k .
A loss function L : Θ → R of e y,k is minimized through the adjustable weights.
An example loss function for regression is (cid:96) p loss (with p ∈ N , p > 0 and even)
L ( θ k ) =
(1 /p ) (cid:107) e y,k (cid:107) pp .
For binary classiﬁcation ( y k ∈ {− 1 , 1 } ) common loss functions include hinge loss L ( θ k )
= max(0 , 1 − y k
ˆ y k ) , and logistic loss
L ( θ k ) =
ln(1 + exp ( − y k
ˆ y k )) .
Additionally, as in empirical risk minimization (ERM) (Vapnik, 1992), the total loss function considered for the purpose of a parameter update may be an average of loss functions over m samples as: (1 /m ) (cid:80) mi
=1 L i ( θ k ) .
The above descriptions make it clear that the structure of the estimation problem in both adaptive control and machine learning are strikingly similar.
In the next section, we examine the nature of the adjustment of θ k .
A.3. Common Update Laws
As previously stated, the goal in adaptive control is to design a rule to adjust θ in an online continuous manner using knowledge of φ and e y such that e y tends toward zero.
Given that the output errors may be corrupted by noise, an iterative, gradient-like update is usually employed.
To do so for the algebraic error model (26), consider the squared loss cost function:
L ( θ ( t ))
= (1 / 2) e 2 y ( t ) .
The gradient of this function with respect to the parameters can be expressed as: ∇ θ L ( θ ( t ))
=
φ ( t ) e y ( t ) .
The standard gradient ﬂow update law (Narendra & Annaswamy, 1989) may be expressed as follows with user-designed gain parameter γ > 0 as
[Equation]
For dynamical error models such as (27), a stability approach rather than a gradient based one is taken using Lyapunov methods, which leads to an adaptive law identical to (29) for a class of dynamic systems W ( s ) that are strictly positive real (Narendra & Annaswamy, 1989; Parks, 1966).
The common update law for supervised machine learning problems, gradient descent 2 , is akin to the time varying regression law (29) in discrete time, and of the form
[Equation]
30) where the “stepsize” γ k is usually chosen as a decreasing function of time (Hazan et al., 2007; 2008; Hazan, 2016; Bubeck, 2015; Zinkevich, 2003), a standard feature of stochastic gradient algorithms.