DDPNO PT : D IFFERENTIAL D YNAMIC P ROGRAMMING N EURAL O PTIMIZER
A BSTRACT
Interpretation of Deep Neural Networks (DNNs) training as an optimal control problem with nonlinear dynamical systems has received considerable attention recently yet the algorithmic development remains relatively limited
In this work , we make an attempt along this line by reformulating the training procedure from the trajectory optimization perspective.
We ﬁrst show that most widely-used algorithms for training DNNs can be linked to the Differential Dynamic Programming (DDP), a celebrated second-order method rooted in the Approximate Dynamic Programming.
In this vein we propose a new class of optimizer, DDP Neural Optimizer (DDPNOpt), for training feedforward and convolution networks.
DDPNOpt features layer-wise feedback policies which improve convergence and reduce sensitivity to hyper-parameter over existing methods.
It outperforms other optimal-control inspired training methods in both convergence and complexity and is competitive against state-of-the-art ﬁrst and second order methods.
We also observe DDPNOpt has surprising beneﬁt in preventing gradient vanishing.
Our work opens up new avenues for principled algorithmic design built upon the optimal control theory
I NTRODUCTION
In this work, we consider the following optimal control problem (OCP) in the discrete-time setting:
[Equation]
where x t ∈ R n and u t ∈ R m represent the state and control at each time step t .
f t ( · , · ) , (cid:96) t ( · , · ) and φ ( · ) respectively denote the nonlinear dynamics, intermediate cost and terminal cost functions.
OCP aims to ﬁnd a control trajectory, ¯ u (cid:44) { u t } T − 1 t =0 , such that the accumulated cost J over the ﬁnite horizon t ∈ { 0 , 1 , · · · , T } is minimized.
Problems with the form of OCP appear in multidisciplinary areas since it describes a generic multi-stage decision making problem (Gamkrelidze, 2013), and have gained commensurate interest recently in deep learning (Weinan, 2017; Liu & Theodorou, 2019).
Central to the research along this line is the interpretation of DNNs as discrete-time nonlinear dynamical systems , where each layer is viewed as a distinct time step (Weinan, 2017).
The dynamical system perspective provides a mathematically-sound explanation for existing DNN models (Lu et al., 2019).
It also leads to new architectures inspired by numerical differential equations and physics (Lu et al., 2017;
Chen et al., 2018; Greydanus et al., 2019).
In this vein, one may interpret the training as the parameter identiﬁcation (PI) of nonlinear dynamics.
However, PI typically involves (i) searching time-independent parameters (ii) given trajectory measurements at each time step (Voss et al., 2004;
Peifer & Timmer, 2007).
Neither setup holds in piratical DNNs training, which instead optimizes time- ( i.e. layer-) varying parameters given the target measurements only at the ﬁnal stage.
An alternative perspective, which often leads to a richer analysis, is to recast network weights as control variables .
Through this lens, OCP describes w.l.o.g.
the training objective composed of layerwise loss ( e.g. weight decay) and terminal loss ( e.g. cross-entropy).
This perspective (see Table 1) has been explored recently to provide theoretical statements for convergence and generalization (Weinan et al., 2018; Seidman et al., 2020).
On the algorithmic side, while OCP has motivated new architectures (Benning et al., 2019) and methods for breaking sequential computation (Gunther et al., 2020; Zhang et al., 2019), OCP-inspired optimizers remain relatively limited, often restricted to either speciﬁc network class ( e.g. discrete weight)
(Li & Hao, 2018) or small-size dataset (Li et al., 2017).
The aforementioned works are primarily inspired by the Pontryagin Maximum Principle (PMP,
Boltyanskii et al.
(1960)), which characterizes the ﬁrst-order optimality conditions to OCP.
Another parallel methodology which receives little attention is the Approximate Dynamic Programming (ADP,
Bertsekas et al.
Despite both originate from the optimal control theory, ADP differs from
PMP in that at each time step a locally optimal feedback policy (as a function of state x t ) is computed.
These policies, as opposed to the vector update from PMP, are known to enhance the numerical stability of the optimization process when models admit chain structures ( e.g. in DNNs) (Liao &
Shoemaker, 1992; Tassa et al., 2012).
Practical ADP algorithms such as the Differential Dynamic
Programming (DDP, Jacobson & Mayne (1970)) appear extensively in modern autonomous systems for complex trajectory optimization (Tassa et al., 2014; Gu, 2017).
However, whether they can be lifted to large-scale stochastic optimization, as in the DNN training, remains unclear.
[Table]
In this work, we make a signiﬁcant advance toward optimal-control-theoretic training algorithms inspired by ADP.
We ﬁrst show that most existing ﬁrst- and second-order optimizers can be derived from DDP as special cases.
Built upon this intriguing connection, we present a new class of optimizer which marries the best of both.
The proposed method, DDP Neural Optimizer (DDPNOpt), features layer-wise feedback policies, which, as we will show through experiments, improve convergence and robustness.
To enable efﬁcient training, DDPNOpt adapts key components including (i) curvature adaption from existing methods, (ii) stabilization techniques used in trajectory optimization, and (iii) an efﬁcient factorization to OCP.
These lift the complexity by orders of magnitude compared with other OCP-inspired baselines, without sacriﬁcing the performance.
In summary, we present the following contributions.
• We draw a novel perspective of DNN training from the trajectory optimization viewpoint, based on a theoretical connection between existing training methods and the DDP algorithm.
• We present a new class of optimizer, DDPNOpt , that performs a distinct backward pass inherited with Bellman optimality and generates layer-wise feedback policies to robustify the training against unstable hyperparameter ( e
g. large learning rate) setups.
• We show that DDPNOpt achieves competitive performance against existing training methods on classiﬁcation datasets and outperforms previous OCP-inspired methods in both training performance and runtime complexity.
We also identify DDPNOpt can mitigate vanishing gradient.
P RELIMINARIES
We will start with the Bellman principle to OCP and leave discussions on PMP in Appendix A.1.
Theorem 1 ( Dynamic Programming (DP) (Bellman, 1954)) .
Deﬁne a value function V t : R n (cid:55)→ R at each time step that is computed backward in time using the Bellman equation
[Equation]
where Γ x t : R n (cid:55)→ R m denotes a set of mapping from state to control space.
Then, we have
V 0 ( x 0 )
=
J ∗ ( x 0 ) be the optimal objective value to OCP.
Further, let µ ∗ t ( x t ) ∈
Γ x t be the minimizer of Eq.
1 for each t , then the policy π ∗ = { µ ∗ t ( x t ) } T − 1 t =0 is globally optimal in the closed-loop system.
Notation: We will always use t as the time step of dynamics and denote a subsequence trajectory until time s as ¯ x s (cid:44) { x t } st =0 , with ¯ x (cid:44) { x t } Tt =0 as the whole.
For any real-valued time-dependent function F t , we denote its derivatives evaluated on a given state-control pair ( x t ∈ R n and u t ∈ R m ) as ∇ x t F t ∈ R n , ∇ 2 x t F t ∈ R n × n , ∇ x t u t F t ∈ R n × m , or simply F t x , F t xx , and F t xu for brevity.
The vector-tensor product, i.e. the contraction mapping on the dimension of the vector space, is denoted by V x · f xx (cid:44) (cid:80) ni
{+=
+}
[ V x ] i [ f xx ] i , where [ V x ] i is the i -th element of the vector V x
and [ f xx ] i is the Hessian matrix corresponding to that element. [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
Hereafter we refer Q t ( x t , u t ) to the Bellman objective . The Bellman principle recasts minimization over a control sequence to a sequence of minimization over each control. The value function V t summarizes the optimal cost-to-go at each stage, provided all afterward stages also being minimized.
Differential Dynamic Programming (DDP).
Despite providing the sufﬁcient conditions to OCP, solving Eq.
1 for high-dimensional problems appears to be infeasible, well-known as the Bellman curse of dimensionality.
To mitigate the computational burden of the minimization involved at each stage, one can approximate the Bellman objective in Eq.
1 with its second-order Taylor expansion.
Such an approximation is central to DDP, a second-order trajectory optimization method that inherits a similar Bellman optimality structure while being computationally efﬁcient.
Alg. 1 summarizes the DDP algorithm.
Given a nominal trajectory (¯ x , ¯ u ) with its initial cost J , DDP iteratively optimizes the objective value, where each iteration consists a backward (lines 2 - 6 ) and forward pass (lines 7 - 11 ).
During the backward pass, DDP performs second-order expansion on the
Bellman objective Q t at each stage and computes the updates through the following minimization,
[Equation]
where Q t x = Q t
[Equation]
We note that all derivatives in Eq.
3 are evaluated at the state-control pair ( x t , u t ) at time t among the nominal trajectory.
The derivatives of Q t follow standard chain rule and the dot notation represents the product of a vector with a 3D tensor.
Γ (cid:48) δ x t = { b t + A t δ x t : b t ∈ R m , A t ∈ R m × n } denotes the set of all afﬁne mapping from δ x t .
The analytic solution to Eq.
2 admits a linear form given by
[Equation]
, where
[Equation]
denote the open and feedback gains, respectively.
δ x t is called the state differential , which will play an important role later in our analysis.
Note that this policy is only optimal locally around the nominal trajectory where the second order approximation remains valid.
Substituting Eq.
4 back to
Eq. 2 gives us the backward update for V x and V xx ,
[Equation]
In the forward pass, DDP applies the feedback policy sequentially from the initial time step while keeping track of the state differential between the new simulated trajectory and the nominal trajectory.
D IFFERENTIAL D YNAMIC P ROGRAMMING N EURAL O PTIMIZER 3. T RAINING DNN S AS T RAJECTORY O PTIMIZATION
Recall that DNNs can be interpreted as dynamical systems where each layer is viewed as a distinct time step.
Consider e.g. the propagation rule in feedforward layers,
[Equation]
[Equation] x t ∈ R n t and x t +1 ∈ R n t +1 represent the activation vector at layer t and t + 1 , with h t ∈ R n t + being the pre-activation vector.
σ t and g t respectively denote the nonlinear activation function and the afﬁne transform parametrized by the vectorized weight u t (cid:44) [vec( W t ) , b t ] T .
Eq.
6 can be seen as a dynamical system (by setting f t ≡ σ t ◦ g t in OCP) propagating the activation vector x t using u t .
Next, notice that the gradient descent (GD) update, denoted δ ¯ u ∗ ≡ − η ∇ ¯ u J with η being the learning rate, can be break down into each layer, i.e. δ ¯ u ∗ (cid:44) { δ u ∗ t } T − [Equation] t = [Equation]
, and computed backward by
[Equation]
where J t ( x t , u t ) (cid:44) (cid:96) t ( u t ) + J t +1 ( f t ( x t [Equation] u t ) , u t +1 ) , J T ( x [Equation]
(cid:44) φ ( x T ) (8) is the per-layer objective 1 at layer t .
It can be readily veriﬁed that p t ≡ ∇ x t J t gives the exact
Back-propagation dynamics.
Eq. 8 suggests that GD minimizes the quadratic expansion of J t with the Hessian ∇ 2 u t J t replaced by 1 η I t .
Similarly, adaptive ﬁrst-order methods, such as RMSprop and
Adam, approximate the Hessian with the diagonal of the covariance matrix.
Second-order methods, such as KFAC and EKFAC (Martens & Grosse, 2015; George et al., 2018), compute full matrices using Gauss-Newton (GN) approximation:
[Equation]
We now draw a novel connection between the training procedure of DNNs and DDP.
Let us ﬁrst summarize the Back-propagation (BP) with gradient descent in Alg. 2 and compare it with DDP (Alg. 1).
At each training iteration, we treat the current weight as the control ¯ u that simulates the activation sequence ¯ x .
Starting from this nominal trajectory (¯ x , ¯ u ) , both algorithms recursively deﬁne some layer-wise objectives ( J t in Eq.
8 vs V t in Eq. 1), compute the weight/control update from the quadratic expansions (Eq. 7 vs Eq. 2), and then carry certain information ( ∇ x t J t vs ( V t x , V t xx ) ) backward to the preceding layer.
The computation graph between the two approaches is summarized in Fig.
In the following proposition, we make this connection formally and provide conditions when the two algorithms become equivalent.
Proposition 2.
Assume Q t ux = 0 at all stages, then the backward dynamics of the value derivative can be described by the Back-propagation,
[Equation]
In this case, the DDP policy is equivalent to the stage-wise Newton, in which the gradient is preconditioned by the block-wise inverse Hessian at each layer:
[Equation]
If further we have Q t uu ≈ 1 η I , then DDP degenerates to Back-propagation with gradient descent.
Proof is left in Appendix A.2.
Proposition 2 states that the backward pass in DDP collapses to BP when Q ux vanishes at all stages.
In other words, existing training methods can be seen as special cases of DDP when the mixed derivatives ( i.e. ∇ x t u t ) of the layer-wise objective are discarded.
3. E FFICIENT A PPROXIMATION AND F ACTORIZATION
Motivated by Proposition 2, we now present a new class of optimizer, DDP Neural Optimizer (DDPNOpt), on training feedforward and convolution networks.
DDPNOpt follows the same procedure in vanilla DDP (Alg. 1) yet adapts several key traits arising from DNN training, which we highlight below.
Evaluate derivatives of Q t with layer dynamics.
The primary computation in DDPNOpt comes from constructing the derivatives of Q t at each layer.
When the dynamics is represented by the layer
[Table]
propagation (recall Eq.
6 where we set f t ≡ σ t ◦ g t ), we can rewrite Eq.
3 as:
Q t x
[Equation]
(12) where V t h (cid:44) σ t T h V t +1 x and V t hh (cid:44) σ t T h
V t +1 xx σ t
h absorb the computation of the non-parametrized activation function σ .
Note that Eq.
12 expands the dynamics only up to ﬁrst order, i.e. we omitt the tensor products which involves second-order expansions on dynamics, as the stability obtained by keeping only the linearized dynamics is thoroughly discussed and widely adapted in practical DDP usages (Todorov & Li, 2005).
The matrix-vector product with the Jacobian of the afﬁne transform ( i.e. g t u , g t x ) can be evaluated efﬁciently for both feedforward (FF) and convolution (Conv) layers:
[Equation]
where ⊗ , ˆ ∗ , and ∗ respectively denote the Kronecker product and (de-)convolution operator.
Curvature approximation.
Next, since DNNs are highly over-parametrized models, u t ( i.e. the layer weight) will be in high-dimensional space.
This makes Q t uu and ( Q t uu ) − 1 computationally intractable to solve; thus requires approximation.
Recall the interpretation we draw in Eq. 8 where existing optimizers differ in approximating the Hessian ∇ 2 u t J t .
DDPNOpt adapts the same curvature approximation to Q t uu .
For instance, we can approximate Q t uu simply with an identity matrix I t , adaptive diagonal matrix diag( (cid:112) E
[ Q t u (cid:12) Q t u ] ) , or the GN matrix:
[Equation] [Equation]
Table 2 summarizes the difference in curvature approximation ( i.e. the precondition M t ) for different methods.
Note that DDPNOpt constructs these approximations using ( V, Q ) rather than J since they consider different layer-wise objectives.
As a direct implication from Proposition 2 , DDPNOpt will degenerate to the optimizer it adapts for curvature approximation whenever all Q t ux vanish.
Outer-product factorization.
When the memory efﬁciency becomes nonnegligible as the problem scales, we make GN approximation to ∇ 2 φ , since the low-rank structure at the prediction layer has been observed for problems concerned in this work (Nar et al., 2019; Lezama et al., 2018).
In the following proposition, we show that for a speciﬁc type of OCP, which happens to be the case of DNN training, such a low-rank structure preserves throughout the DDP backward pass.
Proposition 3 (Outer-product factorization in DDPNOpt) .
Consider the OCP where (cid:96) t ≡ (cid:96) t ( u t ) is independent of x t , If the terminal-stage Hessian can be expressed by the outer product of vector z T x , ∇ 2 φ ( x T ) =
z T x ⊗ z T x (for instance, z T x = ∇ φ for GN), then we have the factorization for all t :
[Equation]
q t u , q t x , and z t x are outer-product vectors which are also computed along the backward pass.
[Equation]
[Equation]
The derivation is left in Appendix A.3.
In other words, the outer-product factorization at the ﬁnal layer can be backward propagated to all proceeding layers.
Thus, large matrices, such as Q t ux , Q t xx ,
V t xx , and even feedback policies K t , can be factorized accordingly, greatly reducing the complexity.
1: Input: dataset
, learning xx 2: Initialize ) 3: for to do 4: Sample batch initial state from dataset, X 0 ≡ { x ( i ) 0 } Bi =1 ∼ D 5:
Forward propagate to generate nominal batch trajectory X t (cid:66)
Forward simulation 6:
Set V T x ( i ) =
∇ x ( i ) Φ( x ( i ) T ) and V T xx ( i ) = ∇ 2 x ( i ) Φ( x ( i ) T ) 7: for t =
T − 1 to 0 do (cid:66) Backward Bellman pass 8:
Compute Q t u , Q t x , Q t xx , Q t ux with Eq.
12 (or Eq.
16-17 if factorization is used) 9:
Compute E [ Q t uu ] with one of the precondition matrices in Table10:
Store the layer-wise feedback policy δ u ∗ t ( δ X t ) =
1 B (cid:80) Bi
=1 k ( i ) t + K ( i ) t
δ x ( i ) t 11:
Compute V t x ( i ) and V t xx ( i ) with Eq.
5 (or Eq.
16-17 if factorization is used) 12:
V t xx ( i ) ← V t xx ( i ) + (cid:15)
V xx
I t if regularization is used 13: end for 14:
Set ˆ x ( i )
= x ( i )15: for t = 0 to T − 1 do (cid:66) Additional forward pass 16: u ∗ t =
u t + δ u ∗ t ( δ X t ) , where δ X t
= { ˆ x ( i ) t − x ( i ) t } B i = 17: ˆ x ( i ) t +1 =
f t (ˆ x ( i ) t , u ∗ t ) 18: end for 19: ¯ u ( k +1)
← { u ∗ t } T − 1 t = 20: end for
Regularization on V xx .
Finally, we apply Tikhonov regularization to the value Hessian V t xx (linein Alg. 3).
This can be seen as placing a quadratic state-cost and has been shown to improve stability on optimizing complex humanoid behavior (Tassa et al., 2012).
For the application of DNN where the dimension of the state ( i.e. the vectorized activation) varies during forward/backward pass, the
Tikhonov regularization prevents the value Hessian from low rank (throught g t T u V t hh g t x ); hence we also observe similar stabilization effect in practice.
T HE R OLE OF F EEDBACK P OLICIES
DDPNOpt differs from existing methods in the use of feedback K t and state differential δ x t .
The presence of these terms result in a distinct backward pass inherited with the Bellman optimality.
As shown in Table 2, the two frameworks differ in computing the update directions d t , where the
Bellman formulation applies the feedback policy through additional forward pass with δ x t .
We have built the connection between these two d t in Proposition 2.
In this section, we further characterize the role of the feedback policy K t and state differential δ x t during optimization.
First we discuss the relation of DDPNOpt with other second-order methods and highlight the role feedback during training.
To do so let us consider the example in Fig.
2a.
Given an objective L expanded at ( x 0 , u 0 ) , standard second-order methods compute the Hessian w.r.t. u then apply the update δu = − L − 1 uu L u (shown as green arrows).
DDPNOpt differs in that it also computes the mixed partial derivatives, i.e. L ux .
The resulting update law has the same intercept but with an additional feedback term linear in δx (shown as red arrows).
Thus, DDPNOpt searches for an update from the afﬁne mapping Γ (cid:48)
δ x t (Eq. 2), rather than the vector space
R m t (Eq. 7).
Next, to show how the state differential δ x t arises during optimization, notice from Alg.
1 that ˆ x t can be compactly expressed as ˆ x t = F t ( x 0 , ¯ u + δ ¯ u ∗ ( δ ¯ x )) 2 .
Therefore, δ x t = ˆ x t − x t captures the state difference when new updates δ ¯ u ∗ ( δ ¯ x ) are applied until layer t − 1 .
Now, consider the 2D example in
Fig 2b.
Back-propagation proposes the update directions (shown as blue arrows) from the ﬁrst-order derivatives expanded along the nominal trajectory (¯ x , ¯ u ) .
However, as the weight at each layer is correlated, parameter updates from previous layers δ ¯ u ∗ s affect proceeding states { x t : t > s } , thus the trustworthiness of their descending directions.
As shown in Fig 2c, cascading these (green) updates may cause an over-shoot w.r.t. the designed target.
From the trajectory optimization perspective, a much stabler direction will be instead ∇ u t J t (ˆ x t , u t ) (shown as orange), where the derivative is evaluated at the new cascading state ˆ x t , which accounts for previous updates, rather than the original state x t .
This is exactly what DDPNOpt proposes, as we can derive the relation (see Appendix A.5),
[Equation]
Thus, the feedback direction compensates the over-shoot by steering the GD update toward ∇ u t J t (ˆ x t , u t ) after observing δ x t .
The difference between ∇ u t J (ˆ x t , u t ) and ∇ u t J ( x t , u t ) cannot be neglected especially during early training when the loss landscape contains nontrivial curvature everywhere (Alain et al., 2019).
In short, the use of feedback K t and state differential δ x t arises from the fact that deep nets exhibit chain structures .
DDPNOpt feedback policies thus have a stabilization effect on robustifying the training dynamics against e.g. improper hyper-parameters which may cause unstable training.
This perspective ( i.e. optimizing chained parameters) is explored rigorously in trajectory optimization, where DDP is shown to be numerically stabler than direct optimization such as Newton method (Liao & Shoemaker, 1992).
Remarks on other optimizers.
Our discussions so far rigorously explore the connection between
DDP and stage/layer-wise Newton, thus include many popular second-order training methods.
General Newton method coincides with DDP only for linear dynamics (Murray & Yakowitz, 1984), despite both share the same convergence rate when the dynamics is fully expanded to second order.
We note that computing layer-wise value Hessians with only ﬁrst-order expansion on the dynamics (Eq. 12) resembles the computation in Gauss-Newton method (Botev et al., 2017).
For other controltheoretic methods, e.g. PID optimizers (An et al., 2018), they mostly consider the dynamics over training iterations.
DDPNOpt instead focuses on the dynamics inherited in the DNN architecture.
E XPERIMENTS 5. P ERFORMANCE ON C LASSIFICATION D ATASET
Networks & Baselines Setup.
We ﬁrst validate the performance of training fully-connected (FCN) and convolution networks (CNN) using DDPNOpt on classiﬁcation datasets.
FCN consists offully-connected layers with the hidden dimension ranging from 10 to 32 , depending on the size of the dataset.
CNN consists of 4 convolution layers (with 3 × 3 kernel, 32 channels), followed byfully-connected layers.
We use ReLU activation on all datasets except Tanh for WINE and DIGITS to better distinguish the differences between optimizers.
The batch size is set to 8 - 32 for datasets trained with FCN, and 128 for datasets trained with CNN.
As DDPNOpt combines strengths from both standard training methods and OCP framework, we select baselines from both sides.
This includes ﬁrst-order methods, i.e. SGD (with tuned momentum), RMSprop, Adam, and second-order method EKFAC (George et al., 2018), which is a recent extension of the popular KFAC (Martens &
Grosse, 2015).
For OCP-inspired methods, we compare DDPNOpt with vanilla DDP and E-MSA (Li et al., 2017), which is also a second-order method yet built upon the PMP framework.
Regarding the curvature approximation used in DDPNOpt ( M t in Table 2), we found that using adaptive diagonal and GN matrices respectively for FCNs and CNNs give the best performance in practice.
We leave the complete experiment setup and additional results in Appendix A.6.
Training Results.
Table 3 presents the results over 10 random trials.
It is clear that DDPNOpt outperforms two OCP baselines on all datasets and network types .
In practice, both baselines suffer from unstable training and require careful tuning on the hyper-parameters.
In fact, we are not able to obtain results for vanilla DDP with any reasonable amount of computational resources when the problem size goes beyond FC networks.
This is in contrast to DDPNOpt which adapts amortized
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
curvature estimation from widely-used methods; thus exhibits much stabler training dynamics with superior convergence.
In Table 4, we provide the analytic runtime and memory complexity among different methods.
While vanilla DDP grows cubic w.r.t. BX , DDPNOpt reduces the computation by orders of magnitude with efﬁcient approximation presented in Sec. 3.
As a result, when measuring the actual computational performance with GPU parallelism, DDPNOpt runs nearly as fast as standard methods and outperforms E-MSA by a large margin.
The additional memory complexity, when comparing DDP-inspired methods with Back-propagation methods, comes from the layer-wise feedback policies.
However, DDPNOpt is much memory-efﬁcient compared with vanilla DDP by exploiting the factorization in Proposition 3.
Ablation Analysis.
On the other hand, the performance gain between DDPNOpt and standard methods appear comparatively small.
We conjecture this is due to the inevitable use of similar curvature adaptation, as the local geometry of the landscape directly affects the convergence behavior.
To identify scenarios where DDPNOpt best shows its effectiveness, we conduct an ablation analysis on the feedback mechanism.
This is done by recalling Proposition 2: when Q t ux vanishes, DDPNOpt degenerates to the method associated with each precondition matrix.
For instance, DDPNOpt with identity ( resp.
adaptive diagonal and GN)
precondition M t will generate the same updates as SGD ( resp.
RMSprop and EKFAC) when all Q t ux are zeroed out.
In other words, these DDPNOpt variants can be viewed as the DDP-extension to existing baselines.
In Fig.
4a we report the performance difference between each baseline and its associated DDPNOpt variant.
Each grid corresponds to a distinct training conﬁguration that is averaged over 10 random trails, and we keep all hyper-parameters ( e.g. learning rate and weight decay) the same between baselines and their DDPNOpt variants.
Thus, the performance gap only comes from the feedback policies, or equivalently the update directions in Table 2.
Blue ( resp. red) indicates an improvement ( resp.
degradation) when the feedback policies are presented.
Clearly, the improvement over baselines remains consistent across most hyper-parameters setups, and the performance gap tends to become obvious as the learning rate increases.
This aligns with the previous study on numerical stability (Liao & Shoemaker, 1992), which suggests the feedback can stabilize the optimization when e.g. larger control updates are taken.
Since larger control corresponds to a further step size in the application of
DNN training, one should expect DDPNOpt to show its robustness as the learning rate increases.
As shown in Fig.
4b, such a stabilization can also lead to smaller variance and faster convergence.
This sheds light on the beneﬁt gained by bridging two seemly disconnected methodologies between DNN training and trajectory optimization.
5. D ISCUSSION ON F EEDBACK P OLICIES
Visualization of Feedback Policies.
To understand the effect of feedback policies more perceptually, in Fig.
5 we visualize the feedback policy when training CNNs.
This is done by ﬁrst conducting
[Figure] [Figure]
singular-value decomposition on the feedback matrices K t , then projecting the leading right-singular vector back to image space (see Alg. 4 and Fig. 7 in Appendix for the pseudo-code).
These feature maps, denoted δx max in Fig.
5, correspond to the dominating differential image that the policy shall respond with during weight update.
Fig.
5 shows that the feedback policies indeed capture non-trivial visual features related to the pixel-wise difference between spatially similar classes, e.g. (8 , 3) or (7 , 1) .
These differential maps differ from adversarial perturbation (Goodfellow et al., 2014) as the former directly links the parameter update to the change in activation; thus being more interpretable.
Vanishing Gradient.
Lastly, we present an interesting ﬁnding on how the feedback policies help mitigate vanishing gradient (VG), a notorious effect when DNNs become impossible to train as gradients vanish along Back-propagation.
Fig.
6a reports results on training a sigmoid-activated
DNN on DIGITS.
We select SGD-VGR, which imposes a speciﬁc regularization to mitigate VG (Pascanu et al., 2013), and EKFAC as our baselines.
While both baselines suffer to make any progress,
DDPNOpt continues to generate non-trivial updates as the state-dependent feedback, i.e. K t δ x t , remains active.
The effect becomes signiﬁcant when dynamics is fully expanded to the second order.
As shown in Fig.
6b, the update norm from DDPNOpt is typically 5 - 10 times larger.
We note that in this experiment, we replace the cross-entropy (CE) with Max-Mahalanobis center (MMC) loss, a new classiﬁcation objective that improves robustness on standard vision datasets (Pang et al., 2019).
MMC casts classiﬁcation to distributional regression, providing denser Hessian and making problems similar to original trajectory optimization.
None of the algorithms escape from VG using CE.
We highlight that while VG is typically mitigated on the architecture basis, by having either unbounded activation function or residual blocks, DDPNOpt provides an alternative from the algorithmic perspective.
C ONCLUSION
In this work, we introduce DDPNOpt, a new class of optimizer arising from a novel perspective by bridging DNN training to optimal control and trajectory optimization.
DDPNOpt features layer-wise feedback policies which improve convergence and robustness to hyper-parameters over existing optimizers.
It outperforms other OCP-inspired methods in both training performance and scalability.
This work provides a new algorithmic insight and bridges between deep learning and optimal control.
A CKNOWLEDGMENTS
The authors would like to thank Chen-Hsuan Lin, Yunpeng Pan, Yen-Cheng Liu and Chia-Wen Kuo for many helpful discussions on the paper.
This research was supported by NSF Award Number
A A PPENDIX A. C ONNECTION BETWEEN P ONTRYAGIN M AXIMUM P RINCIPLE AND DNN S T RAINING
Development of the optimality conditions to OCP can be dated back to 1960s, characterized by both the Pontryagin's Maximum Principle (PMP) and the Dynamic Programming (DP).
Here we review
Theorem of PMP and its connection to training DNNs
Theorem 4 (Discrete-time PMP (Pontryagin et al., 1962))
Let ¯ u ∗ be the optimal control trajectory for OCP and ¯ x
∗ be the corresponding state trajectory.
Then there exists a co-state trajectory ¯ p ∗ (cid:44) { p ∗ t } Tt = such that
[Equation]
19c) where H t : R n × R n × R m (cid:55)→ R is the discrete-time
Hamiltonian given by
[Equation]
and Eq. 19b is called the adjoint equation.
The discrete-time PMP theorem can be derived using KKT conditions, in which the co-state p t is equivalent to the Lagrange multiplier.
Note that the solution to Eq.
19c admits an open-loop process in the sense that it does not depend on state variables.
This is in contrast to the Dynamic Programming principle, in which a feedback policy is considered.
It is natural to ask whether the necessary condition in the PMP theorem relates to ﬁrst-order optimization methods in DNN training.
This is indeed the case as pointed out in Li et al.
Lemma 5 (Li et al.
Back-propagation satisﬁes Eq. 19b and gradient descent iteratively solves Eq. 19c.
Lemma 5 follows by ﬁrst expanding the derivative of Hamiltonian w.r.t. x t ,
[Equation]
[Equation]
Thus, Eq. 19b is simply the chain rule used in the Back-propagation.
When H t is differentiable w.r.t. u t , one can attempt to solve Eq. 19c by iteratively taking the gradient descent.
This will lead to
[Equation]
where k and η denote the update iteration and step size.
Thus, existing optimization methods can be interpreted as iterative processes to match the PMP optimality conditions.
Inspired from Lemma 5, Li et al.
(2017) proposed a PMP-inspired method, named Extended Method of Successive Approximations (E-MSA), which solves the following augmented Hamiltonian ˜
[Equation]
˜ H t is the original Hamiltonian augmented with the feasibility constraints on both forward states and backward co-states.
E-MSA solves the minimization
[Equation] [Equation]
with L-BFGS per layer and per training iteration.
As a result, we consider E-MSA also as second-order method.
A. P ROOF OF P ROPOSITION
Proof.
We ﬁrst prove the following lemma which connects the backward pass between two frameworks in the degenerate case.
Lemma 6.
Assume Q t ux = 0 at all stages, then we have
[Equation]
Proof.
It is obvious to see that Eq.
25 holds at t = T .
Now, assume the relation holds at t + 1 and observe that at the time t , the backward passes take the form of
[Equation]
where we recall J t = (cid:96) t
+ J t
+1 ( f t ) in Eq. 8.
Now, Eq. 11 follows by substituting Eq.
25 to the deﬁnition of Q t u and Q t uu
[Equation]
Consequently, the DDP feedback policy degenerates to layer-wise Newton update.
A. P ROOF OF P ROPOSITION
Proof.
We will prove Proposition 3 by backward induction.
Suppose at layer t + 1 , we have
V t +1 xx =
z t +1 x ⊗ z t +1 x and (cid:96) t ≡ (cid:96) t ( u t ) , then Eq.
3 becomes
[Equation]
Setting q t x :=
f t x T z t +1 x and q t u :=
f t u T z t +1 x will give the ﬁrst part of Proposition 3.
Next, to show the same factorization structure preserves through the preceding layer, it is sufﬁcient to show V t xx =
z t x ⊗ z t x for some vector z t x .
This is indeed the case.
[Equation]
where the last equality follows by observing q t T u ( Q t uu ) − 1 q t u is a scalar.
Set
A. D ERIVATION OF E Q .
For notational simplicity
[Equation] we drop the superscript t and denote V (cid:48) x (cid:48) (cid:44) ∇ x V t +1 ( x t +1 ) as the derivative of the value function at the next state.
[Equation]
The last equation follows by recalling V h (cid:44) σ T h V (cid:48) x
[Equation]
and V hh (cid:44) σ [Equation] h V [Equation]
x
[Equation]
x
[Equation]
σ h .
Follow similar derivation, we have
[Equation]
Remarks. For feedforward networks, the computational overhead in Eq.
12 and 26 can be mitigated by leveraging its afﬁne structure.
Since g is bilinear in x t and u t , the terms g t xx and g t uu vanish.
The tensor g t ux admits a sparse structure, whose computation can be simpliﬁed to
[Equation]
For the coordinate-wise nonlinear transform, σ t h and σ t hh are diagonal matrix and tensor.
In most learning instances, stage-wise losses typically involved with weight decay alone; thus the terms (cid:96) t x , (cid:96) t xx , (cid:96) t ux also vanish.
A. D ERIVATION OF E Q .
Eq.
18 follows by an observation that the feedback policy K t δ x t = − ( Q t uu ) − 1 Q t ux δ x t stands as the minimizer of the following objective
[Equation]
where Γ (cid:48)
( δ x t ) denotes all afﬁne mappings from δ x t to δ u t and (cid:107)·(cid:107) can be any proper norm in the
Euclidean space.
Eq.
28 follows by the Taylor expansion of Q ( x t + δ x t , u t + δ u t ) to its ﬁrst order,
[Equation] When Q = J , we will arrive at Eq. From Proposition 2, we know the equality holds when all Q s xu vanish for s > t .
In other words, the approximation in Eq.
18 becomes equality when all aferward layer-wise objectives s > t are expanded only w.r.t.
u s .
A. E XPERIMENT D ETAIL A.6. S ETUP
Clariﬁcation Dataset.
All networks in the classiﬁcation experiments are composed of 5 - 6 layers.
For the intermediate layers, we use ReLU activation on all dataset, except Tanh on WINE and DIGITS.
We use identity mapping at the last prediction layer on all dataset except WINE, where we use sigmoid instead to help distinguish the performance among optimizers.
For feedforward networks, the dimension of the hidden state is set to 10 - 32 .
On the other hand, we use standard 3 × 3 convolution [Table] kernels for all CNNs.
The batch size is set 8 - 32 for dataset trained with feedforward networks, and 128 for dataset trained with convolution networks.
For each baseline we select its own hyperparameter from an appropriate search space, which we detail in Table 5.
We use the implementation in https://github.com/Thrandis/EKFAC-pytorch for EKFAC and implement our own E-MSA in PyTorch since the ofﬁcial code released from Li et al.
(2017) does not support GPU implementation.
We impose the GN factorization presented in Proposition 3 for all CNN training.
Regarding the machine information, we conduct our experiments on GTX 1080 TI, RTX TITAN, and four Tesla V100 SXM2 16GB.
Procedure to Generate Fig. 5.
First, we perform standard DDPNOpt steps to compute layer-wise policies.
Next, we conduct singular-value decomposition on the feedback matrix ( k t , K t ) .
In this way, the leading right-singular vector corresponding to the dominating that the feedback policy shall respond with.
Since this vector is with the same dimension as the hidden state, which is most likely not the same as the image space, we project the vector back to image space using the techniques proposed in (Zeiler & Fergus, 2014). The pseudo code and computation diagram are included in Alg. 4 and Fig. Algorithm
1: Input: Image x (we drop the time subscript for nota- tional simplicity, i.e. x ≡ x 0 ) 2: Perform backward pass of DDPNOpt. Compute ( k t , K t ) backward 3: SVD on K t 4: Extract the right
singular
corresponding to
largest singular
denoted
max
: Project
decon
A.
[Figure]
[Figure]
A DDITIONAL E XPERIMENT AND D ISCUSSION
Batch trajectory optimization on synthetic dataset.
One of the difference between DNN training and trajectory optimization is that for the former, we aim to ﬁnd an ultimate control law that can drive every data point in the training set, or sampled batch, to its designed target.
Despite seemly trivial from the ML perspective, this is a distinct formulation to OCP since the optimal policy typically varies at different initial state.
As such, we validate performance of DDPNOpt in batch trajectories optimization on a synthetic dataset, where we sample data from k ∈ { 5 , 8 , 12 , 15 } Gaussian clusters in R 30 .
Since conceptually a DNN classiﬁer can be thought of as a dynamical system guiding trajectories of samples toward the target regions belong to their classes, we hypothesize that for the
DDPNOpt to show its effectiveness on batch training, the feedback policy must act as an ensemble policy that combines the locally optimal policy of each class.
Fig.
8 shows the spectrum distribution, sorted in a descending order, of the feedback policy in the prediction layer.
The result shows that the number of nontrivial eigenvalues matches exactly the number of classes in each setup (indicated by the vertical dashed line).
As the distribution in the prediction layer concentrates to k bulks through training, the eigenvalues also increase, providing stronger feedback to the weight update.
[Figure] Ablation analysis on Adam.
Fig.
9 reports the ablation analysis on Adam using the same setup as in Fig.
4a, i.e. we keep all hyper-parameters the same for each experiment so that the performance difference only comes from the existence of feedback policies.
It is clear that the improvements from the feedback policies remain consistent for Adam optimizer.
[Figure]
Ablation analysis on DIGITS compared with best-tuned baselines. Fig. 4 reports the performance difference between baselines and DDPNOpt under different hyperparameter setupts. Here, we report the numerical values when each baseline uses its best-tuned learning rate (which is the values we report in Table 3) and compare with its DDPNOpt counterpart using the same learning rate.
As shown in Tables 6,
[Table]
and 8, for most cases extending the baseline to accept the Bellman framework improves the performance.
[Table]
[Table]
[Table]
[Table] [Table] Numerical absolute values in ablation analysis (DIGITS). Fig.
4a reports the relative performance between each baseline and its DDPNOpt counterpart under different learning rate and regularization setups. In Table 9 and 10, we report the absolute numerical values of this experiment. For instance, the most left-upper grid in Fig. 4a,
i
[Table]
e. the training loss difference between DDPNOpt and SGD with learning rate 0 . 4 and V xx regularization 5 × 10 − 5 , corresponds to 0 . 1974 − 0 . 1662 in Table 9.
All values in these tables are averaged over 10 seeds. [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
ea
[Table] R [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] More experiments on vanishing gradient.
Recall that Fig.
6 reports the training performance using MMC loss on Sigmoid-activated networks.
In Fig.
10a, we report the result when training the same networks but using CE loss (notice the numerical differences in the y axis for different objectives).
None of the presented optimizers were able to escape from vanishing gradient, as evidenced by the vanishing update magnitude.
On the other hands, changing the networks to ReLU-activated networks eliminates the vanishing gradient, as shown in Fig. 10b. Fig.
11 reports the performance with other ﬁrst-order adaptive optimizers including Adam and
RMSprop. In general, adaptive ﬁrst-order optimizers are more likely to escape from vanishing gradient since the diagonal precondition matrix (recall M t = E [ J u t (cid:12) J u t ] in Table 2) rescales the vanishing update to a ﬁxed norm. However, as shown in Fig. 11, DDPNOpt* (the variant of
DDPNOpt that utilizes similar adaptive ﬁrst-order precondition matrix) converges faster compared with these adaptive baselines.
Fig.
12 illustrates the selecting process on the learing-rate tuning when we report Fig. 6.
The training performance for both SGD-VGR and EKFAC remains unchanged when tuning the learning rate.
In practice, we observe unstable training with SGD-VGR when the learing rate goes too large.
On the other hands, DDPNOpt and DDPNOpt2nd are able to escape from VG with all tested learning rates.
Hence [Figure] Fig. 6 combines Fig.
12a (SGD-VGR-lr 0 . 1 ) and Fig.
12c (EKFAC-lr 0 .
03 , DDPNOpt-lr 0 .
DDPNOpt2nd-lr 0 . 03 ) for best visualization
[Figure]
[Figure]
[-R a t e 0 .
-]
[Figure]