F IRST -O RDER O PTIMIZATION A LGORITHMS VIA D ISCRETIZATION OF F INITE -T IME C ONVERGENT F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper the performance of several discretization algorithms for two ﬁrst-order ﬁnite-time optimization ﬂows.
These ﬂows are, namely, the rescaled-gradient ﬂow (RGF) and the signed-gradient ﬂow (SGF), and consist of non-Lipscthiz or discontinuous dynamical systems that converge locally in ﬁnite time to the minima of gradient-dominated functions.
We introduce three discretization methods for these ﬁrst-order ﬁnite-time ﬂows, and provide convergence guarantees.
We then apply the proposed algorithms in training neural networks and empirically test their performances on three standard datasets, namely, CIFAR10, SVHN, and MNIST.
Our results show that our schemes demonstrate faster convergences against standard optimization alternatives, while achieving equivalent or better accuracy.
I NTRODUCTION
Consider the unconstrained minimization problem for a given cost function f : R n → R .
When f is sufﬁciently regular, the standard algorithm in continuous time (dynamical system) is given by
[Equation]
[Equation]
with ˙ x (cid:44) dd t x ( t ) , known as the gradient ﬂow (GF).
Generalizing GF, the q - rescaled GF ( qRGF)
Wibisono et al.
(2016) given by
[Equation]
(2) with c > 0 and q ∈ (1 , ∞ ] has an asymptotic convergence rate f ( x ( t ))
− f ( x (cid:63) )
=
O (cid:0) 1 t q − 1 (cid:1) under mild regularity, for (cid:107) x (0) − x (cid:63) (cid:107) > 0 small enough, where x (cid:63) ∈ R n denotes a local minimizer of f .
However, we recently proved Romero & Benosman (2020) that the q -RGF, as well as our proposed q - signed GF ( q -SGF)
[Equation]
where sign( · ) denotes the sign function, applied element-wise for (real-valued) vectors, are both ﬁnite-time convergent, provided that f is gradient dominated of order p ∈ (1 , q ) .
In particular, if f is strongly convex, then q -RGF and q -SGF is ﬁnite-time convergent for any q ∈ (2 , ∞ ] , since f must be gradient dominated of order p = 2 .
C ONTRIBUTION
In this paper, we explore three discretization schemes for the q -RGF (2) and q -SGF (3) and provide some convergence guarantees using results from hybrid dynamical control theory.
In particular, we explore a forward-Euler discretization of RGF/SGF, followed by an explicit Runge-Kutta discretization, and ﬁnally a novel Nesterov-like discretization.
We then test their performance on both synthetic and real-world data in the context of deep learning, namely, over the well-known datasets CIFAR10, SVHN, and MNIST.
R ELATED W ORK
Propelled by the work of Wang & Elia (2011) and Su et al.
(2014), there has been a recent and signiﬁcant research effort dedicated to analyzing optimization algorithms from the perspective of dynamical systems and control theory, especially in continuous time Wibisono et al.
; Wilson (2018); Lessard et al. (2016); Fazlyab et al.
(2017b)
; Scieur et al.
(2017); Franc¸a et al.
; Fazlyab et al. ; Fazlyab et al.
; Taylor et al.
; Franc¸a et al. (2019a); Orvieto & Lucchi (2019); Muehlebach & Jordan (2019).
A major focus within this initiative is in accceleration , both in terms of trying to gain new insight into more traditional optimization algorithms from this perspective, or even to exploit the interplay between continuous-time systems and their potential discretizations for novel algorithm design Muehlebach & Jordan (2019);
Fazlyab et al.
(2017a)
; Shi et al.
Zhang et al.
; Franc¸a et al. (2019b); Wilson et al.
Many of these papers also focus on explicit mappings and matchings of convergence rates from the continuous-time domain into discrete time.
For older work connecting ordinary differential equations (ODEs) and their numerical analysis, with optimization algorithms, see Botsaris (1978a;b); Zghier (1981); Snyman (1982; 1983); Brockett (1988); Brown (1989).
In Helmke & Moore (1994), the authors studied relationships between linear programming, ODEs, and general matrix theory.
Further, Schropp (1995) and Schropp & Singer (2000) explored several aspects linking nonlinear dynamical systems to gradient-based optimization, including nonlinear constraints.
Tools from Lyapunov stability theory are often employed for this purpose, mainly because there already exists a rich body of work within the nonlinear systems and control theory community for this purpose.
In particular, typically in previous works, one seeks asymptotically Lyapunov stable gradient-based systems with an equilibrium (stationary point) at an isolated extremum of the given cost function, thus certifying local convergence.
Naturally, global asymptotic stability leads to global convergence, though such an analysis will typically require the cost function to be strongly convex everywhere.
For physical systems, a Lyapunov function can often be constructed from ﬁrst principles via some physically meaningful measure of energy ( e.g. , total energy = potential energy + kinetic energy).
In optimization, the situation is somewhat similar in the sense that a suitable Lyapunov function may often be constructed by taking simple surrogates of the objective function as candidates.
For instance,
V ( x ) (cid:44) f
( x ) − f ( x (cid:63) ) can be a good initial candidate.
Further, if f is continuously differentiable and x (cid:63) is an isolated stationary point, then another alternative is V ( x ) (cid:44) (cid:107)∇ f ( x ) (cid:107) 2 .
However, most fundamental and applied research conducted in systems and control regarding Lyapunov stability theory deals exclusively with continuous-time systems.
Unfortunately, (dynamical) stability properties are generally not preserved for simple forward-Euler and sample-and-hold discretizations and control laws Stuart & Humphries (1998).
Furthermore, practical implementations of optimization algorithms in modern digital computers demand discrete-time.
Nonetheless, it has been extensively noted that a vast amount of general Lyapunov-based results appear to have a discrete-time equivalent.
In that sense, we aim here to start from the q -RGF, and q -SGF continuous ﬂows, characterized by their Lyapunov-based ﬁnite-time convergence, and seek discretization schemes, which allow us to ‘shadow’ the solutions of these ﬂows in discrete time, hoping to achieve an acceleration of the discrete methods inspired from the ﬁnite-time convergence characteristics of the continuous ﬂows.
O PTIMIZATION A LGORITHMS AS D ISCRETE -T IME S YSTEMS
Generalizing (1), (2), and (3), consider a continuous-time algorithm (dynamical system) modeled via an ordinary differential equation (ODE)
[Equation]
4) for t ≥ 0 , or, more generally, a differential inclusion
[Equation]
[Equation] for the q = ∞ case), such that x ( t ) → x (cid:63) as t → t (cid:63) .
In the case of the q -RGF (2) and q -SGF (3) for f gradient dominated of order p ∈ (1 , q ) , we have ﬁnite-time convergence, and thus t (cid:63) =
t (cid:63) ( x (0)) < ∞ .
Most of the popular numerical optimization schemes can be written in a state-space form ( i.e. , recursively), as
[Equation]
for k ∈ Z
+ (cid:44) { 0 , 1 , 2 , . . .
and a given X 0 ∈ R m (typically m ≥ n ), where F d : Z + × R m → R n and G : R m → R n .
Naturally, (6) can be seen as a discrete-time dynamical system constructed by discretizing (4) in time.
In particular, we have x k ≈ x ( t k ) , where { 0 = t 0 < t 1 < t 2 < . . .
} denotes a time partition and x ( · ) a solution to (4) or (5) as appropriate.
Therefore, we call X k and x k , respectively, the state and output at time step k .
Whenever F d ( k, X ) does not depend on k , we will drop k and thus write
F d ( X ) instead.
Whenever m = n , we will denote take G ( X ) (cid:44) X and replace X and X k by x and x k , respectively.
Example 1.
The standard gradient descent (GD) algorithm
[Equation]
with step size (learning rate) η > 0 can be readily written in the form (6) by taking m = n ,
F d ( x ) (cid:44) x − η ∇ f ( x ) , and G ( x ) (cid:44) x .
• If the step sizes are adaptive,
i
e. if we replace η by a sequence { η k } with η k > 0 , then we only need to replace F d ( k, x ) (cid:44) x − η k ∇ f ( x ) , provided that { η k } is not computed using feedback from { x k } ( e g. through a line search method).
• If we do wish to use feedback 1 (and no memory past the most recent output and step size), then we can set m = n + 1 , G ([ x ; η ]) (cid:44) x , and F d ([ x ; η ]) (cid:44) [ F (1)d ([ x ; η ]); F (2)d ([ x ; η ])] , where F (1)d ([ x ; η ]) (cid:44) x − η ∇ f ( x ) , and F (2)d is a user-deﬁned function that dictates the updates in the step size.
In particular, an open-loop (no feedback) adaptive step size { η k } may also be achieved under this scenario, provided that it is possible to write η k +1 =
F (2)d ( η k ) .
If this is not possible (and still open-loop step size), then we may take F (2)d ( k, X ) (cid:44) η k +1 , and of course add a k -argument in F d . • If we wish to use individual step sizes for each the n components of { x k } , then it sufﬁces to take η k as an n -dimensional vector (thus m = 2 n ), and make appropriate changes in F d and G .
In each of these cases, GD can be seen as a forward-Euler discretization of the GF (1), i.e. ,
[Equation]
with F GF = −∇ f and adaptive time step ∆ t k (cid:44) t k +1 − t k chosen as ∆ t k = η k .
Example 2.
The proximal point algorithm (PPA)
[Equation]
[Equation] with step size η k > 0 (open loop, for simplicity) can also be written in the form (6), by taking m = n ,
F d ( k, x ) (cid:44) arg min x (cid:48) ∈ R n { f ( x (cid:48) )
+ 12 η k (cid:107) x (cid:48)
− x (cid:107) 2 } , and G ( x ) (cid:44) x .
Naturally, we need to assume sufﬁcient regularity for F d ( k, x ) to exist and we must design a consistent way to choose F d ( k, x ) when multiple minimizers exist in the deﬁnition of F d ( k, x ) .
Alternatively, these conditions must be satisﬁed, at the very least, at every ( k, x ) ∈ { (0 , x 0 ) , (1 , x 1 ) , (2 , x 2 ) , . . .
} for a particular chosen initial x 0 ∈ R n .
Assuming sufﬁcient regularity, we have ∇ x { f ( x ) + 12 η k (cid:107) x − x k (cid:107) 2 }| x = x k +
= 0 , and thus
[Equation]
(10) with ∆ t k = η k , which is precisely the backward-Euler discretization of the GF (1).
Example 3.
The Nesterov accelerated gradient descent (N-AGD)
[Equation]
with step size η k > 0 and momentum coefﬁcient β k > 0 (both open loop, for simplicity), can be written in the form (6) by taking m = 2 n ,
[Equation]
and G ([ y ; x ]) (cid:44) x for y, x ∈ R n .
In other words, X k =
[ y k ; x k ] .
Traditionally, β k = k − 1 k − 2 , but clearly, if we set η k = η > 0 and β k = β ∈ (0 , 1) (in practice, η ≈ 0 and β ≈ 1 ), then we can drop k from F d ( k, [ y ; x ]) .
There exist a few approaches in the literature on the interpretation of N-AGD (11b) as the discretization of a second-order continuous-time dynamical system, namely via a vanishing step size argument Su et al.
(2014), or via symplectic Euler schemes of crafted Hamiltonian systems Muehlebach & Jordan (2019); Franc¸a et al. (2019b).
P ROPOSED A LGORITHMS VIA D ISCRETIZATION
In this section, we propose three classes of optimization algorithms via discretization of the q -RGF (2) and q -SGF (3).
But ﬁrst, we review the necessary conditions to ensure ﬁnite-time convergence of these ﬂows.
Given q ∈ (1 , ∞ ] , let F q − RGF ( x ) and
F q − SGF ( x ) be deﬁned, respectively, by the RHS of (2) and (3).
The hyperparameter c > 0 is not explicitly denoted in F q − RGF , F q − SGF .
Next, borrowing terminology from Wilson et al.
(2019), we say that f (assumed continuously differentiable) is µ - gradient dominated of order p ∈ (1 , ∞ ] (with µ > 0 ) near the local minimizer x (cid:63) if p −
[Equation]
(13) for every x ∈ R n near x = x (cid:63) , where f (cid:63) =
f ( x (cid:63) ) .
When µ > 0 is unknown or unimportant, but known to exist, we will omit it in the previous deﬁnition.
It can be proved that continuously differentiable strongly convex functions are gradient dominated of order p = 2 .
Furthermore, if f is gradient dominated (of any order) w.r.t.
x (cid:63) , then x (cid:63) is an isolated stationary point of f .
Remark 1.
For strongly convex functions, gradient dominance of order p = 2 can be established.
In fact, gradient dominance is usually deﬁned exclusively for order p = 2 , often referred to as the PolyakŁojasiewicz (PL) inequality, which was introduced by Polyak (1963) to relax the (strong) convexity assumption commonly used to show convergence of the GD algorithm (7).
The PL inequality can also be used to relax convexity assumptions of similar gradient and proximal-gradient methods Karimi et al. (2016); Attouch & Bolte (2009). Our adopted generalized notion of gradient dominance is strongly tied to the Łojasiewicz gradient inequality from real analytic geometry, established by Łojasiewicz (1963; 1965) 2 independently and simultaneously from Polyak (1963), and generalizing the PL inequality. More precisely, this inequality is typically written as
[Equation]
14) for every x ∈ R n in a small enough open neighborhood of the stationary point x = x (cid:63) , for some
C > 0 and θ ∈ (cid:0) 12 , 1 (cid:3) .
This inequality is guaranteed for analytic functions Łojasiewicz (1965).
More precisely, when x (cid:63) is a local minimizer of f , the aforementioned relationship is explicitly given by
[Equation]
Therefore, analytic functions are always gradient dominated.
However, while analytic functions are always smooth, smoothness is not required to attain gradient dominance.
We are now ready to state the ﬁnite-time convergence of the q -RGF (2) and q -SGF (3).
Theorem 1.
Romero & Benosman (2020) Suppose that f : R n → R is continuously differentiable and µ -gradient dominated of order p ∈ (1 , ∞ ) near a strict local minimizer x (cid:63) ∈ R n .
Let c > 0 and q ∈ ( p, ∞ ] .
Then, any maximal solution x ( · ) , in the sense of Filippov, to the q - RGF (2) or q - SGF (3) will converge in ﬁnite time to x (cid:63) , provided that (cid:107) x (0) − x (cid:63) (cid:107) > 0 is sufﬁciently small.
More precisely, lim t → t (cid:63) x ( t ) =
x (cid:63) , where the convergence time t (cid:63) < ∞ may depend on which ﬂow is used, but in both cases is upper bounded by
[Equation]
(16) where x 0 = x (0) , C = (cid:16) pp − 1 (cid:17)
[Equation]
µ [Equation] p , θ = p − 1 p , and θ (cid:48) =
q − 1 q .
In particular, given any compact and positively invariant subset S ⊂ D , both ﬂows converge in ﬁnite time with the aforementioned convergence time upper bound (which can be tightened by replacing D with S ) for any x 0 ∈ S .
Furthermore, if D = R n , then we have global ﬁnite-time convergence, i.e. ﬁnite-time convergence to any maximal solution (in the sense of Filippov) x ( · ) with arbitrary x 0 ∈ R n .
In essence, the analysis (introduced in Romero & Benosman (2020)) consists of leveraging the gradient dominance to show that the energy function E ( t ) (cid:44) f
( x ( t )) − f (cid:63) satisﬁes the Lyapunov-like differential inequality ˙ E ( t ) =
O ( E ( t ) α ) for some α < 1 .
The detailed proof is recalled in Appendix
C for completeness.
3. M AIN RESULT 1: D ISCRETIZATION A LGORITHMS AND THEIR C ONVERGENCE A NALYSIS 3.1. F ORWARD -E ULER D ISCRETIZATION
First, we propose a simple forward Euler discretization of the ﬁnite-time convergent ﬂows
[Equation]
(17) where F ∈ { F q − RGF , F q − SGF } .
We show later, in Theorem 2, that this simple method leads, for small enough η > 0 , to solutions that are (cid:15) -close to the ﬁnite-time ﬂows.
3.1. E XPLICIT R UNGE -K UTTA D ISCRETIZATION
We propose to use the following discretization
[Equation]
(18b) for η > 0 , K ∈ { 1 , 2 , 3 , . . . } , and F ∈ { F q − RGF , F q − SGF } .
This method is well-known to be numerically stable under the consistency condition (cid:80) i
= K i
=
α i = 1 , Stuart & Humphries (1996).
However, in our optimization framework, we want to be able to guarantee that the stable numerical solution of (18) remains close to the solution of the continuous ﬂows.
In other words, we seek arbitrarily small global error, also known as shadowing .
This will be discussed in Theorem 2.
N ESTEROV - LIKE D ISCRETIZATION
First, we rewrite Nesterov’s accelerated GD as
[Equation]
where y k now serves as a momentum term.
We argue that Nesterov’s acceleration can be interpreted as actually applying the discretization given by (19) to the GF (1), i.e., by seeing the term − η ∇ f ( x k + βy k ) as a mapping applied to the GF ﬂow (1) at x k + βy k , as − ηF GF ( x k + βy k ) .
Therefore, given any optimization ﬂow represented by the continuous-time system ˙ x = F ( x ) , locally convergent to a local minimizer x (cid:63) ∈ R n of a cost function f : R n → R , we can replicate Nesterov’s acceleration of (1).
More precisely, we obtain the algorithm
[Equation]
Based on this idea, we propose two ‘Nesterov-like’ discrete optimization algorithms.
The ﬁrst one based on the q -RGF continuous ﬂow, is deﬁned as:
[Equation]
The second algorithm is based on the q -SGF continuous ﬂow, and is given by:
[Equation]
3.1. C ONVERGENCE ANALYSIS
We present here some convergence results of the three proposed discretizations.
The analysis summarized in Theorem 2 is based on tools from hybrid control theory, and is detailed in Appendix
D 3 .
Theorem 2.
Suppose that f : R n → R is continuously differentiable, locally L f -Lipschitz, and µgradient dominated of order p ∈ (2 , ∞ ) in a compact neighborhood S of a strict local minimizer x (cid:63) ∈ R n .
Let c > 0 and q ∈ ( p, ∞ ] .
Then, for a given initial condition x 0 ∈ S any maximal solution x ( t ) , x (0) = x 0 , (in the sense of Filippov) to the q - RGF given by (2) or the q - SGF ﬂow (3) , there exists an arbitrarily small (cid:15) > [-
-]0 such that the solution x k of any of the discrete algorithms (17), (18), (21), or (22), with sufﬁciently small η > 0 , are (cid:15) -close to x ( t ) , i.e., (cid:107) x k − x (cid:107) ≤ (cid:15) , and s.t.
the following convergence bound holds
[Equation]
L
[Equation]
k
[Equation]
k [Equation]
[Equation]
where α = θθ (cid:48) , θ
= p − 1 p , θ (cid:48)
=
q − 1 q , ˜ c
= c (cid:18)(cid:16) pp − 1 (cid:17)
[Equation]
, and
k
[Equation]
Theorem 2 thus shows that that (cid:15) -convergence of x k → x (cid:63) can be achieved in a ﬁnite number of steps upper bounded by k (cid:63)
=
( f ( x 0 ) − f ( x (cid:63) ))
(1 − α )
˜ c (1 − α ) η . This is a preliminary convergence result, which is meant to show the existence of discrete solutions obtained via the proposed discretization algorithms, which are (cid:15) -close to the continuous solutions of the ﬁnite-time ﬂows.
We also underline here that after x k reaches an (cid:15) -neighborhood of x (cid:63) , then x k [-+
-]+1 ≈ x k , ∀ k > k
(cid:63) , since x (cid:63) is an equilibrium point of the continuous ﬂows; see Deﬁnition 2 in Appendix B.
M AIN RESULT 2: N UMERICAL E XPERIMENTS 4. N UMERICAL TESTS ON AN ACADEMIC EXAMPLE
Let us show ﬁrst on a simple numerical example that the acceleration in convergence, proven in continuous time for certain range of the hyperparmeters, can translate to some convergence acceleration in discrete time, as shown in Theorem 2.
We consider the Rosenbrock function f : R 2 → R , given by f ( x 1 , x 2 ) =
( a − x 1 ) 2 + b ( x 2 − x 21 ) 2 , with parameters a, b ∈ R .
This function admits exactly one stationary point ( x (cid:63) 1 , x (cid:63) 2 ) =
( a, a 2 ) for b ≥ 0 , and is locally strongly convex, hence locally satisﬁes gradient dominance of order p = 2 , which allows us to select q > 2 in q -RGF
and q -SGF to achieve ﬁnite-time convergence in continuous-time.
We report in Figure 1 the mean performance of all three discretizations for q -RGF and q -SGF 4 with ﬁxed step size 5 , for several values of q , for 10 random initial conditions in [0 , 2] .
We observe for all three discretizations that, as expected from the continuous ﬂow analysis, for q close to 2 , q -RGF behaves similar to GD in terms of convergence rate, whereas for q > 2 the ﬁnite-time convergence in continuous time seems to translate to some acceleration in this simple discretization method.
Similarly for q -SGF, q closer totranslates to less accelerated algorithms, with a behavior similar to GD, whereas larger q values lead to accelerated convergence.
[Figure]
We report here the results of our experiments using deep neural network (DNN) training on three well-known datasets, namely, CIFAR10, MNIST, and SVHN.
We report results on CIFAR10, and
SVHN in the sequel, while results on MNIST can be found in Appendix E. Note that, we use Pytorch platform to conduct all the tests reported in this paper, and do not use GPUs.
We underline here that the DNNs are non-convex globally, however, one could assume at least local convexity, hence local gradient dominance of order p = 2 , thus, we will select q > 2 in our experiments (see (Remark 2,
Appendix E) for more explanations on the choice of q ).
4.2. E XPERIMENT ON CIFAR
In this experiment, we use the proposed algorithms to train a VGG16 CNN model with cross entropy loss, e.g., Simonyan & Zisserman (2015) on the CIFAR10 dataset.
We divided the dataset into a training set spread in 50 batches of 1000 images each, and a test set of 10 batches with 1000 images each.
We ran 20 epochs of training over all the training batches.
Since Nesterov accelerated GD is one of the most efﬁcient methods is DNN applications, to conduct fair comparisons we implemented our Nesterov-like discretization of q -RGF ( c = 1 , q = 3 , η = 0 .
04 , µ = 0 . 9 ), and the Nesterovlike discretization of q -SGF ( c = 10 − 3 , q = 3 , η = 0 .
04 , µ = 0 . 9 ).
We compare against the mainstream algorithms 6 , such as, Nesterov’s accelerated gradient descent (GD), Adam, Adaptive gradient (AdaGrad), per-dimension learning rate method for gradient descent (AdaDelta), and Root
Mean Square Propagation (RMSprop) 7 .
Note that, all algorithms have been implemented in their stochastic version 8 , i.e., using mini-batches implementation, with constant step size.
For instance, in Figures 2, 3 9 , we see the training loss for the different optimization algorithms.
We notice that the proposed Algorithms, RGF, and SGF, quickly separate from the GD, and RMS algorithms in terms of convergence speed, but also ends up with an overall better performance on the test set 84% vs. 83% for GD, and RMS.
We also note that other algorithms, such as, AdaGrad and AdaDelta behave similarly to RGF in terms of convergence speed, but lack behind in terms of ﬁnal performance 75% and 68% , respectively.
Finally, in Figure 3, we notice that Adam is slower in terms of computation time w.r.t SGF, and RGF, with an average lag of 8 min , and 80 min , respectively.
However, to be fair one, has to underline that Adam is an adaptive method based on the extra computations and memory overhead of lower order moments estimate and bias correction, Kingma & Ba (2015).
Furthermore, to better compare the performance of these algorithms, we report in Figure 2 the loss on the test dataset over the learning iterations.
We conﬁrm that RGF and SGF performs better than GD, RMSprop, and
Adam, while avoiding the overﬁtting observed with AdaGrad and AdaDelta.
4.2. E XPERIMENTS ON SVHN D ATASET
We test the proposed algorithms to train the same VGG16 CNN model with cross entropy loss on the SVHN dataset.
We divided the dataset into a training set of 70 batches with 1000 images each, and a test set of 10 of 1000 images each, and ran 20 epochs of training over all the training batches.
We tested the Nesterov-like discretization of q -RGF ( c = 1 , q = 3 , η = 0 .
04 , µ = 0 . 09 ), and the
Nesterov-like discretization of q -SGF ( c = 10 − 3 , q = 11 , η = 0 .
04 , µ = 0 . 09 ) against Nesterov’s accelerated gradient descent (GD), and Adam 10 .
Note from Figures 4, 5 it is clear that RGF and SGF give a good performance in terms of convergence speed, and ﬁnal test performance 93% .
We can also observe in Figure 5 that SGF, RGF are faster than GD, and all three methods are faster (in average 41 min for GD, 58 min for SGF, 75 min for RGF) than Adam as expected since it is an adaptive scheme with more computation steps (see our discussion of Adam in Section 4.2.1).
More numerical results on MNIST, and on SVHN using Euler and Runge-Kutta descretization, showing similar qualitative results, can be found in Appendix E
C ONCLUSION
We studied connections between optimization algorithms and continuous-time representations (dynamical systems) via discretization.
We then reviewed two families of non-Lipschitz or discontinuous ﬁrst-order optimization ﬂows for continuous-time optimization, namely the q -RGF and q -SGF, whose distinguishing characteristic is their ﬁnite-time convergence.
We then proposed three discretization methods for these ﬂows, namely a forward-Euler discretization, followed by an explicit Runge-Kutta discretization, and ﬁnally a novel Nesterov-like discretization.
Based on tools from hybrid systems control theory, we proved a convergence bound for these algorithms.
Finally, we conducted numerical experiments on known deep neural net benchmarks, which showed that the proposed discrete algorithms can outperform some state of the art algorithms, when tested on large DNN models.
A D ISCONTINUOUS S YSTEMS AND D IFFERENTIAL I NCLUSIONS
11 Recall that for an initial value problem (IVP)
[Equation]
with F : R n → R n , the typical way to check for existence of solutions is by establishing continuity of F .
Likewise, to establish uniqueness of the solution, we typically seek Lipschitz continuity.
When
F is discontinuous, we may understand (24a) as the Filippov differential inclusion
[Equation]
where K [ F ] : R n ⇒ R n denotes the Filippov set-valued map given by
[Equation]
where µ denotes the usual Lebesgue measure and co the convex closure, i.e. closure of the convex hull co .
For more details, see Paden & Sastry (1987).
We can generalize (25) to the differential inclusion Bacciotti & Ceragioli (1999)
[Equation]
where F : R n ⇒ R n is some set-valued map.
Deﬁnition 1 (Carath´eodory/Filippov solutions) .
We say that x : [0 , τ ) → R n with 0 < τ ≤ ∞ is a Carath´eodory solution to (27) if x ( · ) is absolutely continuous and (27) is satisﬁed a.e. in every compact subset of [0 , τ ) .
Furthermore, we say that x ( · ) is a maximal Carath´eodory solution if no other Carath ´ eodory solution x (cid:48) ( · ) exists with x = x (cid:48)
| [0 ,τ ) .
If F = K [ F ] , then Carath ´ eodory solutions are referred to as Filippov solutions .
For a comprehensive overview of discontinuous systems, including sufﬁcient conditions for existence (Proposition 3) and uniqueness (Propositions 4 and 5) of Filippov solutions, see the work of Cort ´ es (2008).
In particular, it can be established that Filippov solutions to (24) exist, provided that the following assumption (Assumption 1) holds.
Assumption 1 (Existence of Filippov solutions) .
F : R n → R n is deﬁned almost everywhere (a.e.) and is Lebesgue-measurable in a non-empty open neighborhood U ⊂ R n of x 0 ∈ R n .
Further, F is locally essentially bounded in U , i.e. , for every point x ∈ U , F is bounded a.e.
in some bounded neighborhood of x .
More generally, Carath ´ eodory solutions to (27) exist (now with arbitrary x 0 ∈ R n ), provided that the following assumption (Assumption 2) holds.
Assumption 2 (Existence of Carath ´ eodory solutions) .
F : R n ⇒ R n has nonempty, compact, and convex values, and is upper semi-continuous .
Filippov & Arscott (1988) proved that, for the Filippov set-valued map F = K [ F ] , Assumptionsand 2 are equivalent (with arbitrary x 0 ∈ R n in Assumption 1).
Uniqueness of the solution requires further assumptions.
Nevertheless, we can characterize the
Filippov set-valued map in a similar manner to Clarke’s generalized gradient, as seen in the following proposition.
Proposition 1 (Theorem 1 of Paden & Sastry (1987)) .
Under Assumption 1, we have
[Equation]
for some (Lebesgue) zero-measure set N F ⊂ R n and any other zero-measure set S ⊂ R n .
In particular, if F is continuous at a ﬁxed x , then K [ F ]( x )
= { F ( x ) } .
For instance, for the GF (1), we have K [ −∇ f ]( x ) =
{−∇ f ( x ) } for every x ∈ R n , provided that f is continuously differentiable.
Furthermore, if f is only locally Lipschitz continuous and regular (see
Deﬁnition 3 of Appendix B), then K [ −∇ f ]( x ) = − ∂f ( x ) , where
[Equation]
denotes Clarke’s generalized gradient Clarke (1981) of f , with N f denoting the zero-measure set over which f is not differentiable (Rademacher’s theorem).
It can be established that ∂f coincides with the subgradient of f , provided that f is convex.
Therefore, the GF (1) interpreted as Filippov differential inclusion may also be seen as a continuous-time variant of subgradient descent methods.
B F INITE -T IME S TABILITY OF D IFFERENTIAL I NCLUSIONS
We are now ready to focus on extending some notions from traditional Lipschitz continuous systems to differential inclusions.
Deﬁnition 2.
We say that x (cid:63) ∈ R n is an equilibrium of (27) if x ( t ) ≡ x (cid:63) on some small enough non-degenerate interval is a Carath ´ eodory solution to (27).
In other words, if and only if 0 ∈ F ( x (cid:63) ) .
We say that (27) is (Lyapunov) stable at x (cid:63) ∈ R n if, for every ε > 0 , there exists some δ > 0 such that, for every maximal Carath ´ eodory solution x ( · ) of (27), we have (cid:107) x 0 − x (cid:63) (cid:107) <
δ = ⇒ (cid:107) x
( t ) − x (cid:63) (cid:107) <
ε for every t ≥ 0 in the interval where x ( · ) is deﬁned.
Note that, under Assumption 2, if (27) is stable at x (cid:63) , then x (cid:63) is an equilibrium of (27) Bacciotti & Ceragioli (1999).
Furthermore, we say that (27) is (locally and strongly) asymptotically stable at x (cid:63) ∈ R n if is stable at x (cid:63) and there exists some δ > 0 such that, for every maximal Carath´eodory solution x : [0 , τ ) → R n of (27), if (cid:107) x 0 − x (cid:63) (cid:107) <
δ then x ( t ) → x (cid:63) as t → τ .
Finally, (27) is (locally and strongly) ﬁnite-time stable at x (cid:63) if it is asymptotically stable and there exists some δ > 0 and T :
B δ ( x (cid:63) ) → [0 , ∞ ) such that, for every maximal Carath ´ eodory solution x ( · ) of (27) with x 0 ∈ B δ ( x (cid:63) ) , we have lim t → T ( x 0 ) x ( t ) =
x (cid:63) .
We will now construct a Lyapunov-based criterion adapted from the literature of ﬁnite-time stability of Lipschitz continuous systems.
Lemma 1.
Let E ( · ) be an absolutely continuous function satisfying the differential inequality
[Equation]
a.e. in t ≥ 0 , with c, E (0) > 0 and α < 1 .
Then, there exists some t (cid:63) > 0 such that E ( t ) > 0 for t ∈ [0 , t (cid:63) ) and E ( t (cid:63) ) = 0 .
Furthermore, t (cid:63) > 0 can be bounded by
[Equation]
(31) with this bound tight whenever (30) holds with equality.
In that case, but now with α ≥ 1 , then
E ( t ) > 0 for every t ≥ 0 , with lim t →∞ E ( t ) = 0 .
This will be represented by t (cid:63) = ∞ , with
E ( ∞ ) (cid:44) lim
t →∞ E ( t ) .
Proof.
Suppose that E ( t ) > 0 for every t ∈ [0 , T ] with T > 0 .
Let t (cid:63) be the supremum of all such
T ’s, thus satisfying E ( t ) > 0 for every t ∈ [0 , t (cid:63) ) .
We will now investigate E ( t (cid:63) ) .
First, by continuity of E , it follows that E ( t (cid:63) ) ≥ 0 .
Now, by rewriting
[Equation] [Equation]
a.e. in t ∈ [0 , t (cid:63) ) , we can thus integrate to obtain
[Equation]
33) everywhere in t ∈ [0 , t (cid:63) ) , which in turn leads to
[Equation]
34) and
[Equation]
where the last inequality follows from E ( t ) > 0 for every t ∈ [0 , t (cid:63) ) .
Taking the supremum in (35) then leads to the upper bound (31).
Finally, we conclude that E ( t (cid:63) ) = 0 , since E ( t (cid:63) ) > 0 is impossible given that it would mean, due to continuity of E , that there exists some T > t (cid:63) such that
E ( t ) > 0 for every t ∈ [0 , T ] , thus contradicting the construction of t (cid:63) .
Finally, notice that if E is such that (30) holds with equality, then (34) and the ﬁrst inequality in (35) hold with equality as well.
The tightness of the bound (31) thus follows immediately.
Furthermore, notice that if α ≥ 1 , and E is a tight solution to the differential inequality (30), i.e. E ( t ) =
[ E (0) 1 − α − c (1 − α ) t ] 1 / (1 − α ) , then clearly E ( t ) > 0 for every t ≥ 0 and E ( t ) → 0 as t → ∞ .
(cid:4)
Cort´es & Bullo (2005) proposed (Proposition 2.8) a Lyapunov-based criterion to establish ﬁnitetime stability of discontinuous systems, which fundamentally coincides with our Lemma 1 for the particular choice of exponent α = 0 .
Their proposition was, however, directly based on Theoremof Paden & Sastry (1987).
Later, Cort´es (2006) proposed a second-order Lyapunov criterion, which, on the other hand, fundamentally translates to E ( t ) (cid:44) V ( x ( t )) being strongly convex.
Finally, Hui et al.
(2009) generalized Proposition 2.8 of Cort´es & Bullo (2005) in their Corollary 3.1, to establish semistability.
Indeed, these two results coincide for isolated equilibria.
We now present a novel result that generalizes the aforementioned ﬁrst-order Lyapunov-based results, by exploiting our Lemma 1.
More precisely, given a Laypunov candidate function V ( · ) , the objective is to set E ( t ) (cid:44) V ( x ( t )) , and we aim to check that the conditions of Lemma 1 hold.
To do this, and assuming V to be locally Lipschitz continuous, we ﬁrst borrow and adapt from Bacciotti & Ceragioli (1999) the deﬁnition of set-valued time derivative of V : D → R w.r.t.
the differential inclusion (27), given by
[Equation]
for each x ∈ D .
Notice that, under Assumption 2 for Filippov differential inclusions F = K [ F ] , the set-valued time derivative of V thus coincides with with the set-valued Lie derivative L F V ( · ) .
Indeed, more generally ˙ V could be seen as a set-valued Lie derivative L F V w.r.t.
the set-valued map
F .
Deﬁnition 3.
V ( · ) is said to be regular if every directional derivative, given by
[Equation]
exists and is equal to
[Equation]
known as Clarke’s upper generalized derivative Clarke (1981).
In practice, regularity is a fairly mild and easy to guarantee condition.
For instance, it would sufﬁce that V is convex or continuously differentiable to ensure that it is Lipschitz and regular.
Assumption 3.
V : D → R is locally Lipscthiz continuous and regular, with D ⊆ R n open.
Under Assumption 3, Clarke’s generalized gradient
[Equation]
is non-empty for every x ∈ D , and is also given by
[Equation]
where N V denotes the set of points in D ⊆ R n where V is not differentiable (Rademacher’s theorem) Clarke (1981).
Through the following lemma (Lemma 2), we can formally establish the correspondence between the set-valued time-derivative of V and the derivative of the energy function E ( t ) (cid:44) V ( x ( t )) associated with an arbitrary Carath ´ eodory solution x ( · ) to the differential inclusion (27).
Lemma 2 (Lemma 1 of Bacciotti & Ceragioli (1999)) .
Under Assumption 3, given any Carath´eodory solution x : [0 , τ ) → R n to (27) , then E ( t ) (cid:44) V ( x ( t )) is absolutely continuous and ˙ E ( t ) =
dd t V ( x ( t )) ∈ ˙
V ( x ( t ))
a.e. in t ∈ [0 , τ ) .
We are now ready to state and prove our Lyapunov-based sufﬁcient condition for ﬁnite-time stability of differential inclusions.
Theorem 3.
Suppose that Assumptions 2 and 3 hold for some set-valued map F : R n ⇒ R n and some function V : D → R , where D ⊆ R n is an open and positively invariant neighborhood of a point x (cid:63) ∈ R n .
Suppose that V is positive deﬁnite w.r.t.
x (cid:63) and that there exist constants c > 0 and α <
such that
[Equation]
a.e. in x ∈ D .
Then, (27) is ﬁnite-time stable at x (cid:63) , with settling time upper bounded by
[Equation]
where x (0) = x 0 .
In particular, any Carath´eodory solution x ( · ) with x (0) =
x 0 ∈ D will converge in ﬁnite time to x (cid:63) under the upper bound (42) .
Furthermore, if D = R n , then (27) is globally ﬁnite-time stable.
Finally, if ˙ V ( x ) is a singleton a.e. in x ∈ D and (41) holds with equality, then the bound (42) is tight.
Proof.
Note that, by Proposition 1 of Bacciotti & Ceragioli (1999), we know that (27) is Lyapunov stable at x (cid:63) .
All that remains to be shown is local convergence towards x (cid:63) (which must be an equilibrium) in ﬁnite time.
Indeed, given any maximal solution x : [0 , t (cid:63) )
→ R n to (27) with x (0) =
x 0 (cid:54) =
x (cid:63) , we know by Lemma 2, that E ( t )
=
V ( x ( t )) is absolutely continuous with ˙ E ( t ) ∈ ˙
V ( x ( t ))
a.e. in t ∈ [0 , t (cid:63) ] .
Therefore, we have
[Equation]
a.e. in t ∈ [0 , t (cid:63) ] .
Since E (0) =
V ( x 0 ) > 0 , given that x 0 (cid:54) =
x (cid:63) , the result then follows by invoking
Lemma 1 and noting that E
[-that E ( t (cid:63) ) = 
 ⇐⇒ V ( t (cid:63) , x ( t (cid:63) )) = 
-][Equation]
t
[Equation]
Finite-time stability still follows without Assumption 2, provided that x (cid:63) is an equilibrium of (27).
In practical terms, this means that trajectories starting arbitrarily close to x (cid:63) may not actually exist, but nevertheless there exists a neighborhood D of x (cid:63) over which, any trajectory x ( · ) that indeed exists and starts at x (0) = x 0 ∈ D must converge in ﬁnite time to x (cid:63) , with settling time upper bounded by
T ( x 0 ) (the bound still tight in the case that (41) holds with equality).
C P ROOF OF T HEOREM
Let us focus on the q -RGF (2) (the case of q -SGF (3) follows exactly the same steps) with the candidate Lyapunov function V (cid:44)
f − f (cid:63) .
Clearly, V is Lipschitz continuous and regular (given that it is continuously differentiable).
Furthermore, V is positive deﬁnite w.r.t.
x (cid:63) .
Notice that, due to the dominated gradient assumption, x (cid:63) must be an isolated stationary point of f .
To see this, notice that, if x (cid:63) were not an isolated stationary point, then there would have to exist some ˜ x (cid:63) sufﬁciently near x (cid:63) such that ˜ x (cid:63) is both a stationary point of f , and satisﬁes f (˜ x (cid:63) ) >
f (cid:63) , since x (cid:63) is a strict local minimizer of f .
But then, we would have
[Equation]
[Equation]
and subsequently 0 > 0 , which is absurd.
Therefore, F ( x ) (cid:44) − c ∇ f ( x ) /
(cid:107)∇ f ( x ) (cid:107) q − 2 q − 1 is continuous for every x ∈ D \ { x (cid:63) } , for some small enough open neighborhood D of x (cid:63) .
Let us assume that D is positively invariant w.r.t.
(2), which can be achieved, for instance, by replacing D with its intersection with some small enough strict sublevel set of f .
Notice that (cid:107) F ( x ) (cid:107) =
c (cid:107)∇ f ( x ) (cid:107) 1 q − 1 with q ∈ ( p, ∞ ] ⊂ (1 , ∞ ] , i.e. , 1 q − 1 ∈ [0 , ∞ ) .
If q = ∞ , which results in the normalized gradient ﬂow ˙ x = − ∇ f ( x ) (cid:107)∇ f ( x ) (cid:107) proposed by Cort ´ es (2006), then (cid:107) F ( x ) (cid:107) =
c > 0 .
We can thus show that F ( x ) is discontinuous at x = 0 for q = ∞ .
On the other hand, if q ∈ ( p, ∞ ) ⊂ (1 , ∞ ) , then we have (cid:107) F ( x ) (cid:107) → 0 as x → x (cid:63) , and thus F ( x ) is continuous (but not Lipschitz) at x = x (cid:63) .
Regardless, we may freely focus exclusively on D \ { x (cid:63) } since { x (cid:63) } is obviously a zero-measure set.
Let F (cid:44) K
[ F ] .
We thus have, for each x ∈ D \ { x (cid:63) } ,
[Equation]
Since θθ (cid:48) < 1 , given that s > 1 (cid:55)→ s − 1 s is strictly increasing, then the conditions of Theorem 3 are satisﬁed.
In particular, we have ﬁnite-time stability at x (cid:63) with a settling time t (cid:63) upper bounded by t (cid:63) ≤ ( f ( x 0 ) − f (cid:63) ) 1 −
[Equation]
for each x 0 ∈ D , which completes the proof.
D P ROOF OF T HEOREM
To prove Theorem 2, we borrow some tools and results from hybrid control systems theory.
Hybrid control systems are characterized by continuous ﬂows with discrete jumps between the continuous ﬂows.
They are often modeled by differential inclusions added to discrete mappings to model the jumps between the differential inclusions.
We see the case of the optimization ﬂows proposed here as a simple case of a hybrid systems with one differential inclusion, with a possible jump or discontinuity at the optimum.
Based on this, we will use the tools and results of Sanfelice & Teel (2010), which study how a certain class of hybrid systems behave after discretization with a certain class of discretization algorithms.
In other words, Sanfelice & Teel (2010) quantiﬁes, under some conditions, how close are the solutions of the discretized hybrid dynamical system to the solutions of the original hybrid system.
In this section we will denote the differential inclusion of the continuous optimization ﬂow by
F : R n ⇒ R n , and its discretization in time by F d : R n ⇒ R n .
We ﬁrst recall a deﬁnition, which we will adapt from the general case of jumps between multiple differential inclusions (Deﬁnition 3.2,
Sanfelice & Teel (2010)) to our case of one differential inclusion or ﬂow.
Deﬁnition 4.
( ( T, (cid:15) ) -closeness).
Given T > 0 , (cid:15) > 0 , η > 0 , two solutions x t : [0 , T → R n , and x k : { 0 , 1 ,
→ R n are ( T, (cid:15) ) -close
if:
(a) for all t ≤ T there exists k ∈ { 1 , 2 , ... } such that | t − kη | < (cid:15) , and (cid:107) x t ( t ) − x k ( k ) (cid:107) <
(cid:15) , (b) for all k ∈ { 1 , 2 , ... } there exists t ≤ T such that | t − kη | < (cid:15) , and (cid:107) x t ( t ) − x k ( k ) (cid:107) < (cid:15) .
Next, we will recall Theorem 5.2 in Sanfelice & Teel (2010), while adapting it to our special case of a hybrid system with one differential inclusion 12 .
Theorem 4.
(Closeness of continuous and discrete solutions on compact domains) Consider the differential inclusion
[Equation]
for a given set-valued mapping F : R m ⇒ R m assumed to be outer semicontinuous, locally bounded, nonempty, and with convex values for every x ∈ C , for some closed set C ⊆ R m .
Consider the discrete-time system represented by the ﬂow F d : R n ⇒ R n , such that, for each compact set K ⊂ R n , there exists ρ ∈ K ∞ , and η (cid:63) > 0 such that for each x ∈ K and each η ∈ (0 , η (cid:63) ] ,
[Equation]
Then, for every compact set K ⊂ R n , every (cid:15) > 0 , and every time horizon T ∈ R ≥ 0 there exists η (cid:63) > 0 such that: for any η ∈ (0 , η (cid:63) ] and any discrete solution x k with x k (0) ∈ K + δ B , δ > 0 , there exists a continuous solution x t with x t (0) ∈
K such that x k and x t are ( T, (cid:15) ) -close.
To prove Theorem 2 we will use the results of Theorem 4, where we will have to check that condition (48) is satisﬁed for the three proposed discretizations.
We are now ready to prove Theorem 2.
First, note that outer semicontinuity follows from the upper semicontinuity and the closedness of the Filippov differential inclusion map.
Furthermore, local boundedness follows from continuity everywhere outside stationary points, which are isolated.
Now, let us examine their discretization by the three proposed algorithms:
F ORWARD -E ULER D ISCRETIZATION
The mapping F d in this case is a singleton, given by
[Equation]
[Equation]
where η > 0 , which clearly satisﬁes condition (48).
R UNGE -K UTTA D ISCRETIZATION
Once again, the mapping F d is singleton, this time given by
[Equation]
[Equation]
(50b) where η, α 1 , . .
. , α K , β 1 , . . .
, β K − 1 > 0 are such that (cid:80)
Ki
=1 α i = 1 .
By equation (50b) one can establish a function ρ ∈ K ∞ such that for each x k ∈ K ⊂ R n , for each η > 0 , y ik ∈ x k + ηρ ( η ) B .
Next, by equation (50a) together with the condition (cid:80) i
= K i
=
α
i
= 1 , one can write that for any x k ∈ K and η > 0 , F d ( x k ) ⊂ x k + η con F ( y ik ) ⊂ x k + η con F
( x k + ηρ ( η ) B )
+ ηρ ( η ) B .
N ESTEROV - LIKE D ISCRETIZATION
In this case the discrete-time ﬂow F d is deﬁned as
[Equation]
51b)
In this case, to take into account the integral effect of the Nesterov-like discretization, let us extend the continuous-time ﬂow as
[Equation]
we then compare the solution of the extended continuous-time system (52) with the extended discretetime system [Equation] η
[Equation]
which could be rearranged as
[Equation]
[Equation]
where M = (cid:20)µ 0 − 1 + µ (cid:21) , and ˜ F t (˜ x k ) = (cid:20) F (˜ x k )
F (˜ x k ) (cid:21) , which shows that for any ˜ x k ∈ K ⊂ R 2 n , and η > 0 , we have (using similar recursive reasoning as above) that F d (˜ x k ) ⊂
˜ x k + η con ˜ F t (˜ x k + ηρ ( η ) B ) + ηρ ( η ) B .
Then, using Theorem 4 we conclude about the ( T, (cid:15) )
-closeness between the continuous-time solutions of the ﬂows F : q -RGF (2), q -SGF (3), and the discrete-time solutions of their respective discretization by any of the three discretization methods.
Furthermore, for the
Nesterov-like discretization
, we can also conclude about the ( T, (cid:15) ) − closeness of the integral of the continuous-time solutions ˜ x t (2) and its discretization ˜ x k (2) .
Finally, using the Lyapunov function V = f − f (cid:63) as deﬁned in the proof of Theorem 1, together with inequalities (45g), (34), and a local Lipschitz bound on f , one can derive the convergence bound given by (23), as follows:
[Equation]
) (cid:107) − (cid:107) ( f ( x t ) − f ( x (cid:63) ))
(cid:107) ≤ (cid:107) f ( x k ) − f ( x t ) (cid:107) ≤ ˜ (cid:15), (cid:107) f ( x k ) − f ( x (cid:63) ) (cid:107) ≤ ˜ (cid:15) + (cid:107) f ( x t ) − f ( x (cid:63) )
(cid:107) , (cid:107) f
( x k ) − f ( x (cid:63) ) (cid:107) ≤ ˜ (cid:15) + [( f ( x 0 ) − f ( x (cid:63) ))
(1 − α ) − ˜ c (1 − α ) ηk ] 1 / (1 − α ) , for k ≤ k (cid:63) , where α = θθ (cid:48) , θ = p − 1 p , θ (cid:48) =
q − 1 q , ˜ c
= c (cid:18)(cid:16) pp − 1 (cid:17) p − 1 p µ 1 p (cid:19) 1 θ (cid:48) , k (cid:63) =
( f ( x 0 ) [Equation] f ( x (cid:63) )) (
[Equation]
[Equation] α ) η .
E A DDITIONAL DETAILS AND NUMERICAL RESULTS
In this section, we will expand upon the numerical results experiments discussed in the paper.
In particular, we report more details on the hyper-parameters values used 13 for the numerical tests, and report some results for the MNIST experiments.
E. H YPER PARAMETERS VALUES USED IN THE TESTS OF S ECTION 4.
• GD ﬁxed step size: η = 10 −• RGF Euler disc.
w/ﬁxed step size: q = 2 . 2 , η = 10 −• RGF Euler disc. w/ﬁxed step size: q = 3 , η = 10 −• RGF Euler disc.
w/ﬁxed step size: q = 6 , η = 10 −• RGF Euler disc.
w/ﬁxed step size: q = 10 , η = 10 −• GD Nesterov acceleration ﬁxed step size: η = 10 − 4 ; µ = 0 .•
SGF Nesterov-like disc.
w/ﬁxed step size: q = 2 . 2 , η = 10 − 4 , µ = 0 .•
SGF Nesterov-like disc.
w/ﬁxed step size: q = 3 , η = 10 − 3 , µ = 0 .•
SGF Nesterov-like disc.
w/ﬁxed step size: q = 6 , η = 10 − 3 , µ = 0 .•
SGF Nesterov-like disc.
w/ﬁxed step size: q = 10 , η = 10 − 2 , µ = 0 .•
RGF Runge Kutta disc.
w/ﬁxed step size: q = 2 . 2 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .•
RGF Runge Kutta disc.
w/ﬁxed step size: q = 3 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .•
RGF Runge Kutta disc.
w/ﬁxed step size: q = 6 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .•
RGF Runge Kutta disc.
w/ﬁxed step size: q = 10 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .
E. H YPER PARAMETERS VALUES USED IN THE TESTS OF S ECTION 4.
Note that the description of the coefﬁcients for each of the prior art methods can be found in: https://pytorch.org/docs/stable/optim.html.
• GD: η = 4 .
10 − 2 , µ = 0 .
9 , Nesterov=
True • RGF: η = 4 .
10 − 2 , µ = 0 .•
SGF: η = 4 .
10 − 3 , µ = 0 .•
ADAM:
η = 8 .
10 − 4 (remaining coefﬁcients=nominal values) • RMS:
η = 10 − 3 (remaining coefﬁcients=nominal values) • ADAGRAD: η = 10 − 3 (remaining coefﬁcients=nominal values) • ADADELTA:
η = 4 .
10 − 2 , ρ = 0 .
9 , (cid:15) =
10 − 6 , weight decay =
E. M ORE TESTS ON THE R OSENBROCK FUNCTION
Due to space limitations, we have decided to report in the main paper only one test for q -RGF with
Euler discretization, one test for q -SGF with Nesterov-like discretization, and one test for q -RGF with Runge Kutta discretization.
For the sake of completeness we report here the remaining tests for each algorithm.
One can observe similar qualitative behavior in Figure 6 as the one noticed in the results of Section 4.5.
The step-size and other hyper-parameters for each test are given below:
• GD ﬁxed step size: η = 10 −• SGF Euler disc.
w/ﬁxed step size: q = 2 . 1 , η = 10 −• SGF Euler disc. w/ﬁxed step size: q = 2 . 5 , η = 10 −• SGF Euler disc.
w/ﬁxed step size: q = 2 . 8 , η = 10 −• SGF Euler disc.
w/ﬁxed step size: q = 100 , η = 10 −• GD Nesterov acceleration ﬁxed step size: η = 10 − 4 ; µ = 0 .•
RGF Nesterov-like disc.
w/ﬁxed step size: q = 2 . 2 , η = 10 − 4 , µ = 0 .•
RGF Nesterov-like disc.
w/ﬁxed step size: q = 3 , η = 10 − 3 , µ = 0 .•
RGF Nesterov-like disc.
w/ﬁxed step size: q = 6 , η = 10 − 3 , µ = 0 .•
RGF Nesterov-like disc.
w/ﬁxed step size: q = 10 , η = 10 − 3 , µ = 0 .•
SGF Runge Kutta disc.
w/ﬁxed step size: q = 2 . 2 , K = 2 , η = 10 − 3 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .•
SGF Runge Kutta disc.
w/ﬁxed step size: q = 3 , K = 2 , η = 10 − 2 , β 1 = 0 .
9 , α 1 =
α 2 = 0 .•
SGF Runge Kutta disc.
w/ﬁxed step size: q = 6 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .•
SGF Runge Kutta disc.
w/ﬁxed step size: q = 10 , K = 2 , η = 10 − 2 , β 1 = 0 .
09 , α 1 =
α 2 = 0 .
Remark 2.
Choice of q : The settling time upper bound (15) decreases as q → ∞ , which appears to lead to faster convergence when discretized.
On the other hand, the larger q is, the stiffer the ODE, so more prone to numerical instability, so q cannot be too large.
Therefore, assuming p to be not too large, it appears that q ∈ ( p, p + δ ] works best, with δ > 0 as small as needed to avoid numerical issues.
For instance, if we know the cost function to be strongly convex (locally), then we search for q slightly larger than p = 2 at ﬁrst, but continue to increase until performance deteriorates.
If, on the other hand, we don’t know the order p > 1 , then it’s currently unclear how to choose q .
We will investigate this further in future work.
Furthermore, there is evidence that gradient dominance does hold locally in many deep learning contexts (Zhou and Liang, 2017, https://arxiv.org/abs/1710.06910).
Indeed, since convexity readily leads to gradient dominance of order p = ∞ , it sufﬁces that a slightly stronger form of it holds (but weaker than strong convexity), in order to have p < ∞ , and thus for us to be able to choose q > p .
[Figure]
E XPERIMENT 3: MNIST CNN M ODEL
In this experiment we optimize the CNN network described using a Pytorch code sequence in
MODEL 1 with a negative log likelihood loss, on the MNIST dataset.
We use 10 epochs of training, with 60 batches of 1000 images for training, and 10 batches ofimages for testing.
We tested Algorithm 1 RGF ( c = 1 , q = 3 , η = 0 .
06 , µ = 0 . 9 ), and Algorithm 2 SGF ( c = 0 .
001 , q = 2 . 1 , η = 0 .
06 , µ = 0 . 9 ) against Nesterov’s accelerated gradient descent (GD) ( η = 0 .
06 , µ = 0 .
9 , Nesterov=True), and Adam ( η = 0 .
004 , remaining coefﬁcients=nominal values of Torch.
Optim).
We also tested other algorithms such as RMSprop, AdaGrad, and AdaDelta, but since their convergence performance, as well as, test performance were not very competitive w.r.t.
GD, on this experiment, we decided not to report them here, to avoid overloading the graphs.
In
Figures 7, 8 we can see the training loss over the training iterations, where we see that GD, RGF and SGF perform better than Adam in terms of convergence speed ( 20 sec lead in average), and in terms of test performance 98% for Adam, 98% for GD, and 99% for both RGF and SGF.
The
RGF and SGF perform slightly better than GD in terms of convergence speed.
The gain is relatively small ( 5 sec to 10 sec in average) which is expected in such a small DNN network (please refer to VGG16-CIFAR10, and VGG16-SVHN test results for larger computation-time gains).
We also notice, in Figure 7 that all algorithms behave well in terms of avoiding overﬁtting the model.
[Figure]
[Table]
[Figure]
[Table]
E. E XPERIMENT 4: E ULER DISCRETIZATION ON SVHN DATASET
In the main paper
[Figure]
due to space limitation
[Figure]
we decided to report the results of the Nesterov-like discretization only
[Figure]
which also seems like a fair comparison since our Nesterov-like discretization of q -RGF and q -SGF ﬂows can be compared against Nesterov implementation of GD algorithm.
However, we also wanted to test the performance of the simple Euler discretization of the proposed ﬂows against the simple GD algorithm, to do so we run some extra tests on the SVHN dataset.
These results are presented below.
We tested the proposed Euler algorithms to train the same VGG16 CNN model with cross entropy loss.
We divided the dataset into a training set of 70 batches with 1000 images each, and a test set of 10 of 1000 images each, and ran 20 epochs of training over all the training batches.
We tested the Euler
In Figures 9 , 10 we can see that both algorithms, Euler q -RGF and Euler q -SGF, converge faster ( 40 min lead in average) than SGD and Adam for these tests, and reach an overall better performance on the test-set.
E. E XPERIMENT 5: R UNGE -K UTTA DISCRETIZATION ON SVHN DATASET
Finally, for compleetness, we also wanted to test the performance of the Runge-Kutta discretization of the proposed ﬂows against SGD, to do so we run some extra tests on the SVHN dataset.
These results are presented below.
We tested the proposed Runge-Kutta algorithms to train the same VGG16 CNN model with cross entropy loss.
We divided the dataset into a training set of 70 batches with 1000 images each, and a test set of 10 ofimages each, and ran 20 epochs of training over all the training batches.
We tested the Runge-Kutta discretization of q -RGF ( c = 1 , q = 2 . 1 , K = 2 , η = 10 − 2 , β 1 = 10 − 2 , α 1 = α 2 = 0 . 5 ), and the Runge-Kutta discretization of q -SGF ( c = 10 − 3 , q = 2 . 1 , K = 2 , η = 10 − 2 , β 1 = 10 − 2 , α 1 = α 2 = 0 . 5 ) against
In Figures 11, 12 we can see that both algorithms, Runge-Kutta q -RGF and Runge-Kutta q -SGF converge faster ( 40 min lead in average) than SGD and Adam for these tests.