Non Vanishing Gradients for Arbitrarily Deep Neural Networks: a Hamiltonian System Approach
Abstract
Deep Neural Networks (DNNs) training can be difﬁcult due to vanishing or exploding gradients during weight optimization through backpropagation.
To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stems from the discretization of continuous-time Hamiltonian systems.
Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth.
This is obtained by proving that, using a semi-implicit
Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic
Introduction
Consider a N -layer deep neural network (DNN)
[Equation]
where y j ∈ R n j and θ j ∈ R n θj denote the j -th layer feature vector and set of of parameters, respectively.
Moreover, consider a training set for a classiﬁcation task { ( y k 0 , c k ) ∈ R n 0 × { 1 , . .
. , n c }} sk =1 , where y k 0 are the input data and c k are the corresponding labels.
In the framework of Deep Learning (DL), it is common practice to train the DNN (1) by minimizing the empirical risk, i.e.
[Equation]
where L ( · , · ) is the loss function.
When gradient descent methods are used, the update of the vectors θ j is performed using
[Equation]
where k is the iteration number and γ ( k ) > 0 is the optimization step size at each iteration.
Even if DL has achieved remarkable success in various ﬁelds like computer vision, speech recognition and natural language processing [16, 25], the training of DNNs still presents several challenges such as the occurrence of vanishing or exploding gradients.
Both situations are very critical as they imply that the learning process either stops prematurely or becomes unstable.
At each iteration k of (3), the gradient of the loss function with respect to the parameters needs to be calculated.
By using the chain rule, the gradient for parameter i of layer j is computed through backpropagation [12] as:
[Equation]
The problem of vanishing/exploding gradients is related to the backward sensitivity matrices (BSM), i.e. the terms (cid:81) N − 1 l = j +1 ∂ y l +1 ∂ y l .
Indeed, (4) implies that [Equation] vanishes if the term [Equation]
is very small.
In other words, the training of a DNN might stop before achieving a good prediction performance.
Vice-versa, if (cid:107) ∂ y N ∂
y N − j (cid:107) is very large, the derivative ∂ L ∂θ i,N − j becomes very sensitive to perturbations in the vectors ∂
y j +1 ∂θ i,N − j and ∂ L ∂ y N , and this can make the learning process unstable or cause overﬂow issues.
Since the BSMs are related to the network depth, both problems are generally exacerbated when the number of layers N is large [12].
Heuristic methods for dealing with these problems leverage subtle weight initialization or gradient clipping [12].
More recent approaches, instead, focus on the study of DNN architectures and associated training algorithms for which exploding/vanishing gradients can be avoided or mitigated by design [1, 18, 24, 17, 6, 26, 13, 14, 4, 21, 9].
For instance, in [1, 24, 17, 6] unitary and orthogonal weight matrices are used to control the magnitude of BSMs during backpropagation.
Moreover, in [26, 18], methods based on clipping singular values of weight matrices are utilized to constrain the magnitude of BSMs.
These approaches, however, require expensive computations during training [26, 18, 6], introduce perturbations in gradient descent [26, 18] or use restricted classes of weight matrices [1, 24, 17].
Recently, it has been argued that speciﬁc classes of DNNs stemming from the time discretization of Ordinary Differential Equations (ODEs) [13, 14, 4, 21, 9] are less affected by vanishing and exploding gradients.
The arguments provided in [13] rely on the stability properties of the underlying continuous-time nonlinear systems for characterizing relevant behaviors of the corresponding DNNs obtained after discretization, suggesting to use DNN architectures based on dynamical systems that are marginally stable , i.e. that produce bounded and non-vanishing state trajectories.
Examples are provided by ﬁrst-order ODEs based on skew-symmetric maps, which have been used in [13, 4] for deﬁning anti-symmetric DNNs , and by Hamiltonian-inspired DNNs in [13].
However, these approaches consider only restricted classes of skew-symmetric weight matrices or particular Hamiltonian functions, which limit the representation power of the resulting DNNs.
Moreover, the behavior of BSMs arising in backpropagation has been analyzed only in [4], which however focuses on
DNNs with identical weights in all layers and relies on hard-to-compute quantities such as kinematic eigenvalues [22, 2].
To address these open points, we propose a class of DNNs stemming from the symplectic discretization of Hamiltonian systems.
We analyze the sensitivity dynamics in discrete-time and prove that
the
BSM is symplectic.
As a result, the norm of the BSM cannot vanish, irrespective of the network depth and the choice of layer-varying weights.
H-DNN architecture
We propose H-DNNs as the time-discretization of continuous-time Hamiltonian dynamical systems, as per the standard deﬁnition given, for instance, in [-[22]: 
-][23]: [Equation]
[Equation]
where y ( t ) ∈ R n for all t , J ∈ R n × n is skew-symmetric, i.e., J = − J (cid:62) .
The continuously differentiable function H : R n × R → R is denoted as the Hamiltonian function .
We consider,
[Equation]
where 1 n denotes the vector of all ones of length n , ˜ σ : R → R is differentiable and applied elementwise, and its derivative σ ( · ) =
˜ σ (cid:48) ( · ) acts as the DNN’s activation function .
The system (5) can be rewritten as
[Equation]
The H-DNN layer equations are obtained after selecting a discretization method.
In this work, we use semi-implicit Euler (S-IE) [3] as it preserves useful geometric properties of (5).
In particular, we will be interested in the preservation of the symplectic property of its ﬂow.
Assuming that the number of features n ∈ N is even, 2 we split the feature vector at each layer j = 0 , . . .
, N as y j =
( p j , q j ) where p j , q j ∈ R n 2 .
Then, S-IE discretization of (7) leads to the layer equation
[Equation]
where j = 0 , 1 , . . .
, N − 1 and h = T/N .
In general, computing the updates ( p j +1 , q j +1 ) as per (8) involves solving a system of nonlinear equations, that is, ( p j +1 , q j +1 ) are not explicit functions of ( p j , q j ) .
We refer the interested reader to [10, 20] for a discussion of deep learning with implicit layers equations.
To make the updates (8) easily computable, one can further assume that
[Equation]
which yields the explicit layer equations
[Equation]
where one can ﬁrst compute p j +1 through (10), while q j +1 is obtained as a function of p j +1 through (11).
3 Appendix A also presents the layer equations obtained when using forward Euler (FE) and it illustrates how H-DNNs generalize several architectures recently appeared in the literature [13, 5].
Non-vanishing gradients of H-DNNs
Our main result states that the H-DNNs described by (10)-(11) have BSMs lower-bounded in norm by the value 1 irrespectively of the number of layers .
For this purpose, we ﬁrst introduce the notion of symplectic matrix.
Then, we show that, for any choice of weights in the form (9), the BSM obtained from S-IE discretization belong to this class.
Finally we use symplecticity to show our main result.
Deﬁnition 1 (Symplectic matrix) Let Q ∈ R n × n be a skew-symmetric matrix.
A matrix M is symplectic with respect to Q if M (cid:62) QM = Q .
Symplectic matrices are usually deﬁned [15] by assuming n ∈ N to be an even integer and Q = (cid:104) 0 n 2 I n 2 − I n 2 0 n 2 (cid:105) .
In this respect, Deﬁnition 1 provides a slightly generalized notion of symplecticity.
Lemma 1 Consider the system (8) and assume that J has the block structure in (9) .
Then, the matrices ∂ y k +1 ∂ y k for j = 1 . .
. , N − 1 are symplectic with respect to J , i.e.
[Equation]
The proof of Lemma 1 can be found in Appendix B and is built upon Poincaré Theorem (see, e.g.
Theorem 3.3 of Section VI in [15]) and the deﬁnition of extended Hamiltonian systems [7].
This result allows us to prove that the BSMs of H-DNNs are always lower bounded in norm by 1 .
Theorem 1 Consider the H-DNN in (8) and assume that J has the block structure in (9) .
Then,
[Equation]
for all j = 0 , . . .
, N − 1 , where (cid:107)·(cid:107) denotes any sub-multiplicative norm.
Proof: One has ∂ y N ∂ y N
[Equation]
=
[Equation]
[Equation]
N
[Equation] [Equation]
N − j .
Then, by applying iteratively (12)
[Equation]
Hence, we have
[Equation]
for all j = 0 , . . .
, N − 1 .
This inequality implies (13).
The inequality (13) shows that the H-DNN architecture (10)-(11) guarantees non-vanishing BSMs by construction, irrespectively of the network depth.
For a heuristic method to control the growth of the BSMs, we refer the reader to our work [11].
Numerical experiments
We provide a numerical validation 4 of the result in Theorem 1.
We consider a classiﬁcation problem over the 2D “Double moons” dataset (shown in Appendix C) and analyze the norm of BSMs during the training of a H-DNN with 32 layers (Figure 1a) and fully connected multilayer perceptron networks (MLPs) with 8 and 32 layers (Figure 1b).
Detailed information about the architectures and the training parameters can be found in Appendix C.
[Figure]
∂ y N ∂ y N − ℓ
|| [Figure] While the H-DNN and the 8-layer MLP achieve good performance at the end of the training ( 100% and 99 . 7% accuracy over the test set, respectively), the 32-layer MLP completely fails to classify the data ( 50% accuracy) due to rapidly vanishing gradients.
Figure 1a validates (13) since no BSM norm is smaller than 1 at any iteration.
However, this is not the case for the 32-layer MLP.
Figure 1b (bottom) shows that, after 400 iterations, only a few gradient Despite the BSM not vanishing, we verify that a stationary point θ (cid:63) such that ∇ θ L ( θ (cid:63) norms are different from zero, since (cid:13)(cid:13)(cid:13) ∂ y N ∂ y N − (cid:96) (cid:13)(cid:13)(cid:13)
≈ 0 for (cid:96) = 7 , . .
The main cause of such a premature stop of the learning is therefore directly linked to the phenomenon of vanishing gradients.
We refer the reader to Appendix D for further experiments demonstrating the potential of H-DNNs.
Conclusions
We introduced a class of H-DNNs obtained from the time-discretization of Hamiltonian dynamics and proved that H-DNNs stemming from S-IE discretization do not suffer from vanishing gradients.
Although we limited our analysis to S-IE discretization, one can leverage the rich literature on symplectic integration [27] for deﬁning even broader classes of H-DNNs with similar properties
Acknowledgments
Research supported by the Swiss National Science Foundation under the NCCR Automation (grant agreement 51NF40_180545).
A H-DNNs and existing architectures
We ﬁrst introduce H-DNNs obtained using forward Euler (FE) discretization.
Then, we show how existing architectures proposed in [13], [4] and [5] are included in our framework.
Consider the step size h = T/N .
Then, the FE discretization of the ODE (7) leads to a N -layer
H-DNN architecture deﬁned as
[Equation]
Having set two different architectures for H-DNNs, i.e. using S-IE (10)-(11) and FE (14), we list the existing architectures [13, 4, 5] and show that they are special cases of H-DNNs.
These architectures stem from the discretization of Marginally Stable (MS) systems with constant weights across layers.
Hence, we call them MS i -DNN ( i = 1 , 2 , 3 ).
MS 1 -DNN: In [13], the authors propose to use Verlet integration method to discretize
[Equation]
where y , z ∈ R n 2 .
Then, the layer equations are given by
[Equation]
Note that (15) is an instance of H-DNN with S-IE discretization when K j is assumed to be invertible and where we set J j K (cid:62)
j = I n , and K j = (cid:104)
[Equation]
for all j = [Equation]
N − 1 .
MS 2 -DNN: In [13, 4], the authors propose to use FE to discretize
[Equation]
where K ( t ) is skew-symmetric ∀ t ∈ [0 , T ] .
Then, the layer equation is given by
[Equation]
In this case, (16) is an instance of H-DNN with FE discretization when K j is assumed to be invertible and where we set J j K (cid:62)
j = I n and K j = − K (cid:62) j for all j = 0 , . . .
, N − 1 .
MS 3 -DNN: In [5], the authors propose to use Verlet integration method to discretize
[Equation]
, where y , z ∈ R n 2 .
Then, the layer equations are given by
[Equation]
Note that (17) is an instance of H-DNN with S-IE discretization where we set
[Equation]
and J j =
[Equation]
n
[Equation]
In [13] and [5], the DNNs MS 1 and MS 3 are called Hamiltonian-inspired in view of their similarities with Hamiltonian models, although a precise Hamiltonian function for the corresponding ODE has not been provided.
Moreover, note that the Verlet discretization used by the authors coincides with
S-IE.
We highlight that a necessary condition for the skew-symmetric n × n matrix K j to be invertible is that the size n of input features is even.
6 If n is odd, however, one can perform input-feature augmentation by adding an extra state initialized at zero to satisfy the previous condition [8].
B Proof of Lemma
The proof of Lemma 1 is built upon the result of Theorem 3.3 of Section VI in [15] and the deﬁnition of extended Hamiltonian systems [7].
The former proves that the numerical ﬂow of a time-invariant
Hamiltonian system with J = (cid:2)
I − I 0 (cid:3) is symplectic.
The latter will allow us to embed the study of a time-dependent Hamiltonian function into the time-independent case by deﬁning an extended phase space of dimension n + 2 instead of n .
We study the Hamiltonian system (5) in the extended phase space [7], i.e., we deﬁne an extended state vector ˜ y
= ( p , q , ε, t ) , 7 an extended interconnection matrix ˜ J = (cid:104)
J
[Equation]
(cid:105) , Ω =
[Equation]
and an extended Hamiltonian function
[Equation]
such that
[Equation]
[Equation]
Note that the extended Hamiltonian system deﬁned by ˜ H is time-invariant by construction, i.e. d ˜ Hdt = 0 .
Then, following Theorem 3.3 in Section VI of [15], it can be seen that ∂ ˜ y j +1 ∂ ˜ y
j is a symplectic matrix, i.e. it satisﬁes
[Equation]
Next, we show that (19) implies symplecticity for the BSM of the original time-varying system (8).
Considering the Hamiltonian (18), we obtain the S-IE layer equations for the extended Hamiltonian dynamics as per:
[Equation]
[Equation]
Then, we differentiate each of the equations of the system (20) with respect to each subvector of ˜ y
j = ( p j , q j , e j , t j )
and we rearrange the terms.
We obtain[-
-]
[Equation]
˜
[Equation]
where
[Equation]
and x, y indicate any combination of two variables in the set { p, q, t } .
It remains to verify that (21) implies symplecticity of ∂ y j +1 ∂ y j , i.e. it satisﬁes (12) for:
[Equation]
+1 ∂ p j [Equation] [Equation]
q [Equation] ∂ q [Equation]
+
[Equation]
(cid:35) . By denoting Γ = (cid:0) I n − h (cid:2) H pp H qp0 (cid:3) J (cid:62) (cid:1) and Λ = (cid:0) I n + h (cid:2)0 H pq H qq (cid:3) J (cid:62) (cid:1) , where Γ and Λ are invertible for almost every choice of step size h , the part of (21) concerning ∂ y j +1 ∂
y j reads as
[Equation]
The above implies
[Equation]
where the equality ΛJΛ =
ΓJΓ can be veriﬁed by direct inspection.
We conclude that ∂ y j +1 ∂
y j is a symplectic matrix.
It is worth remarking that similar conclusions can be obtained by performing the analysis in continuous-time before selecting a discretization scheme.
Hence, one can prove that the continuoustime counter-part of H-DNNs has non-vanishing BSMs.
C Implementation details
The DNN architectures and training algorithms are implemented using the PyTorch library.
We use the “Double moons” dataset shown in Figure 2.
It consists of 16,000 2-dimensional points, that are equally split to obtain the train and test datasets.
The layer equations of the H-DNNs are described in (10)-(11) where we set X to be the identity matrix.
For the MLPs, their layer equation is given by:
[Equation]
with trainable parameters K j and b j .
We use tanh( · ) as activation function for both architectures.
Moreover, we complement each of them with a soft-max output layer.
Training is performed using coordinate gradient descent, i.e. a modiﬁed version of stochastic gradient descent (SGD) with Adam ( β 1 = 0 .
9 , β 2 = 0 .
999 ) [13] and minibatches of size 125 .
We use standard cross-entropy as the loss function L to minimize in (2), together with the regularization term
[Equation]
The term R K is deﬁned as h 2 (cid:80) N − 1 j =1 (cid:16) (cid:107) K j − K j − 1 (cid:107) 2 F + (cid:107)
b j − b j − 1 (cid:107) 2 (cid:17) which favours smooth weight variations across consecutive layers [13, 5].
The term R N ( · ) refers to a standard L 2 regularization for the output layer.
We set α N = 1 × 10 − 4 and α = 5 × 10 − 4 .
Following [13], in every iteration of the algorithm, the optimal weights of the output layer are computed given the last updated parameters of the hidden layers.
Then, a step update of the hidden layers’ parameters is performed by keeping ﬁxed the output parameters.
The training consists of 50 epochs and each of them has maximum 10 iterations to compute the output layer weights.
The learning rate, or optimization step size as per γ in (3), is set to 2 .
5 × 10 − 2 .
D Experiments with MNIST dataset
We evaluate our methods on a more complex example: the image classiﬁcation benchmark MNIST.
The dataset consists of 28 × 28 digital images in gray scale of hand-written digits from 0 to 9 with their corresponding labels.
It contains 60,000 training examples and 10,000 test examples.
Following [13], we use a network architecture consisting of a convolutional layer followed by a
Hamiltonian DNN and an output classiﬁcation layer.
The convolutional layer is a linear transformation that expands the data from 1 to 8 channels, and the network output is a vector in R 10 representing the probabilities of an image to belong to each of the 10 classes.
We compare the performance of MS 1 -DNNs (see Appendix A) [13] and H-DNNs (using S-IE discretization) with 2, 4, 6 and 8 layers, and tanh( · ) as activation function.
We set h = 0 .
4 for
MS 1 -DNNs and h = 0 .
45 for H-DNNs.
The training consists of 40 epochs with mini-batches of size 100.
For the optimization algorithm we use SGD with Adam [19].
The learning rate, or optimization step size as per γ in (3), is initialized to be 0 .
04 with a decay rate of 0 .
8 at each epoch.
We utilize cross-entropy loss function L and we add a regularization term given by
[Equation]
The term R K is deﬁned as h 2 (cid:80) N − 1 j =1 (cid:16) (cid:107) K j − K j − 1 (cid:107) 2 F + (cid:107)
b j − b j − 1 (cid:107) 2 (cid:17) which favours smooth weight variations across consecutive layers [13, 5].
The term R L 2 ( · ) refers to a standard L 2 reg-
[Table]
In Table 1, we summarize the accuracies of the considered networks on train and test data.
The ﬁrst row of the table provides, as a baseline, the results obtained when omitting the Hamiltonian DNN block, i e., when using only a convolutional layer followed by the output layer.
We observe that both
MS 1 -DNN and H-DNN achieve similar performance.
Note that, while the training errors are almost zero, the test errors are reduced when increasing the number of layers, hence showing the beneﬁt of using deeper networks.
Moreover, these results are in line with test accuracies obtained when using standard convolutional layers instead of H-DNNs [5].